Every CSV Lied to Us, So We Gave It a Language

Somewhere out there, right now, a device is quietly measuring something temperature, vibration, pressure, water level, who knows and every few minutes it politely drops a CSV file into a folder and goes back to sleep. Thousands of these devices, across hundreds of sites, all doing the same simple job.

And here’s the part nobody warns you about before you sign up to build the system that ingests all of it: every single one of those files looks a little different.

Not “different data.” Different shape. Different column order, different header names, different number of junk rows at the top, different delimiters, different date formats, different everything. Same job, same intent, completely different handwriting.

Chapter 1: The Hardcoded Parser Era

The first version of the story is the one every engineer has lived at least once: we just wrote a parser for it. New sensor type shows up, someone opens the parser file, adds an if block, ships it. Simple. Works. Feels good for exactly as long as you only have a handful of these.

Then it stops being a handful.

Now imagine that same pattern repeated across roughly a hundred different parsers, one per client or sensor variant, all hardcoded, all living in the same codebase, all one accidental edit away from breaking someone else’s file quietly. Someone adds a new column for one client. Somewhere else, a value silently stops being read correctly. Nobody notices immediately, because the file still “parses” it just parses wrong.

This is the “it compiled, ship it” energy, except the bug shows up three weeks later as a missing alert.

And that’s the part that actually hurts. This isn’t a dashboard nobody looks at the values coming out of these files sometimes need to trigger an alert. If a hardcoded parser quietly breaks and a critical reading never gets read, the alert that was supposed to fire never fires. Nobody gets paged. Nobody gets an email. The system just sits there, looking healthy, saying nothing, which is the worst possible way for a monitoring system to fail.

So on top of the parsers themselves, we had to build reception limits a way of saying “if this file hasn’t arrived on schedule, someone needs to know.” Which sounds simple, until you have to answer the much more political question underneath it:

“Is this our parser being wrong, or is it their sensor being wrong?”

Because those two failures look identical from the outside no data showed up and they have very different owners. So we built a whole second layer just to notify site engineers: “hey, your file might be corrupted, or your sensor might be misbehaving, or okay fine, maybe it’s on us too.” Diplomacy, encoded in a notification service.

Chapter 2: We Stopped Writing Parsers. We Wrote a Language.

At some point the honest answer wasn’t “write parser #101.” It was: stop writing parsers.

So we built a small domain-specific language for describing a CSV file’s shape as a JSON document. Not a parser. A grammar. You describe the rules: which lines to skip and when to skip them dynamically, how to map a column to a value, how to transform it, what to do when a value is missing, what “this row is garbage, ignore it” looks like. Then a single engine reads that grammar and runs it against any file that matches it.

Think of it less like a parser and more like a tiny programming language whose only job is describing CSV files its own syntax, its own rules, and one engine underneath that actually executes it. A new sensor variant stopped being a code change. It became configuration.

One problem that looked small on paper and turned out to be genuinely nasty: timestamps. Converting a local reading to UTC sounds like a one-line function until your clients are spread across the entire globe, and some of them live in places that helpfully shift between two different timezones depending on the season. Summer time, winter time, same site, same sensor, different offset depending on what month it is. The DSL had to know how to handle that too not as a special case bolted on somewhere, but as a rule you could actually express in the grammar itself.

Chapter 3: When the Language Learned Too Many Tricks

Once the grammar proved itself, it didn’t stay small for long. People kept finding one more weird file shape it needed to handle, so it kept growing another operation to handle it. Skip a line, sure. But also: trim this, regex that, look this value up in a table, convert this unit, do a bit of math on two columns, branch depending on what a header says, fall back to a default when a cell is empty. One thing led to another, and the grammar quietly crossed 35+ distinct operations it could perform on a file.

And here’s the part that made it genuinely fun: you could chain them. Stack five or six operations on a single column, combine them in whatever order the file demanded, and there was almost no CSV weird enough to survive contact with it. Nested headers, split timestamps across two columns, a stray footer row some sensor firmware insists on adding didn’t matter. The grammar could describe its way through basically anything.

Which meant our tiny language could now parse almost anything you threw at it.

It also meant our tiny language was no longer tiny.

A JSON config with enough nested operations stops looking like configuration and starts looking suspiciously like actual code minus the debugger, minus the autocomplete, plus about four times the brackets anyone signed up for.

We didn’t feel the downside of this yet. At this point we mostly felt powerful look at this thing, it can parse anything. That feeling had an expiration date, and it was closer than we thought. (More on that soon. If you’ve ever had to explain a 200-line config to someone whose job title does not contain the word “engineer” you already know. IYKYK.)

Chapter 4: No More Deploys for a New Parser

The next piece was making the DSL itself dynamic end-to-end. We built a small UI where a grammar could be written, saved, and stored as a JSON object in Postgres pulled back out at runtime and handed to the engine whenever a matching file showed up. Full CRUD over an API. No redeploy, no code review for a config change, no engineer required to add a client.

And because one grammar can describe more than one source, a single parser could be attached to many folders at once hundreds of them, each just a mapping of “this folder uses that grammar.” Adding a new site became an API call, not a pull request.

Chapter 5: The File That Broke Kubernetes’ Feelings

Just when the grammar was working nicely, reality delivered its next gift: a file over 33 MB.

Parsing it wasn’t impossible it was just heavy. CPU-heavy, sitting-on-the-event-loop heavy. And our service happened to live inside Kubernetes, which periodically likes to check in and ask, politely, “hey, are you still alive?” While the main thread was buried in parsing a 33 MB file, it had no room left to answer that question. Kubernetes assumed the worst and did what Kubernetes does when it stops trusting you.

This is fine dog sitting in a burning room — The pod, mid-parse, insisting everything is fine.

The fix wasn’t clever, just correct: move the CPU-heavy parsing off the main thread entirely, into worker threads, so the process could keep breathing answering health checks, accepting new work while the actual parsing happened in parallel elsewhere. Throughput went up because we were finally using more than one core for the part of the job that actually needed it.

By the time all of this had settled into place, the system was quietly chewing through around 15 million records a day.

But This Is Software, So Obviously That Wasn’t the End

Here’s the thing about solving a problem cleanly: it buys you exactly enough time to discover the next one. The DSL solved “stop hardcoding parsers.” It did not solve, as it turns out, several other things and this is where I’ll stop, on purpose, because that’s a whole story of its own.

Coming up, in no particular order of pain:

How a system designed so anyone could write a new parser somehow still needed the backend team for every single one of them
Why teaching a 35-operation JSON language to support staff turned into its own bottleneck (see Chapter 3 yes, it was exactly as fun as it sounds)
How CPU costs eventually pushed part of this stack toward Go (yes, really)
A dataset I quietly started building on the side input CSVs paired with their correct output JSON that nobody knew about for a while
What happens when you go looking for an open-source model to fine-tune on that dataset, buy yourself an RTX 3060, and just… leave your PC running for days

One does not simply hardcode 100 parsers — One does not simply hardcode a hundred parsers.

There’s a lot more to this story. As we say around here picture abhi baaki hai, mere dost.