← All work

JAVIS Technologies

Schema-Driven Ingestion Compiler

A multi-tenant ingestion platform that compiles configuration into executable profiles

JavaSpring BootTemporalMySQL
Problem
Every new client meant new ingestion code. Column mappings, validation rules, and industry-specific quirks were expressed as branching logic, so the codebase grew linearly with the customer list and no two onboardings looked alike.
My role
Led the design. Owned the compiler, the profile format, and the execution model end to end.
Approach
Treated onboarding as a compilation problem rather than a coding one. Column metadata, versioned rulesets, and per-tenant and per-industry overrides compile into a single immutable execution profile, which a generic engine interprets. The cost is that the hard cases move into the compiler, where they are harder to write; the gain is that every client after that is a configuration change.
Outcome
Eliminated per-client code entirely and reduced the codebase 95% — from roughly 12,000 lines to 600. Onboarding a client stopped being an engineering task.

The shape of the problem

What made the old approach break down. Be concrete about the scale and the kind of variation between tenants.

Compiling instead of coding

The core idea: what goes into a profile, what the profile looks like, and why immutability matters. A small illustrative example of a rule compiling into a profile fragment would carry a lot here.

Dataflow optimization

The compiler pass that derives the operation dependency graph from declared read/write sets — how declaration enables auto-parallelization of independent foreign-key lookups and coalescing of identical operations into batched calls. The headline number: a 1M-row file resolving against each master table in a handful of queries instead of a million.

Execution

The zero-business-logic interpreter on Temporal: crash-resumable mid-file, multi-leg parallel execution, row-level error accumulation, version-pinned profiles for reproducible replay.

What I’d do differently

Optional, and disproportionately convincing. One honest thing.