What is new here, in five lines
- DuckDB-WASM turns the browser into an analytical engine: the diff of a million rows is computed on the client and the server only receives the result.
- Row-level optimistic concurrency via Delta Row Tracking, using the engine’s own mechanism instead of an
updated_atcolumn every writer has to remember to maintain. - An Excel parser written in Rust and compiled to WASM, at 436 KB, under an explicit bundle budget.
- End-to-end Arrow streaming, never touching JSON anywhere between the warehouse and browser memory.
- An LLM agent with tool-calling that creates projects and provisions tables from a natural-language instruction — and dispatches to the existing domain services, so it inherits their validation and audit trail.
The problem it solves
An internal SaaS that lets business teams create, edit and audit master tables in Databricks Unity Catalog without writing SQL or having direct warehouse access. Every catalog change used to go through the data engineering team; now the loop is closed by the people who actually know the data.
Role: full-stack design and development — frontend, backend, WASM module and data model.
Architecture
| Layer | Technology |
|---|---|
| Frontend | React 18 + Vite + TypeScript + Tailwind 4, TanStack Query |
| In-browser compute | DuckDB-WASM (embedded analytical engine) |
| Excel parser | Rust (calamine) compiled to WASM — 436 KB |
| Backend | FastAPI (Python 3.13) + Uvicorn, hexagonal architecture |
| Metadata and audit | MongoDB (change streams) |
| Data warehouse | Databricks Unity Catalog / Delta Lake |
| Deployment | Databricks Apps; logging to Google Cloud Logging |
The monorepo splits apps/client (UI), packages/core (domain and shared adapters), packages/ui, backend and wasm_modules. The backend defines an IDatabricksAdapter port with its concrete adapter, so services and business rules never depend on the Databricks SDK.
The technically interesting parts
1. Bulk editing with row-level optimistic concurrency
The classic “download a spreadsheet, edit it, upload it” problem: between download and upload, someone else may have changed the same rows. The usual fix — an updated_at column — demands discipline from every writer and lies outright if anything writes outside the app.
Instead I use Delta Row Tracking pseudo-columns (_metadata.row_id, _metadata.row_commit_version): the engine maintains them automatically, they add no physical columns and no storage bytes, and they survive external writes. The apply runs as a single MERGE INTO with a per-row version guard:
WHEN MATCHED AND s.__op__='update'
AND t._metadata.row_commit_version = s.__row_version__
THEN UPDATE SET ...If the version moved, the row is left alone and reported as a conflict. A preceding anti-join (~1-3 s over 500k rows) returns the exact list of conflicting PKs, so the user is told which rows failed and not merely how many. Enabling the feature is lazy and idempotent, so tables registered before it existed migrate themselves on first use.
The same pattern covers row-by-row editing in the UI: a versioned UPDATE/DELETE becomes a MERGE joining against the target itself filtered on row_commit_version; if the version does not match, the join yields nothing, zero rows are affected, and an optimistic-locking error is raised.
2. The diff is computed in the browser, not on the server
The Excel template carries a hidden _meta sheet with row_id, row_version and a SHA-256 per row — not a duplicate of the data, which for a million rows would weigh 200 MB. On upload:
- The Rust→WASM module converts the sheet to RFC 4180 CSV in the browser itself.
- DuckDB-WASM ingests both sheets and computes inserts, updates and deletes in plain SQL: anti-joins plus hash comparison, recomputed with the same formula as the backend (
sha256(CONCAT_WS(CHR(31), ...))with columns in alphabetical order). - It exports the changeset as ZSTD-compressed Parquet and uploads it multipart.
The server only drops the Parquet into a Unity Catalog Volume and fires a MERGE INTO ... USING read_files(...). No row-by-row processing, no giant SQL statements with thousands of VALUES: the diff scales with the client’s CPU and the apply is a single statement. The staging file is always deleted in the finally, success or failure.
3. End-to-end Arrow streaming, no JSON
Dimension tables are read from Databricks as ARROW_STREAM with EXTERNAL_LINKS: the backend downloads the chunks over presigned URLs, concatenates them and hands back an Arrow IPC stream that DuckDB-WASM inserts straight into memory (insertArrowFromIPCStream), preserving the warehouse’s typed schema. Nothing is serialised to JSON anywhere along the path. On that in-browser table the user filters, sorts and edits at millisecond latency.
4. Generic keyset pagination over composite keys
For fact tables, which do not fit in memory, I implemented cursor pagination instead of OFFSET, supporting composite PKs and mixed sort directions: when every column sorts ascending it uses the compact tuple form (a, b) > (?, ?); with mixed directions it generates the disjunctive lexicographic expansion. Client filters go through an operator allowlist and travel as typed statement parameters, with identifier quoting — no user strings concatenated into SQL.
5. Project-creation agent with tool-calling
One endpoint takes a natural-language instruction plus attached spreadsheets and runs a tool-calling loop — 12 turns maximum — against a Databricks Model Serving endpoint through its OpenAI-compatible HTTP API. The tools exposed to the model are not new wrappers: they dispatch to the same domain services the UI uses (create_project, provision_table_from_excel, register_table_in_project, add_users_to_project), so the agent inherits the existing validation and audit logging. Every step — thought, tool call, result — is emitted as an SSE event and rendered live by the frontend.
Provisioning a table from a spreadsheet checks that the target does not already exist, runs CREATE TABLE ... USING DELTA with Row Tracking on from birth, uploads the Parquet to the Volume and inserts with read_files.
6. Cascading authorisation wired to corporate identity
Identity arrives through the Databricks Apps proxy headers (X-Forwarded-Email). The role is resolved centrally against Databricks SCIM groups with a cache and a live fallback when the cache comes back empty, so a miss never denies access. Permission is evaluated in a project → table cascade, inherited when the table declares no restrictions of its own, by email or by group. Every mutating operation lands in an audit collection after role and action validation.
7. Real-time observability from a single fanned-out change stream
The operations dashboard is fed by a WebSocket over MongoDB change streams. Rather than opening one stream per client, a single shared stream fans out to everyone connected, with per-project filtering on the server and a normalised envelope that insulates the frontend from Mongo’s internal format. KPIs and time series come from aggregation pipelines scoped to the projects the user can reach.
8. WASM assets served from a Databricks Volume
The environment blocks external CDNs, so the DuckDB-WASM binaries are served from a Unity Catalog Volume through a proxy endpoint that streams them with the right Content-Type and CORS/CORP headers (application/wasm, Cross-Origin-Resource-Policy), and the worker is instantiated via a blob to sidestep cross-origin restrictions.
The decisions that define the project
- Push compute to the client. DuckDB-WASM makes the browser a real analytical engine: a million-row diff never touches the server.
- Pick the engine’s mechanism before inventing your own. Row Tracking instead of hand-rolled timestamps;
MERGEover a Volume instead of N queries. - An explicit bundle budget. Arrow IPC from WASM (+500 KB) and an XLSX writer in Rust (+300 KB) were both dropped; CSV with
ALL_VARCHARandopenpyxlin write-only mode achieve the same thing while keeping the module at 436 KB. - The agent reuses the domain, it does not duplicate it. The LLM tools are a façade over the existing services.
Alternatives considered and rejected
| Idea | Why it was dropped |
|---|---|
A physical updated_at column for concurrency | Demands discipline from every writer plus a manual backfill; Row Tracking is automatic and survives external writes |
A _meta sheet duplicating all the data | Doubles the weight of the XLSX (~200 MB for 1M rows); the hash is enough to detect updates |
| A hand-written diff engine in Rust/WASM | DuckDB-WASM was already integrated; an ANTI JOIN in SQL is trivial and fast |
| Arrow IPC from the WASM module | +500 KB of bundle; CSV with an implicit cast in the MERGE gives the same result |
Inline MERGE INTO ... USING (VALUES ...) | Practical ceiling of ~1000 rows per SQL statement; does not scale |
Computing the diff in a backend /preview | Wastes the client’s compute and adds a round-trip per iteration |