Cutting Data Processing Time by 80% for a Top Logistics Platform
Updated: Nov 16, 2026

Discover how Aays moved a logistics technology provider from hand-built streaming jobs to a config-driven CDC platform on Databricks, so inserts, updates and deletes now stay in sync across 82 live tables.
About the Client
Our client is a logistics technology provider that helps businesses track and manage shipments. Its platform runs on PostgreSQL and generates a steady stream of data on shipment tracking, delivery exceptions and customer signups. Hundreds of tables feed analytics and reporting teams, so fresh, accurate data is core to the business.
The Problem
Could One Custom Streaming Framework Keep Up with 200 Tables?
The client’s data moved from PostgreSQL through AWS DMS into S3 as Parquet files. A custom PySpark Structured Streaming framework then loaded those files into Delta tables. It worked at first, but it did not scale well. Every table had its own long-running job, its own notebook and its own hand-written merge logic. Updates and deletes were not being handled correctly, so the tables could drift away from the source.
The setup struggled because of:
- 200 always-on streams: One job per table meant a lot of monitoring and a lot to go wrong.
- Fragile schema changes: Schemas were hard-coded, so a new column meant a code change.
- Hard to debug: Checkpoints were managed by hand, and finding the cause of a failure was slow.
- Incorrect updates and deletes: The merge logic didn’t keep target tables in step with the source.
The framework was built to move data in, not to keep it correct or easy to run. A simpler, standard design was needed.
The Solution
Building a Config-Driven CDC Platform on Databricks
Instead of patching the old framework, we replaced it with Lakeflow Spark Declarative Pipelines. Each table is described in a single configuration table, and the pipeline builds itself from that. Adding a table now means adding a row, not writing a notebook.
Each table is loaded in two phases:
Historical Load (Runs Once)
The full history is read from a bronze staging table. S3 only keeps 7–10 days of change files, so it can’t serve as a source for a full backfill.
Continuous CDC Streaming
New change files are picked up from S3 using Auto Loader with managed file events, with no directory scanning.
Every change is applied the same way:
Upserted into the target table, with the latest record winning.
The row is removed from the target.
Other design choices:
- Two pipelines by priority: 35 priority tables run continuously. 47 non-priority tables run hourly.
- Automatic schema evolution: New columns are picked up without code changes, and unexpected data is captured in a rescue column.
- Built-in data quality checks: Records with a missing key, a missing sequence value or an invalid operation are dropped and tracked.
- Optimized bronze layer: Predictive optimization, clustering and data skipping make merges and reads faster.
- Built-in monitoring: Table-level status, a pipeline graph, automatic retries and managed checkpoints replace custom logs.
We also built an independent validation framework to check the pipelines against S3, the source of truth. It runs three checks at primary-key level:
- Are any live records missing from the target?
- Are any deleted records still in the target?
- Are there any orphan records in the target?
It also reports lag between source and target, and each table gets a pass, warn or fail status.
The Impact
Business and analytics teams now work from tables that match the source, without depending on engineers to babysit streams.
Why It Matters
Moving 200 streaming jobs onto a shared framework is hard to do without breaking things. We made it low risk by keeping the design simple with one config table, one standard pattern and a validation layer that proves the data is right. The client’s team now spends its time using the data instead of fixing jobs.
