Cutting Data Processing Time by 80% for a Top Logistics Platform

Author : Team Aays3-4 min read

Updated: Nov 16, 2026

Table of contents
  1. Overview
  2. About the Client
  3. The Problem
  4. The Solution
  5. The Impact
  6. Why It Matters
Black-and-white warehouse aisle with two workers and magenta stripes, with the text “Faster, Reliable Data for Logistics”

Discover how Aays moved a logistics technology provider from hand-built streaming jobs to a config-driven CDC platform on Databricks, so inserts, updates and deletes now stay in sync across 82 live tables.

82
Tables
Synced with full insert, update and delete handling
200 → 2
Pipelines
One stream per table replaced by two pipelines
~4 min
Fastest Triggered Run
On serverless compute

About the Client

Our client is a logistics technology provider that helps businesses track and manage shipments. Its platform runs on PostgreSQL and generates a steady stream of data on shipment tracking, delivery exceptions and customer signups. Hundreds of tables feed analytics and reporting teams, so fresh, accurate data is core to the business.

The Problem

Could One Custom Streaming Framework Keep Up with 200 Tables?

The client’s data moved from PostgreSQL through AWS DMS into S3 as Parquet files. A custom PySpark Structured Streaming framework then loaded those files into Delta tables. It worked at first, but it did not scale well. Every table had its own long-running job, its own notebook and its own hand-written merge logic. Updates and deletes were not being handled correctly, so the tables could drift away from the source.

The setup struggled because of:

  • 200 always-on streams: One job per table meant a lot of monitoring and a lot to go wrong.
  • Fragile schema changes: Schemas were hard-coded, so a new column meant a code change.
  • Hard to debug: Checkpoints were managed by hand, and finding the cause of a failure was slow.
  • Incorrect updates and deletes: The merge logic didn’t keep target tables in step with the source.

The framework was built to move data in, not to keep it correct or easy to run. A simpler, standard design was needed.

The Solution

Building a Config-Driven CDC Platform on Databricks

Instead of patching the old framework, we replaced it with Lakeflow Spark Declarative Pipelines. Each table is described in a single configuration table, and the pipeline builds itself from that. Adding a table now means adding a row, not writing a notebook.

Each table is loaded in two phases:

Historical Load (Runs Once)

The full history is read from a bronze staging table. S3 only keeps 7–10 days of change files, so it can’t serve as a source for a full backfill.

Continuous CDC Streaming

New change files are picked up from S3 using Auto Loader with managed file events, with no directory scanning.

Every change is applied the same way:

Insert / Update

Upserted into the target table, with the latest record winning.

Delete

The row is removed from the target.

Other design choices:

  • Two pipelines by priority: 35 priority tables run continuously. 47 non-priority tables run hourly.
  • Automatic schema evolution: New columns are picked up without code changes, and unexpected data is captured in a rescue column.
  • Built-in data quality checks: Records with a missing key, a missing sequence value or an invalid operation are dropped and tracked.
  • Optimized bronze layer: Predictive optimization, clustering and data skipping make merges and reads faster.
  • Built-in monitoring: Table-level status, a pipeline graph, automatic retries and managed checkpoints replace custom logs.

We also built an independent validation framework to check the pipelines against S3, the source of truth. It runs three checks at primary-key level:

  • Are any live records missing from the target?
  • Are any deleted records still in the target?
  • Are there any orphan records in the target?

It also reports lag between source and target, and each table gets a pass, warn or fail status.

The Impact

Business and analytics teams now work from tables that match the source, without depending on engineers to babysit streams.

82
Tables
Synced across both pipelines, each with full insert, update and delete handling.
200 → 2
Pipelines
One long-running job per table replaced by two pipelines.
~4 min
Fastest Triggered Run
On serverless compute.
30+ hrs
Continuous Serverless Run
With no pipeline failures.
0
Code Changes
New columns are picked up automatically through schema evolution.

Why It Matters

Moving 200 streaming jobs onto a shared framework is hard to do without breaking things. We made it low risk by keeping the design simple with one config table, one standard pattern and a validation layer that proves the data is right. The client’s team now spends its time using the data instead of fixing jobs.