Deadline-bound batch computation: how to guarantee it finishes on time

“It has to finish by the 5th of the next month, otherwise commissions don't go out on time and sellers go on the warpath.” That's deadline-bound batch — it isn't enough that it finishes once; it has to finish always, on time, even with 40% data growth this year.

Same applies to payroll (law mandates payout within X days), month-end closes, invoicing cycles, regulatory reporting. If you're in that world, here's how.

1. Measure, don't estimate

Step one: how long does the computation take on real data, and how does it scale. Not “should be fine.” Concrete numbers.

For JUST we measured:

• 1 000 sellers × commissions = 12 seconds
• 3 000 sellers = 38 seconds (super-linear — O(N log N) somewhere inside)
• 10 000 sellers = 5.5 minutes (extrapolation)
• Projected growth over 3 years: 1 000 → 8 000 sellers

Three years in, the run would take 3 minutes. Deadline is 5 days, 3 minutes fits. But if growth were 1 000 → 30 000, it would be 30+ minutes, and if the run has downstream dependencies (bank invoice generation), that starts biting.

Measuring upfront = realistic view. If you don't measure, you design for today, and a year later you have a bug.

2. Parallelise at the level of independent units

Batch computation is typically a loop over N items where each is independent. For JUST, one seller = one independent unit. Results don't depend on what another seller does (except for downline overrides, which can be split into their own preprocessing stage).

Architecture:

• Preprocessing: validate data, prepare context
• Partition: split sellers into N batches (e.g. 100 batches of 80 sellers for an 8 000-seller fleet)
• Parallel execute: N workers, each processes its batch independently
• Aggregate: combine results into the final report

For JUST we ran 10 parallel workers on Azure Functions (later 20). Scaling was near-linear until we hit the DB connection pool.

3. Deadline monitoring with pre-alerting

The run starts on day 1 of the month. Deadline is day 5. “Is it going to make it?” on day 5 is too late. Instead:

• Day 1, hour 0: batch start
• Day 1, hour 2: if not done yet, informational pre-alert
• Day 1, hour 6: high-priority alert (something is seriously slow)
• Day 2, hour 0: escalation (human intervention required)
• Day 3: rollback plan, manual computation, etc.

Pre-alerting gives 3-4 days of runway to diagnose and fix. Without it you find out on the morning of day 5, and it's already too late.

4. Idempotent retry + checkpoint

The batch crashes halfway. Now what? Two extremes:

• Start from scratch — simple, but on a 30-min run it's 30 minutes of delay
• Resume from last checkpoint — more complex, but saves time

For a deadline-bound system checkpoints are mandatory. After each batch a batch_progress row is written (batch_id, completed_sellers, timestamp). On retry we load the last checkpoint and continue from there.

Critical: the per-seller computation must be idempotent. Running it twice yields the same result. Enforce that by doing DB writes as UPSERTs on (seller_id, month).

5. Full dry-run on a real data copy before production

A staging test with 100 sellers tells you nothing about behaviour with 10 000. Before every quarter we do a complete dry-run on a clone of production data:

• Clone production DB into staging
• Run the batch end-to-end
• Compare results with the production history
• Measure timing — does it match your growth model?

3 times in the last year the dry-run caught a problem before it reached production — one memory leak, two N+1 query bugs that would have inflated runtime 10×.

6. Degradation strategy for edge cases

What if some data is corrupt and blocks the computation for the whole batch? Two strategies:

• Fail-fast: first error stops the batch, human intervention
• Skip-and-log: problematic seller is skipped, batch continues, the error is logged for post-processing

Choice depends on business context. For commissions, fail-fast doesn't make sense — 5 sellers with bad data would mean 10 000 others don't get paid. Skip-and-log is better — everyone gets paid except those 5, who get resolved manually.

JUST outcome

After 3 years of operation:

• Seller count grew from 2 800 to 6 200 (+120%)
• Monthly commission run: used to be ~8 min, now ~14 min (scales well)
• 0 missed deadlines in 3 years
• 2 incidents where pre-alert triggered intervention 4-6 hours in, resolved before deadline without stress

Generalising

Deadline-bound batch is a common pattern and still gets done ad-hoc. The rules repeat:

• Measure, don't estimate
• Parallelise at the level of independent units
• Pre-alerting on several time horizons
• Idempotent retry + checkpoint for resume
• Dry-run on a real data copy before every critical cycle
• Degradation strategy for edge cases

Payroll, month-end closes, regulatory reports, clearing cycles, billing runs — all deadline-bound batch. If you're in one of those and your pipeline lacks at least 4 of these 6, it's a question of when, not if, you miss a deadline.