← /fixes
perf · 41× faster

Pattern-based bulk schedule generator

ProjectSecurity workforce management SaaS
TypePerformance
OriginAI-generated, shipped unverified
DjangoMySQLbulk_createquery profiling
TL;DR

An AI-generated schedule generator re-checked every rule against the database inside a per-date loop and wrote rows one at a time, so its cost grew with every schedule it created. I moved the checks out of the loop, answered them from one prefetch, and batched the writes. A year of schedules went from 8,879 → 53 queries and 12.7s → 311ms (41×), with output proven identical to the original.

What was broken
  • For roughly 1,460 date × slot iterations per shift position, the loop re-queried licences, certificates, leave, bank holidays, conflicts, duplicates and working-time rules, plus holiday calendars, break times and status lookups.
  • Almost all of that was the same answer every iteration, or answerable from a single read.
  • Every schedule, officer assignment and readiness record was written with its own .create() or get_or_create(), so database round-trips grew linearly with the number of schedules.
The fix
  • Validation caching: static qualification checked once per tile; only expiry-sensitive checks stay per date.
  • Hoisting: holiday calendar, readiness configs, break times and status types loaded once per call.
  • Prefetch + in-memory maps: existing schedules and assignments become dict/set lookups, updated as new rows are queued.
  • Batched writes: bulk_create in batches of 200, flushed at date and config boundaries, with a per-row savepoint fallback so one bad row can't sink a batch.
  • Shipped behind an opt-in flag next to the original, so both paths could be compared on the same input.
Database queries per run, by date range original rewrite
3k 6k 9k 30 days · original: 761 queries 30 days · rewrite: 46 queries 76146 30 days23 schedules 90 days · original: 2,219 queries 90 days · rewrite: 46 queries 2,21946 90 days68 schedules 180 days · original: 4,385 queries 180 days · rewrite: 49 queries 4,38549 180 days135 schedules 365 days · original: 8,879 queries 365 days · rewrite: 53 queries 8,87953 365 days274 schedules
View as table
RangeSchedulesOriginal queriesRewrite queries
30 days2376146
90 days682,21946
180 days1354,38549
365 days2748,87953

The original grows with every schedule it creates; the rewrite stays flat. Measured with Django's query capture on local MySQL, median of repeated runs, each in a rolled-back transaction.

How I proved it
  • Differential check: original and rewrite run on the same seed, each in a rolled-back savepoint, with every result field compared. Re-run after every change.
  • Stress test: two shift positions on the same site and slot over 250 days, larger than one batch, so the second config has to merge with rows the first already flushed. Expected 500 results; both paths matched.
  • Fault injection: forced every bulk_create to fail and a later row insert to fail; the fallback isolated the bad row (22 created, 1 error) and the counts stayed correct.
Latent bugs found on the way
  • Crash on first call: a local import shadowed a module-level name, raising UnboundLocalError before any work happened.
  • Wrong model: the generated code wrote fields to a model they didn't belong to.
  • Every officer treated as relief staff: a check tested for an attribute that never exists, so it was always true.
Result

8,879 → 53 queries, 12.7s → 311ms median for a year of schedules, cost now flat as the range grows, and 3 latent bugs fixed before they reached users.

full case study →Security workforce management SaaS
Want to talk about this fix? → get in touch