A multi-company SaaS that schedules, tracks and reports on security officers across sites. Managers use a web console and officers use a mobile app, with the same data live on both. Scope: scheduling, attendance, leave, SIA compliance, licence deferrals, biometrics, invoicing, payroll and notifications.
- Backend at scale: 11 Django apps, 31 DRF ViewSets, and role-based access for officers, site managers and admins.
- Async work: 24 Celery task types and 12 Beat jobs for attendance polling, auto-checkout, licence expiry, compliance and SIA sync. CSV imports and offer processing moved off the request path, eliminating upload timeouts.
- Realtime delivery: Redis-backed SSE where each event gets a global id, goes into a per-channel replay buffer, then is published live. On reconnect a client subscribes first, replays everything after its Last-Event-ID, and skips duplicates by id. Firebase push covers 14 event types, with notification fan-out deduplicated per recipient and sent only after the database commit.
- Caching: signal-driven cache invalidation for frequently read officer and shift data.
- Production memory leak: APScheduler was starting inside each of the 3 Gunicorn workers, so every job ran three times and memory grew until instances had to be terminated. Moved 7 jobs to a singleton Celery Beat, capped Redis and per-worker Celery memory, chunked the SSE health refresh, and added per-process memory metrics in CloudWatch.
- Slow bulk scheduling: rewrote pattern-based generation to answer its validation checks
from one prefetch and write in
bulk_createbatches, taking a year of schedules from 8,879 → 53 queries and 12.7s → 311ms (41×). Re-running an already-generated range dropped from ~9s → 0.06s (thousands of queries down to 16) after moving it to Celery with a check that skips already-assigned tiles. Batches of up to 1,000 shift tiles run with Redis-backed progress polling, and the schedule table moved to yearly MySQL range partitions. → full write-up - Costly distance lookups: the officer list used while scheduling fetched distances from Google's Distance Matrix once per officer on every load, so Maps costs grew with every roster view. I redesigned it around a precomputed officer-to-site distance table, recalculated only when an officer's or site's location changes. Roster radius highlighting and carbon estimates now need 0 Maps calls, the list loads faster, and an estimated $1.25k–2.6k a month in Maps charges is avoided. Estimate: 150+ duties/day × 75–150 officers per list load × 30 days ≈ 340k–675k elements a month, at Google's list price of $5 per 1,000 (first 100k) and $4 per 1,000 after, less the $200 monthly credit Google offered at the time.
- Duplicate check-calls: moved the check-call engine onto a proper scheduler with latest-record filtering and select_related, ending duplicate firing and N+1 queries as patrol backlogs grew.
Owned production on AWS Elastic Beanstalk: Gunicorn, nginx, Celery worker and Beat, Redis, deploy hooks and CloudWatch. Wrote a pre-deploy script that stopped crashes during vertical scaling, and shipped fortnightly through GitHub Actions and S3/CloudFront.
- Team lead and co-tech lead for 8 engineers across backend, frontend, mobile and QA; created the leave management module (requests, approvals, entitlements, cover and escalations), and built the Vonage video-calling proof of concept that won client approval, then mentored and reviewed the engineer who integrated it.
- First responder for production incidents, wrote design docs, set PR review rules and CI quality gates, ran client demos, estimated work for management, and interviewed candidates.
The daily operational backbone for enterprise clients, backed by 1,200+ automated tests.