Loading Austin Makasare's portfolio

Architecture Journey — how a request travels through QuizBuzz

  1. You are looking at QuizBuzz. Every contest begins as a single entity. Everything else grows out of it.
  2. A participant joins. Registration is paid and verified before a single question is served. Money clears via the Razorpay webhook before any compute is spent — no free load on the system.
  3. The doors open. 10,000 WebSocket connections open at once, on one synchronized clock. Redis holds the shared live state, so a socket survives across any server instance.
  4. An answer is submitted. It locks idempotently, then hands off to a background queue. Heavy work runs async on BullMQ so the API stays under 100 ms during the peak.
  5. The system holds. Redis keeps the live state and every job; Postgres makes the result durable.
  6. Results, automatically. One worker fans out: score, leaderboard, certificate, and messaging — per participant.
  7. That’s the whole system.. Payments, real-time, queues, workers, infrastructure — assembled in front of you.

System components

Infrastructure scaling — from idle to 7,500 concurrent users

  1. Idle mode. Between contests, one small instance runs everything. Near-zero cost. A single t3.medium hosts backend, worker and a local Docker Redis — about $35/month.
  2. Registration opens. The first few hundred participants arrive. The load balancer comes online.
  3. 2,500 participants. Numbers climb toward the live contest. Utilisation rises but holds.
  4. The contest goes live. 7,000 connect at once. CPU pins past its threshold, latency balloons — under stress. WebSocket load is memory- and IO-bound, so the crunch shows up as heap + latency, not only CPU.
  5. Auto Scaling responds. CPU crosses 80%, the policy trips, and fresh t3.medium instances boot. The ALB spreads the load. Peak demand is known in advance from registrations, so capacity is pre-warmed — not reactively chased.
  6. Steady at scale. Eight instances, load balanced near 55%, latency flat — 7,500 live participants.
  7. Now you drive it. Take control: set the participant count, simulate a spike, kill an instance, or drop Redis — and watch it survive.

Architecture diagrams

← Back to work
BACKEND · REAL-TIME · INFRA · 2024–2026

QUIZBUZZ

// PROJECT

QUIZBUZZ

A multi-tenant SaaS platform for running large-scale, real-time, proctored online quiz contests. Built end-to-end solo — backend, frontend, infrastructure, CI/CD, load testing, and incident response — with an engineering target of 10,000 concurrent WebSocket users.

Organizations sign up, create branded contests, collect paid registrations, run a live timed quiz to thousands of simultaneous participants over WebSockets, auto-evaluate answers, publish a leaderboard, and issue certificates — with live facial detection proctoring throughout.

Node.jsTypeScriptExpressSocket.IOBullMQRedisPostgreSQLPrismaNext.jsAWSTerraformk6RazorpayDockerGitHub Actions
// THE CHALLENGE

SCALE WITHOUT COST.

The core constraint: support up to 10,000 simultaneous WebSocket connections during a live contest — which lasts 30 minutes to 2 hours — on a cost-conscious AWS footprint that costs near zero when no contest is running.

WORKLOAD SHAPE
Contest traffic is fundamentally spiky. Near-zero between events, then a hard burst of sustained WebSocket connections during a live quiz. Reactive autoscaling doesn't fit a scheduled, known-peak event — reactive tooling is too slow, and always-on capacity wastes money.
REAL-TIME CONSTRAINT
All 10,000 participants must receive the same events simultaneously — quiz start, time warnings, question delivery, force-submit — over persistent WebSocket connections that survive across server instances. State must be shared, not local.
INTEGRITY CONSTRAINT
A participant's in-progress answers must survive crashes, reconnects, and infrastructure mode switches. Zero data loss is the acceptance criterion. 'Probably fine' isn't good enough when someone has paid a registration fee.
// START HERE — WATCH IT ASSEMBLE

THE ARCHITECTURE JOURNEY.

The whole system, built in front of you. Scroll to fly the camera across the architecture as one real request travels through it — registration and payment, the real-time engine, background workers, and the infrastructure underneath. The detailed diagrams and decisions that follow are the proof.

// SCROLL TO LOAD ARCHITECTURE JOURNEY...
// PRODUCTION ENGINEERING — TAKE THE CONTROLS

FROM ONE USER TO TEN THOUSAND.

The Journey showed how QuizBuzz works. This shows how it survives. Scroll to watch a single idle instance meet a live contest — CPU pins, Auto Scaling launches capacity, the load balancer spreads it out — then take the controls and push the system yourself.

// SCROLL TO LOAD SIMULATOR...
// INFRASTRUCTURE

DUAL-MODE INFRA.

The central architectural bet: two completely different infrastructure stacks for two completely different load profiles.

IDLE MODE — ~$35/MONTH
A single EC2 instance (t3.small) runs everything: Next.js frontend, Express backend, BullMQ worker, and a local Docker Redis container. Handles admin work, registration, payment webhooks, and result pages between contests.
LIVE MODE — +$14–30/CONTEST DAY
Terraform spins up an ALB, an ElastiCache Redis replication group (primary + replica, HA), and an Auto Scaling Group of quiz-serving EC2 instances (2–10, sized by expected participant count). Route53 switches from an A record to an ALB ALIAS record.

Mode switching is orchestrated by two scripts: go-live.sh (Terraform apply → DNS propagation wait → health-check polling → smoke tests) and go-idle.sh (drain BullMQ queues → Terraform destroy live resources → verify DNS reverted).

The backend enforces strict module boundaries: every domain (contest, participant, quiz, payment, proctoring) has its own routes/controller/service/repository layer. Only primitive identifiers cross module boundaries — no cross-domain joins.

// VISUAL ARCHITECTURE

HOW IT'S BUILT.

Five diagrams covering every layer of the system — from AWS infrastructure to the WebSocket wire protocol to the database schema. All diagrams are generated from the actual source architecture documentation.

// ENGINEERING DECISIONS

WHAT I BUILT.

These are the engineering decisions that defined this project — each one with a concrete problem, the approach taken, and the measured outcome.

01

630× faster test-data seeding

SHIPPED
PROBLEM

Seeding 10,000 participant rows for load testing took 42 minutes — sequential Prisma upserts over an SSH tunnel at ~100ms latency per round-trip.

APPROACH

Rewrote as bulk INSERT ... VALUES (...), ... ON CONFLICT DO NOTHING via a raw pg.Pool (bypassing Prisma's raw-query layer, which has a parameter-binding bug at large VALUES tables). Batch size 500, 4 concurrent batches, index-deterministic registrationRef keys.

OUTCOME

42 minutes → 4.1 seconds. A 630× improvement. The key insight: for bulk operations, round-trip count dominates over query complexity.

02

Zero job loss across a dual-Redis switch

SHIPPED
PROBLEM

BullMQ jobs scheduled before a go-live switch were enqueued into the local Docker Redis. After the switch to ElastiCache, workers listened to a different Redis — jobs were stranded, invisible, and contest start events never fired.

APPROACH

Built redis-migrate.js — a DUMP/RESTORE migration tool (preserves types and TTLs) that copies every key between Redis instances in both directions, wired into both go-live.sh and go-idle.sh. Always executes inside the live backend container so it inherits the container's actual REDIS_HOST.

OUTCOME

Zero job loss across mode switches. The lesson: 'always remember to do X' is not a control — move the execution context so the system guarantees correctness.

03

Closing the auto-submit coverage gap

SHIPPED
PROBLEM

AUTO_SUBMIT produced only 140 submissions from a contest with 1,323 peak IN_QUIZ participants. The force-submit logic only read from the active Redis SET — participants whose connections were dropped by an OOM crash had their socket disconnect handlers skipped, landing them in the disconnected SET that AUTO_SUBMIT never checked.

APPROACH

Fixed handleTimeExpiry() to union both active and disconnected participant sets before submitting, batched in groups of 50 with Promise.allSettled so one failure can't abort the batch.

OUTCOME

Full submission coverage regardless of crash history. The principle: predict the symmetric bug once you've found the first instance of a failure class.

04

Debugging the Socket.IO protocol layer

SHIPPED
PROBLEM

Load test k6 client sent plain JSON over raw WebSockets. Socket.IO uses Engine.IO v4 framing on top of the WS transport — the server silently ignored every message. WS success rate: 50%. Admin live monitor showed 0 participants.

APPROACH

Manually verified the infrastructure layer with curl (confirmed clean 101 upgrade and EIO4 open packet). Diagnosed the bug as above-transport: reimplemented the k6 client as an explicit state machine speaking the correct EIO4 handshake, namespace CONNECT with auth, framed event packets, and ping/pong keepalive.

OUTCOME

WS success rate 50% → ~100%. 50/50 VUs appeared in the live monitor simultaneously. Lesson: a successful transport connection tells you nothing about the application protocol above it.

05

Rethinking autoscaling for WebSocket load

DESIGNED
PROBLEM

The ASG's scale-out policy triggered on CPU at 60%. WebSocket load is memory/IO-bound — CPU stays near 20% while heap climbs to 95%. The scaler sees 'no scaling needed' right up until an OOM crash causes a brief CPU spike. A new instance still needs ~7 minutes to become healthy — far too slow for a quiz with a hard start time.

APPROACH

Three options designed: (A) memory-percentage CloudWatch alarm as a more direct signal, (B) custom connection-count metric published from the backend itself, (C) pre-warm capacity from registered-participant count in go-live.sh before the event starts — most accurate since this workload's peak demand is always known in advance.

OUTCOME

Designed and documented. Pre-warming selected as primary approach, CloudWatch as defense-in-depth. Validated on the next full load-test cycle.

06

Auditing every rate limiter, not just the broken one

SHIPPED
PROBLEM

OTP rate limiter middleware was attached to the participant-login route, which doesn't use OTP. Under load testing, all k6 VUs shared a single source IP — 5 req/window instantly exhausted the limiter. A subsequent audit found a second bug: a separate limiter used windowMs: config.rateLimit.window without ×1000 conversion — giving it a 600ms window instead of 600 seconds.

APPROACH

Removed the limiter from the non-OTP route. Audited every rate limiter in the codebase (not just the one that triggered). Fixed the unit mismatch in two locations.

OUTCOME

Rate limiting correct across all routes. Principle: finding one instance of a bug class is the right moment to audit for siblings.

// LOAD TESTING

24 BUGS. DOCUMENTED.

A real, dated (June 2026) attempt to validate 10,000 concurrent WebSocket users against production AWS infrastructure. 24 distinct documented bugs across tooling, protocol, infrastructure, and architecture.

SEED TOOLING
Before a single test could run: ts-node not in the build, Prisma v7 requiring a new driver adapter, SSH tunnel IPv4/IPv6 mismatch, interactive transactions timing out over a 100ms tunnel, and a seeding script that took 42 minutes for 10,000 rows. Each fixed individually, root-cause first.
MEMORY VS CPU
At 500 concurrent WebSocket connections, Node.js hit its default ~512MB heap ceiling — not from the ~9MB of actual session data, but from socket objects, closures, Redis pub/sub subscriptions, and BullMQ state. Fixed by raising the heap ceiling and container memory limit. Lesson: WebSocket load is memory-bound, not CPU-bound.
INFRASTRUCTURE DRIFT
HTTPS listener and routing rules missing from Terraform state after an interrupted apply. ElastiCache attached to the wrong parameter group (volatile-lru instead of noeviction — would cause silent BullMQ job loss under memory pressure). Stale COOKIE_DOMAIN from a prior domain migration breaking auth on quiz-serving instances. Each found and fixed via direct code audit.
// RESULTS

WHAT WAS ACHIEVED.

7500
PEAK CONCURRENT IN-QUIZ
630×
SEED SPEED IMPROVEMENT
24
PRODUCTION BUGS DOCUMENTED

The gap between 7500 peak concurrent and the 10,000-user architectural target is documented explicitly — not glossed over. Outstanding: apply pre-warmed scaling, run from an external non-AWS network, re-validate the mode-switch migration end-to-end.

The full pipeline was confirmed working end-to-end: login → WebSocket join with EIO4 protocol → answer recording per question → auto-submit on timeout → evaluation worker → leaderboard.

// RETROSPECTIVE

WHAT I'D DO DIFFERENTLY.

DUAL-REDIS IS A SHARP EDGE
The idle/live dual-Redis design was the right call for cost, but 7 of 24 documented bugs trace directly to the infrastructure mode switch itself. The redis-migrate.js tool was the right permanent fix — but the lesson is that any system with 'two different backends depending on mode' needs automatic data migration at the switch boundary, not operational discipline.
AUTOSCALING FOR SCHEDULED WORKLOADS
Reactive CPU-based autoscaling is wrong for a workload where peak demand is knowable in advance. This system always knows registered-participant count before the contest starts — the right approach is pre-warming, not reacting. This is a design insight that applies to any system with scheduled, registration-gated bursts.
PROTOCOL BOUNDARIES MATTER
The Socket.IO EIO4 debugging story is the most portable lesson from this project: a successful TCP handshake and WebSocket upgrade tell you nothing about the application-protocol layer above them. Testing each layer independently — transport, framing, namespace, application — is the discipline that found a bug that would otherwise have looked like 'random connection failures.'