Loading Austin Makasare's portfolio

Architecture Journey — how a request travels through QuizBuzz

  1. You are looking at QuizBuzz. Every contest begins as a single entity. Everything else grows out of it.
  2. A participant joins. Registration is paid and verified before a single question is served. Money clears via the Razorpay webhook before any compute is spent — no free load on the system.
  3. The doors open. 10,000 WebSocket connections open at once, on one synchronized clock. Redis holds the shared live state, so a socket survives across any server instance.
  4. An answer is submitted. It locks idempotently, then hands off to a background queue. Heavy work runs async on BullMQ so the API stays under 100 ms during the peak.
  5. The system holds. Redis keeps the live state and every job; Postgres makes the result durable.
  6. Results, automatically. One worker fans out: score, leaderboard, certificate, and messaging — per participant.
  7. That’s the whole system.. Payments, real-time, queues, workers, infrastructure — assembled in front of you.

System components

Infrastructure scaling — from idle to 7,500 concurrent users

  1. Idle mode. Between contests, one small instance runs everything. Near-zero cost. A single t3.medium hosts backend, worker and a local Docker Redis — about $35/month.
  2. Registration opens. The first few hundred participants arrive. The load balancer comes online.
  3. 2,500 participants. Numbers climb toward the live contest. Utilisation rises but holds.
  4. The contest goes live. 7,000 connect at once. CPU pins past its threshold, latency balloons — under stress. WebSocket load is memory- and IO-bound, so the crunch shows up as heap + latency, not only CPU.
  5. Auto Scaling responds. CPU crosses 80%, the policy trips, and fresh t3.medium instances boot. The ALB spreads the load. Peak demand is known in advance from registrations, so capacity is pre-warmed — not reactively chased.
  6. Steady at scale. Eight instances, load balanced near 55%, latency flat — 7,500 live participants.
  7. Now you drive it. Take control: set the participant count, simulate a spike, kill an instance, or drop Redis — and watch it survive.

Architecture diagrams

← Back to work
BACKEND · REAL-TIME · INFRA · 2024–2026

QUIZBUZZ

// PROJECT

QUIZBUZZ

A multi-tenant SaaS platform for running large-scale, real-time, proctored online quiz contests. Built end-to-end solo — backend, frontend, infrastructure, CI/CD, load testing, and incident response — with an engineering target of 10,000 concurrent WebSocket users.

Organizations sign up, create branded contests, collect paid registrations, run a live timed quiz to thousands of simultaneous participants over WebSockets, auto-evaluate answers, publish a leaderboard, and issue certificates — with live facial detection proctoring throughout.

Node.jsTypeScriptExpressSocket.IOBullMQRedisPostgreSQLPrismaNext.jsAWSTerraformk6RazorpayDockerGitHub Actions
// THE CHALLENGE

SCALE WITHOUT COST.

The core constraint: support up to 10,000 simultaneous WebSocket connections during a live contest — which lasts 30 minutes to 2 hours — on a cost-conscious AWS footprint that costs near zero when no contest is running.

WORKLOAD SHAPE
Contest traffic is fundamentally spiky. Near-zero between events, then a hard burst of sustained WebSocket connections during a live quiz. Reactive autoscaling doesn't fit a scheduled, known-peak event — reactive tooling is too slow, and always-on capacity wastes money.
REAL-TIME CONSTRAINT
All 10,000 participants must receive the same events simultaneously — quiz start, time warnings, question delivery, force-submit — over persistent WebSocket connections that survive across server instances. State must be shared, not local.
INTEGRITY CONSTRAINT
A participant's in-progress answers must survive crashes, reconnects, and infrastructure mode switches. Zero data loss is the acceptance criterion. 'Probably fine' isn't good enough when someone has paid a registration fee.
// START HERE — WATCH IT ASSEMBLE

THE ARCHITECTURE JOURNEY.

The whole system, built in front of you. Scroll to fly the camera across the architecture as one real request travels through it — registration and payment, the real-time engine, background workers, and the infrastructure underneath. The detailed diagrams and decisions that follow are the proof.

// SCROLL TO LOAD ARCHITECTURE JOURNEY...
// PRODUCTION ENGINEERING — TAKE THE CONTROLS

FROM ONE USER TO TEN THOUSAND.

The Journey showed how QuizBuzz works. This shows how it survives. Scroll to watch a single idle instance meet a live contest — CPU pins, Auto Scaling launches capacity, the load balancer spreads it out — then take the controls and push the system yourself.

// SCROLL TO LOAD SIMULATOR...
// INFRASTRUCTURE

DUAL-MODE INFRA.

The central architectural bet: two completely different infrastructure stacks for two completely different load profiles.

IDLE MODE — ~$35/MONTH
A single EC2 instance (t3.small) runs everything: Next.js frontend, Express backend, BullMQ worker, and a local Docker Redis container. Handles admin work, registration, payment webhooks, and result pages between contests.
LIVE MODE — +$14–30/CONTEST DAY
Terraform spins up an ALB, an ElastiCache Redis replication group (primary + replica, HA), and an Auto Scaling Group of quiz-serving EC2 instances (2–10, sized by expected participant count). Route53 switches from an A record to an ALB ALIAS record.

Mode switching is orchestrated by two scripts: go-live.sh (Terraform apply → DNS propagation wait → health-check polling → smoke tests) and go-idle.sh (drain BullMQ queues → Terraform destroy live resources → verify DNS reverted).

The backend enforces strict module boundaries: every domain (contest, participant, quiz, payment, proctoring) has its own routes/controller/service/repository layer. Only primitive identifiers cross module boundaries — no cross-domain joins.

// VISUAL ARCHITECTURE

HOW IT'S BUILT.

Five diagrams covering every layer of the system — from AWS infrastructure to the WebSocket wire protocol to the database schema. All diagrams are generated from the actual source architecture documentation.

// ENGINEERING DECISIONS

WHAT I BUILT.

These are the engineering decisions that defined this project — each one with a concrete problem, the approach taken, and the measured outcome.

01

Pay for 10,000 users only on contest day

SHIPPED
PROBLEM

Contests are scheduled, short and spiky. Running a load balancer, an auto-scaling fleet and a Redis cluster all month for a few hours of peak would be paying for idle capacity.

APPROACH

One Terraform variable switches modes. Idle is a single EC2 with Redis in Docker; live adds an ALB, an auto-scaling group and an ElastiCache primary + replica — created by go-live and torn down by go-idle.

OUTCOME

~70% lower infra cost. The peak is known before it arrives, so capacity can be decided in advance instead of chased.

02

One question cache, not ten thousand reads

SHIPPED
PROBLEM

At the start signal every participant asks for the same questions at once — a thundering herd against Postgres, at the worst possible moment.

APPROACH

The question set is cached once in Redis for the whole contest; each participant stores only their own shuffled order. The answer key is stripped before anything reaches a client.

OUTCOME

~2.7 KB per participant. Share what is identical, store per-user only what actually differs.

03

Scoring moved off the request path

SHIPPED
PROBLEM

Scoring answers, ranking a leaderboard and scoring proctoring inline would make every submit as slow as the slowest of them, exactly when everyone submits together.

APPROACH

A submit only enqueues a BullMQ job. Separate workers score answers (concurrency 50), aggregate proctoring (30), and trigger the leaderboard once every submission is evaluated.

OUTCOME

< 100 ms API at peak. The request path records intent; everything heavy happens behind a queue.

04

Exactly-once submissions across instances

SHIPPED
PROBLEM

A reconnect, a retry and the contest-end auto-submit can all try to submit the same participant at once — possibly on different instances.

APPROACH

A Redis SET NX lock per participant (10 s TTL) guards submission; only the holder writes and enqueues scoring. A 5 s lock does the same for the question fetch.

OUTCOME

0 answers scored twice. When work can arrive twice, make the second arrival a no-op.

05

Heap exhaustion at 500 sockets an instance

SHIPPED
PROBLEM

Both auto-scaled instances died together at 1,000 and 2,000 users: “JavaScript heap out of memory”. Redis state was only ~18.6 KB a participant — the heap was full of socket objects and closures.

APPROACH

Raised the heap to 1.5 GB and the container limit to 2 GB in the launch template — then found running instances had booted from the old template and never got the fix.

OUTCOME

512 MB → 1.5 GB heap. Measure where memory actually lives before raising a limit, and check the fix reached the machines.

06

A connection pool sized for a quiet day

SHIPPED
PROBLEM

1,000 users arriving in 30 s is ~33 logins a second; a pool of five connections queued instantly and requests timed out.

APPROACH

Pool raised to 20 (minimum 5) with a 10 s query timeout, and the load ramp slowed to a realistic 120 s — eight arrivals a second.

OUTCOME

pool 5 → 20. Arrival rate and pool size are one variable, not two.

07

Redis could silently evict the job queue

SHIPPED
PROBLEM

Workers kept warning about the eviction policy. The no-eviction parameter group was defined in Terraform but never attached, so under pressure Redis could drop BullMQ keys and lose jobs.

APPROACH

Attached the parameter group to the replication group, so every live cluster is created with noeviction.

OUTCOME

volatile-lru → noeviction. A resource defined in code is not a resource applied — read the warning.

08

Rethinking autoscaling for WebSocket load

DESIGNED
PROBLEM

The ASG's scale-out policy triggered on CPU at 60%. WebSocket load is memory/IO-bound — CPU stays near 20% while heap climbs to 95%. The scaler sees 'no scaling needed' right up until an OOM crash causes a brief CPU spike. A new instance still needs ~7 minutes to become healthy — far too slow for a quiz with a hard start time.

APPROACH

Three options designed: (A) memory-percentage CloudWatch alarm as a more direct signal, (B) custom connection-count metric published from the backend itself, (C) pre-warm capacity from registered-participant count in go-live.sh before the event starts — most accurate since this workload's peak demand is always known in advance.

OUTCOME

Designed and documented. Pre-warming selected as primary approach, CloudWatch as defense-in-depth. Validated on the next full load-test cycle.

09

Closing the auto-submit coverage gap

SHIPPED
PROBLEM

AUTO_SUBMIT produced only 140 submissions from a contest with 1,323 peak IN_QUIZ participants. The force-submit logic only read from the active Redis SET — participants whose connections were dropped by an OOM crash had their socket disconnect handlers skipped, landing them in the disconnected SET that AUTO_SUBMIT never checked.

APPROACH

Fixed handleTimeExpiry() to union both active and disconnected participant sets before submitting, batched in groups of 50 with Promise.allSettled so one failure can't abort the batch.

OUTCOME

Full submission coverage regardless of crash history. The principle: predict the symmetric bug once you've found the first instance of a failure class.

10

Zero job loss across a dual-Redis switch

SHIPPED
PROBLEM

BullMQ jobs scheduled before a go-live switch were enqueued into the local Docker Redis. After the switch to ElastiCache, workers listened to a different Redis — jobs were stranded, invisible, and contest start events never fired.

APPROACH

Built redis-migrate.js — a DUMP/RESTORE migration tool (preserves types and TTLs) that copies every key between Redis instances in both directions, wired into both go-live.sh and go-idle.sh. Always executes inside the live backend container so it inherits the container's actual REDIS_HOST.

OUTCOME

Zero job loss across mode switches. The lesson: 'always remember to do X' is not a control — move the execution context so the system guarantees correctness.

11

Late joiners stuck in the waiting room

SHIPPED
PROBLEM

Anyone joining after the contest went live was parked in the waiting room for a start signal that had already fired once and was gone.

APPROACH

Joining now checks the contest status; if it is already live, the participant starts the quiz directly instead of waiting.

OUTCOME

START_IMMEDIATELY. One-shot events need a path for whoever arrives after them.

12

Login worked, then every request was a 401

SHIPPED
PROBLEM

Admin login returned 200, then /me and /refresh failed. The auto-scaled instances set cookies for the old domain — and the load balancer was routing admin logins to them.

APPROACH

Fixed the domain in both instance templates, patched the running instances, and traced why the request landed on that fleet at all.

OUTCOME

200 → 401. Know which machine actually served the request before debugging the one you expected.

13

Debugging the Socket.IO protocol layer

SHIPPED
PROBLEM

Load test k6 client sent plain JSON over raw WebSockets. Socket.IO uses Engine.IO v4 framing on top of the WS transport — the server silently ignored every message. WS success rate: 50%. Admin live monitor showed 0 participants.

APPROACH

Manually verified the infrastructure layer with curl (confirmed clean 101 upgrade and EIO4 open packet). Diagnosed the bug as above-transport: reimplemented the k6 client as an explicit state machine speaking the correct EIO4 handshake, namespace CONNECT with auth, framed event packets, and ping/pong keepalive.

OUTCOME

WS success rate 50% → ~100%. 50/50 VUs appeared in the live monitor simultaneously. Lesson: a successful transport connection tells you nothing about the application protocol above it.

14

Recovering drifted Terraform without touching live traffic

SHIPPED
PROBLEM

An interrupted apply left the HTTPS listener and its routing rules out of state — the site timed out — while one DNS record from that run already existed in AWS.

APPROACH

Read the plan first, imported the existing record instead of recreating it, and applied only once the diff showed six additions and nothing destroyed.

OUTCOME

6 add · 0 destroy. Understand the diff before you apply it, especially on infrastructure serving traffic.

15

Auditing every rate limiter, not just the broken one

SHIPPED
PROBLEM

OTP rate limiter middleware was attached to the participant-login route, which doesn't use OTP. Under load testing, all k6 VUs shared a single source IP — 5 req/window instantly exhausted the limiter. A subsequent audit found a second bug: a separate limiter used windowMs: config.rateLimit.window without ×1000 conversion — giving it a 600ms window instead of 600 seconds.

APPROACH

Removed the limiter from the non-OTP route. Audited every rate limiter in the codebase (not just the one that triggered). Fixed the unit mismatch in two locations.

OUTCOME

Rate limiting correct across all routes. Principle: finding one instance of a bug class is the right moment to audit for siblings.

// LOAD TESTING

24 BUGS. DOCUMENTED.

A real, dated (June 2026) attempt to validate 10,000 concurrent WebSocket users against production AWS infrastructure. 24 distinct documented bugs across tooling, protocol, infrastructure, and architecture.

SEED TOOLING
Before a single test could run: ts-node not in the build, Prisma v7 requiring a new driver adapter, an SSH tunnel IPv4/IPv6 mismatch, and interactive transactions timing out over a 100ms tunnel. Each fixed individually, root-cause first.
MEMORY VS CPU
At 500 concurrent WebSocket connections, Node.js hit its default ~512MB heap ceiling — not from the ~9MB of actual session data, but from socket objects, closures, Redis pub/sub subscriptions, and BullMQ state. Fixed by raising the heap ceiling and container memory limit. Lesson: WebSocket load is memory-bound, not CPU-bound.
INFRASTRUCTURE DRIFT
HTTPS listener and routing rules missing from Terraform state after an interrupted apply. ElastiCache attached to the wrong parameter group (volatile-lru instead of noeviction — would cause silent BullMQ job loss under memory pressure). Stale COOKIE_DOMAIN from a prior domain migration breaking auth on quiz-serving instances. Each found and fixed via direct code audit.
// RESULTS

WHAT WAS ACHIEVED.

7500
PEAK CONCURRENT IN-QUIZ
0
ANSWERS SCORED TWICE
24
PRODUCTION BUGS DOCUMENTED

The gap between 7500 peak concurrent and the 10,000-user architectural target is documented explicitly — not glossed over. Outstanding: apply pre-warmed scaling, run from an external non-AWS network, re-validate the mode-switch migration end-to-end.

The full pipeline was confirmed working end-to-end: login → WebSocket join with EIO4 protocol → answer recording per question → auto-submit on timeout → evaluation worker → leaderboard.

// RETROSPECTIVE

WHAT I'D DO DIFFERENTLY.

DUAL-REDIS IS A SHARP EDGE
The idle/live dual-Redis design was the right call for cost, but 7 of 24 documented bugs trace directly to the infrastructure mode switch itself. The redis-migrate.js tool was the right permanent fix — but the lesson is that any system with 'two different backends depending on mode' needs automatic data migration at the switch boundary, not operational discipline.
AUTOSCALING FOR SCHEDULED WORKLOADS
Reactive CPU-based autoscaling is wrong for a workload where peak demand is knowable in advance. This system always knows registered-participant count before the contest starts — the right approach is pre-warming, not reacting. This is a design insight that applies to any system with scheduled, registration-gated bursts.
PROTOCOL BOUNDARIES MATTER
The Socket.IO EIO4 debugging story is the most portable lesson from this project: a successful TCP handshake and WebSocket upgrade tell you nothing about the application-protocol layer above them. Testing each layer independently — transport, framing, namespace, application — is the discipline that found a bug that would otherwise have looked like 'random connection failures.'