Design Meta Serverless Platform: A System Design Interview

The Interview Scenario

Interviewer: "Design a serverless platform like AWS Lambda or Meta's internal XFaaS. Target: handle millions of function invocations per second."

Candidate: "Interesting! Let me understand the requirements first."

---

Phase 1: Requirements Clarification (5 minutes)

Candidate: "Key questions:

1. **Function types** - Short HTTP handlers or long-running jobs?
2. **Execution environment** - Containers, VMs, or language-specific runtimes?
3. **Triggers** - HTTP, events, scheduled?
4. **Scale target** - What's our invocations per second goal?
5. **Multi-tenancy** - Shared infrastructure across customers?"

Interviewer: "Design for:

• Short functions (typically <30 seconds)
• Container-based isolation
• HTTP and event triggers
• Target: 10+ million invocations per second
• Multi-tenant with strict isolation
• Cold start under 500ms"

Candidate: "Got it. Additional requirements I'll assume:

• **Pay-per-use** - Bill only for actual execution time
• **Auto-scaling** - 0 to millions without user intervention
• **High availability** - 99.99% uptime
• **Security** - Complete isolation between tenants

Interviewer: "Yes, those are critical."

---

Phase 2: Back-of-Envelope Calculations (5 minutes)

Candidate: "Let's size this:

Invocation volume:

• Target: 10 million invocations/second
• Average duration: 200ms
• Concurrent executions: 10M × 0.2s = **2 million concurrent containers**

Worker capacity:

• Each worker node: 64 cores, 256GB RAM
• Functions average 256MB RAM, 0.5 vCPU
• Functions per worker: ~200 concurrent
• Workers needed: 2M / 200 = **10,000 worker nodes**

Scheduler throughput:

• 10M scheduling decisions per second
• Single scheduler bottleneck at ~100K/s
• Need: **100+ scheduler instances** (partitioned)

Cold start impact:

• 10% cold starts = 1M/second new containers
• Container start time: 200ms (optimized)
• Container creation rate: 1M/second = significant challenge"

Interviewer: "What's the hardest scaling challenge?"

Candidate: "Cold starts at scale. Creating 1 million containers per second is extremely hard. We need aggressive pre-warming strategies."

---

Phase 3: High-Level Architecture (10 minutes)

Candidate: "Here's the architecture:"


Interviewer: "Walk me through an invocation."

Candidate: "Step by step:

1. **Request arrives** at API Gateway with function identifier
2. **Auth & rate limiting** - Verify API key, check quotas
3. **Scheduler receives request** - Consistent hashing routes to shard
4. **Scheduler checks placement cache:**
5. Warm instance available? → Route directly
6. No warm instance? → Cold start path
7. **Cold start (if needed):**
8. Scheduler requests container from Worker pool
9. Worker pulls function image (or uses cached)
10. Container initialized with runtime
11. Function code loaded
12. **Execute function** - Worker runs function, streams response
13. **Return result** - Response back through Gateway
14. **Container kept warm** - Available for next invocation

Key optimization: Keep containers warm for 5-15 minutes after last invocation."

---

Phase 4: Deep Dive - Cold Start Optimization (10 minutes)

Interviewer: "Cold start is critical. How do you get it under 500ms?"

Candidate: "This is the hardest problem. Multiple strategies:

Strategy 1: Pre-warming

```

// Predictive model based on historical patterns

function predictCapacity(functionId, timeOfDay) {

// ML model predicting invocations per minute

// Pre-warm containers before expected traffic spike

}

// Keep pool of 'generic' warm containers

// Initialize with runtime, inject function code on demand

```

Strategy 2: Snapshot/Restore (Firecracker approach)

```

Traditional cold start:

1. Create container (100ms)
2. Start runtime (200ms)
3. Load function code (50ms)
4. Initialize dependencies (100-500ms)

Total: 450-850ms

Snapshot approach:

1. Pre-create snapshot of initialized function
2. On cold start: restore from snapshot (50ms)
3. Resume execution

Total: ~50-100ms

```

AWS Firecracker does this with microVMs!

Strategy 3: Tiered container pools

```

Pool Hierarchy:

├── Hot Pool (function-specific warm containers)

│ └── Instant routing, 0ms overhead

├── Warm Pool (runtime-initialized, no function)

│ └── Just load function code, ~100ms

├── Cold Pool (empty containers)

│ └── Full initialization, ~300ms

└── Creation (new container)

└── Full cold start, ~500ms

```

Interviewer: "How do you decide how many to pre-warm?"

Candidate: "Prediction model with multiple signals:

1. **Historical patterns** - Same time yesterday/last week
2. **Recent trend** - Invocations in last 5 minutes
3. **External signals** - Marketing campaigns, launches
4. **User hints** - Provisioned concurrency setting

```

// Simple exponential smoothing with time-of-day seasonality

predicted = α × recent + (1-α) × historical_same_time

// Buffer = predicted × (1 + safety_margin)

// safety_margin = 0.2 for normal functions

// safety_margin = 0.5 for spiky functions

```

Cost trade-off: Pre-warming costs money (idle resources) but improves latency. Let users choose their trade-off with 'provisioned concurrency' setting."

---

Phase 5: Multi-tenancy & Isolation (8 minutes)

Interviewer: "How do you isolate tenants on shared infrastructure?"

Candidate: "Security isolation is non-negotiable:

Level 1: Container Isolation

```

Each function runs in:

• Separate container with own filesystem
• Resource limits (CPU, memory, network)
• Seccomp profiles limiting syscalls
• Read-only root filesystem

```

Level 2: Network Isolation

```

┌─────────────────────────────────────┐

│ Worker Node │

│ ┌─────────────────────────────┐ │

│ │ Virtual Network (per tenant)│ │

│ │ ┌─────┐ ┌─────┐ ┌─────┐ │ │

│ │ │Fn 1 │ │Fn 2 │ │Fn 3 │ │ │

│ │ └─────┘ └─────┘ └─────┘ │ │

│ └─────────────────────────────┘ │

│ ┌─────────────────────────────┐ │

│ │ Virtual Network (tenant B) │ │

│ │ ┌─────┐ ┌─────┐ │ │

│ │ │Fn A │ │Fn B │ │ │

│ │ └─────┘ └─────┘ │ │

│ └─────────────────────────────┘ │

└─────────────────────────────────────┘

```

Level 3: Noisy Neighbor Prevention

```

// Per-tenant resource quotas

tenant_limits:

cpu_cores: 1000

memory_gb: 2000

concurrent_executions: 10000

invocations_per_second: 100000

// Fair scheduling within worker

// If Tenant A is CPU-bound, shouldn't starve Tenant B

// Use Linux cgroups v2 for CPU bandwidth limiting

```

Interviewer: "What about data isolation?"

Candidate: "Multiple layers:

1. **Memory** - Container isolation, no shared memory
2. **Storage** - Ephemeral /tmp per invocation, cleared after
3. **Environment** - Secrets injected at container start, encrypted at rest
4. **Logs** - Routed to tenant-specific streams
5. **Metrics** - Tagged with tenant ID, filtered at query time

For extra-sensitive workloads: dedicated worker pools (at premium price)."

---

Phase 6: Scheduler Design (5 minutes)

Interviewer: "How does scheduling work at 10M/second?"

Candidate: "Distributed scheduling is key:

Sharded Schedulers:

```

// Consistent hashing by function_id

scheduler_shard = hash(function_id) % num_schedulers

// Each scheduler handles ~100K invocations/second

// 100 schedulers = 10M/second capacity

```

Scheduler State:

```

// In-memory cache per scheduler

warm_instances: Map>

placement_cache: Map

function_metadata: Map

// Cached from central store, refreshed every second

```

Scheduling Decision Flow:

```

• Check warm_instances[function_id]
• If available: route to warm container (fast path)

• Check placement_cache[function_id]
• If exists: try preferred workers first

• Query Placement Service
• Find workers with capacity
• Prefer workers with cached function image
• Consider locality (same region as data)

• Create container on selected worker
• Update warm_instances cache

```

Load balancing across warm instances:

• Round-robin for equal distribution
• Weighted based on recent response times
• Power-of-two-choices: sample 2 random, pick least loaded"

---

Phase 7: Trade-offs (2 minutes)

Candidate: "Key decisions:

| Decision | Chose | Alternative | Trade-off |

|----------|-------|-------------|-----------|

| Isolation | Containers | VMs (Firecracker) | Speed vs security (VMs more secure but slower cold start) |

| Scheduling | Sharded | Centralized | Complexity vs scalability |

| Pre-warming | Predictive | Reactive | Cost vs latency |

| State | Stateless functions | Stateful | Simplicity vs capability |

What I'd add with more time:

1. Durable execution (checkpointing for long functions)
2. Step functions (workflow orchestration)
3. Edge deployment (functions at CDN edge)"

---

Key Interview Takeaways

1. **Cold start is THE challenge** - Spend time on optimization strategies
2. **Pre-warming vs cost** - It's a trade-off; give users control
3. **Sharded schedulers** - Single scheduler can't do 10M/s
4. **Multi-tenancy** - Isolation is non-negotiable; layers of defense
5. **Container pools** - Tiered pools balance latency and resource usage