Running APIs: Gateways, Mesh, Rate Limits, Canaries

How to run an API in production: gateway pipelines, service mesh, rate limit algorithms and quotas, SLOs and error budgets, canary releases and observability.

Running APIs in production: common questions

What is the difference between an API gateway and a service mesh?

A gateway handles north–south traffic at the edge: API keys, user tokens, quotas and routing for outside callers. A mesh handles east–west traffic between your own services with sidecar proxies that add mTLS, timeouts, retries and traffic splitting.

What is the difference between a rate limit and a quota?

A rate limit, such as 10 requests a second, protects the system from bursts. A quota, such as 100,000 calls a month, is part of the pricing plan you sell, and the same metering counter becomes the bill.

Related chapters

Keep learning

Overview

TLS termination Rate limit / quota Request transform Response cache John books a seat POST /bookings\nAuthorization: Bearer eyJhbGci… (John) No token POST /bookings\n(no Authorization header) Partner over quota GET /events?city=pune\nX-Api-Key: pk_live_… (TixPal, 100,001st call this month) Typo in path GET /evnets/123\nAuthorization: Bearer eyJhbGci… Injection attempt GET /events?city=pune' OR '1'='1 Popular event page GET /events/123 (asked 2,000 times a minute) never reached Without TLS at the edge the token crosses the internet readable. An anonymous booking reached the bookings service. Unless the service checks again, it books a seat for nobody. TixPal keeps calling past its plan, on your servers, for free. Routing is the one plugin a gateway cannot run without. gateway itself no route table: the gateway knows no service, so nothing gets through off: the service must parse the JWT itself plugin off, request passes unchecked gateway’s WAF rule matched a SQL injection pattern in ?city no attack signature found gateway’s auth check POST /bookings needs a token public route, no token needed API key belongs to partner TixPal JWT signature valid, user u_77 gateway’s quota check monthly quota of 100,000 used up counter under the limit gateway’s router no service owns /evnets adds X-User-Id and X-Request-Id headers gateway’s cache cache hit, served by the gateway in 2 ms POST is not cacheable miss, forwarded access log line written, rejections included off: this request leaves no trace No log line, so a spike of these would be invisible. Every one of the 2,000 calls a minute hits the events service. bookings service events service Lab: one request through the gateway pipeline Plugins switched on Pipeline trace The events service never saw it. Only safe for public data: a shared cache must never serve one user’s bookings to another. Every check passed and the request did its job. Stopped at the cheapest possible point: the services behind never spent a cycle on it. (15).fill(0.95), ...Array 25 at once Inside a single window all three behave the same: the first 10 pass, the other 15 get 429. Steady 8 a second for 3 s Under the limit nobody is refused. The algorithms only differ at the edges, which is exactly where bots and retry loops live. (10).fill(0), ...Array Fixed window and sliding log refuse the whole second burst until the second is over. The bucket refilled 5 tokens in half a second, so 5 get in: smoother for a real user, and the long-run rate is still 10 a second. Math.floor(x) === Math.floor(t)).length t - 1 + 1e-9).length Fixed window counter per clock second Sliding log count of the last 1 s Token bucket 10 tokens, +1 every 100 ms Lab: the same limit, 10 requests a second, three algorithms most in any 1 s Lab: canary for bookings v2 · gates: errors under 1%, p99 under 400 ms v2 is healthy Inject a regression into v2, now or mid-rollout rolled back Start: send 5% to v2 Metrics look fine: next step traffic on v2 now v2 error rate (v1: 0.3%) v2 p99 latency (v1: 210 ms) At 100% everyone met it, which is why v1 is kept warm for a day. ) : stage === STAGES.length ? ( v2 serves everyone Each step held for its analysis window and every gate stayed green. v1 stays around for a day as the rollback target. The canary is compared with v1 serving the same minutes of traffic, not with last week, so a busy evening doesn’t look like a regression. Its bug: payments is called without a timeout, so bookings hang and fail.

Running APIs: gateways, mesh, quotas, releases and signals

A well-designed API still has to survive production: a front door that checks every call, a safe network between services, limits that protect the system and define the product, releases that can fail small, and signals that tell you something is wrong before John tweets about it.

Two kinds of traffic: north–south and east–west

North–south traffic crosses the edge: John’s phone, Asha’s venue dashboard or a partner’s server calling in. East–west traffic stays inside: the bookings service calling payments and seats. Each has its own tool. North–south: one public call enters through the gateway. John’s app POST /bookings Load balancer L4, spreads connections API gateway auth, quota, routing bookings service East–west: the same booking fans out inside, proxy to proxy. sidecar proxy mTLS, retries, timeout checks caller identity API gateway (edge) Faces strangers. Cares about API keys, user tokens, plans and quotas, public paths and versions. Usually a few shared instances. Service mesh (inside) Faces your own services. Cares about service identity, encryption, retries, timeouts and traffic splits. One proxy next to every service.

The gateway is a pipeline of plugins

A gateway runs every request through an ordered chain. Cheap checks go first, so a bad request dies before it costs anything: decrypt, look for attack patterns, identify the caller, count, then route. Turn plugins off and watch where each request ends up. Two lessons hide here. Order matters: rate limiting after authentication can count per user or per key, while anonymous traffic needs a per-IP limit placed earlier. And the gateway is a first filter, not the only one: services still check ownership ( who may do what ), because one misconfigured route is all it takes.

Gateway pitfalls, and the backend for frontend

Gateway as ESB logic creep First a field rename, then “refund if the event was cancelled”. Soon business rules live in gateway scripts that no service team owns or tests. Keep it to cross-cutting concerns. Loopback calls Bookings calls payments through the public gateway because the gateway knows where everything is. Traffic leaves and re-enters the network: extra latency, egress cost, and every internal call now depends on the edge. Turtles all the way down layered gateways Each team adds its own gateway behind the company one. A request crosses three gateways, each with its own timeouts and auth. Debugging means reading three configs. Single point of failure blast radius Every call goes through it, so a bad config push takes all of ticketing down at once. Run several instances across zones and roll out config like code, with a canary. A backend for frontend (BFF) is a small API owned by a client team, such as the mobile app. It calls events, seats and bookings and returns exactly one screen’s data. That is product logic, so it belongs in a BFF service and not in the shared gateway. The aggregation trade-offs are in network-efficient APIs

Service mesh: a proxy next to every service

mTLS everywhere Each service gets a certificate. Payments can tell the caller is bookings, and the traffic is encrypted, without a line of app code. Timeouts, retries, circuit breaking The proxy gives up after 800 ms, retries on another instance, and stops calling a payments instance that keeps failing. Traffic splitting Send 5% of seat lookups to seats v2. This is how the canary in section 7 works east–west. Uniform telemetry Every proxy reports rate, errors and latency for every caller and callee pair, in one format, whatever language the service is written in. Only bookings may call payments. Asha’s reporting job may read bookings but not write them. The cost An extra hop per call, a proxy per pod, and a control plane to run. For five services, a library and a load balancer may be enough. The famous mesh failure is the retry storm. Payments slows down. Each layer above it tries three times, so one tap becomes many calls, all landing on the service that is already struggling. John taps “Pay” the app, the gateway and bookings each try 3 times calls reach payments for one tap idempotency keys

Rate limits, quotas, plans and SLAs

These four sound alike but answer different questions. A rate limit per second protects the system from bursts. A quota per month is the product you sell. A pricing plan bundles both with a price, and an SLA is the promise, with penalties, that comes with the paid plans. rate limit monthly quota Starter (TixPal today) 10 req/s 100,000 calls none, best effort 50 req/s 5 million calls ₹25,000 a month 99%, credits if missed 99.5%, credits and named support Metering is the hidden part: every call is counted per key, so the counter that enforces the quota is also the bill. Counting must survive a gateway restart, and a client should be able to ask how much it has left. Now the algorithms. Same limit, different behaviour at the edges: Too many requests Starter plan allows 10 requests a second. Try again in 30 s.

SLOs and error budgets: how good is good enough

of POST /bookings succeed, over 30 days p99 latency for POST /bookings 43 min of full outage that 99.9% allows per 30 days failed bookings allowed out of 10 million An SLI is what you measure (share of successful bookings), an SLO is the target you hold yourself to (99.9%), and the SLA is the looser promise in the contract (99.5% for Enterprise above): promise less than you aim for, so one bad day doesn’t cost credits. Measure at the gateway, as callers see it, and count 5xx and timeouts as failures, not 4xx caused by the client. downtime per 30 days 7.2 hours 43.2 minutes 21.6 minutes 4.3 minutes The gap between 100% and the SLO is the error budget, and it turns arguments into arithmetic. Budget left: ship faster, try the risky migration. Budget burnt: freeze features and fix reliability. Track p99, not the average: an average of 120 ms can hide the 1 in 100 bookings that takes four seconds, and on a busy sale that is thousands of people.

Releases: deploy is not release

Put bookings v2 on servers. It runs, takes health checks, and serves nobody yet. A technical step you can repeat ten times a day. Let users reach v2: 5% of traffic, one venue, or everyone. A business decision you can undo without redeploying. Feature flag in the code The new seat-map endpoint ships dark and is switched on for Asha’s venue first. Doubles as a kill switch. Delete old flags, or they pile up as untested branches. at the router A small share of real traffic goes to v2 while its metrics are compared with v1. Rollback is automatic when a gate fails. two environments Traffic mirroring The gates must be automatic, because nobody watches dashboards at 2 a.m., and they must compare like with like. Schema and contract changes need extra care: both versions run side by side during every one of these strategies, so changes follow the expand-then-contract rules from evolving an API

Signals: metrics, traces, logs and health

Requests per second for each endpoint. POST /bookings at 40/s on a Tuesday, 900/s when a concert opens. Share of 5xx and timeouts per endpoint. Watch 4xx separately: they are the client’s fault, but a surge still means something. Latency as p50, p95 and p99 per endpoint, never one average for the whole API. POST /bookings payments timed out That second block is a structured log: JSON with fixed fields instead of free text, so you can search “all 504s for trace 4bf9…” across every service. Never log tokens, card numbers or full request bodies. OpenTelemetry is the vendor-neutral standard for producing all three signals. Liveness: am I alive? GET /livez checks only the process itself. Failing means “restart me”. Never check the database here, or one database blip restarts every pod at once. Readiness: can I take traffic? GET /readyz checks that dependencies and caches are ready. Failing means “take me out of the load balancer for now”, without a restart. A wall of 401s from one partner key Someone is guessing tokens or replaying old ones, or TixPal’s key leaked. Alert, then rotate the key. 404s walking through booking IDs Booking 1001, 1002, 1003… from one user is an object-level authorization probe. See the OWASP chapter. 404s on paths you never had Valid calls, inhuman pace Each of those is a 2xx or 4xx, so an error-rate alert stays green. Alert on the shape (401s per key, distinct IDs per user per minute) and link the response to the gateway: throttle the key, demand a fresh login, or block. The attack patterns themselves are in the OWASP API Top 10

The interview line

You now have every piece. The last chapter puts them together: a full design review of the ticketing API, from resources to rollout. Spotify sells access by tier: development mode (5 users) vs extended quota mode, with a rolling 30-second rate window. Spotify quota modes GitHub exposes x-ratelimit-remaining and x-ratelimit-reset on every response. Rate-limit headers compared

More courses

Jobs, linked to lessons — Browse jobs where every requirement links to the chapter that teaches it. See how ready you are, how long the gap takes to close, and which listings look stale.
Match a job post to courses — Paste a job post. Each skill is matched to a track here and put in learning order, foundations first.
React from open source with patterns — Patterns extracted from Material UI, Radix, React Aria, React Hook Form, Redux Toolkit, TanStack, and Next.js — not slides about hooks.
Database in depth — How a query becomes work. Indexes, isolation, locks, and the plans that explain why a page is slow — not an ORM tour.
Build a real API with NestJS — One ticket-booking API, built in order: a plain Node server first, then NestJS modules, validation, a real database, and the features real clients lean on.
Microservices — Microservice patterns one diagram at a time: how to split a monolith, how services talk, and how data stays consistent without distributed transactions.
Kafka — A beginner-to-senior path through event logs, reliable delivery, production operations, and disaster recovery.
AI engineering for software engineers — Eight free reads, in order, for software engineers moving into AI: when to build an agent, how to build one, how to test it, and how the model works underneath.
Frontend system design — How a UI is sliced when it is a product, not a page: ownership of state, cache, routes, and what happens when the network lies.
TypeScript in depth — The type system as a design tool. Inference, narrowing, generics, and the errors that mean the model is wrong — not a catalog of utility types.
CSS in depth — See the box, then the page. Visual chapters from the box model through Flexbox surprises, Grid, selectors, and units, then backgrounds, shapes, and visual effects, each with a playground.
Accessibility — Foundations first (WCAG in plain words, semantic HTML, lists, alt text), then hands-on HTML labs, a full pass of WAI-ARIA 1.3, and a final fix-the-page challenge.
Docker — What an image is, why layers cache, and how containers find each other on a network.
Redis — Fifteen Redis patterns, one picture each: cache, session, lock, counter, rate limiter, queue, and when Redis is the wrong tool.
Kubernetes — Kubernetes from zero to operators: why it exists, Pods, Deployments and Services, then configuration, networking, storage, scheduling, scaling and security.