Chapter 27 of 27 · Part 12: Hands-on: FlixTube

FlixTube and running in production

Chapters 10 to 12 of Bootstrapping Microservices: the finished FlixTube app, three ways to deploy it, keeping it healthy and reliable, scaling it, and moving an existing system toward microservices.

key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral

FlixTube: the finished application

Chapter 10 ships the complete FlixTube. It reuses the services from earlier chapters and adds a gateway and an upload service. A typical microservice has the same layout everywhere, which makes every project feel familiar.

FlixTube services in the clusterBrowserGatewayUI + REST APIVideo streamingAzure storagecloud filesVideo uploadHistoryviewed videosMetadatavideo detailsOne database server hosts a database per service. RabbitMQ carries the indirect messages.
ServicePurpose
Gatewayentry point that serves the web UI and a REST API
Video streamingstreams videos from storage to the viewer
Azure storagestores and retrieves videos in external cloud storage
Historyrecords the viewing history
Metadatarecords details about each video
Video uploadorchestrates upload of new videos
Mock storagestand-in for Azure storage that uses the local disk
Project structure: service and monorepoOne microservice folderDockerfile-dev, Dockerfile-prodpackage.json (scripts)src/index.js, index.test.jsjest.config.js, .dockerignoreThe monorepogateway/ video-streaming/ ...scripts/ (local, production, cd).github/workflows/ (one per service)docker-compose.yaml, fixtures/Every service has virtually the same layout. All of them live in one repository.

A monorepo keeps things simple at the start. It once defeated the point of microservices, because everything deployed together. With one GitHub Actions pipeline per subfolder, each service can again deploy on its own schedule.

From Bootstrapping Microservices, ch. 10 (sections 10.4).

Running and testing in development

A service is booted alone with npm install and npm start while you focus on it. The entire app boots with docker compose up --build. Wait for the gateway to print that it is online, then browse to the UI, upload a video, watch it and look at the history page.

FlixTube booted with ComposeDocker Compose on the developer computerGatewayStreamingUploadMetadataHistoryMongoDB + fixturesRabbitMQMock storage + shared volumeIn development the Azure storage service is replaced by a mock that uses local files.

Testing reuses lessons 14 and 15. The same word npm test means Jest inside a service and Playwright at the repository root. That convention means you never need to remember which framework a project uses.

Wherenpm test runsScope
metadata folderJest unit tests (database library mocked)one service
repository rootPlaywright, in three browserswhole running app
npm run test:watchJest in live reload modewhile coding

From Bootstrapping Microservices, ch. 10 (sections 10.5 and 10.6).

Deep dive: fixtures, mocks, gateway, UI, streams

Database fixtures load realistic data before a test, a manual check or a demo. They can be JSON for static data or JavaScript for generated data. A small REST API for loading fixtures runs beside the app during Playwright tests.

Loading a database fixtureNamed fixturee.g. two-videosFixture REST APIloads the dataDatabaserealistic contentTest or demoknown dataA meaningful fixture name tells you at a glance what state the app starts from.

Mocking the storage microservice swaps a whole service for a simpler one. Mock storage keeps videos in the local filesystem through a shared Docker volume. The other services cannot tell the difference. Development no longer needs the network or a cloud account, and developers cannot disturb each other.

Replacing cloud storage with a mock serviceDeveloper computerVideo streamingMock storagesame HTTP interfaceLocal filesDocker volumeSame contract as the Azure service, so nothing else changes.

The gateway is the single door into the cluster. It serves the UI and a REST API and forwards requests inward. Authentication would go here. If web, mobile and admin frontends have different needs, give each its own gateway (backends for frontends) on its own host name.

Backends for frontendsWebMobileAdminGateway: webGateway: mobileGateway: adminServicesinside the clusterKeep one gateway if you can. Add more only when frontends truly differ.

The UI is deliberately plain: server-rendered pages from Express and Handlebars templates, vanilla JavaScript and Tailwind CSS for styling. The home route fetches the video list from the metadata service and renders it. A real product would probably use a single-page framework, but UI is not the topic of the book.

From Bootstrapping Microservices, ch. 10 (sections 10.7.1 to 10.7.4).

Following a video: streaming passes through three services. Each pipes the stream it receives into its own response, so the video moves byte by byte from cloud storage to the video element in the page.

Streaming path through FlixTubeCloud storageblobAzure storageGET /videoVideo streamingpipes streamGateway/api/videoThe HTML video element requests the gateway route and receives the same stream.

Video streaming also broadcasts a "video viewed" message, so the history service can record it. A new consumer can be added later without editing the streaming service.

Upload path through FlixTubeUpload pageuser picks fileGatewayforwards uploadVideo uploadorchestratesAzure storagestreams to blobUpload is the mirror image of streaming: an HTTP POST body is piped through to storage.

From Bootstrapping Microservices, ch. 10 (sections 10.7.5 and 10.7.6).

Deployment flows: local, manual, automatic

Deployment is practiced in three steps of rising difficulty. If one fails, step back to the easier one. You cannot automate what you cannot do by hand.

Three stages of deployment1 Local Kubernetesmock storage, no registry2 Manual productionregistry + cloud cluster3 Automatic CDpush to deployDay to day, the author still prefers Compose for development. Local Kubernetes is for practice.

Local: a script builds each image and applies each service's YAML to the Kubernetes bundled with Docker Desktop. The gateway appears on a NodePort at or above 30000. No container registry is needed, and a delete script removes it.

Manual production: log in to the private registry, set the registry address as an environment variable, then run the deploy script. It builds and pushes each image, uses envsubst to fill in the templated YAML, and pipes the result to kubectl.

One CD pipeline per microserviceGit pushto mainGitHub Actionsgateway/** onlyBuild + pushimage to registrykubectl applydeploy to clusterSecrets: container registry, registry user and password, base64 kube configOne workflow file per service. Only services whose folder changed are redeployed.

Each workflow is scoped to its service folder, and each can also be started by hand from the GitHub UI. Shell scripts do the work, driven by environment variables. MongoDB and RabbitMQ are deployed once, manually, with an infrastructure script, since they rarely change. For real production the book still advises an external managed database.

From Bootstrapping Microservices, ch. 10 (sections 10.8 to 10.10).

Healthy microservices: seeing what happens

A healthy microservice is not crashing, buggy or starved of CPU and memory. To keep the app healthy you observe it, protect customers when it fails, triage, and debug. Without logs you cannot fix what you cannot see.

The path of a log lineServiceconsole outputDocker or K8scaptures streamsCollectorone place to readYousearch, alert, actStandard output carries normal events. Standard error carries errors.
WhereHow to read the logs
Development, one serviceterminal output
Docker Composeone merged stream; pipe it through tee to also save a file
Kuberneteskubectl logs with the pod name; a pod can have several containers
A growing clusterinstall an aggregator; Kubernetes has none built in

Log meaningful events and the success or failure of important operations. Do not log secrets, personal details or facts easy to find elsewhere. More logs help more than fewer, because you cannot add logging after the problem has happened.

Enterprise monitoring toolsFluentd, Elasticsearch, KibanaPrometheus and GrafanaFluentd collectsElasticsearch storesKibana searches, alertsPrometheus scrapes metricsGrafana draws dashboardsPowerful but heavy. Do not rush to adopt them.

Observability goes further. Services output rich telemetry (OpenTelemetry is the standard), and a tool such as Honeycomb connects the dots. A trace shows one request crossing services and points to the span that failed.

From Bootstrapping Microservices, ch. 11 (sections 11.1 to 11.2.7).

Error handling, health checks, debugging

Errors are unavoidable: runtime exceptions, bad input, a broken dependency such as RabbitMQ or storage. Choose a strategy. The default recommended here is abort and restart: log the error, exit with a nonzero code, and let the platform restart the process. Use resume only where aborting loses work, such as video uploads in progress.

Abort and restartUnhandled erroruncaught exceptionLog itseen by opsExit nonzeroabort the processK8s restartsfresh instanceWatch for repeated restarts: they signal a bug that still needs fixing.

Kubernetes already restarts crashed pods. For a custom definition of healthy, give each service two routes. The readiness probe says the service is started and can take requests. The liveness probe says it is still alive. Probes also solve the start-up ordering problem from lesson 26, and what happens if RabbitMQ restarts later.

Readiness and liveness probesKubeletagent on each nodeService: /readyService: /aliveready to accept traffic?still healthy? restart if notA 200 response means yes. Rolling updates also wait for readiness (below).

Debugging a distributed app is harder because the problem hides among many small processes. A repeatable process helps:

The seven debugging stepsTriageworst firstEvidencelogs, metricsMitigateprotect usersIsolatewhich serviceReproduceideally a testFixthe root causeReflectprevent the classA failing test that reproduces the bug also proves when it is fixed.

In production, check the logs first, including those of the previous instance of a crashed pod. List the pods: an image pull error in the status means the cluster cannot reach the registry, usually a connection set-up mistake.

SymptomFirst move
Service crashedkubectl logs for the pod, and the previous instance
Pod in an error state or restartingkubectl get pods to read status and restart count
Image pull errorscheck the registry connection and image name
Cannot find the problemadd experiments in production only if you cannot reproduce it elsewhere

From Bootstrapping Microservices, ch. 11 (sections 11.2.2, 11.2.8, 11.3).

Reliability: surviving failure

Fault tolerance starts inside each service with defensive programming: expect bad inputs, hidden bugs, future bugs and unreliable dependencies. Defensive testing then attacks the code, checking that errors are reported and recovered from.

Protect data first. Source data captured from customers matters more than data that can be regenerated. Automate backups and practice restoring. Keep the capture code minimal and well tested, and never write code that overwrites or deletes source data.

Replication gives redundancy. Run at least three instances behind a load balancer; when one fails, others take over while Kubernetes restarts it. It is one field in the deployment YAML. Still log and investigate every failure.

Replicas behind a load balancerLoad balancerReplica 1Replica 2Replica 3Replicas set to 3. One request goes to one replica; the dashed one may be restarting.

Fault isolation is a strength of microservices, if you use it. If video upload breaks, the gateway should catch the error and tell the user that upload is unavailable, so the rest of the app keeps working. This is the bulkhead idea, taken from ship compartments: a leak is contained.

Containing a fault at the gatewayWithout isolationUploadGatewayUser sees errorWith isolationUploadGateway catchesNotice, rest OKGraceful degradation: lose one feature, keep the application.

Simple techniques you can use now:

Fault tolerance toolboxTimeoutgive up after waitRetrytry againJob queuesmooth heavy workCircuit breakerfail fast when downAxios waits forever by default, so set a timeout. Failing fast beats failing slowly.
Circuit breaker statesClosed (on)requests pass throughOpen (off)fails requests fasta request failsUnlike a timeout, it already knows the upstream is failing, so it does not even wait.

A job queue stores heavy tasks (such as a thumbnail per uploaded video) in a database-backed queue service. Workers pull one job when free, jobs survive a crash and failed jobs are tried again. The load is spread over time, avoiding a burst of extra compute.

From Bootstrapping Microservices, ch. 11 (sections 11.4 to 11.4.7).

Scaling the team: repositories and environments

One developer can push straight from a laptop to customers. A growing team cannot: code should not go directly to customers, developers must not trip over each other, and one repository eventually becomes too much. Microservice boundaries give natural seams for splitting teams, each owning its services from development through production.

From monorepo to repository per serviceMonorepoAll services + infrastructureOne pipeline per service folderIndependent repositoriesgatewayhistorymetadatainfrastructure (Terraform)Splitting: git init for each folder, or keep history with git filter-branch.

A meta-repo (the meta tool and a .meta file) ties the separate repositories back into one virtual repository, so cloning and running everything feels like a monorepo again. A team can keep its own meta-repo for the set of services it works on.

Environments and branchesDeveloperown computerDevelopmentintegrate dailyTestweekly mergeProduction1 to 2 weeksOne branch per environment. A merge triggers deployment to that environment.

Environments reuse the Terraform from lesson 19, parameterized by an environment name, with Terraform state stored in cloud storage per environment. Pushing to the development branch runs CI tests and deploys to development. The less time between merges, the fewer integration surprises.

Finally, application configuration can move to its own repository that records which version of each service is deployed. Deployment then has two phases: the service pipeline builds and publishes an image, then a change to the config repository deploys it.

Do not split early: the separate config repository adds steps and chances for error, such as typing an older version number. Use it only for needs like promoting the same version between environments, one place to view history, or restricted access.

From Bootstrapping Microservices, ch. 12 (sections 12.2.1 to 12.2.7).

Scaling performance

Microservices let you tune performance part by part, instead of only scaling a whole monolith. Measure first with the Kubernetes dashboard or cloud portal, then use the lightest option that works. Most of these are one-line changes in Terraform or YAML.

Ways to scaleCluster, verticalbigger VMsCluster, horizontalmore VMsService, horizontalmore replicasCluster, elasticautoscaler with min/maxService, elasticreplicas follow demandDatabasesplit, then shardMany small VMs usually cost less than a few big ones.
Vertical and horizontal scaling of the clusterVertical: bigger machinesLarge VMLarge VMLarge VMHorizontal: more machinesVMVMVMVMVMVMSame cluster, two directions. Raising the VM size or the node count is a Terraform change.

For one overloaded service, raise its replicas. The same field serves redundancy and performance. Elastic scaling adds and removes capacity automatically: the cluster autoscaler grows the node pool between a minimum and a maximum, and a horizontal pod autoscaler grows a service between its own minimum and maximum, so you pay only for what the load needs.

Scaling the databaseShared database: avoidServiceServiceServiceOne database, bottleneckOne database per serviceServiceServiceServiceDBDBDBSeparate databases may share a server. Later: more servers, then sharding for huge data.
Do not scale too early. You cannot predict how you will need to scale, so future-proofing often blocks the scaling you actually need. Keep code and architecture simple, and scale when real load shows the need.

From Bootstrapping Microservices, ch. 12 (sections 12.3 to 12.3.7).

Safe changes and basic security

Many production problems come from pushing broken code. Catch them early, where they are cheaper, with a chain of safeguards.

Layers of change protectionPull requesttests + reviewTest environmentshake it downRolling updateone by oneBlue-greenwhole new stackBranch protection forbids direct pushes to the production branch.

A rolling update replaces replicas one by one, moving on only when the new replica passes its readiness probe. If it fails, the rollout halts and you can roll back. Settings cap how many extra and unavailable pods are allowed at once.

Blue-green deploymentBluecustomers use itGreennew stack, risky workSwitch DNSblue to greenIf green misbehaves, flip DNS back. Works best when the cluster holds no data.

Use blue-green for risky infrastructure changes, such as the scaling changes above. Build the whole second environment from Terraform, test it, then repoint DNS. A stateless cluster makes this safe.

Security is a book of its own, but the basics matter. FlixTube exposes only the gateway; RabbitMQ and MongoDB were closed to the outside after early experiments. Planned upgrades include authentication at the gateway and HTTPS. Think like an attacker: how would someone break this?

Trust modelsInternal trustGatewayauthenticatesServices trusteach otherZero trustGatewayauthenticatesAuthenticateevery callStart with internal trust. Zero trust suits higher needs or several clusters.

Keep passwords, tokens and API keys out of the code. Use GitHub secrets, Kubernetes secrets, or a tool such as Vault. Simple security has fewer hiding places for flaws.

From Bootstrapping Microservices, ch. 12 (sections 12.4 and 12.5).

Refactoring to microservices and the spectrum

Converting an existing monolith is done like everything else here: small tested steps that keep it working, while features still ship. Before you start, ask whether microservices are truly needed. Then follow the tips.

Refactoring a monolith step by stepPlanshared vision, teamKnow the codetests, risksAutomatedeploy and testPlatformcluster, templateFind seamsnatural componentsPrioritizemost benefit firstExtract + testkeep it workingRepeatoften for yearsIf you cannot afford automation, you probably cannot afford microservices.

Carve along natural seams: components that already look like the target services. Without clear seams, expect to refactor first, with tests. Extract first what changes often or would gain most in speed or scale, so benefits appear early.

The spectrum from monolith to functionsSpectrum of possibilitiesMonolithone big processHybridmonolith + helpersMicroservicesmany servicesFunctionsmany tiny processesThere is no right place on the line. Pick what suits your team, business and customers.

Returns diminish: the first extractions bring big wins, later ones less, and services that are too small cost more than they give. It is fine to stop. The hybrid approach, a monolith surrounded by a constellation of helper services for jobs like thumbnail generation, is the author's default when unsure.

Microservices on a budget
Start with a monorepo and one CD pipeline per service; add a meta-repo after splitting
One database server, one database per service
A one-VM cluster, a single instance per service, simple pipelines
External file storage and database so the cluster stays stateless
Without automated tests, script a fast boot with Compose and fixtures; invest early in automated deployment

From Bootstrapping Microservices, ch. 12 (sections 12.6 to 12.9).

Recap

Quick check

The upload service is down. How do users keep watching videos?
Show answer

The gateway catches the error from upload (fault isolation), shows a message that upload is unavailable, and keeps serving the other routes. This is graceful degradation.

A rollout of the gateway shows its first new replica never becomes ready. What happens?
Show answer

With a rolling update that waits on readiness, the rollout stops after the first replica, the other replicas keep serving, and you debug and roll back.

Why can a one-VM cluster with single replicas still be a sound start?
Show answer

Early on you rarely need redundancy or performance, and it costs little. The design leaves a path to scale by changing replicas and node counts later.