Continuous integration and delivery pipelines
From commit to production: continuous integration, build pipelines, artifacts and how code maps to services.
key idea / coordinatorservice / good pathin progress / watch outproblem / failureneutral
Continuous integration, honestly
CI keeps everyone in sync: a CI server notices a commit, checks the code out, compiles it, runs the tests and, if all is well, builds an artifact for later stages. Because the build is scripted and versioned, you can rebuild any artifact and trace a deployed one back to its code.
Owning a CI tool is not the same as doing CI. Jez Humble's three questions are a quick test:
From Building Microservices, ch. 7, "A Brief Introduction to Continuous Integration" and "Are You Really Doing CI?".
Branching: integrate early, integrate often
A long-lived feature branch delays integration, so the merge at the end is bigger and riskier. Newman prefers trunk-based development: everyone commits to one trunk and hides unfinished work behind feature flags (which also help progressive delivery, lesson 19).
Build pipelines and continuous delivery
Split a build into stages: fast tests first, slow tests after. If the fast stage fails there is no point running the slow one. Each commit becomes a release candidate that moves through the pipeline, so you can always see how far a version has got.
| Term | Meaning |
|---|---|
| Continuous delivery | Every check-in is a release candidate; you know how production-ready each one is. Deploying stays a decision. |
| Continuous deployment | Every change that passes automated checks is deployed with no human step. Builds on continuous delivery. |
Pick a tool that models pipelines directly, including manual stages such as user acceptance testing, instead of bending a plain CI tool into one. Human steps in the path to production are usually the bottleneck to automate.
From Building Microservices, ch. 7, "Build Pipelines and Continuous Delivery" and "Tooling".
CI/CD pipelines and GitHub Actions
A build pipeline splits the build into stages: fast small tests first, then slower ones, then deployment. A good CD tool models the whole path to production, even a manual acceptance step. Trading fast feedback against production-like checks is a choice you revisit.
From Bootstrapping Microservices, ch. 8. Values such as the registry address and version come from environment variables, and secrets live in GitHub secrets, not in the repo. The tool envsubst fills them into the Kubernetes file before it is piped to kubectl. Keep deployment code simple, since failures in the cloud are harder to debug. Reproduce an error locally by copying the workflow environment.
Environments and artifacts
Early stages want speed; late stages want realism. Your laptop is fastest and least like production; a full production copy is realistic but slow and costly. Finding the balance is an ongoing choice, and it is one reason teams also test in production.
Two rules about artifacts: build it once, and deploy exactly what you tested. Rebuilding for production means you are not shipping the thing you verified. Store the artifact in a registry (Artifactory, Nexus or a container registry) and keep environment-specific settings outside it.
From Building Microservices, ch. 7, "Trade-Offs and Environments" and "Artifact Creation".
Promoting one artifact through environments
A microservice runs in several environments on its way to users. The same service may have one instance on a laptop and many in production. From Building Microservices, ch. 7 and 8.
Environments trade feedback speed against realism: a laptop is fastest but least like production, a full copy is realistic but slow and costly. Some environments hold only a few services, with fakes for the rest. This is partly why teams also test in production.
Mapping code and builds to services
How you lay out source code decides which builds a commit triggers. Three options (a fuller comparison with Git workflows is in One repository or many? below):
| Giant repo + one build | One repo per service | Monorepo | |
|---|---|---|---|
| Good | Simplest start | Clear commit-to-build map; per-repo ownership | Atomic commits, fine-grained code reuse, visibility |
| Bad | One red build blocks everyone; encourages deploying together | Cross-repo changes are awkward; shared code needs library versions | Needs tooling (e.g. Bazel) at scale; blurs ownership |
| Newman | Rarely used beyond day one | Preferred as you scale | Fine for 10 to 20 developers or very large, tool-rich firms |
From Building Microservices, ch. 7, "Mapping Source Code and Builds to Microservices".
One repository or many?
Newman: giants such as Google and Microsoft make monorepos work by investing heavily, and small groups of 10 to 20 developers can too. The middle suffers. He prefers one repository per microservice at scale, because the benefits of reuse and atomic commits do not outweigh cross-service changes and blurred ownership. From Building Microservices, ch. 7.
Bootstrapping Microservices keeps all FlixTube services in one monorepo, with a separate CD workflow file for each microservice.
Recap
- CI means integrating to mainline daily, with tests, and fixing red builds first.
- Prefer short-lived branches or trunk-based development with feature flags.
- A pipeline runs fast stages before slow ones; every commit is a release candidate. Continuous deployment removes the human gate.
- Build the artifact once, test that artifact, and keep configuration outside it.
- One repo per service is simplest at scale; monorepos need tooling and ownership discipline. Atomic commits are not atomic deploys.
- Build an artifact once, keep it environment-agnostic, and promote it. Container images are immutable and tagged per version.
Quick check
Show answer
Otherwise you ship something you never verified. Build once, store it in a registry, and vary only external configuration.
Show answer
The artifact you tested is then exactly the one that reaches production. Images are immutable, so earlier tests and security checks stay valid, and only the environment-specific configuration changes.