You inherit a codebase with few tests, a deployment script that usually works, and a request to “put CI/CD in place.” The popular default is to make every check mandatory immediately. I would do the opposite: make the pipeline expose risk before asking it to veto releases. An untrusted red build does not protect users; it teaches developers to ignore red builds.
A mandatory gate is premature until its failures mean something
The first pipeline for an untested codebase should answer a narrow question: what happens when we run the same build and checks on every change? That is different from asking whether the change is safe. A successful build proves that the build succeeded under its configured conditions; it cannot establish that an undocumented billing rule or webhook retry still behaves as intended.
You can read Cheap CI/CD vs Doing It Right What a CTO Should Buy or Build and still reject the apparent choice between cheap automation and a fully enforced pipeline. An evidence-gathering pipeline is a useful third option because it gives QA a repeatable baseline without pretending that the baseline is a release verdict.
Start by recording the commit SHA, build result, test result, duration, and deployment outcome for each run. GitHub Actions can produce the run record; JUnit XML can preserve individual test results; Sentry or an existing error tracker can show what happened after deployment. Keep those records linked to the same commit. Otherwise, a production exception and a green build may describe different versions of the application.
Use a provisional 14-day observation window before turning an inherited suite into a required check; that duration is a setting to adjust around your release frequency, not an industry benchmark. During it, distinguish a failure caused by the current commit from one already present on the base branch. For a flaky failure, record the same test’s result on an unchanged commit rather than repeatedly rerunning the pull request until it passes. Reruns without a fixed commit erase the evidence you need to diagnose the problem.
I would not set a global coverage threshold on day one, because coverage.py can report executed lines while missing the assertion that would catch a changed business rule. I would also avoid making an unstable legacy suite a required GitHub branch-protection check: developers who cannot distinguish new failures from old ones will eventually work around the gate. Keep the build visible, investigate failures, and promote individual checks to required status only after they have shown that they can give an actionable answer.
Characterization tests should precede coverage targets
For QA, the most valuable first tests are often characterization tests: checks that preserve a behavior the team has observed but cannot yet confidently redesign. Choose a seam with a visible input and output, such as an HTTP endpoint, a CSV import, or a webhook handler. Ask what a user or another system would notice if it changed. A test of that boundary can protect a release even while most internal functions remain untested.
Pick cases from evidence rather than from an imagined ideal specification. A recent bug report might identify a malformed payload that must return HTTP 400 without creating a database row. A support ticket might identify a duplicate webhook that must not create a second charge. Mark assumptions explicitly in the test name or review discussion; otherwise, an accidentally observed behavior can harden into a permanent requirement. When a domain owner disagrees with the observed behavior, resolve that disagreement before making its test a release gate.
Here is a minimal GitHub Actions workflow for a Python repository after you add a tests/characterization directory. The Python 3.12 and pytest 8.3.5 versions are pinned configuration choices in this example, not claims that those versions improve reliability. If the tests import the application, install its dependencies in the workflow as well.
name: characterize
on: [push, pull_request]
jobs:
changed-behavior:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with:
python-version: '3.12'
- run: python -m pip install pytest==8.3.5
- run: python -m pytest tests/characterization -q
This workflow runs the selected tests, but it does not by itself prove that their assertions are good. Review a new test by asking whether it fails for the regression it names. One practical check is to temporarily change the relevant condition on a local branch and confirm that pytest fails for the expected reason. Revert that mutation before merging. This is more informative than watching coverage.py’s –cov-branch percentage rise, because the failed assertion demonstrates that the test can detect a particular wrong result.
A characterization test also needs controlled dependencies. Use Docker Compose or Testcontainers when a database interaction is part of the behavior under test, and record the PostgreSQL image tag so local and CI runs use the same major version. Use a recorded payload when the external service is not the subject of the test; a live third-party API makes a pull-request result depend on a service and credentials outside the commit. Playwright is appropriate for a critical browser journey, but putting every UI path into the first required gate makes diagnosis slow when the application and test fixtures are still changing.
Changed-code gates beat an inherited-suite ultimatum
There are two workable options for a pull-request gate. Option A, a global coverage minimum, wins when a suite is already stable and the team agrees that its coverage figure represents useful protection; it costs test maintenance and can reward low-value assertions written to meet the number. Option B, a changed-behavior gate, wins when you have inherited an untested application and can identify risky seams; it costs QA and developers time to review which behaviors each change could affect. For most teams in that position, I would choose Option B first because its failure is easier to connect to the proposed change.
Make that choice enforceable without claiming to test everything. Require the application to build. Require the stable characterization tests for touched boundaries. For a change to a payment retry handler, a reviewer should see a duplicate-event test or a written reason that the existing test still covers the change. A CODEOWNERS entry can route the review to someone who knows the boundary, but ownership is not a substitute for an assertion that would fail on a regression.
Keep unstable tests running in a separate, visible job rather than deleting them or letting them block every pull request. A test should leave quarantine when its failure is reproducible, its fixture is controlled, and its owner knows what action to take when it turns red. As an initial rule to tune, investigate a check that fails on 2 unchanged-commit runs before treating its next red result as evidence against a new change. That rule is deliberately local: a deterministic security or data-loss check deserves a different response.
Do not confuse a short pipeline with a safe one. An adjustable target of 8 minutes for the required pull-request checks may encourage useful prioritization, but it should not force a critical database test out of the gate merely to hit the target. Move slower, broader tests to a scheduled run and give someone responsibility for its failures. If nobody reads that result, scheduling it has only delayed the moment at which the team admits it lacks protection.
Judge the gate by whether it changes decisions. Track how often a required check identifies a defect before merge, how often a failure cannot be reproduced on the same commit, and how often a release incident was in an area the gate claimed to cover. Those are team measurements to collect, not numbers to borrow from another project. A rising coverage percentage alongside repeated missed regressions is a reason to change test selection, because the percentage measures execution rather than the quality of the assertions.
Release evidence matters more than a permanently green dashboard
Some failures cannot be caught cheaply before merge, especially when the inherited codebase depends on production data shapes or interactions the test environment does not reproduce. That does not justify shipping blind. For each release, identify a small set of observable behaviors: an API success rate, a queue backlog, or a specific error signature. OpenTelemetry traces and Prometheus metrics can help connect those signals to a deployment, provided the application emits a version identifier. Without that identifier, a spike is harder to attribute to the release you just made.
Agree on a rollback trigger before deploying. For example, set a trial threshold for the payment endpoint’s HTTP 500 rate, then revise it using the endpoint’s normal variation; a threshold chosen without a baseline will either fire constantly or miss the incident. Confirm that git revert and the deployment process can actually restore the previous application version. If a release includes a destructive database migration, reverting code alone may not restore service, so test the migration’s recovery path separately.
You might read CI/CD Will Eat Your Weekends Unless You Plan This Work before scheduling that work, but a calendar reservation alone will not make a red build trustworthy. Give each failed check an owner and a decision: fix the product, fix the test, quarantine the test with an issue, or document why the result is expected. Record that decision beside the run so the next person does not repeat the investigation.
Use DORA’s change failure rate and time to restore service as retrospective measures, because they reflect outcomes after deployment rather than the appearance of the CI dashboard. They need consistent definitions to be useful: decide what counts as a failed change and when restoration ends. If releases are infrequent, individual incidents and their timelines may be more informative than a percentage calculated from a small sample.
Your first gate should be one you can defend
Tomorrow, select one recently broken behavior and write down its input, expected output, and commit that fixed it. Add a characterization test that fails against the broken commit, then run it in CI alongside the build. Keep the remaining legacy checks visible but non-blocking for now. You will have one gate whose red result you can explain—and a concrete basis for choosing the next one.
