A CI/CD platform can pass every demo and still leave you unable to answer a production question: which change is deploying, with whose authority, and what happens if the runner disappears? I would choose a vendor or framework by testing those failure paths, not by comparing pipeline editors. A fast happy-path build is cheap to demonstrate; predictable behavior during an interrupted release is harder to fake.
A demo pipeline is evidence of almost nothing
Ask each candidate to run the same release-shaped workload in a disposable environment. Use your repository, your container registry, and a deployment target configured like production. Keep the application small enough that its build time does not dominate the test, but retain the controls you actually support: a private dependency, an approval boundary, an artifact, and a deployment record. Otherwise, the trial measures how well the vendor’s example repository works rather than how your releases work.
Define the acceptance record before anyone starts building the pipeline. For every attempt, capture the commit SHA, artifact digest, triggering identity, approval identity, start and finish times, and final environment state. Prometheus can store queue and execution timings; OpenTelemetry traces can connect a build to its deployment if your tooling emits a common trace ID. Neither substitutes for the platform’s audit log, because metrics alone do not establish who authorized a production change.
Run the same workload with a failed test, an unavailable registry, and a deployment process killed after it starts. Inspect what an operator sees without administrator access. If the interface says “failed” but cannot distinguish “no artifact published” from “artifact published, deployment uncertain,” it creates investigation work during an incident. Treat that ambiguity as a product defect, even if the vendor calls the run successful after a retry.
I disagree with Cheap CI/CD vs Doing It Right What a CTO Should Buy or Build as the primary purchasing frame: the decisive cost for an on-call engineer is often the time spent establishing what happened after a partial failure, because that work arrives when the normal pipeline has stopped providing answers. Price still matters, but compare it only after the candidates have passed the same failure test.
Use a 30-day trial window as a proposed sampling period, not as a claim that a month guarantees reliability. Record measured p95 queue delay, measured p95 recovery time, and the share of failed runs whose outcome an operator can classify from available evidence. Keep these measurements separate from vendor-published availability figures: an available control plane does not prove that your runners, registry, or deployment path can complete a release.
Concurrency rules decide whether a retry is safe
Release scheduling deserves its own test because “serial deployments” can describe different behaviors. Trigger a release, trigger another while it is running, then trigger a third. Ask which run executes, which waits, and which is discarded. GitHub Actions documents a significant constraint for a concurrency group: it permits at most one running and one pending run, and a new pending run replaces the existing pending run. That vendor-published rule makes the group useful for preventing overlap, but it is not a durable first-in, first-out release queue.
This GitHub Actions workflow is a runnable probe. Put it in .github/workflows/queue-probe.yml and dispatch it repeatedly from the Actions tab:
name: Queue probe
on: workflow_dispatch
permissions: {contents: read}
jobs:
probe:
runs-on: ubuntu-24.04
timeout-minutes: 15
concurrency:
group: release-probe
cancel-in-progress: false
steps:
- run: echo "start $(date -u +%FT%TZ)"; sleep 90; echo "end $(date -u +%FT%TZ)"
The 90-second sleep is an adjustable hold chosen to make overlapping dispatches easy; the 15-minute job timeout is a trial setting to tune against your own failure-recovery policy. Check the run history rather than assuming that all dispatches will execute. Then repeat the experiment with a cancelled run, a runner that stops responding, and a deployment script that exits after making its first change. A scheduling rule is useful only if its behavior remains legible under those conditions.
GitHub Actions versus GitLab CI/CD: GitHub Actions wins when the repository and approval process already live in GitHub and its concurrency semantics fit the release policy, because fewer cross-system handoffs need maintaining. Its cost is that a concurrency group must not be mistaken for a queue that preserves every requested release. GitLab CI/CD wins when resource_group can express the deployment lock you need and your team can administer its process mode and runners alongside the repository. Its cost is operating those runner and resource-group settings and verifying their behavior rather than inheriting a safe default. Neither wins merely because its YAML is shorter.
I would not add automatic retries around an uncertain deployment, because rerunning a non-idempotent database or infrastructure change can compound the original failure. Require the candidate to show how an operator can determine whether the change took effect before retrying. For Kubernetes releases, that means checking the resulting Deployment and image digest, not treating a successful kubectl exit code as proof that the intended version is serving.
Identity and artifact evidence must survive the pipeline
A vendor’s secrets screen is not enough to evaluate its trust model, because the critical question is what a compromised job can do after it starts. Test a forked pull request, an internal branch, and an approved production release with separate credentials. For GitHub Actions, inspect the job’s permissions and grant id-token: write only where an OpenID Connect (OIDC) exchange is needed. For GitLab CI/CD, inspect the equivalent job-token and OIDC configuration. In both cases, verify the cloud trust policy’s audience and repository or project claims; a short-lived token is still dangerous if the receiving policy accepts too many callers.
BuildKit can produce the container image, Syft can generate its software bill of materials, and Sigstore Cosign can sign or attest to the resulting digest. Those tools are useful evaluation fixtures because they let you ask whether evidence follows the artifact out of the CI system. Use SLSA v1.0 as vocabulary for provenance requirements, then verify the particular fields your release decision needs: source repository, revision, builder identity, and subject digest. A badge saying “supply-chain secure” is weaker evidence than a verifier rejecting an artifact built from the wrong revision.
Make the test adversarial. Replace a tag in the registry so that it points to a different digest; the deployment should reject it if approval was tied to the original digest, because tags are mutable names. Remove the signing credential or make the OIDC provider unavailable; the pipeline should stop before publishing a release rather than silently drop its attestation. Feed an image with a known finding through Trivy, then check whether an Open Policy Agent policy fails on the intended condition rather than on an unrelated scanner error. These are evaluations of enforcement, not a request to collect more security logos.
Set a proposed gate of zero deployments whose approved digest differs from the deployed digest; this is a policy value to validate, not a vendor reliability statistic. Also measure how long an on-call engineer takes to retrieve provenance after the CI run has been deleted or aged out. If the artifact cannot be checked independently of the vendor’s web UI, the evidence may disappear exactly when an older release needs investigation.
The exit test exposes who really operates the release
Before signing, ask the candidate to move one release path away from its preferred deployment mechanism. Export the artifact and provenance, then deploy them through a second controller, such as Argo CD for Kubernetes. This is not a demand that every team adopt Argo CD; it is a check that build output has a stable identity and that deployment intent can be inspected outside the pipeline. Terraform-managed runner and environment configuration can make the exercise repeatable, provided the state and credentials remain under your control.
Perform a smaller exit test for observability as well. Can you export run status and timing to Prometheus without scraping a human-facing page? Can an operator find the build log after a runner is replaced? Can a revoked user still have a pending approval or cached credential that affects production? Each question has an operational consequence, which makes it more useful than a feature-list answer.
I treat CI/CD Will Eat Your Weekends Unless You Plan This Work as a warning, but planning alone is too weak an acceptance criterion: rehearse the handoff to an on-call engineer who did not configure the pipeline, because undocumented assumptions become visible when their author is absent. Give that engineer a failed release, the available dashboards, and the runbook. Watch where they need privileged access or a vendor support ticket to establish the deployment state.
Set a two-hour maximum as an initial, tunable recovery objective for the exercise, then adjust it to your service’s actual release risk. Measure from the injected failure to a verified decision to continue, roll back, or stop; do not stop the clock when a support case is opened. A candidate that requires vendor intervention for routine uncertainty may still be acceptable, but its support terms and escalation path belong in the operating design, not in sales fine print.
Start with a failure the vendor cannot narrate away
Tomorrow, create a disposable release target and write down the artifact digest you expect to deploy. Run one candidate’s pipeline, interrupt deployment halfway through, and hand the result to the engineer who would be paged. Record what they can prove from logs, cluster state, and provenance without help from the person who built the pipeline. That record is a stronger first purchasing requirement than another polished demo.
