CI/CD Pipelines Field Notes

Reusable workflows reduce copy-paste. They also centralize failure, permissions, and release risk. Treat a reusable workflow like an internal platform API: version it, document its inputs, and preserve the evidence callers need during review or incident response. Pin The Interface Call reusable workflows by a reviewed ref: jobs: terraform-plan: uses: org/platform-workflows/.github/workflows/terraform-plan.yml@v1 with: working-directory: infra/prod environment: prod Avoid calling shared workflow logic from a moving branch such as main for production-impacting work. A central workflow change should not silently alter every repository’s deployment behavior. ...

GitHub Actions credentials are production access if the workflow can mutate production. The safe pattern is short-lived, scoped access tied to the repository, branch, environment, and workflow that needs it. Long-lived cloud keys in repository secrets should be the exception, not the default. Start With Permissions Set workflow permissions explicitly: permissions: contents: read Then add only what a job needs: permissions: contents: read id-token: write id-token: write enables OIDC token issuance. It does not by itself grant cloud access; the cloud trust policy still decides what the token can assume. ...

GitHub Actions makes it easy to run automation on almost every repository event. That convenience is also the failure mode. A workflow should say what kind of decision it represents: validation, packaging, release, deployment, or scheduled maintenance. Mixing those decisions into one trigger set makes incidents harder to explain. Separate Trigger Intent Use different workflows or clearly separated jobs for different events: pull_request -> validate proposed change push to main -> build or publish reviewed artifact tag -> release or promote immutable version workflow_dispatch -> operator-controlled action schedule -> maintenance or drift detection Do not let a documentation-only PR, force-push, or branch experiment accidentally run production-impacting automation. ...

A Packer image pipeline produces more than a template. It produces evidence. If the only durable output is “template exists in vSphere,” operators lose the ability to answer the questions that matter during rollback or incident review: What source produced this template? Which Packer and plugin versions built it? Which variables were used, excluding secrets? Which validation gates passed? Which template was promoted to current? What changed from the previous image? Artifact storage is the operating memory of the image factory. ...

Blue-green deployment keeps two environments running. At any time, one environment serves production traffic and the other waits for the next release. For a runnable lab, see the blue-green-simulation directory in the IaC repository. Version 1 demonstrates a manual NGINX upstream switch. Version 2 adds weighted routing and named containers. The switch is the critical moment. How it happens determines whether the pattern is fast rollback or just extra complexity. The NGINX Upstream Switch The simplest blue-green switch is an NGINX upstream block with one active server and one standby: ...

A canary deployment sends a small fraction of traffic to a new version while the stable version handles the rest. If the canary fails, only the test fraction is affected. For a runnable lab, see the canary-deployment directory in the IaC repository. It uses HAProxy weighted routing with raw C API servers. HAProxy makes this pattern visible and controllable through weighted backend servers. Weight Ratio backend servers balance leastconn server v1 api_v1:8080 weight 10 check server v2 api_v2:8080 weight 1 check With weights 10 and 1, approximately 9% of requests reach v2. The weight proportion directly controls the blast radius: ...

Concourse requires a set of RSA keys for TSA (Transport Security Authority) authentication between web and worker nodes. Managing these keys is a bootstrapping problem: Concourse needs keys to start, but the keys should live in a secrets store. For a runnable lab, see the concourse-terraform-unix directory in the IaC repository. TSA Key Architecture Concourse uses four key pairs: TSA host key (tsa_host_key + tsa_host_key.pub): identifies the web node to workers. Worker key (worker_key + worker_key.pub): identifies workers to the web node. Authorized worker keys (authorized_worker_keys): the public keys of permitted workers. Session signing key (session_signing_key): signs session tokens for the ATC API. The web node holds the TSA host key and authorized worker keys. Workers connect using their worker key. If any key pair mismatches, the worker cannot authenticate and stays disconnected. ...

Environment variable toggles are the simplest form of feature flag. No SDK, no external service, no runtime dependency. The application reads an env var at startup and enables or disables behavior accordingly. For a runnable lab, see the feature-toggle directory in the IaC repository. It demonstrates the same toggle pattern in both C and Python. The Pattern Python: import os feature_enabled = os.getenv("FEATURE_ENABLED", "false").lower() == "true" if feature_enabled: # new behavior else: # old behavior C: ...

Make Git The Only Entry Point If a change can be made without opening a pull request, it will eventually be made without a pull request. The rule is simple: no PR, no change. This applies to: Terraform and Ansible runs. Image template version bumps. Pipeline configuration changes. DNS and load balancer records. Monitoring and alerting rules. Pipeline Shapes Change Pipeline PR → lint → validate → plan → plan review → apply non-prod → apply prod → verify Plan output must be retained as an artifact. Apply stages must be serialized per state backend. ...

The boundary between Terraform and Helm is a common source of confusion. Terraform provisions infrastructure. Helm deploys applications. Terraform’s helm_release resource bridges them, but the chart templates stay in the application repository. For a runnable lab, see the helm-terraform-js-app directory in the IaC repository. The Pattern Terraform manages the Helm release with set blocks that inject environment-specific values: resource "helm_release" "my_app" { name = "my-app" chart = "${path.module}/../helm/myapp" namespace = kubernetes_namespace.my_app.metadata[0].name set { name = "image.repository" value = var.docker_image_repository } set { name = "image.tag" value = var.docker_image_tag } set { name = "replicaCount" value = var.replica_count } } The Helm chart stays portable. Environment-specific values live in Terraform variables. ...