CI/CD Pipelines Field Notes

Blue-green deployment keeps two environments running. At any time, one environment serves production traffic and the other waits for the next release. For a runnable lab, see the blue-green-simulation directory in the IaC repository. Version 1 demonstrates a manual NGINX upstream switch. Version 2 adds weighted routing and named containers. The switch is the critical moment. How it happens determines whether the pattern is fast rollback or just extra complexity. The NGINX Upstream Switch The simplest blue-green switch is an NGINX upstream block with one active server and one standby: ...

A canary deployment sends a small fraction of traffic to a new version while the stable version handles the rest. If the canary fails, only the test fraction is affected. For a runnable lab, see the canary-deployment directory in the IaC repository. It uses HAProxy weighted routing with raw C API servers. HAProxy makes this pattern visible and controllable through weighted backend servers. Weight Ratio backend servers balance leastconn server v1 api_v1:8080 weight 10 check server v2 api_v2:8080 weight 1 check With weights 10 and 1, approximately 9% of requests reach v2. The weight proportion directly controls the blast radius: ...

Concourse requires a set of RSA keys for TSA (Transport Security Authority) authentication between web and worker nodes. Managing these keys is a bootstrapping problem: Concourse needs keys to start, but the keys should live in a secrets store. For a runnable lab, see the concourse-terraform-unix directory in the IaC repository. TSA Key Architecture Concourse uses four key pairs: TSA host key (tsa_host_key + tsa_host_key.pub): identifies the web node to workers. Worker key (worker_key + worker_key.pub): identifies workers to the web node. Authorized worker keys (authorized_worker_keys): the public keys of permitted workers. Session signing key (session_signing_key): signs session tokens for the ATC API. The web node holds the TSA host key and authorized worker keys. Workers connect using their worker key. If any key pair mismatches, the worker cannot authenticate and stays disconnected. ...

Environment variable toggles are the simplest form of feature flag. No SDK, no external service, no runtime dependency. The application reads an env var at startup and enables or disables behavior accordingly. For a runnable lab, see the feature-toggle directory in the IaC repository. It demonstrates the same toggle pattern in both C and Python. The Pattern Python: import os feature_enabled = os.getenv("FEATURE_ENABLED", "false").lower() == "true" if feature_enabled: # new behavior else: # old behavior C: ...

Make Git The Only Entry Point If a change can be made without opening a pull request, it will eventually be made without a pull request. The rule is simple: no PR, no change. This applies to: Terraform and Ansible runs. Image template version bumps. Pipeline configuration changes. DNS and load balancer records. Monitoring and alerting rules. Pipeline Shapes Change Pipeline PR → lint → validate → plan → plan review → apply non-prod → apply prod → verify Plan output must be retained as an artifact. Apply stages must be serialized per state backend. ...

The boundary between Terraform and Helm is a common source of confusion. Terraform provisions infrastructure. Helm deploys applications. Terraform’s helm_release resource bridges them, but the chart templates stay in the application repository. For a runnable lab, see the helm-terraform-js-app directory in the IaC repository. The Pattern Terraform manages the Helm release with set blocks that inject environment-specific values: resource "helm_release" "my_app" { name = "my-app" chart = "${path.module}/../helm/myapp" namespace = kubernetes_namespace.my_app.metadata[0].name set { name = "image.repository" value = var.docker_image_repository } set { name = "image.tag" value = var.docker_image_tag } set { name = "replicaCount" value = var.replica_count } } The Helm chart stays portable. Environment-specific values live in Terraform variables. ...

Rollback is a deployment strategy that gets rehearsed less often than it should. A rollback plan that has never been tested is not a rollback plan. For a runnable lab, see the rollback-deployment directory in the IaC repository. It uses a sentinel file to trigger Puppet-driven dpkg rollback. The Sentinel File Pattern A sentinel file marks a failure condition. When it exists, automation triggers a rollback. In a Puppet-based lab: exec { 'rollback-to-v1': command => 'dpkg --force-depends -i /opt/v1/c-app.deb', onlyif => 'test -f /tmp/simulate_failure', } The sentinel file is a teaching proxy. In production, the sentinel would be a health check failure, a metrics threshold breach, or a monitoring alert. ...

Local infrastructure labs often start with hardcoded passwords, localhost endpoints, and convenience tokens. That is normal for learning, but dangerous when the lab pattern becomes a production pattern without review. The useful distinction is not “lab bad, production good.” The useful distinction is knowing which shortcuts are temporary and what must change before the pattern is reused. Common Lab Shortcuts Terraform-managed Docker labs often include: Grafana admin credentials in container environment variables. Concourse local users such as admin:admin. Vault dev server tokens in shell environment files. database passwords pulled into Terraform state. generated private keys written to local files. privileged containers for CI workers or system exporters. localhost endpoints that assume a single operator workstation. Each shortcut may be acceptable in a disposable lab. None should cross into shared infrastructure by accident. ...