Discussion space for CI/CD pipeline design work.
Running Concourse CI/CD on Windows Docker is uncommon enough that the setup patterns deserve their own reference. The Terraform provider, key generation, and security hardening steps differ significantly from Linux-based deployments. For a complete working example, see the concourse-terraform-windows directory in the IaC repository. Docker Transport Windows Docker uses named pipes instead of Unix sockets: provider "docker" { host = "npipe:////./pipe/docker_engine" } This is the first thing to verify when a Terraform Docker provider fails on Windows. The connection string is different, and not all provider features work identically on the Windows engine. ...
Concourse works well for infrastructure when the pipeline graph tells the truth about the change flow. Jobs should be small enough to understand and resources should capture the actual inputs. Pipeline Shape Common jobs: lint. validate. plan. plan review or approval. apply non-production. apply production. post-apply verification. Use serial groups for stateful targets so two applies do not compete for the same backend or environment. Task Design Tasks should: run from versioned scripts. print the exact target environment. fail loudly on missing variables. keep secrets out of logs. publish plan or verification artifacts. Resource Patterns Useful resources: ...
Deployment strategy labs are useful when they make failure handling visible. The goal is not to prove that blue-green, canary, feature toggles, or rollback scripts are fashionable. The goal is to understand which failure mode each strategy reduces and which operational cost it adds. Strategy Comparison Useful lab scenarios include: blue-green switching for fast rollback to a known environment. canary rollout for limiting user exposure. feature toggles for disabling behavior without redeploying. rollback packages or manifests for restoring a previous version. health checks that block promotion or trigger investigation. Each strategy should be tested with a deliberate failure. If the lab only demonstrates a successful deployment, it misses the point. ...
GitOps for a platform team is not about syncing a Kubernetes manifest directory. It is about making Git the source of truth for infrastructure state and using pipelines to enforce that state across environments, sites, and provider boundaries. This page collects patterns, pipeline shapes, and operating model decisions for GitOps in infrastructure teams. Scope These patterns apply to: Multi-site RKE2 cluster fleets. vSphere and cloud provider resource lifecycle. Template and image pipeline workflows. Platform tenant onboarding. Secrets and certificate lifecycle management. Configuration drift detection and remediation. The common thread is that Git is the single entry point for change, and pipelines are the only path to production. ...
When Terraform manages Helm releases, validation needs to happen at more than one layer. Terraform can show that the helm_release resource is configured. Helm can render templates. Kubernetes can reject invalid manifests. Production can still fail if the workload has no useful health checks or the image tag points at the wrong build. Validation Layers A practical validation sequence is: terraform fmt -> terraform validate -> helm lint -> helm template -> kubectl dry-run -> terraform plan Each step answers a different question: ...
A local Kafka Streams pipeline managed by Terraform is useful for understanding the full data path before production dependencies are involved. Terraform handles the broker, topics, stream processor image, and container lifecycle — all through the Docker provider. For a complete working example, see the kafka-streams-pipeline directory in the IaC repository. The errors.txt file in the repository contains the exact build failure and resolution for the librdkafka C extension, which is the most common issue when containerizing Python Kafka applications. ...
Pipelines are privileged automation. Treat their credentials like production access, because that is what they are. Credential design should match the environment and action being performed. Credential Handling Use separate credentials for: read-only validation. non-production deploys. production plans. production applies. artifact publication. Prefer short-lived credentials from Vault, cloud identity federation, or platform-native workload identity. Avoid storing long-lived credentials in pipeline definitions. Resource Design Pipeline resources should make flow visible: source repository. versioned artifact. environment configuration. approval gate. deployment target. Do not let a pipeline secretly fetch mutable inputs without recording what it used. ...
Infrastructure pipelines should make change intent visible before they mutate anything. The stage design should separate fast feedback from approval and execution. Stage Model A practical infrastructure pipeline usually has: format -> validate -> security checks -> plan -> review -> apply -> verify Each stage should produce evidence that the next stage can trust. Validation Validation should be fast and safe: formatting. static syntax validation. provider initialization without touching production state when possible. unit-style checks for generated manifests. policy checks for obvious violations. Planning Plan output is the core review artifact. ...