RKE2 clusters across multiple sites had inconsistent DNS search domain behavior. Some nodes appended internal domains to all lookups, some appended data center-specific domains, and a few were clean. In a Kubernetes cluster, uncontrolled search domain expansion causes:

  • Pod DNS lookups that should resolve as-is getting unexpected suffix expansion.
  • Inconsistent behavior across sites for the same service name.
  • Debugging sessions that waste time on DNS before finding the real issue.

Desired State

Layer Target
Netplan search: []
systemd-resolved No per-link or global domains
Active resolver (/etc/resolv.conf) search .

search . is systemd-resolved shorthand for “no search domains” and is the correct end state.

Operating Pattern

1. Audit

Run against all nodes to produce a CSV of current state:

./scripts/fix-dns-search.sh --audit-only <site> <env> <inventory>

The audit must also prove coverage. Compare the targeted Ansible host count with the CSV data row count, and print raw Ansible failures when a host does not return a row. Otherwise SSH or sudo failures can disappear behind a sed filter that only keeps successful CSV|... markers.

Example output:

resolv_conf_search netplan_search Meaning
. [] Clean
internal.corp.example [] Netplan staged, not applied
dc.corp.example NONE Link-level injection (DHCP/systemd-networkd)

2. Stage

For netplan-managed nodes, validate configs without applying:

./scripts/fix-dns-search.sh --stage <site> <env> <inventory>

Uses netplan generate only — no runtime impact.

3. Runtime Fix (If Needed)

For nodes with per-link injection (netplan shows NONE):

resolvectl domain eth0 ""

Do not restart systemd-resolved afterward.

4. Apply (Maintenance Window)

./scripts/fix-dns-search.sh --apply <site> <env> <inventory>

5. Verify Drift

awk -F, 'NR==1 || $3!="." || $4!="[]"' <audit-csv> | column -s, -t

Header-only output = clean.

Edge Cases Encountered

  • Subiquity YAML indentation: Ubuntu installer sometimes writes search: at wrong indentation. Add a YAML normalization step to any netplan automation.
  • set -o pipefail in Ansible: /bin/sh on Ubuntu does not support it. Use set -eu inside shell tasks.
  • stale /etc/resolv.conf after netplan fix: If the file is a symlink to stub-resolv.conf, do not edit it directly. The change comes from systemd-resolved, not the file.
  • short audit CSV: If the host list has 13 nodes but the CSV has 6 rows, the audit is incomplete. Capture raw Ansible output, print unreachable/failed hosts, and warn on host-count versus row-count mismatch before reading drift results.