Rancher cluster-agent errors can look like Rancher is the root cause. Sometimes Rancher is only the first component sensitive enough to report a degraded network path.

Use this note when Rancher-managed clusters show large management-plane traffic, intermittent disconnects, or websocket errors.

Common Symptoms

In downstream cluster-agent logs:

Failed to dial steve aggregation server
Remotedialer proxy error
websocket: close 1006 (abnormal closure)
context deadline exceeded
i/o timeout
connection reset by peer

In Rancher server logs:

tunnel disconnect
ReverseProxy read error during body copy
unable to decode event from watch stream
Failed to watch Namespace
Failed to watch Secret
TLS handshake timeout

In ingress or load balancer logs:

499
502
504
upstream timed out
client disconnected
websocket EOF

These messages prove connection disruption. They do not prove Rancher generated the original traffic spike.

Confirm The Rancher Endpoint

Resolve the Rancher endpoint and identify whether it is a VIP, load balancer, or DNS round-robin target:

nslookup rancher.example.internal
dig rancher.example.internal
dig -x <rancher-vip>

If the VIP fronts HAProxy, Keepalived, ingress, or multiple Rancher nodes, inspect each layer separately.

Find Cluster-Agent Placement

On the downstream cluster:

kubectl get pods -A -o wide | egrep 'cattle|fleet|rancher|system-agent'
kubectl -n cattle-system get pods -o wide

Then check the noisy agent:

kubectl -n cattle-system logs <cattle-cluster-agent-pod> --tail=300 \
  | egrep -i 'error|warn|websocket|disconnect|reconnect|timeout|reset|steve|remotedialer'

Compare replicas. One noisy replica and one quiet replica can point to node placement, pod network, or a backend path issue.

Test From The Agent Pod

Connectivity to Rancher should work from the same pod that reports errors:

kubectl -n cattle-system exec <cattle-cluster-agent-pod> -- \
  curl -sk -o /dev/null -w 'http=%{http_code} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}\n' \
  https://rancher.example.internal/v3/connect/config

An HTTP 401 can be a healthy unauthenticated response. Timeouts, slow connects, TLS delays, or intermittent failures are the important signals.

If the VIP has known backend members, test each one:

for ip in <lb-member-1> <lb-member-2>; do
  kubectl -n cattle-system exec <cattle-cluster-agent-pod> -- \
    curl -sk --resolve rancher.example.internal:443:${ip} \
    -o /dev/null -w "${ip} http=%{http_code} connect=%{time_connect} tls=%{time_appconnect} total=%{time_total}\n" \
    https://rancher.example.internal/v3/connect/config
done

One slow or failing backend can create intermittent remotedialer churn.

Check The Load Balancer Layer

On HAProxy or the Rancher load balancer:

sudo journalctl -u haproxy --since '24 hours ago'
sudo journalctl -u keepalived --since '24 hours ago'

Summarize top clients:

sudo journalctl -u haproxy --since '24 hours ago' \
  | awk '/rancher/ {print $6}' \
  | cut -d: -f1 \
  | sort | uniq -c | sort -nr | head -20

Check websocket-related timeouts:

grep -i timeout /etc/haproxy/haproxy.cfg

For Rancher, pay attention to long-lived connection settings such as timeout tunnel, timeout client, and timeout server.

Correlate The Time Window

Do not accept nearby logs as proof. Convert UTC and local time carefully.

For a suspected event window:

kubectl -n cattle-system logs <cattle-cluster-agent-pod> \
  --since-time='<event-start-utc>' \
  | egrep -i 'timeout|websocket|remotedialer|disconnect|reset'

Then compare with:

  • backup schedules.
  • HAProxy logs.
  • ingress logs.
  • flow collector timestamps.
  • node-level network counters.
  • Rancher server logs.

If the Rancher errors happened hours after the traffic spike, treat them as a separate symptom until proven otherwise.

Operating Rule

Rancher remotedialer errors are a signal to inspect the management path, not a final diagnosis.

Before blaming Rancher, check whether another workload saturated the network and Rancher simply reported the pain first.