Title: Advertised layer2 primary UDN drops pod traffic on some nodes until ovnkube-node is restarted
Component: Networking / ovn-kubernetes
If a RouteAdvertisements and the layer2 primary ClusterUserDefinedNetwork it selects are created a few seconds apart, some workers end up advertising the network over BGP but do not forward traffic to its pods. Nothing reports an error: the CUDN is NetworkCreated=True and the RouteAdvertisements is Accepted=True. Restarting ovnkube-node on the worker fixes it.
4.21.35, 4.22.16 and 4.23 CI payloads. Seen on AWS IPI and ROSA HCP.
Intermittent. Running the script below for 12 attempts on AWS 4.21.35, 6 attempts left at least one worker broken.
- Use a cluster with at least three workers. Enable FRR and route advertisements:
oc patch network.operator.openshift.io cluster --type=merge -p '{"spec":{"additionalRoutingCapabilities":{"providers":["FRR"]},"defaultNetwork":{"ovnKubernetesConfig":{"routeAdvertisements":"Enabled"}}}}'
Wait for the frr-k8s pods to be ready and the ovnkube-node rollout to finish.
- Save these two files.
repro.yaml, the FRRConfiguration and RouteAdvertisements. The BGP neighbour does not need to exist.
apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
name: repro
namespace: openshift-frr-k8s
spec:
bgp:
routers:
- asn: 65001
neighbors:
- address: 10.0.0.250
asn: 65000
disableMP: true
---
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
name: repro
spec:
advertisements:
- PodNetwork
frrConfigurationSelector: {}
nodeSelector: {}
networkSelectors:
- networkSelectionType: ClusterUserDefinedNetworks
clusterUserDefinedNetworkSelector:
networkSelector:
matchLabels:
advertise: "true"network.yaml, the namespace and the CUDN:
apiVersion: v1
kind: Namespace
metadata:
name: repro
labels:
k8s.ovn.org/primary-user-defined-network: ""
cluster-udn: repro
---
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
name: repro
labels:
advertise: "true"
spec:
namespaceSelector:
matchLabels:
cluster-udn: repro
network:
topology: Layer2
layer2:
role: Primary
ipam:
lifecycle: Persistent
subnets:
- 10.100.0.0/16- Save and run this script, which creates the RouteAdvertisements, then the network five seconds later, then checks every worker. It deletes both and repeats. You can run it again without restarting anything.
#!/bin/bash
# Usage: repro.sh <attempts>
ns=openshift-ovn-kubernetes
for i in $(seq "$1"); do
oc delete -f network.yaml --ignore-not-found --wait=true >/dev/null
oc delete routeadvertisements repro --ignore-not-found --wait=true >/dev/null
sleep 10
oc apply -f repro.yaml >/dev/null 2>&1
sleep 5
oc apply -f network.yaml >/dev/null
oc wait routeadvertisements repro --for=condition=Accepted --timeout=120s >/dev/null
sleep 20
for node in $(oc get nodes -l node-role.kubernetes.io/worker -o name | cut -d/ -f2); do
pod=$(oc -n $ns get pod -l app=ovnkube-node --field-selector spec.nodeName=$node -o name)
flows=$(oc -n $ns exec $pod -c ovn-controller -- ovs-ofctl dump-flows br-ex |
grep -cE 'priority=(300|104),.*10.100.0.0/16 actions=output')
port=$(oc -n $ns exec $pod -c nbdb -- ovn-nbctl --no-leader-only --bare --columns=name \
find Logical_Switch_Port name=stotr-cluster_udn_repro_ovn_layer2_switch)
snat=$(oc -n $ns exec $pod -c nbdb -- ovn-nbctl --no-leader-only \
lr-nat-list GR_cluster_udn_repro_$node | grep ' 10.100.0.0/16' | grep -c '==')
r=OK
[ "$flows" = 2 ] || r="$r BUG1:br-ex-flows-missing"
[ -n "$port" ] || r="$r BUG2:stotr-port-missing"
[ "$snat" -ge 1 ] || r="$r BUG3:snat-unconditional"
echo "attempt $i $node ${r/#OK /}"
done
done./repro.sh 10
On some attempts, one or more workers report a fault. Excerpt from ./repro.sh 12 on AWS 4.21.35:
attempt 1 ip-10-0-52-204.us-east-2.compute.internal OK
attempt 1 ip-10-0-6-104.us-east-2.compute.internal OK
attempt 1 ip-10-0-94-245.us-east-2.compute.internal OK
attempt 2 ip-10-0-52-204.us-east-2.compute.internal BUG1:br-ex-flows-missing BUG3:snat-unconditional
attempt 2 ip-10-0-6-104.us-east-2.compute.internal BUG1:br-ex-flows-missing BUG3:snat-unconditional
attempt 2 ip-10-0-94-245.us-east-2.compute.internal BUG1:br-ex-flows-missing BUG3:snat-unconditional
attempt 9 ip-10-0-52-204.us-east-2.compute.internal OK
attempt 9 ip-10-0-6-104.us-east-2.compute.internal OK
attempt 9 ip-10-0-94-245.us-east-2.compute.internal BUG3:snat-unconditional
BUG1:br-ex-flows-missing: br-ex has no forwarding flows for 10.100.0.0/16, so traffic to the pods stops at the node.BUG2:stotr-port-missing: the logical switch port joining the network's switch to its transit router does not exist, and ovnkube-controller logsfailed to get stor porton every gateway sync.BUG3:snat-unconditional: the network gateway router's SNAT for 10.100.0.0/16 has no match, so replies leave with the masquerade address instead of the pod's address. It is usually seen together with BUG1, and sometimes on its own.
BUG2 is rare on the stock code: once in 40 attempts on AWS 4.22.16.
Every worker reports OK on every attempt.
There are three separate bugs in ovnkube-node:
- The Accepted event arrives while the network controller is still starting, and is never applied.
- A node's first add, when retried, syncs only the gateway and never creates the transit router port.
- A gateway sync that started before the advertisement changed clears the request for another sync.
Fixes, one commit per bug: openshift/ovn-kubernetes#3513 (release-4.23) and openshift/ovn-kubernetes#3518 (release-4.22). With these, the same sequence left no worker faulty in 120 attempts on AWS, Azure and GCP across 4.21, 4.22 and 4.23.
Workaround: restart ovnkube-node on the affected worker.