Skip to content

Instantly share code, notes, and snippets.

@frobware
Last active October 12, 2026 15:33
Show Gist options
  • Select an option

  • Save frobware/3561697d5f602cdeee4767a7d86ccbd4 to your computer and use it in GitHub Desktop.

Select an option

Save frobware/3561697d5f602cdeee4767a7d86ccbd4 to your computer and use it in GitHub Desktop.
OCPBUGS draft: advertised layer2 primary UDN drops pod traffic until ovnkube-node restart

Title: Advertised layer2 primary UDN drops pod traffic on some nodes until ovnkube-node is restarted

Component: Networking / ovn-kubernetes

Description of problem

If a RouteAdvertisements and the layer2 primary ClusterUserDefinedNetwork it selects are created a few seconds apart, some workers end up advertising the network over BGP but do not forward traffic to its pods. Nothing reports an error: the CUDN is NetworkCreated=True and the RouteAdvertisements is Accepted=True. Restarting ovnkube-node on the worker fixes it.

Version-Release number of selected component

4.21.35, 4.22.16 and 4.23 CI payloads. Seen on AWS IPI and ROSA HCP.

How reproducible

Intermittent. Running the script below for 12 attempts on AWS 4.21.35, 6 attempts left at least one worker broken.

Steps to reproduce

  1. Use a cluster with at least three workers. Enable FRR and route advertisements:
oc patch network.operator.openshift.io cluster --type=merge -p '{"spec":{"additionalRoutingCapabilities":{"providers":["FRR"]},"defaultNetwork":{"ovnKubernetesConfig":{"routeAdvertisements":"Enabled"}}}}'

Wait for the frr-k8s pods to be ready and the ovnkube-node rollout to finish.

  1. Save these two files.

repro.yaml, the FRRConfiguration and RouteAdvertisements. The BGP neighbour does not need to exist.

apiVersion: frrk8s.metallb.io/v1beta1
kind: FRRConfiguration
metadata:
  name: repro
  namespace: openshift-frr-k8s
spec:
  bgp:
    routers:
    - asn: 65001
      neighbors:
      - address: 10.0.0.250
        asn: 65000
        disableMP: true
---
apiVersion: k8s.ovn.org/v1
kind: RouteAdvertisements
metadata:
  name: repro
spec:
  advertisements:
  - PodNetwork
  frrConfigurationSelector: {}
  nodeSelector: {}
  networkSelectors:
  - networkSelectionType: ClusterUserDefinedNetworks
    clusterUserDefinedNetworkSelector:
      networkSelector:
        matchLabels:
          advertise: "true"

network.yaml, the namespace and the CUDN:

apiVersion: v1
kind: Namespace
metadata:
  name: repro
  labels:
    k8s.ovn.org/primary-user-defined-network: ""
    cluster-udn: repro
---
apiVersion: k8s.ovn.org/v1
kind: ClusterUserDefinedNetwork
metadata:
  name: repro
  labels:
    advertise: "true"
spec:
  namespaceSelector:
    matchLabels:
      cluster-udn: repro
  network:
    topology: Layer2
    layer2:
      role: Primary
      ipam:
        lifecycle: Persistent
      subnets:
      - 10.100.0.0/16
  1. Save and run this script, which creates the RouteAdvertisements, then the network five seconds later, then checks every worker. It deletes both and repeats. You can run it again without restarting anything.
#!/bin/bash
# Usage: repro.sh <attempts>
ns=openshift-ovn-kubernetes
for i in $(seq "$1"); do
  oc delete -f network.yaml --ignore-not-found --wait=true >/dev/null
  oc delete routeadvertisements repro --ignore-not-found --wait=true >/dev/null
  sleep 10
  oc apply -f repro.yaml >/dev/null 2>&1
  sleep 5
  oc apply -f network.yaml >/dev/null
  oc wait routeadvertisements repro --for=condition=Accepted --timeout=120s >/dev/null
  sleep 20
  for node in $(oc get nodes -l node-role.kubernetes.io/worker -o name | cut -d/ -f2); do
    pod=$(oc -n $ns get pod -l app=ovnkube-node --field-selector spec.nodeName=$node -o name)
    flows=$(oc -n $ns exec $pod -c ovn-controller -- ovs-ofctl dump-flows br-ex |
      grep -cE 'priority=(300|104),.*10.100.0.0/16 actions=output')
    port=$(oc -n $ns exec $pod -c nbdb -- ovn-nbctl --no-leader-only --bare --columns=name \
      find Logical_Switch_Port name=stotr-cluster_udn_repro_ovn_layer2_switch)
    snat=$(oc -n $ns exec $pod -c nbdb -- ovn-nbctl --no-leader-only \
      lr-nat-list GR_cluster_udn_repro_$node | grep ' 10.100.0.0/16' | grep -c '==')
    r=OK
    [ "$flows" = 2 ] || r="$r BUG1:br-ex-flows-missing"
    [ -n "$port" ]   || r="$r BUG2:stotr-port-missing"
    [ "$snat" -ge 1 ] || r="$r BUG3:snat-unconditional"
    echo "attempt $i $node ${r/#OK /}"
  done
done
./repro.sh 10

Actual results

On some attempts, one or more workers report a fault. Excerpt from ./repro.sh 12 on AWS 4.21.35:

attempt 1 ip-10-0-52-204.us-east-2.compute.internal OK
attempt 1 ip-10-0-6-104.us-east-2.compute.internal OK
attempt 1 ip-10-0-94-245.us-east-2.compute.internal OK
attempt 2 ip-10-0-52-204.us-east-2.compute.internal BUG1:br-ex-flows-missing BUG3:snat-unconditional
attempt 2 ip-10-0-6-104.us-east-2.compute.internal BUG1:br-ex-flows-missing BUG3:snat-unconditional
attempt 2 ip-10-0-94-245.us-east-2.compute.internal BUG1:br-ex-flows-missing BUG3:snat-unconditional
attempt 9 ip-10-0-52-204.us-east-2.compute.internal OK
attempt 9 ip-10-0-6-104.us-east-2.compute.internal OK
attempt 9 ip-10-0-94-245.us-east-2.compute.internal BUG3:snat-unconditional
  • BUG1:br-ex-flows-missing: br-ex has no forwarding flows for 10.100.0.0/16, so traffic to the pods stops at the node.
  • BUG2:stotr-port-missing: the logical switch port joining the network's switch to its transit router does not exist, and ovnkube-controller logs failed to get stor port on every gateway sync.
  • BUG3:snat-unconditional: the network gateway router's SNAT for 10.100.0.0/16 has no match, so replies leave with the masquerade address instead of the pod's address. It is usually seen together with BUG1, and sometimes on its own.

BUG2 is rare on the stock code: once in 40 attempts on AWS 4.22.16.

Expected results

Every worker reports OK on every attempt.

Additional info

There are three separate bugs in ovnkube-node:

  1. The Accepted event arrives while the network controller is still starting, and is never applied.
  2. A node's first add, when retried, syncs only the gateway and never creates the transit router port.
  3. A gateway sync that started before the advertisement changed clears the request for another sync.

Fixes, one commit per bug: openshift/ovn-kubernetes#3513 (release-4.23) and openshift/ovn-kubernetes#3518 (release-4.22). With these, the same sequence left no worker faulty in 120 attempts on AWS, Azure and GCP across 4.21, 4.22 and 4.23.

Workaround: restart ovnkube-node on the affected worker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment