[bug] wireguard with cillium example doesn't work

[csantanapr]

Description

When using the example wireguard-with-cilium the example pod client is not communicating because is configured before cilium wireguard is established.

we need to put some type of sleep/delay before creating the client pod

• ✋ I have searched the open/closed issues and my issue is not listed.

Steps to reproduce the behavior:

git clone branch
cd examples/wireguard-with-cilium/
terraform init
terraform apply -auto-approve

follow readme and run tcpdump -A -c 3 -i cilium_wg0 command

Expected behaviour

For tcpdump -A -c 3 -i cilium_wg0 to show encrypted traffic

Actual behaviour

The tcpdump -A -c 3 -i cilium_wg0coammand doesn't show any output

Additional context

Thank you to @aaroniscode and the blog Transparent encryption of node to node traffic on Amazon EKS using WireGuard and Cilium he authored to debug the issue

The steps on the blog uses eksctl and there is more time between installing cilium and deploying the client pod, this is why it works if you use the blog instructions

[csantanapr]

It looks like is not a race condition, is about not allowing UDP/51871 communication between nodes, cilium wireguard is configured to tunnel all encryption data thru this port

I tested the example disabling the creation of security groups in eks module and it worked out of the box

  create_node_security_group = false

Now testing adding the udp/51871 ingress, because I don't we want to recommend to open everything up between the nodes

  # Extend node-to-node security group rules
  node_security_group_additional_rules = {

    ingress_cilium_wireguard = {
      description = "Allow Cilium Wireguard node to node"
      protocol    = "udp"
      from_port   = 51871
      to_port     = 51871  # Cilium Wireguard Port https://github.com/cilium/cilium/blob/main/Documentation/security/network/encryption-wireguard.rst
      type        = "ingress"
      self        = true
    }
  }

cc @askulkarni2 @aaroniscode

[csantanapr]

Allowing traffic is not enough to get encrypted traffic working.

I tried with with all default security groups and additional one for udp/51871 with the client/server pods deployed with terraform together with cilium and the traffic was flowing but NOT thru the wrireguard tunnel.

Then I tried with with all default security groups and additional one for udp/51871 without the client/server pods deployed with terraform together with cilium. Then waited ~5mins then deploy the client/server pods and then traffic was flowing thru wireguard tunnel.

[bryantbiggs]

closed in #1619