refactor: rate limit policy enforced status controller

[KevFan]

Description

Closes: #587

Introduces a new controller for reconciling enforcement status for RLPs as with the default / overrides work, the enforecement status of multiple RLPs need to be calculated from a singular RLP event.

Superseds #591

• Fixes RLP GW policy status is not updated when Route RLP is deleted and vice versa

Verification

Code changes are tested with the integration test changes, so passing integration tests should be sufficient.

If you want verify manually:

• Checkout this branch
• Deploy

make local-setup

• Deploy kuadrant

kubectl apply  -f - <<EOF               
---                                     
apiVersion: kuadrant.io/v1beta1
kind: Kuadrant
metadata:
  name: kuadrant-sample
spec: {}
EOF

• Deploy toystore

kubectl apply -f examples/toystore/toystore.yaml
kubectl wait --timeout=300s --for=condition=Available deployment toystore

• Create HTTP Route

kubectl apply -f - <<EOF
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata:
  name: toystore
spec:
  parentRefs:
  - name: istio-ingressgateway
    namespace: istio-system
  hostnames:
  - api.toystore.com
  rules:
  - matches:
    - method: GET
      path:
        type: PathPrefix
        value: "/toys"
    backendRefs:
    - name: toystore
      port: 80
  - matches: # it has to be a separate HTTPRouteRule so we do not rate limit other endpoints
    - method: POST
      path:
        type: Exact
        value: "/toys"
    backendRefs:
    - name: toystore
      port: 80
EOF

• Watch RLP statues

watch -n 0.1 "kubectl get ratelimitpolicy -A -o yaml | yq '.items[].status'"

• Create GW RLP with defaults

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta2
kind: RateLimitPolicy
metadata:
  name: gw-rlp
  namespace: istio-system
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: Gateway
    name: istio-ingressgateway
  defaults:
    limits:
      "gateway":
        rates:
        - limit: 5
          duration: 10
          unit: second
EOF

• Verify GW Policy is accepted and enforced

• Port forward gateway service

kubectl port-forward -n istio-system service/istio-ingressgateway-istio 9080:80 2>&1 >/dev/null &
export GATEWAY_URL=localhost:9080

• Ensure HTTP Route is rate limit at 5 requests per 10 seconds

while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Host: api.toystore.com' http://$GATEWAY_URL/toys -X POST | grep -E --color "\b(429)\b|$"; sleep 1; done

• Create HTTPRoute RLP with implicit default limits

kubectl apply -f - <<EOF
apiVersion: kuadrant.io/v1beta2
kind: RateLimitPolicy
metadata:
  name: toystore
spec:
  targetRef:
    group: gateway.networking.k8s.io
    kind: HTTPRoute
    name: toystore
  limits:
    "route":
      rates:
      - limit: 8
        duration: 10
        unit: second
EOF

• Verify Route policy is accepted and enforced
• Verify GW policy is accepted and not enforced - overridden by route policy

• Ensure HTTP Route is rate limted at 8 requests per second (Route RLP defaults takes precedence over gateway defaults) (Note: you might need to port forward the gateway service again)

while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Host: api.toystore.com' http://$GATEWAY_URL/toys -X POST | grep -E --color "\b(429)\b|$"; sleep 1; done

• Delete route RLP

kubectl delete ratelimitpolicy toystore

• Verify GW policy goes back to enforced true

• Ensure HTTP Route is rate limits at 5 requests per second again (GW RLP defaults)

while :; do curl --write-out '%{http_code}\n' --silent --output /dev/null -H 'Host: api.toystore.com' http://$GATEWAY_URL/toys -X POST | grep -E --color "\b(429)\b|$"; sleep 1; done

[guicassolato]

LGTM

Fun fact – In the verification steps, I actually got the Enforced status condition added to the RLP before the Accepted one. It was of course a matter of milliseconds, but enough to spot it while watching the status of the policy.

I guess it's nothing to worry about, but it does mean RFC 0004 is no longer being strictly implemented.

cc @didierofrivia