Istio-Kafka Authentication and Authorization Setup

This repository contains the necessary configurations and deployment files to set up a Kafka consumer application with Istio, including JWT authentication and authorization policies.

Prerequisites

• Kubernetes cluster (Minikube, kind, or managed Kubernetes service)
• kubectl configured to interact with your cluster
• Helm installed
• Istio CLI (istioctl) installed
• Calico for network policies
• Docker installed for building images
• GitHub repository with secrets configured for Docker Hub

Step-by-Step Instructions

1. Install Istio

1. Download Istio:

   curl -L https://istio.io/downloadIstio | sh -
   cd istio-1.14.1
   export PATH=$PWD/bin:$PATH

2. Install Istio Base:

   istioctl install --set profile=demo -y

3. Enable Ingress and Egress Gateways:

   kubectl apply -f samples/addons

2. Install Calico

1. Install Calico for Network Policies:

   kubectl apply -f https://docs.projectcalico.org/manifests/calico.yaml

3. Deploy Zookeeper

1. Create Kafka Namespace:

   kubectl create namespace kafka

2. Deploy Zookeeper:

   kubectl apply -f deployments/zookeeper-deployment.yaml

4. Deploy Kafka

1. Deploy Kafka:

   kubectl apply -f deployments/kafka-deployment.yaml
   kubectl apply -f deployments/kafka-service.yaml

5. Deploy Kafka Consumer

1. Build and Push Docker Image:

   • Configure GitHub repository secrets as described in the setup section.
   • Push your changes to the main branch.

2. Deploy Kafka Consumer:

   kubectl apply -f deployments/kafka-consumer-deployment.yaml
   kubectl apply -f deployments/kafka-consumer-service.yaml

6. Deploy JWT Issuer

1. Build and Push Docker Image:

   docker build -t jwt-issuer:latest ./jwt-issuer
   docker tag jwt-issuer:latest <your-dockerhub-username>/jwt-issuer:latest
   docker push <your-dockerhub-username>/jwt-issuer:latest

2. Deploy JWT Issuer:

   kubectl apply -f deployments/jwt-issuer-deployment.yaml
   kubectl apply -f deployments/jwt-issuer-service.yaml

7. Configure Istio for Kafka

1. Label Kafka Namespace for Istio Injection:

   kubectl label namespace kafka istio-injection=enabled

2. Create VirtualService and DestinationRule:

   kubectl apply -f istio/kafka-virtualservice.yaml
   kubectl apply -f istio/kafka-destinationrule.yaml

3. Create Gateway and External VirtualService:

   kubectl apply -f istio/kafka-gateway.yaml
   kubectl apply -f istio/kafka-external-virtualservice.yaml

8. Implement JWT Authentication

1. Create JWT Authentication Policy:

   kubectl apply -f istio/kafka-jwt-authentication.yaml

9. Implement Authorization Policy

1. Create Authorization Policy:

   kubectl apply -f istio/kafka-consumer-authorization.yaml

10. Restrict Access to Producer and Use JWT

1. Create AuthorizationPolicy to restrict access to the Kafka broker:

   kubectl apply -f istio/kafka-authorization.yaml

11. Enable Mutual TLS

1. Create PeerAuthentication Policy:

   kubectl apply -f istio/peer-authentication.yaml

12. Observability with Istio

1. Deploy Prometheus, Grafana, and Jaeger:

   kubectl apply -f istio/samples-addons/prometheus.yaml
   kubectl apply -f istio/samples-addons/grafana.yaml
   kubectl apply -f istio/samples-addons/jaeger.yaml

2. Access Grafana Dashboard:

   istioctl dashboard grafana

3. Access Jaeger Tracing:

   istioctl dashboard jaeger

13. Configure External Kafka Producer

1. Build and Run Kafka Producer Docker Image:

   docker build -t kafka-producer:latest ./kafka-producer
   docker run --rm -it kafka-producer:latest

Verifying the Setup

Test JWT Authentication

1. Ensure you have a valid JWT token from the specified issuer.
2. Make a request to the Kafka consumer service with the JWT token in the Authorization header:

   curl --header "Authorization: Bearer <your-jwt-token>" http://<kafka-consumer-service-url>

Test Authorization

1. Ensure that only requests matching the authorization policy (e.g., GET requests to /) are allowed:

   curl --header "Authorization: Bearer <your-jwt-token>" http://<kafka-consumer-service-url>

Verify External Producer

1. Check the logs of the Kafka consumer in Kubernetes to ensure it received messages from the external producer:

   kubectl logs <kafka-consumer-pod-name> -n kafka

Files

• deployments/kafka-consumer-deployment.yaml: Deployment configuration for Kafka consumer.
• deployments/kafka-consumer-service.yaml: Service configuration for Kafka consumer.
• deployments/kafka-service.yaml: Service configuration for Kafka.
• deployments/zookeeper-deployment.yaml: Deployment configuration for Zookeeper.
• deployments/kafka-deployment.yaml: Deployment configuration for Kafka.
• deployments/jwt-issuer-deployment.yaml: Deployment configuration for JWT issuer.
• deployments/jwt-issuer-service.yaml: Service configuration for JWT issuer.
• istio/kafka-jwt-authentication.yaml: JWT authentication policy.
• istio/kafka-consumer-authorization.yaml: Authorization policy for Kafka consumer.
• istio/kafka-virtualservice.yaml: VirtualService for Kafka.
• istio/kafka-destinationrule.yaml: DestinationRule for Kafka.
• istio/kafka-gateway.yaml: Gateway configuration for external access.
• istio/kafka-external-virtualservice.yaml: VirtualService for external access.
• istio/kafka-authorization.yaml: AuthorizationPolicy for restricting access.
• istio/peer-authentication.yaml: PeerAuthentication policy to enable mutual TLS.
• istio/samples-addons/prometheus.yaml: Prometheus addon configuration.
• istio/samples-addons/grafana.yaml: Grafana addon configuration.
• istio/samples-addons/jaeger.yaml: Jaeger addon configuration.
• jwt-issuer/Dockerfile: Dockerfile for JWT issuer.
• jwt-issuer/package.json: Package.json for JWT issuer.
• jwt-issuer/server.js: Server script for JWT issuer.
• jwt-issuer/jwks.json: JWKS file for JWT issuer.
• kafka-producer/Dockerfile: Dockerfile for Kafka producer.
• kafka-producer/kafka_producer.py: Kafka producer script to run in the Docker container.
• kafka-consumer/Dockerfile: Dockerfile for Kafka consumer.
• kafka-consumer/kafka_consumer.py: Kafka consumer script to run in the Docker container.