Feature Request: Remove limitation for ingress controller service being a LoadBalancer service type

[cawirawa]

Overview

The current implementation of Caddy Ingress restricts the controller service to operate only with a LoadBalancer service type. This feature request aims to enhance flexibility by allowing the Caddy Ingress controller service to run within the Kubernetes cluster, eliminating the dependency on an external load balancer in certain deployment scenarios.

Use Cases

• Environments where a dedicated load balancer is not necessary or available.
• Development and testing setups where simplicity and resource efficiency are prioritized.
• Deployments in which the cluster itself performs load balancing.
• When using AWS EKS, in order to only expose the ingress to internal VPC subnets, AWS Load Balancer Controller is required. It sits on top of caddy ingress controller which eliminates the need for caddy ingress controller to be a LoadBalancer. In this scenario, the ideal type should be ClusterIP and the external IP will be issued by AWS LB Controller which routes the traffic to Caddy ingress controller.

Rationale

This change allows users to optimize resource utilization and simplifies the setup, particularly in scenarios where a separate load balancer introduces unnecessary complexity.