Skip to content

Commit

Permalink
Fully support generation of K8s RBAC resources
Browse files Browse the repository at this point in the history
These changes address a long-time issue in regards of K8s RBAC resources (see related issues).

These changes allow to generate custom Roles, ClusterRoles, ServiceAccount, and RoleBindings.
Plus, it allows the Kubernetes Client and Kubernetes Config extensions to configure the role binding to generate.

Fix #16612
Fix #19286
Fix #15422
  • Loading branch information
Sgitario committed Mar 17, 2023
1 parent d08d31f commit aea3b57
Show file tree
Hide file tree
Showing 50 changed files with 1,734 additions and 153 deletions.
59 changes: 57 additions & 2 deletions docs/src/main/asciidoc/deploying-to-kubernetes.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -860,14 +860,69 @@ implementation("io.quarkus:quarkus-kubernetes-client")
----

To access the API server from within a Kubernetes cluster, some RBAC related resources are required (e.g. a ServiceAccount, a RoleBinding).
So, when the `kubernetes-client` extension is present, the `kubernetes` extension is going to create those resources automatically, so that application will be granted the `view` role.
If more roles are required, they will have to be added manually.
To ease the usage of the `kubernetes-client` extension, the `kubernetes` extension is going to generate a RoleBinding resource that binds a cluster role named "view" to the application ServiceAccount resource. It's important to note that the cluster role "view" won't be generated automatically, so it's expected that you have this cluster role with name "view" already installed in your cluster.

On the other hand, you can fully customize the roles, subjects and role bindings to generate using the properties under `quarkus.kubernetes.rbac.role-bindings`, and if present, the `kubernetes-client` extension will use it and hence won't generate any RoleBinding resource.

[NOTE]
====
You can disable the RBAC resources generation using the property `quarkus.kubernetes-client.generate-rbac=false`.
====

=== Generating RBAC resources

In some scenarios, it's necessary to generate additional https://kubernetes.io/docs/reference/access-authn-authz/rbac/[RBAC] resources that are used by Kubernetes to grant or limit access to other resources. For example, in our use case, we are building https://kubernetes.io/docs/concepts/extend-kubernetes/operator/#operators-in-kubernetes[a Kubernetes operator] that needs to read the list of the installed deployments. To do this, we would need to assign a service account to our operator and link this service account with a role that grants access to the Deployment resources. Let's see how to do this using the `quarkus.kubernetes.rbac` properties:

[source,properties]
----
# Generate the Role resource with name "my-role" <1>
quarkus.kubernetes.rbac.roles.my-role.policy-rules.0.api-groups=extensions,apps
quarkus.kubernetes.rbac.roles.my-role.policy-rules.0.resources=deployments
quarkus.kubernetes.rbac.roles.my-role.policy-rules.0.verbs=list
----

<1> In this example, the role "my-role" will be generated with a policy rule to get the list of deployments.

By default, if one role is configured, a RoleBinding resource will be generated as well to link this role with the ServiceAccount resource.

Moreover, you can have more control over the RBAC resources to be generated:

[source,properties]
----
# Generate Role resource with name "my-role" <1>
quarkus.kubernetes.rbac.roles.my-role.policy-rules.0.api-groups=extensions,apps
quarkus.kubernetes.rbac.roles.my-role.policy-rules.0.resources=deployments
quarkus.kubernetes.rbac.roles.my-role.policy-rules.0.verbs=get,watch,list
# Generate ServiceAccount resource with name "my-service-account" in namespace "my_namespace" <2>
quarkus.kubernetes.rbac.service-accounts.my-service-account.namespace=my_namespace
# Bind Role "my-role" with ServiceAccount "my-service-account" <3>
quarkus.kubernetes.rbac.role-bindings.my-role-binding.subjects.my-service-account.kind=ServiceAccount
quarkus.kubernetes.rbac.role-bindings.my-role-binding.subjects.my-service-account.namespace=my_namespace
quarkus.kubernetes.rbac.role-bindings.my-role-binding.role-name=my-role
----

<1> In this example, the role "my-role" will be generated with the specified policy rules.
<2> Also, the service account "my-service-account" will be generated.
<3> And we can configure the generated RoleBinding resource by selecting the role to be used and the subject.

Finally, we can also generate the cluster wide role resource of "ClusterRole" kind as follows:

[source,properties]
----
# Generate ClusterRole resource with name "my-cluster-role" <1>
quarkus.kubernetes.rbac.cluster-roles.my-cluster-role.policy-rules.0.api-groups=extensions,apps
quarkus.kubernetes.rbac.cluster-roles.my-cluster-role.policy-rules.0.resources=deployments
quarkus.kubernetes.rbac.cluster-roles.my-cluster-role.policy-rules.0.verbs=get,watch,list
# Bind the ClusterRole "my-cluster-role" with the application service account
quarkus.kubernetes.rbac.role-bindings.my-role-binding.role-name=my-cluster-role <2>
----

<1> In this example, the cluster role "my-cluster-role" will be generated with the specified policy rules.
<2> As we have configured only one role, this property is not really necessary.

=== Deploying to Minikube

https://github.com/kubernetes/minikube[Minikube] is quite popular when a Kubernetes cluster is needed for development purposes. To make the deployment to Minikube
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -54,7 +54,7 @@
import io.quarkus.kubernetes.client.runtime.KubernetesClientBuildConfig;
import io.quarkus.kubernetes.client.runtime.KubernetesClientProducer;
import io.quarkus.kubernetes.client.runtime.KubernetesConfigProducer;
import io.quarkus.kubernetes.spi.KubernetesRoleBindingBuildItem;
import io.quarkus.kubernetes.client.spi.KubernetesClientCapabilityBuildItem;
import io.quarkus.maven.dependency.ArtifactKey;

public class KubernetesClientProcessor {
Expand All @@ -69,6 +69,7 @@ public class KubernetesClientProcessor {
private static final DotName KUBE_SCHEMA = DotName.createSimple(KubeSchema.class.getName());
private static final DotName VISITABLE_BUILDER = DotName.createSimple(VisitableBuilder.class.getName());
private static final DotName CUSTOM_RESOURCE = DotName.createSimple(CustomResource.class.getName());
private static final String SERVICE_ACCOUNT = "ServiceAccount";

private static final DotName JSON_FORMAT = DotName.createSimple(JsonFormat.class.getName());
private static final String[] EMPTY_STRINGS_ARRAY = new String[0];
Expand Down Expand Up @@ -106,13 +107,13 @@ public void process(ApplicationIndexBuildItem applicationIndex, CombinedIndexBui
BuildProducer<ReflectiveClassBuildItem> reflectiveClasses,
BuildProducer<ReflectiveHierarchyBuildItem> reflectiveHierarchies,
BuildProducer<IgnoreJsonDeserializeClassBuildItem> ignoredJsonDeserializationClasses,
BuildProducer<KubernetesRoleBindingBuildItem> roleBindingProducer,
BuildProducer<ServiceProviderBuildItem> serviceProviderProducer) {
BuildProducer<ServiceProviderBuildItem> serviceProviderProducer,
BuildProducer<KubernetesClientCapabilityBuildItem> kubernetesClientCapabilityProducer) {

featureProducer.produce(new FeatureBuildItem(Feature.KUBERNETES_CLIENT));
if (kubernetesClientConfig.generateRbac) {
roleBindingProducer.produce(new KubernetesRoleBindingBuildItem("view", true));
}

kubernetesClientCapabilityProducer
.produce(new KubernetesClientCapabilityBuildItem(kubernetesClientConfig.generateRbac));

// register fully (and not weakly) for reflection watchers, informers and custom resources
final Set<DotName> watchedClasses = new HashSet<>();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -170,7 +170,9 @@ public class KubernetesClientBuildConfig {
public Optional<String[]> noProxy;

/**
* Enable the generation of the RBAC manifests.
* Enable the generation of the RBAC manifests. If enabled and no other role binding are provided using the properties
* `quarkus.kubernetes.rbac.`, it will generate a default role binding using the role "view" and the application
* service account.
*/
@ConfigItem(defaultValue = "true")
public boolean generateRbac;
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
package io.quarkus.kubernetes.client.spi;

import io.quarkus.builder.item.SimpleBuildItem;

public final class KubernetesClientCapabilityBuildItem extends SimpleBuildItem {

private final boolean generateRbac;

public KubernetesClientCapabilityBuildItem(boolean generateRbac) {
this.generateRbac = generateRbac;
}

public boolean isGenerateRbac() {
return generateRbac;
}
}
Original file line number Diff line number Diff line change
@@ -1,6 +1,5 @@
package io.quarkus.kubernetes.config.deployment;

import java.util.Collections;
import java.util.List;

import org.jboss.logmanager.Level;
Expand All @@ -15,12 +14,22 @@
import io.quarkus.kubernetes.config.runtime.KubernetesConfigBuildTimeConfig;
import io.quarkus.kubernetes.config.runtime.KubernetesConfigRecorder;
import io.quarkus.kubernetes.config.runtime.KubernetesConfigSourceConfig;
import io.quarkus.kubernetes.config.runtime.SecretsRoleConfig;
import io.quarkus.kubernetes.spi.KubernetesClusterRoleBuildItem;
import io.quarkus.kubernetes.spi.KubernetesRoleBindingBuildItem;
import io.quarkus.kubernetes.spi.KubernetesRoleBuildItem;
import io.quarkus.kubernetes.spi.KubernetesServiceAccountBuildItem;
import io.quarkus.kubernetes.spi.PolicyRule;
import io.quarkus.runtime.TlsConfig;

public class KubernetesConfigProcessor {

private static final String ANY_TARGET = null;
private static final List<PolicyRule> POLICY_RULE_FOR_ROLE = List.of(new PolicyRule(
List.of(""),
List.of("secrets"),
List.of("get")));

@BuildStep
@Record(ExecutionTime.RUNTIME_INIT)
public RunTimeConfigurationSourceValueBuildItem configure(KubernetesConfigRecorder recorder,
Expand All @@ -36,15 +45,28 @@ public RunTimeConfigurationSourceValueBuildItem configure(KubernetesConfigRecord
public void handleAccessToSecrets(KubernetesConfigSourceConfig config,
KubernetesConfigBuildTimeConfig buildTimeConfig,
BuildProducer<KubernetesRoleBuildItem> roleProducer,
BuildProducer<KubernetesClusterRoleBuildItem> clusterRoleProducer,
BuildProducer<KubernetesServiceAccountBuildItem> serviceAccountProducer,
BuildProducer<KubernetesRoleBindingBuildItem> roleBindingProducer,
KubernetesConfigRecorder recorder) {
if (buildTimeConfig.secretsEnabled) {
roleProducer.produce(new KubernetesRoleBuildItem("view-secrets", Collections.singletonList(
new KubernetesRoleBuildItem.PolicyRule(
Collections.singletonList(""),
Collections.singletonList("secrets"),
List.of("get")))));
roleBindingProducer.produce(new KubernetesRoleBindingBuildItem("view-secrets", false));
SecretsRoleConfig roleConfig = buildTimeConfig.secretsRoleConfig;
String roleName = roleConfig.name;
if (roleConfig.generate) {
if (roleConfig.clusterWide) {
clusterRoleProducer.produce(new KubernetesClusterRoleBuildItem(roleName,
POLICY_RULE_FOR_ROLE,
ANY_TARGET));
} else {
roleProducer.produce(new KubernetesRoleBuildItem(roleName,
roleConfig.namespace.orElse(null),
POLICY_RULE_FOR_ROLE,
ANY_TARGET));
}
}

serviceAccountProducer.produce(new KubernetesServiceAccountBuildItem(true));
roleBindingProducer.produce(new KubernetesRoleBindingBuildItem(roleName, roleConfig.clusterWide));
}

recorder.warnAboutSecrets(config, buildTimeConfig);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,4 +12,9 @@ public class KubernetesConfigBuildTimeConfig {
*/
@ConfigItem(name = "secrets.enabled", defaultValue = "false")
public boolean secretsEnabled;

/**
* Role configuration to generate if the "secrets-enabled" property is true.
*/
public SecretsRoleConfig secretsRoleConfig;
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
package io.quarkus.kubernetes.config.runtime;

import java.util.Optional;

import io.quarkus.runtime.annotations.ConfigGroup;
import io.quarkus.runtime.annotations.ConfigItem;

@ConfigGroup
public class SecretsRoleConfig {

/**
* The name of the role.
*/
@ConfigItem(defaultValue = "view-secrets")
public String name;

/**
* The namespace of the role.
*/
@ConfigItem
public Optional<String> namespace;

/**
* Whether the role is cluster wide or not. By default, it's not a cluster wide role.
*/
@ConfigItem(defaultValue = "false")
public boolean clusterWide;

/**
* If the current role is meant to be generated or not. If not, it will only be used to generate the role binding resource.
*/
@ConfigItem(defaultValue = "true")
public boolean generate;
}
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@
import io.quarkus.deployment.pkg.builditem.ArtifactResultBuildItem;
import io.quarkus.deployment.pkg.builditem.OutputTargetBuildItem;
import io.quarkus.deployment.util.ExecUtil;
import io.quarkus.kubernetes.client.spi.KubernetesClientCapabilityBuildItem;
import io.quarkus.kubernetes.deployment.AddPortToKubernetesConfig;
import io.quarkus.kubernetes.deployment.DevClusterHelper;
import io.quarkus.kubernetes.deployment.KubernetesCommonHelper;
Expand All @@ -32,6 +33,7 @@
import io.quarkus.kubernetes.spi.CustomProjectRootBuildItem;
import io.quarkus.kubernetes.spi.DecoratorBuildItem;
import io.quarkus.kubernetes.spi.KubernetesAnnotationBuildItem;
import io.quarkus.kubernetes.spi.KubernetesClusterRoleBuildItem;
import io.quarkus.kubernetes.spi.KubernetesCommandBuildItem;
import io.quarkus.kubernetes.spi.KubernetesDeploymentTargetBuildItem;
import io.quarkus.kubernetes.spi.KubernetesEnvBuildItem;
Expand All @@ -46,6 +48,7 @@
import io.quarkus.kubernetes.spi.KubernetesResourceMetadataBuildItem;
import io.quarkus.kubernetes.spi.KubernetesRoleBindingBuildItem;
import io.quarkus.kubernetes.spi.KubernetesRoleBuildItem;
import io.quarkus.kubernetes.spi.KubernetesServiceAccountBuildItem;

public class KindProcessor {

Expand Down Expand Up @@ -97,6 +100,7 @@ public List<DecoratorBuildItem> createDecorators(ApplicationInfoBuildItem applic
KubernetesConfig config,
PackageConfig packageConfig,
Optional<MetricsCapabilityBuildItem> metricsConfiguration,
Optional<KubernetesClientCapabilityBuildItem> kubernetesClientConfiguration,
List<KubernetesInitContainerBuildItem> initContainers,
List<KubernetesJobBuildItem> jobs,
List<KubernetesAnnotationBuildItem> annotations,
Expand All @@ -111,16 +115,16 @@ public List<DecoratorBuildItem> createDecorators(ApplicationInfoBuildItem applic
Optional<KubernetesHealthReadinessPathBuildItem> readinessPath,
Optional<KubernetesHealthStartupPathBuildItem> startupPath,
List<KubernetesRoleBuildItem> roles,
List<KubernetesClusterRoleBuildItem> clusterRoles,
List<KubernetesServiceAccountBuildItem> serviceAccounts,
List<KubernetesRoleBindingBuildItem> roleBindings,
Optional<CustomProjectRootBuildItem> customProjectRoot) {

return DevClusterHelper.createDecorators(KIND, applicationInfo, outputTarget, config, packageConfig,
metricsConfiguration, initContainers, jobs, annotations, labels, envs, baseImage, image, command, ports,
portName,
livenessPath,
readinessPath,
startupPath,
roles, roleBindings, customProjectRoot);
metricsConfiguration, kubernetesClientConfiguration, initContainers, jobs, annotations, labels, envs,
baseImage, image, command, ports, portName,
livenessPath, readinessPath, startupPath,
roles, clusterRoles, serviceAccounts, roleBindings, customProjectRoot);
}

@BuildStep
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@
import io.quarkus.deployment.metrics.MetricsCapabilityBuildItem;
import io.quarkus.deployment.pkg.PackageConfig;
import io.quarkus.deployment.pkg.builditem.OutputTargetBuildItem;
import io.quarkus.kubernetes.client.spi.KubernetesClientCapabilityBuildItem;
import io.quarkus.kubernetes.deployment.AddPortToKubernetesConfig;
import io.quarkus.kubernetes.deployment.DevClusterHelper;
import io.quarkus.kubernetes.deployment.KubernetesCommonHelper;
Expand All @@ -29,6 +30,7 @@
import io.quarkus.kubernetes.spi.CustomProjectRootBuildItem;
import io.quarkus.kubernetes.spi.DecoratorBuildItem;
import io.quarkus.kubernetes.spi.KubernetesAnnotationBuildItem;
import io.quarkus.kubernetes.spi.KubernetesClusterRoleBuildItem;
import io.quarkus.kubernetes.spi.KubernetesCommandBuildItem;
import io.quarkus.kubernetes.spi.KubernetesDeploymentTargetBuildItem;
import io.quarkus.kubernetes.spi.KubernetesEnvBuildItem;
Expand All @@ -43,6 +45,7 @@
import io.quarkus.kubernetes.spi.KubernetesResourceMetadataBuildItem;
import io.quarkus.kubernetes.spi.KubernetesRoleBindingBuildItem;
import io.quarkus.kubernetes.spi.KubernetesRoleBuildItem;
import io.quarkus.kubernetes.spi.KubernetesServiceAccountBuildItem;

public class MinikubeProcessor {

Expand Down Expand Up @@ -93,6 +96,7 @@ public List<DecoratorBuildItem> createDecorators(ApplicationInfoBuildItem applic
KubernetesConfig config,
PackageConfig packageConfig,
Optional<MetricsCapabilityBuildItem> metricsConfiguration,
Optional<KubernetesClientCapabilityBuildItem> kubernetesClientConfiguration,
List<KubernetesInitContainerBuildItem> initContainers,
List<KubernetesJobBuildItem> jobs,
List<KubernetesAnnotationBuildItem> annotations,
Expand All @@ -107,15 +111,15 @@ public List<DecoratorBuildItem> createDecorators(ApplicationInfoBuildItem applic
Optional<KubernetesHealthReadinessPathBuildItem> readinessPath,
Optional<KubernetesHealthStartupPathBuildItem> startupPath,
List<KubernetesRoleBuildItem> roles,
List<KubernetesClusterRoleBuildItem> clusterRoles,
List<KubernetesServiceAccountBuildItem> serviceAccounts,
List<KubernetesRoleBindingBuildItem> roleBindings,
Optional<CustomProjectRootBuildItem> customProjectRoot) {

return DevClusterHelper.createDecorators(MINIKUBE, applicationInfo, outputTarget, config, packageConfig,
metricsConfiguration, initContainers, jobs, annotations, labels, envs, baseImage, image, command, ports,
portName,
livenessPath,
readinessPath,
startupPath,
roles, roleBindings, customProjectRoot);
metricsConfiguration, kubernetesClientConfiguration, initContainers, jobs, annotations, labels, envs,
baseImage, image, command, ports, portName,
livenessPath, readinessPath, startupPath,
roles, clusterRoles, serviceAccounts, roleBindings, customProjectRoot);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,43 @@
package io.quarkus.kubernetes.spi;

import java.util.List;

import io.quarkus.builder.item.MultiBuildItem;

/**
* Produce this build item to request the Kubernetes extension to generate
* a Kubernetes {@code ClusterRole} resource.
*/
public final class KubernetesClusterRoleBuildItem extends MultiBuildItem {
/**
* Name of the generated {@code ClusterRole} resource.
*/
private final String name;
/**
* The {@code PolicyRule} resources for this {@code ClusterRole}.
*/
private final List<PolicyRule> rules;

/**
* The target manifest that should include this role.
*/
private final String target;

public KubernetesClusterRoleBuildItem(String name, List<PolicyRule> rules, String target) {
this.name = name;
this.rules = rules;
this.target = target;
}

public String getName() {
return name;
}

public List<PolicyRule> getRules() {
return rules;
}

public String getTarget() {
return target;
}
}
Loading

0 comments on commit aea3b57

Please sign in to comment.