akka.actor.ActorInitializationException when trying to send API requests

[razzu]

Hi,

This is a follow-up to a previous issue (#59).

After updating skuber to 2.7.2, all API requests sent by Skuber fail with an Akka initialization exception. I've tried this inside a Kubernetes cluster, with default configuration settings. Full log of my dummy app:

[INFO] [07/29/2021 09:28:18.485] [main] [skuber.api] Using following context for connecting to Kubernetes cluster: Context(Cluster(v1,https://192.168.0.1:443,false,Some(Left(/var/run/secrets/kubernetes.io/serviceaccount/ca.crt))),TokenAuth(token=<redacted>),Namespace(Namespace,v1,ObjectMeta(test-ns,,,,,,None,None,None,Map(),Map(),List(),0,None,None),None,None))
[INFO] [07/29/2021 09:28:19.382] [main] [skuber.api] [ { reqId=d69cff65-6a8b-4515-a6bb-47acb4c297c7} } - about to send HTTP request: GET https://192.168.0.1/api/v1/namespaces/test-ns/configmaps/test-config]
[ERROR] [07/29/2021 09:28:19.781] [default-akka.actor.default-dispatcher-5] [akka://default/system/Materializers/StreamSupervisor-0/TLS-for-flow-1-1] Client/Server mode has not yet been set.
akka.actor.ActorInitializationException: akka://default/system/Materializers/StreamSupervisor-0/TLS-for-flow-1-1: exception during creation
	at akka.actor.ActorInitializationException$.apply(Actor.scala:196)
	at akka.actor.ActorCell.create(ActorCell.scala:664)
	at akka.actor.ActorCell.invokeAll$1(ActorCell.scala:514)
	at akka.actor.ActorCell.systemInvoke(ActorCell.scala:536)
	at akka.dispatch.Mailbox.processAllSystemMessages(Mailbox.scala:295)
	at akka.dispatch.Mailbox.run(Mailbox.scala:230)
	at akka.dispatch.Mailbox.exec(Mailbox.scala:243)
	at java.base/java.util.concurrent.ForkJoinTask.doExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.scan(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinPool.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ForkJoinWorkerThread.run(Unknown Source)
Caused by: java.lang.IllegalStateException: Client/Server mode has not yet been set.
	at java.base/sun.security.ssl.SSLEngineImpl.beginHandshake(Unknown Source)
	at akka.stream.impl.io.TLSActor.<init>(TLSActor.scala:165)
	at akka.stream.impl.io.TLSActor$.$anonfun$props$1(TLSActor.scala:40)
	at akka.actor.TypedCreatorFunctionConsumer.produce(IndirectActorProducer.scala:91)
	at akka.actor.Props.newActor(Props.scala:226)
	at akka.actor.ActorCell.newActor(ActorCell.scala:616)
	at akka.actor.ActorCell.create(ActorCell.scala:643)
	... 10 more

Note that this works if I downgrade Skuber to 2.7.0.

My build.sbt file is quite simple, only Skuber and a few extra things:

libraryDependencies += "io.github.hagay3"           %% "skuber"          % "2.7.2"
libraryDependencies += "ch.qos.logback"              % "logback-classic" % "1.2.3"
libraryDependencies += "com.typesafe.scala-logging" %% "scala-logging"   % "3.9.3"
libraryDependencies += "com.github.pureconfig"      %% "pureconfig"      % "0.16.0"

assembly / assemblyJarName := "dummy-skuber.jar"

ThisBuild / assemblyMergeStrategy := {
  case PathList("module-info.class")         => MergeStrategy.discard
  case x if x.endsWith("/module-info.class") => MergeStrategy.discard
  case x =>
    val oldStrategy = (ThisBuild / assemblyMergeStrategy).value
    oldStrategy(x)
}

The actual code doesn't do much, just tries to retrieve a ConfigMap and then start a watch:

private def run(appConfig: AppConfig): Unit = {
    val k8s                = k8sInit
    val getConfigMapFuture = k8s.getInNamespace[ConfigMap](appConfig.configMapName, appConfig.namespace)
    getConfigMapFuture.onComplete {
      case Success(configMap) =>
        logger.info(s"Succesfully read ConfigMap: $configMap")
        k8s.watchContinuously(configMap).runWith(configMapMonitor)
      case Failure(e) =>
        logger.warn(s"Failed to read ConfigMap, due to: $e")
    }
  }

Note that it doesn't even create the Future (so it doesn't reach the Failure branch in onComplete.

Let me know if there's anything else I can try! I've run out of ideas.

Thanks!

[hagay3]

Hi @razzu , thank you for following up on that matter.
I think your issue is related to the fact that you are trying to reach the cluster with https with IP address.
In order to make https request, you need to use a DNS name with a valid certificate.

Can you try using http instead of https ?
You can also set up a proper cname and make sure you have installed kubernetes https certificate:
https://kubernetes.io/docs/setup/best-practices/certificates/

[INFO] [07/29/2021 09:28:18.485] [main] [skuber.api] Using following context for connecting to Kubernetes cluster: Context(Cluster(v1,https://192.168.0.1:443,false,So

By the way, I tested with Kubernetes cluster and the connection is established.
You might have SSL issues since akka HTTP version is upgraded from 10.1.12 to 10.2.4 .
I saw that many changes have been done around SSL between those releases.
https://doc.akka.io/docs/akka-http/current/release-notes/10.2.x.html

[razzu]

Thanks for the input! Unfortunately I'm not sure how to test that or whether I can actually change this behaviour.

The app runs on a pod inside a managed K8s cluster and I can't really modify anything related to the Kubernetes installation. I'm letting Skuber do the Configuration.inClusterConfig as usual, and the KUBERNETES_SERVICE_HOST environment variable that it uses is an IP. I checked in an EKS cluster as well and it's the same. Never had this issue with previous versions, so I can see that it might be related to something which has changed in Akka, but I'm not really sure how to troubleshoot this further.

Have you also tested this running inside a cluster? Or just from the outside?

[hagay3]

@razzu
Let's move the discussion to skuber discord channel
https://discord.gg/9VnWmnX2

[razzu]

The invite doesn't seem to be working for me.

[hagay3]

@razzu
I fixed the link
https://discord.gg/byEh56vFJR

[brycechesternewman]

I am running into the same issue. Did anything come out of the discord conversation about how you resolved this?

[brycechesternewman]

@hagay3 and @razzu I think this issue is with KubernetesClientImpl code where engine.setUseClientMode(true) is not being set.

 val connectionContext = sslContext
      .map { ssl =>
        ConnectionContext.httpsClient { (host, port) =>
          val engine = ssl.createSSLEngine(host, port)
          engine.setEnabledProtocols(Array("TLSv1.2", "TLSv1"))
          engine
        }
      }
      .getOrElse(Http().defaultClientHttpsContext)

[hagay3]

@brycechesternewman
nice catch, thanks for the PR
I merged it and released skuber v2.7.3
it will be available in the next 24 hours
you can check here
https://search.maven.org/search?q=g:io.github.hagay3

Please let me know if it resolves the issue

[brycechesternewman]

2.7.3 resolved this issue. Thanks for the quick turn around on the merge.