skaffold debug not adding JAVA_TOOL_OPTIONS env variable to deployment

[rmohammed-xtime]

Expected behavior

JAVA_TOOL_OPTIONS env variable added to deployment yaml when running skaffold debug

Actual behavior

JAVA_TOOL_OPTIONS env variable not added to deployment yaml when running skaffold debug

Information

skaffold debug --namespace=skaffold --skip-tests --port-forward

time="2020-03-09T10:54:11-07:00" level=debug msg="no debug transformation for: deployment.extensions/mypod-pod"

Same result with

1. skaffold debug --namespace=skaffold --skip-tests
2. skaffold debug --namespace=skaffold --skip-tests --port-forward --cache-artifacts=false
3. explicitly adding empty JAVA_TOOL_OPTIONS env var to deployment

• Skaffold version: version...
  time="2020-03-09T10:54:08-07:00" level=info msg="Skaffold &{Version:v1.5.0 ConfigVersion:skaffold/v2alpha4 GitVersion: GitCommit:6a9247
  5a4aa07180b1340c5bb1aa2b18ae5058ca GitTreeState:clean BuildDate:2020-03-06T20:20:58Z GoVersion:go1.13.4 Compiler:gc Platform:darwin/amd
  64}"
• kustomize version:
  {Version:kustomize/v3.5.4 GitCommit:3af514fa9f85430f0c1557c4a0291e62112ab026 BuildDate:2020-01-11T03:12:59Z GoOs:darwin GoArch:amd64}
• jib-maven-plugin.version
  2.1.0
• Operating system: ...
  macosx Darwin Kernel Version 18.7.0
• Contents of skaffold.yaml:

apiVersion: skaffold/v2alpha4
kind: Config
metadata:
  name: mypod-service
build:
  artifacts:
  - image: mypod
    context: mypod-server
    jib:
      project: com:mypod-server
deploy:
  kustomize:
    paths:
    - kustomize/skaffold
portForward:
  - resourceType: deployment
    resourceName: mypod-pod
    namespace: skaffold
    port: 8080
    localPort: 9000

• kubectl -n skaffold describe deployment mypod-pod

Name:                   mypod-pod
Namespace:              skaffold
CreationTimestamp:      Mon, 09 Mar 2020 10:51:35 -0700
Labels:                 app=mypod-pod
                        app.kubernetes.io/managed-by=skaffold-v1.5.0
                        skaffold.dev/builder=local
                        skaffold.dev/cleanup=true
                        skaffold.dev/deployer=kustomize
                        skaffold.dev/docker-api-version=1.40
                        skaffold.dev/namespace=skaffold
                        skaffold.dev/run-id=5a1bd733-1b80-4c89-98a1-ee20617b6854
                        skaffold.dev/tag-policy=git-commit
                        skaffold.dev/tail=true
Annotations:            deployment.kubernetes.io/revision: 3
                        kubectl.kubernetes.io/last-applied-configuration:
                          {"apiVersion":"extensions/v1beta1","kind":"Deployment","metadata":{"annotations":{},"labels":{"app":"mypod-pod","app.kubernetes.io/managed...
Selector:               app=mypod-pod
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 1 max surge
Pod Template:
  Labels:       app=mypod-pod
                app.kubernetes.io/managed-by=skaffold-v1.5.0
                skaffold.dev/builder=local
                skaffold.dev/cleanup=true
                skaffold.dev/deployer=kustomize
                skaffold.dev/docker-api-version=1.40
                skaffold.dev/namespace=skaffold
                skaffold.dev/run-id=5a1bd733-1b80-4c89-98a1-ee20617b6854
                skaffold.dev/tag-policy=git-commit
                skaffold.dev/tail=true
  Containers:
   mypod-pod:
    Ports:       8080/TCP, 8181/TCP
    Host Ports:  0/TCP, 0/TCP
    Environment:
      override_mypod_dot_store:                                           map
      JAVA_MEM_OPTS:                                                      -Xms2048M -Xmx2048M -XX:MaxPermSize=512M -XX:+ExitOnOutOfMemoryError -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/heapdumps/${HOSTNAME}_$(date +%Y%m%d_%H_%M_%S).hprof
      JAVA_GC_OPTS:                                                       -Dnogclogging=true

[dgageot]

cc @briandealwis

[briandealwis]

(Oops sorry — I mis-clicked on the close button.)

@rmohammed-xtime could you please tell me more about your image: what are you using for the base image? What is your image's entrypoint or command? Basically skaffold debug looks at the image environment for JAVA_TOOL_OPTIONS or JAVA_VERSION, or at the entrypoint to determine if the image seems to be running a JVM. You show that the deployment has JAVA_MEM_OPTS and JAVA_GC_OPTS set, which isn't used by skaffold debug.

[jkleckner]

I also had this situation occur just now which I chalked up to having multiple releases defined in the configuration. But quickly scanning #1702 I don't see anything trying to limit the deployments.

I am running skaffold v1.6.0 on MacOs High Sierra 10.13.6 for a Java deployment of a play framework set of applications.

[briandealwis]

@rmohammed-xtime could you please attach your pom.xml?

@jkleckner could you please open a new issue, and include a small reproducible example?

[briandealwis]

@rmohammed-xtime it would also help to see a debug log from skaffold debug -v debug ... too. Thanks!

[jkleckner]

@briandealwis The topic #2203 should address the underlying issue I saw.

[jkleckner]

@rmohammed-xtime Would #2203 solve your issue as well?

[jkleckner]

From conversation with @briandealwis in slack chat, I'm including more information on our image.
Here is the output of docker inspect with various hashes redacted:
inspect.txt

The base image is openjdk:8-jre-slim that introduces the env vars of JAVA_HOME and JAVA_VERSION.

[briandealwis]

@jkleckner I'm puzzled as to why your image isn't being picked up: the JVM debug transformer should be activated by the JAVA_VERSION envvar.

[jkleckner]

@briandealwis I'm happy to try to debug a bit. I haven't studied Go at all so am at a disadvantage.

[jkleckner]

@briandealwis I added some debug println statements and find that ApplyDebuggingTransforms is not getting called.

• https://github.com/GoogleContainerTools/skaffold/blob/master/pkg/skaffold/debug/debug.go#L52

This append doesn't "step into" with the debugger:

• https://github.com/GoogleContainerTools/skaffold/blob/master/pkg/skaffold/deploy/transformations.go#L31

Happy to set up a brief screen sharing debug session if you like.

[briandealwis]

@jkleckner could you open a new issue for this? And please include the logs of what you're seeing, with the command invocation. Honestly, it sounds like you're either encountering build or deploy errors, or running skaffold dev rather than skaffold debug — that ApplyDebuggingTransforms is registered by the debug implementation.

[jkleckner]

@briandealwis Ok, will do. But I assure you that a breakpoint entered the doDebug function and that the even though registered, the debugging.ApplyDebuggingTransforms printed no trace (that I added manually).

But I can easily believe there is some complexity with the helm deployment that stimulates this problem.

Am a bit pressed at the moment so probably won't be until next week at the earliest.

[jkleckner]

@briandealwis Before creating another issue, I noticed that JAVA_VERSION is defined in the container image and if I connect to a pod, I see it in the environment. However, I don't see it in the env as obtained with kubectl get -n ns pod/pod-id -oyaml

Does skaffold look into the image env or just the pod spec ?

We are using helm for deploy if that matters.

Cc: @aboyett

[briandealwis]

@jkleckner Oh! Debug isn't supported with Helm yet. It's being tracked in #2350.

[jkleckner]

Ahh... That explains it.

[dgageot]

@jkleckner @briandealwis should we close this issue then?

[jkleckner]

@dgageot For my use case, yes.

@rmohammed-xtime Appears to be using jib deployment.

[briandealwis]

Closing — will re-open if OP responds.