Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Karpenter added new ec2 node, but never joined to the cluster #6659

Closed
davidt-gh opened this issue Aug 6, 2024 · 2 comments
Closed

Karpenter added new ec2 node, but never joined to the cluster #6659

davidt-gh opened this issue Aug 6, 2024 · 2 comments
Labels
bug Something isn't working needs-triage Issues that need to be triaged

Comments

@davidt-gh
Copy link

davidt-gh commented Aug 6, 2024

Description

Observed Behavior:
Following the "Getting started guide", I've reached to "inflate" step.
I'm trying to inflate deployment, I can see the log of karpenter trying to do so:

{"level":"INFO","time":"2024-08-06T11:36:52.502Z","logger":"controller","message":"found provisionable pod(s)","commit":"490ef94","controller":"provisioner","Pods":"default/inflate-6876fc65ff-jhwbk, default/inflate-6876fc65ff-lsfnf, default/inflate-6876fc65ff-6kv45, default/inflate-6876fc65ff-r7z6f","duration":"162.720012ms"}
{"level":"INFO","time":"2024-08-06T11:36:52.502Z","logger":"controller","message":"computed new nodeclaim(s) to fit pod(s)","commit":"490ef94","controller":"provisioner","nodeclaims":1,"pods":4}
{"level":"INFO","time":"2024-08-06T11:36:52.527Z","logger":"controller","message":"created nodeclaim","commit":"490ef94","controller":"provisioner","NodePool":{"name":"default"},"NodeClaim":{"name":"default-fxdpz"},"requests":{"cpu":"4150m","pods":"6"},"instance-types":"m3.2xlarge, m4.2xlarge, m4.4xlarge, m5.2xlarge, m5.4xlarge and 55 other(s)"}
{"level":"INFO","time":"2024-08-06T11:36:54.890Z","logger":"controller","message":"launched nodeclaim","commit":"490ef94","controller":"nodeclaim.lifecycle","controllerGroup":"karpenter.sh","controllerKind":"NodeClaim","NodeClaim":{"name":"default-fxdpz"},"namespace":"","name":"default-fxdpz","reconcileID":"f9ca3146-bb47-421d-b93f-3249b77ca19a","provider-id":"aws:///us-east-1b/i-07435d79ed36de8c3","instance-type":"t3a.2xlarge","zone":"us-east-1b","capacity-type":"on-demand","allocatable":{"cpu":"7910m","ephemeral-storage":"17Gi","memory":"29317Mi","pods":"58"}}

Next step I can see the EC2 node is starting (with no name not sure why)
But it's never joined to the cluster, so the pods keep pending.

Expected Behavior:
Getting new node into my cluster

Reproduction Steps (Please include YAML):
values.yaml

  settings:
    clusterName: "<cluster-name>"
    interruptionQueue: "<cluster-name>"
  serviceAccount:
    annotations:
      "eks.amazonaws.com/role-arn": "arn:aws:iam::<acc-num>:role/role-name"
  controller:
    resources:
      requests:
        cpu: 1
        memory: 1Gi
      limits:
        cpu: 1
        memory: 1Gi

nodeClass.yaml

apiVersion: karpenter.k8s.aws/v1beta1
kind: EC2NodeClass
metadata:
  name: default
spec:
  amiFamily: AL2
  role: <role-name>
  subnetSelectorTerms:
    - tags:
        karpenter.sh/discovery: {{ .Values.settings.clusterName }}
    - id: subnet-***
    - id: subnet-***
  securityGroupSelectorTerms:
    - tags:
        karpenter.sh/discovery: {{ .Values.settings.clusterName }}
  amiSelectorTerms:
    - id: "ami-***"

nodePool.yaml

apiVersion: karpenter.sh/v1beta1
kind: NodePool
metadata:
  name: default
spec:
  template:
    spec:
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: kubernetes.io/os
          operator: In
          values: ["linux"]
        - key: karpenter.sh/capacity-type
          operator: In
          values: ["on-demand"]
        - key: karpenter.k8s.aws/instance-category
          operator: In
          values: ["t", "m", "r"]
        - key: karpenter.k8s.aws/instance-generation
          operator: Gt
          values: ["2"]
        - key: "topology.kubernetes.io/zone"
          operator: In
          values: [ "us-east-1a","us-east-1b" ]
      nodeClassRef:
        name: default
  limits:
    cpu: 1000
  disruption:
    consolidationPolicy: WhenUnderutilized
    expireAfter: 720h # 30 * 24h = 720h

Versions:

  • Chart Version: 0.37.0
  • Kubernetes Version (kubectl version):
Client Version: v1.30.3
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.30.2-eks-db838b0

I can see logs in the new EC2 that looks like #1067
Here is it:

<13>Aug  7 07:12:48 user-data: + true
<13>Aug  7 07:12:48 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:12:48 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:12:49 user-data: 
<13>Aug  7 07:12:49 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:12:49 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:12:49 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:12:49 user-data: + '[' 10 -ge 20 ']'
<13>Aug  7 07:12:49 user-data: + ATTEMPT=11
<13>Aug  7 07:12:49 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:12:49 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:12:49 user-data: + echo 2024-08-07T07:12:49+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:12:49 user-data: 2024-08-07T07:12:49+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:12:49 user-data: + sleep 6
<13>Aug  7 07:12:55 user-data: + true
<13>Aug  7 07:12:55 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:12:55 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:12:56 user-data: 
<13>Aug  7 07:12:56 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:12:56 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:12:56 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:12:56 user-data: + '[' 11 -ge 20 ']'
<13>Aug  7 07:12:56 user-data: + ATTEMPT=12
<13>Aug  7 07:12:56 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:12:56 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:12:56 user-data: + echo 2024-08-07T07:12:56+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:12:56 user-data: 2024-08-07T07:12:56+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:12:56 user-data: + sleep 6
<13>Aug  7 07:13:02 user-data: + true
<13>Aug  7 07:13:02 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:13:02 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:13:03 user-data: 
<13>Aug  7 07:13:03 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:13:03 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:13:03 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:13:03 user-data: + '[' 12 -ge 20 ']'
<13>Aug  7 07:13:03 user-data: + ATTEMPT=13
<13>Aug  7 07:13:03 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:03 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:13:03 user-data: + echo 2024-08-07T07:13:03+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:03 user-data: 2024-08-07T07:13:03+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:13:03 user-data: + sleep 6
<13>Aug  7 07:13:09 user-data: + true
<13>Aug  7 07:13:09 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:13:09 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:13:10 user-data: 
<13>Aug  7 07:13:10 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:13:11 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:13:11 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:13:11 user-data: + '[' 13 -ge 20 ']'
<13>Aug  7 07:13:11 user-data: + ATTEMPT=14
<13>Aug  7 07:13:11 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:11 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:13:11 user-data: + echo 2024-08-07T07:13:11+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:11 user-data: 2024-08-07T07:13:11+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:13:11 user-data: + sleep 6
<13>Aug  7 07:13:17 user-data: + true
<13>Aug  7 07:13:17 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:13:17 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:13:18 user-data: 
<13>Aug  7 07:13:18 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:13:18 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:13:18 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:13:18 user-data: + '[' 14 -ge 20 ']'
<13>Aug  7 07:13:18 user-data: + ATTEMPT=15
<13>Aug  7 07:13:18 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:18 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:13:18 user-data: + echo 2024-08-07T07:13:18+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:18 user-data: 2024-08-07T07:13:18+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:13:18 user-data: + sleep 6
<13>Aug  7 07:13:24 user-data: + true
<13>Aug  7 07:13:24 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:13:24 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:13:25 user-data: 
<13>Aug  7 07:13:25 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:13:25 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:13:25 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:13:25 user-data: + '[' 15 -ge 20 ']'
<13>Aug  7 07:13:25 user-data: + ATTEMPT=16
<13>Aug  7 07:13:25 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:25 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:13:25 user-data: + echo 2024-08-07T07:13:25+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:25 user-data: 2024-08-07T07:13:25+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:13:25 user-data: + sleep 6
<13>Aug  7 07:13:31 user-data: + true
<13>Aug  7 07:13:31 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:13:31 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:13:32 user-data: 
<13>Aug  7 07:13:32 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:13:33 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:13:33 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:13:33 user-data: + '[' 16 -ge 20 ']'
<13>Aug  7 07:13:33 user-data: + ATTEMPT=17
<13>Aug  7 07:13:33 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:33 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:13:33 user-data: + echo 2024-08-07T07:13:33+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:33 user-data: 2024-08-07T07:13:33+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:13:33 user-data: + sleep 6
<13>Aug  7 07:13:39 user-data: + true
<13>Aug  7 07:13:39 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:13:39 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:13:40 user-data: 
<13>Aug  7 07:13:40 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:13:40 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:13:40 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:13:40 user-data: + '[' 17 -ge 20 ']'
<13>Aug  7 07:13:40 user-data: + ATTEMPT=18
<13>Aug  7 07:13:40 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:40 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:13:40 user-data: + echo 2024-08-07T07:13:40+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:40 user-data: 2024-08-07T07:13:40+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:13:40 user-data: + sleep 6
<13>Aug  7 07:13:46 user-data: + true
<13>Aug  7 07:13:46 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:13:46 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:13:47 user-data: 
<13>Aug  7 07:13:47 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:13:47 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:13:47 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:13:47 user-data: + '[' 18 -ge 20 ']'
<13>Aug  7 07:13:47 user-data: + ATTEMPT=19
<13>Aug  7 07:13:47 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:47 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:13:47 user-data: + echo 2024-08-07T07:13:47+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:47 user-data: 2024-08-07T07:13:47+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:13:47 user-data: + sleep 6
<13>Aug  7 07:13:53 user-data: + true
<13>Aug  7 07:13:53 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:13:53 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:13:55 user-data: 
<13>Aug  7 07:13:55 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:13:55 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:13:55 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:13:55 user-data: + '[' 19 -ge 20 ']'
<13>Aug  7 07:13:55 user-data: + ATTEMPT=20
<13>Aug  7 07:13:55 user-data: + log 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:55 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:13:55 user-data: + echo 2024-08-07T07:13:55+0000 '[private-dns-name]' 'WARN: PrivateDnsName is not available, waiting for 6 seconds...'
<13>Aug  7 07:13:55 user-data: 2024-08-07T07:13:55+0000 [private-dns-name] WARN: PrivateDnsName is not available, waiting for 6 seconds...
<13>Aug  7 07:13:55 user-data: + sleep 6
<13>Aug  7 07:14:01 user-data: + true
<13>Aug  7 07:14:01 user-data: ++ jq -r '.Reservations[].Instances[].PrivateDnsName'
<13>Aug  7 07:14:01 user-data: ++ aws ec2 describe-instances --region us-east-1 --instance-ids i-0c415807407980e01
<13>Aug  7 07:14:02 user-data: 
<13>Aug  7 07:14:02 user-data: Unable to locate credentials. You can configure credentials by running "aws configure".
<13>Aug  7 07:14:02 user-data: + PRIVATE_DNS_NAME=
<13>Aug  7 07:14:02 user-data: + '[' '!' '' = '' ']'
<13>Aug  7 07:14:02 user-data: + '[' 20 -ge 20 ']'
<13>Aug  7 07:14:02 user-data: + break
<13>Aug  7 07:14:02 user-data: + '[' '' = '' ']'
<13>Aug  7 07:14:02 user-data: + log 'ERROR: failed to retrieve PrivateDnsName after 20 attempts!'
<13>Aug  7 07:14:02 user-data: ++ date +%Y-%m-%dT%H:%M:%S%z
<13>Aug  7 07:14:02 user-data: + echo 2024-08-07T07:14:02+0000 '[private-dns-name]' 'ERROR: failed to retrieve PrivateDnsName after 20 attempts!'
<13>Aug  7 07:14:02 user-data: 2024-08-07T07:14:02+0000 [private-dns-name] ERROR: failed to retrieve PrivateDnsName after 20 attempts!
<13>Aug  7 07:14:02 user-data: + exit 1
<13>Aug  7 07:14:02 user-data: Exited with error on line 546
[  161.177079] cloud-init[3139]: Aug 07 07:14:02 cloud-init[3139]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
[  161.179881] cloud-init[3139]: Aug 07 07:14:02 cloud-init[3139]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
[  161.180510] cloud-init[3139]: Aug 07 07:14:02 cloud-init[3139]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed
ci-info: no authorized ssh keys fingerprints found for user ec2-user.
[  161.189321] cloud-init[3139]: ci-info: no authorized ssh keys fingerprints found for user ec2-user.
<14>Aug  7 07:14:02 ec2: 
<14>Aug  7 07:14:02 ec2: #############################################################
<14>Aug  7 07:14:02 ec2: -----BEGIN SSH HOST KEY FINGERPRINTS-----
<14>Aug  7 07:14:02 ec2: 256 SHA256:RlUTqGwm1xuEmqfQJz0C/wq3YoFC8DrBxcEt/oud1Vg no comment (ECDSA)
<14>Aug  7 07:14:02 ec2: 256 SHA256:eIUYHDeVMCDbEhsQORQjXiGLUpvTGTZvBYaQXGVf18s no comment (ED25519)
<14>Aug  7 07:14:02 ec2: 2048 SHA256:BFf1DgAG5rCpuxOky090bXmhGMjBACCfJeE8OSsGxEQ no comment (RSA)
<14>Aug  7 07:14:02 ec2: -----END SSH HOST KEY FINGERPRINTS-----
<14>Aug  7 07:14:02 ec2: #############################################################
-----BEGIN SSH HOST KEY KEYS-----
**********
-----END SSH HOST KEY KEYS-----
[  161.249864] cloud-init[3139]: Cloud-init v. 19.3-46.amzn2.0.2 finished at Wed, 07 Aug 2024 07:14:02 +0000. Datasource DataSourceEc2.  Up 161.24 seconds

@davidt-gh davidt-gh added bug Something isn't working needs-triage Issues that need to be triaged labels Aug 6, 2024
@davidt-gh davidt-gh changed the title Added new ec2 node, but never joined to the cluster Karpenter added new ec2 node, but never joined to the cluster Aug 6, 2024
@ibalat
Copy link

ibalat commented Aug 9, 2024

same problem. I tried 0.36.2 and 0.37.0

@davidt-gh
Copy link
Author

same problem. I tried 0.36.2 and 0.37.0

I solve it and I had 2 problem:

  1. In the values I've used the karpenter role instead the node role as they describe on start guide.
  2. In the policy of karpetner role, I was need to do that:
{
              "Sid": "AllowPassingInstanceRole",
              "Effect": "Allow",
              "Resource": "arn:aws:iam::${var.account_id}:role/KarpenterNodeRole-*",
              "Action": "iam:PassRole",
              "Condition": {
                "StringEquals": {
                  "iam:PassedToService": "ec2.amazonaws.com"
                }
              }
            }

instead:

{
              "Sid": "AllowPassingInstanceRole",
              "Effect": "Allow",
              "Resource": "arn:aws:iam::${var.account_id}:role/Karpenter-Role",
              "Action": "iam:PassRole",
              "Condition": {
                "StringEquals": {
                  "iam:PassedToService": "ec2.amazonaws.com"
                }
              }
            },

Both issues made the main issue for me.
Hope that's can be helpful.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working needs-triage Issues that need to be triaged
Projects
None yet
Development

No branches or pull requests

2 participants