Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

service available replicas below target #33

Closed
andy108369 opened this issue Jun 28, 2022 · 6 comments
Closed

service available replicas below target #33

andy108369 opened this issue Jun 28, 2022 · 6 comments
Assignees
@andy108369
Labels
repo/provider Akash provider-services repo issues sev2

Comments

@andy108369
Copy link
Contributor

akash v0.16.4

I'm often seeing providers to hit service available replicas below target message despite the pod/svc/ing seem to be running well.

Related code path https://github.com/ovrclk/akash/blob/v0.16.4/provider/cluster/monitor.go#L185

provider logs

D[2022-06-28|13:12:52.338] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash1angnwmnsv5vkp4jpgkdcunar8efpvennxwwcev/6194991/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=tezos cmp=deployment-monitor service=tezos available=0 target=1

root@node1:~# NS=$(akash provider show-cluster-ns --dseq 6194991 --owner akash1angnwmnsv5vkp4jpgkdcunar8efpvennxwwcev --provider akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0)

root@node1:~# kubectl -n lease get manifest $NS -o yaml
apiVersion: akash.network/v2beta1
kind: Manifest
metadata:
  creationTimestamp: "2022-06-06T01:14:19Z"
  generation: 2
  labels:
    akash.network: "true"
    akash.network/lease.id.dseq: "6194991"
    akash.network/lease.id.gseq: "1"
    akash.network/lease.id.oseq: "1"
    akash.network/lease.id.owner: akash1angnwmnsv5vkp4jpgkdcunar8efpvennxwwcev
    akash.network/lease.id.provider: akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
    akash.network/namespace: i8srn0jevt07jq3ittrgrg1q3ja3srij6t7efeftq3fi4
  name: i8srn0jevt07jq3ittrgrg1q3ja3srij6t7efeftq3fi4
  namespace: lease
  resourceVersion: "5343337"
  uid: 7b8f4027-c978-44d0-ae94-2f971a846a6e
spec:
  group:
    name: tezos
    services:
    - count: 1
      env:
      - SNAPSHOT_NETWORK=mainnet
      - SNAPSHOT_TYPE=rolling
      - SNAPSHOT_PATH=/var/run/tezos
      expose:
      - external_port: 80
        global: true
        hosts:
        - eu-central.nulldao.xyz
        http_options: {}
        port: 8732
        proto: TCP
      image: ghcr.io/sfdotstudio/tezos-quikn:latest
      name: tezos
      unit:
        cpu: 1000
        memory: "4294967296"
        storage:
        - name: default
          size: "10737418240"
  lease_id:
    dseq: "6194991"
    gseq: 1
    oseq: 1
    owner: akash1angnwmnsv5vkp4jpgkdcunar8efpvennxwwcev
    provider: akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0

root@node1:~# kubectl -n $NS get ing 
NAME                                           CLASS                 HOSTS                                          ADDRESS        PORTS   AGE
brgb45eveddpv99qqqbgosldf4.ingress.akash.pro   akash-ingress-class   brgb45eveddpv99qqqbgosldf4.ingress.akash.pro   65.108.6.185   80      22d
eu-central.nulldao.xyz                         akash-ingress-class   eu-central.nulldao.xyz                         65.108.6.185   80      22d

root@node1:~# kubectl -n $NS get svc
NAME    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
tezos   ClusterIP   10.233.48.214   <none>        80/TCP    22d
@andy108369
Copy link
Contributor Author

Interesting, once I bounced the akash-provider, that lease terminated due to deployment failed. closing lease., see below in the logs:

# kubectl -n akash-services logs $(kubectl -n akash-services get pods -l app=akash-provider --output jsonpath='{.items[-1].metadata.name}') | grep -Ev 'running check|check result' | grep 6489621
D[2022-06-28|14:57:56.716] found existing hostname                      module=provider-cluster cmp=service hostname=h0oufm2j6ldhj1v6hooc2nm2e0.ingress.akash.pro id=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
D[2022-06-28|14:57:56.716] found existing hostname                      module=provider-cluster cmp=service hostname=hkfdsh.fans id=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
I[2022-06-28|14:57:56.718] hostnames withheld                           module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cnt=0
D[2022-06-28|14:57:56.798] no services                                  module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=brwpdb
D[2022-06-28|14:57:56.802] no services                                  module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=brwpf
D[2022-06-28|14:57:56.818] no services                                  module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service=dns
I[2022-06-28|14:57:56.820] declaring hostname                           module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service-name=cms external-port=80
I[2022-06-28|14:57:56.827] declaring hostname                           module=provider-cluster-kube lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 service-name=cms external-port=80
D[2022-06-28|14:57:56.829] deploy complete                              module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|14:58:07.136] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:58:18.480] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:58:31.328] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:58:49.678] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:58:56.744] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:59:11.938] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:59:21.488] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:59:33.548] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|14:59:50.719] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:01.556] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:14.627] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:30.558] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:36.478] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:50.019] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:00:58.803] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:12.285] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:23.590] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:28.275] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:36.042] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:01:54.820] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:08.357] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:17.871] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:24.385] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:35.412] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:02:48.164] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:04.027] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:14.691] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:24.546] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:34.115] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:44.495] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:03:52.101] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:03.612] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:10.769] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:28.467] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:47.203] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:04:56.890] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:01.079] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:08.390] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:18.091] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:23.101] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
D[2022-06-28|15:05:36.299] service available replicas below target      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor service=cms available=0 target=1
E[2022-06-28|15:05:36.299] deployment failed.  closing lease.           module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor
I[2022-06-28|15:05:36.319] bidding on lease closed                      module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor
D[2022-06-28|15:05:37.616] teardown request                             module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.616] shutting down                                module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor
D[2022-06-28|15:05:37.616] shutdown complete                            module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-monitor
D[2022-06-28|15:05:37.644] teardown complete                            module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] shutting down                                module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] waiting on withdrawal                        module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] shutting down                                module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-withdrawal
D[2022-06-28|15:05:37.644] shutdown complete                            module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash cmp=deployment-withdrawal
D[2022-06-28|15:05:37.644] waiting on dm.wg                             module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
I[2022-06-28|15:05:37.644] shutdown complete                            module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] hostnames released                           module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
D[2022-06-28|15:05:37.644] sending manager into channel                 module=provider-cluster cmp=service cmp=deployment-manager lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0 manifest-group=akash
I[2022-06-28|15:05:37.644] manager done                                 module=provider-cluster cmp=service lease=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1/akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0
D[2022-06-28|15:05:37.644] unreserving capacity                         module=provider-cluster cmp=service cmp=inventory-service order=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1
I[2022-06-28|15:05:37.644] attempting to removing reservation           module=provider-cluster cmp=service cmp=inventory-service order=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1
I[2022-06-28|15:05:37.644] removing reservation                         module=provider-cluster cmp=service cmp=inventory-service order=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1
I[2022-06-28|15:05:37.644] unreserve capacity complete                  module=provider-cluster cmp=service cmp=inventory-service order=akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua/6489621/1/1

@andy108369
Copy link
Contributor Author

And that lease looked okay, i.e. did not get overdrawn, was running for less than 2 days:

$ PROVIDER=akash1nxq8gmsw2vlz3m68qvyvcf3kh6q269ajvqw6y0; HEIGHT=$(akash query block | jq -r '.block.header.height'); akash query market lease list --height $HEIGHT --provider $PROVIDER --gseq 0 --oseq 0 --page 1 --limit 10000 --state active -o json | jq --argjson h $HEIGHT -r '["owner","dseq/gseq/oseq","rate","monthly","withdrawn","consumed","days"], (.leases[] | [(.lease.lease_id | .owner, (.dseq|tostring) + "/" + (.gseq|tostring) + "/" + (.oseq|tostring)), (.escrow_payment | (.rate.amount|tonumber), (.rate.amount|tonumber), (.withdrawn.amount|tonumber)), (($h-(.lease.created_at|tonumber))*(.escrow_payment.rate.amount|tonumber)/pow(10;6)), (($h-(.lease.created_at|tonumber))/((60/6)*60*24))]) | @csv' | awk -F ',' '{if (NR==1) {$1=$1; printf $0"\n"} else {block_time=6; rate_akt=(($4*((60/block_time)*60*24*30.436875))/10^6); $4=rate_akt; withdrawn_akt=($5/10^6); $5=withdrawn_akt; $6; $7; print $0}}' | column -t
"owner"                                         "dseq/gseq/oseq"  "rate"   "monthly"  "withdrawn"  "consumed"          "days"
"akash18c330sepgqsmqdfpsygdpm2fm7weskhp9zseua"  "6489621/1/1"     105      46.0206    1.47473      2.583945            1.7089583333333334
...

@tidrolpolelsef
Copy link

This can happen when the docker container is unavailable, or has an error in it. Is it possible ghcr.io/sfdotstudio/tezos-quikn:latest crashes at some point?

@andy108369
Copy link
Contributor Author

Could be, I've entirely forgot to check the pod itself.
Going to try kubectl -n $NS get pods next time.
Thanks for the pointer.

@troian troian transferred this issue from akash-network/node Feb 17, 2023
@troian troian added sev2 repo/provider Akash provider-services repo issues and removed sev1 labels Feb 17, 2023
@andy108369
Copy link
Contributor Author

haven't seen these messages for quite some time.

@andy108369
Copy link
Contributor Author

andy108369 commented Dec 18, 2023 •
edited
Loading

FWIW, the service available replicas below target followed by deployment failed. closing lease. message relates to the monitorMaxRetries counter (ref #14 ) ; that means the pods in the K8s ReplicaSet were unable to get into the READY state.

More details:

In Akash deployments, the process involves converting the deployment into the Kubernetes (K8s) Deployment kind, except in the case of persistent storage deployments, which use the StatefulSet kind. Here's how the 'Deployment' kind functions:

  1. Kubernetes creates a ReplicaSet based on the Deployment (the amount of replicas is typically set to 1);
  2. If any issues arise during deployment, Kubernetes will not achieve the target replica availability. Consequently, the Akash Provider logs the status as service available replicas below target. This also implies that the deployment attempt count will continue to increase until it reaches 40. At that point, the akash-provider automatically closes the lease.

When the (monitorMaxRetries counter in the code or attempt counter in the logs) reaches 40, the next message you see is:

E[2022-12-08|16:46:15.299] deployment failed.  closing lease.           module=provider-cluster cmp=provider cmp=service cmp=deployment-manager lease=akash1h2adh8s6ptsx33m6hda7p9kahcdwy09dhr5x90/8816000/1/1/akash1vky0uh4wayh9npd74uqesglpaxwymynnspf6a4 manifest-group=dcloud cmp=deployment-monitor

I.e. the incrementing attempt counter along with these lines indicate there is some issue due to which the K8s cannot reach the active replicas (i.e. deploy the deployment) 1/1 (DESIRED/READY)

service available replicas below target
deployment failed.  closing lease.

image

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@andy108369 andy108369

Labels
repo/provider Akash provider-services repo issues sev2
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@troian @chandadharap @tidrolpolelsef @andy108369