2024-04-26T20:53:09.944585Z info FLAG: --concurrency="0" 2024-04-26T20:53:09.944660Z info FLAG: --domain="tetrate-istio-system.svc.cluster.local" 2024-04-26T20:53:09.944679Z info FLAG: --help="false" 2024-04-26T20:53:09.944690Z info FLAG: --log_as_json="false" 2024-04-26T20:53:09.944704Z info FLAG: --log_caller="" 2024-04-26T20:53:09.944715Z info FLAG: --log_output_level="default:info" 2024-04-26T20:53:09.944724Z info FLAG: --log_rotate="" 2024-04-26T20:53:09.944734Z info FLAG: --log_rotate_max_age="30" 2024-04-26T20:53:09.944743Z info FLAG: --log_rotate_max_backups="1000" 2024-04-26T20:53:09.944753Z info FLAG: --log_rotate_max_size="104857600" 2024-04-26T20:53:09.944762Z info FLAG: --log_stacktrace_level="default:none" 2024-04-26T20:53:09.944776Z info FLAG: --log_target="[stdout]" 2024-04-26T20:53:09.944786Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2024-04-26T20:53:09.944795Z info FLAG: --outlierLogPath="" 2024-04-26T20:53:09.944806Z info FLAG: --profiling="true" 2024-04-26T20:53:09.944821Z info FLAG: --proxyComponentLogLevel="misc:error" 2024-04-26T20:53:09.944831Z info FLAG: --proxyLogLevel="warning" 2024-04-26T20:53:09.944841Z info FLAG: --s2a_enable_appengine_dialer="false" 2024-04-26T20:53:09.944851Z info FLAG: --s2a_timeout="3s" 2024-04-26T20:53:09.944862Z info FLAG: --serviceCluster="istio-proxy" 2024-04-26T20:53:09.944871Z info FLAG: --stsPort="0" 2024-04-26T20:53:09.944881Z info FLAG: --templateFile="" 2024-04-26T20:53:09.944891Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange" 2024-04-26T20:53:09.944900Z info FLAG: --vklog="0" 2024-04-26T20:53:09.944911Z info Version 1.18.1-tetrate-v0-726d52e068bcbb7eb08c7fc93b1c81a43d33dfbb-Clean 2024-04-26T20:53:09.947169Z info Maximum file descriptors (ulimit -n): 1048576 2024-04-26T20:53:09.947423Z info Proxy role ips=[10.0.150.133] type=router id=custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system domain=tetrate-istio-system.svc.cluster.local 2024-04-26T20:53:09.947522Z info Apply proxy config from env {"discoveryAddress":"istiod-1-18-1.tetrate-istio-system.svc:15012","tracing":{"zipkin":{"address":"zipkin.tetrate-istio-system:9411"}}} 2024-04-26T20:53:09.948964Z info cpu limit detected as 8, setting concurrency 2024-04-26T20:53:09.949239Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 8 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-1-18-1.tetrate-istio-system.svc:15012 drainDuration: 45s proxyAdminPort: 15000 serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s tracing: zipkin: address: zipkin.tetrate-istio-system:9411 2024-04-26T20:53:09.949261Z info JWT policy is third-party-jwt 2024-04-26T20:53:09.949272Z info using credential fetcher of JWT type in cluster.local trust domain 2024-04-26T20:53:09.951001Z info platform detected is AWS 2024-04-26T20:53:09.953185Z info Workload SDS socket not found. Starting Istio SDS Server 2024-04-26T20:53:09.953215Z info CA Endpoint istiod-1-18-1.tetrate-istio-system.svc:15012, provider Citadel 2024-04-26T20:53:09.953191Z info Opening status port 15020 2024-04-26T20:53:09.953265Z info Using CA istiod-1-18-1.tetrate-istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2024-04-26T20:53:09.975903Z info ads All caches have been synced up in 34.763337ms, marking server ready 2024-04-26T20:53:09.976270Z info xdsproxy Initializing with upstream address "istiod-1-18-1.tetrate-istio-system.svc:15012" and cluster "Kubernetes" 2024-04-26T20:53:09.976354Z info sds Starting SDS grpc server 2024-04-26T20:53:09.976833Z info starting Http service at 127.0.0.1:15004 2024-04-26T20:53:09.978221Z info Pilot SAN: [istiod-1-18-1.tetrate-istio-system.svc] 2024-04-26T20:53:09.984503Z info Starting proxy agent 2024-04-26T20:53:09.984562Z info starting 2024-04-26T20:53:09.984602Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields --log-format %Y-%m-%dT%T.%fZ %l envoy %n %g:%# %v thread=%t -l warning --component-log-level misc:error --concurrency 8] 2024-04-26T20:53:10.181751Z warn ca ca request failed, starting attempt 1 in 97.645256ms 2024-04-26T20:53:10.280076Z warn ca ca request failed, starting attempt 2 in 202.535619ms 2024-04-26T20:53:10.483750Z warn ca ca request failed, starting attempt 3 in 407.375951ms 2024-04-26T20:53:10.892038Z warn ca ca request failed, starting attempt 4 in 737.274706ms 2024-04-26T20:53:11.239317Z info xdsproxy connected to upstream XDS server: istiod-1-18-1.tetrate-istio-system.svc:15012 2024-04-26T20:53:11.256953Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-1 2024-04-26T20:53:11.261073Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-2 2024-04-26T20:53:11.636322Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T20:53:11.706057Z warn ca ca request failed, starting attempt 1 in 106.788978ms 2024-04-26T20:53:11.813438Z warn ca ca request failed, starting attempt 2 in 191.11485ms 2024-04-26T20:53:12.004783Z warn ca ca request failed, starting attempt 3 in 427.295336ms 2024-04-26T20:53:12.432346Z warn ca ca request failed, starting attempt 4 in 727.917243ms 2024-04-26T20:53:13.167170Z warn cache slow generate secret lock: 1.90594857s 2024-04-26T20:53:13.167178Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-1 terminated 2024-04-26T20:53:13.167388Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:13.390550Z warn ca ca request failed, starting attempt 1 in 91.154239ms 2024-04-26T20:53:13.393727Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-3 2024-04-26T20:53:13.482222Z warn ca ca request failed, starting attempt 2 in 217.815808ms 2024-04-26T20:53:13.700648Z warn ca ca request failed, starting attempt 3 in 390.746283ms 2024-04-26T20:53:14.092268Z warn ca ca request failed, starting attempt 4 in 759.283989ms 2024-04-26T20:53:14.851846Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-2 terminated 2024-04-26T20:53:14.851839Z warn cache slow generate secret lock: 2.738831579s 2024-04-26T20:53:14.852038Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:14.973188Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-4 2024-04-26T20:53:15.003471Z warn ca ca request failed, starting attempt 1 in 92.271975ms 2024-04-26T20:53:15.096801Z warn ca ca request failed, starting attempt 2 in 180.410097ms 2024-04-26T20:53:15.278199Z warn ca ca request failed, starting attempt 3 in 433.448188ms 2024-04-26T20:53:15.712501Z warn ca ca request failed, starting attempt 4 in 777.235548ms 2024-04-26T20:53:16.497499Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T20:53:16.497542Z warn cache slow generate secret lock: 3.103690954s 2024-04-26T20:53:16.702663Z warn ca ca request failed, starting attempt 1 in 94.69273ms 2024-04-26T20:53:16.797963Z warn ca ca request failed, starting attempt 2 in 207.416803ms 2024-04-26T20:53:17.006390Z warn ca ca request failed, starting attempt 3 in 420.828013ms 2024-04-26T20:53:17.427793Z warn ca ca request failed, starting attempt 4 in 748.03589ms 2024-04-26T20:53:18.176764Z warn cache slow generate secret lock: 3.203423564s 2024-04-26T20:53:18.176784Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-3 terminated 2024-04-26T20:53:18.176936Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:18.274311Z warn ca ca request failed, starting attempt 1 in 92.894826ms 2024-04-26T20:53:18.367621Z warn ca ca request failed, starting attempt 2 in 189.043943ms 2024-04-26T20:53:18.425181Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-5 2024-04-26T20:53:18.557842Z warn ca ca request failed, starting attempt 3 in 408.777954ms 2024-04-26T20:53:18.967490Z warn ca ca request failed, starting attempt 4 in 856.283983ms 2024-04-26T20:53:19.824637Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-4 terminated 2024-04-26T20:53:19.824647Z warn cache slow generate secret lock: 2.681957622s 2024-04-26T20:53:19.824761Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:20.045231Z warn ca ca request failed, starting attempt 1 in 98.815796ms 2024-04-26T20:53:20.055205Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-6 2024-04-26T20:53:20.144775Z warn ca ca request failed, starting attempt 2 in 196.936936ms 2024-04-26T20:53:20.348335Z warn ca ca request failed, starting attempt 3 in 390.066325ms 2024-04-26T20:53:20.738871Z warn ca ca request failed, starting attempt 4 in 737.028137ms 2024-04-26T20:53:21.476794Z warn cache slow generate secret lock: 3.051485298s 2024-04-26T20:53:21.476803Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T20:53:21.534946Z warn ca ca request failed, starting attempt 1 in 104.884955ms 2024-04-26T20:53:21.640239Z warn ca ca request failed, starting attempt 2 in 209.570242ms 2024-04-26T20:53:21.850663Z warn ca ca request failed, starting attempt 3 in 380.912635ms 2024-04-26T20:53:22.232256Z warn ca ca request failed, starting attempt 4 in 828.41537ms 2024-04-26T20:53:23.063777Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-5 terminated 2024-04-26T20:53:23.063790Z warn cache slow generate secret lock: 3.00841305s 2024-04-26T20:53:23.063962Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:23.297650Z warn ca ca request failed, starting attempt 1 in 101.287254ms 2024-04-26T20:53:23.399955Z warn ca ca request failed, starting attempt 2 in 190.350587ms 2024-04-26T20:53:23.591248Z warn ca ca request failed, starting attempt 3 in 406.443088ms 2024-04-26T20:53:23.763341Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-7 2024-04-26T20:53:23.997942Z warn ca ca request failed, starting attempt 4 in 789.121193ms 2024-04-26T20:53:24.787990Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-6 terminated 2024-04-26T20:53:24.787987Z warn cache slow generate secret lock: 1.947598023s 2024-04-26T20:53:24.788151Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:24.940955Z warn ca ca request failed, starting attempt 1 in 107.535572ms 2024-04-26T20:53:25.048804Z warn ca ca request failed, starting attempt 2 in 190.385219ms 2024-04-26T20:53:25.240215Z warn ca ca request failed, starting attempt 3 in 419.250273ms 2024-04-26T20:53:25.659793Z warn ca ca request failed, starting attempt 4 in 754.95195ms 2024-04-26T20:53:25.804770Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-8 2024-04-26T20:53:26.421577Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T20:53:26.421677Z warn cache slow generate secret lock: 2.658156982s 2024-04-26T20:53:26.558823Z warn ca ca request failed, starting attempt 1 in 105.383477ms 2024-04-26T20:53:26.665135Z warn ca ca request failed, starting attempt 2 in 182.90497ms 2024-04-26T20:53:26.848370Z warn ca ca request failed, starting attempt 3 in 431.424176ms 2024-04-26T20:53:27.279957Z warn ca ca request failed, starting attempt 4 in 804.44451ms 2024-04-26T20:53:28.085577Z warn cache slow generate secret lock: 2.280694817s 2024-04-26T20:53:28.085635Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-7 terminated 2024-04-26T20:53:28.085814Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:28.276198Z warn ca ca request failed, starting attempt 1 in 93.257088ms 2024-04-26T20:53:28.370620Z warn ca ca request failed, starting attempt 2 in 189.845082ms 2024-04-26T20:53:28.561061Z warn ca ca request failed, starting attempt 3 in 423.697887ms 2024-04-26T20:53:28.985810Z warn ca ca request failed, starting attempt 4 in 791.239069ms 2024-04-26T20:53:29.633033Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-9 2024-04-26T20:53:29.777816Z warn cache slow generate secret lock: 1.654652615s 2024-04-26T20:53:29.777816Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-8 terminated 2024-04-26T20:53:29.777966Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:29.978961Z warn ca ca request failed, starting attempt 1 in 102.562482ms 2024-04-26T20:53:30.082246Z warn ca ca request failed, starting attempt 2 in 204.412935ms 2024-04-26T20:53:30.287672Z warn ca ca request failed, starting attempt 3 in 372.892353ms 2024-04-26T20:53:30.661264Z warn ca ca request failed, starting attempt 4 in 850.412061ms 2024-04-26T20:53:31.512278Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T20:53:31.512305Z warn cache slow generate secret lock: 1.879149291s 2024-04-26T20:53:31.676340Z warn ca ca request failed, starting attempt 1 in 91.886401ms 2024-04-26T20:53:31.768640Z warn ca ca request failed, starting attempt 2 in 207.504841ms 2024-04-26T20:53:31.977091Z warn ca ca request failed, starting attempt 3 in 380.732581ms 2024-04-26T20:53:32.001167Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-10 2024-04-26T20:53:32.359006Z warn ca ca request failed, starting attempt 4 in 745.734489ms 2024-04-26T20:53:33.105056Z warn cache slow generate secret lock: 1.103755298s 2024-04-26T20:53:33.105061Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-9 terminated 2024-04-26T20:53:33.105238Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:33.219688Z warn ca ca request failed, starting attempt 1 in 106.488261ms 2024-04-26T20:53:33.327012Z warn ca ca request failed, starting attempt 2 in 180.836045ms 2024-04-26T20:53:33.508374Z warn ca ca request failed, starting attempt 3 in 428.424351ms 2024-04-26T20:53:33.936953Z warn ca ca request failed, starting attempt 4 in 742.23074ms 2024-04-26T20:53:34.679395Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-10 terminated 2024-04-26T20:53:34.679591Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:34.811830Z warn ca ca request failed, starting attempt 1 in 103.498271ms 2024-04-26T20:53:34.916154Z warn ca ca request failed, starting attempt 2 in 188.68264ms 2024-04-26T20:53:35.105478Z warn ca ca request failed, starting attempt 3 in 362.115774ms 2024-04-26T20:53:35.281205Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-11 2024-04-26T20:53:35.467911Z warn ca ca request failed, starting attempt 4 in 800.78054ms 2024-04-26T20:53:36.269631Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T20:53:36.435424Z warn ca ca request failed, starting attempt 1 in 107.704139ms 2024-04-26T20:53:36.543735Z warn ca ca request failed, starting attempt 2 in 186.638921ms 2024-04-26T20:53:36.731127Z warn ca ca request failed, starting attempt 3 in 366.690376ms 2024-04-26T20:53:37.098762Z warn ca ca request failed, starting attempt 4 in 878.20138ms 2024-04-26T20:53:37.983759Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-11 terminated 2024-04-26T20:53:37.983945Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:39.537488Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-12 2024-04-26T20:53:39.988540Z warn ca ca request failed, starting attempt 1 in 92.505264ms 2024-04-26T20:53:40.081836Z warn ca ca request failed, starting attempt 2 in 218.596903ms 2024-04-26T20:53:40.301658Z warn ca ca request failed, starting attempt 3 in 418.313425ms 2024-04-26T20:53:40.720297Z warn ca ca request failed, starting attempt 4 in 871.654327ms 2024-04-26T20:53:41.592438Z warn cache slow generate secret lock: 1.886060343s 2024-04-26T20:53:41.592470Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-12 terminated 2024-04-26T20:53:41.592674Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:41.954964Z warn ca ca request failed, starting attempt 1 in 100.327224ms 2024-04-26T20:53:42.056206Z warn ca ca request failed, starting attempt 2 in 198.084568ms 2024-04-26T20:53:42.254597Z warn ca ca request failed, starting attempt 3 in 406.936554ms 2024-04-26T20:53:42.661884Z warn ca ca request failed, starting attempt 4 in 818.362147ms 2024-04-26T20:53:43.480898Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T20:53:48.690419Z warn ca ca request failed, starting attempt 1 in 104.639513ms 2024-04-26T20:53:48.795689Z warn ca ca request failed, starting attempt 2 in 212.71457ms 2024-04-26T20:53:49.009127Z warn ca ca request failed, starting attempt 3 in 379.404885ms 2024-04-26T20:53:49.389598Z warn ca ca request failed, starting attempt 4 in 748.946927ms 2024-04-26T20:53:50.139656Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T20:53:51.763573Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-13 2024-04-26T20:53:51.870374Z warn ca ca request failed, starting attempt 1 in 104.18215ms 2024-04-26T20:53:51.975676Z warn ca ca request failed, starting attempt 2 in 211.02392ms 2024-04-26T20:53:52.187039Z warn ca ca request failed, starting attempt 3 in 375.858031ms 2024-04-26T20:53:52.563336Z warn ca ca request failed, starting attempt 4 in 796.051446ms 2024-04-26T20:53:53.049353Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-14 2024-04-26T20:53:53.360193Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-13 terminated 2024-04-26T20:53:53.360533Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:53.731133Z warn ca ca request failed, starting attempt 1 in 105.19081ms 2024-04-26T20:53:53.837446Z warn ca ca request failed, starting attempt 2 in 196.791244ms 2024-04-26T20:53:54.034865Z warn ca ca request failed, starting attempt 3 in 403.98862ms 2024-04-26T20:53:54.439493Z warn ca ca request failed, starting attempt 4 in 754.540252ms 2024-04-26T20:53:55.200514Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-dmvpx.tetrate-istio-system-14 terminated 2024-04-26T20:53:55.200649Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=20 2024-04-26T20:53:58.209031Z warn ca ca request failed, starting attempt 1 in 95.937979ms 2024-04-26T20:53:58.305348Z warn ca ca request failed, starting attempt 2 in 212.210781ms 2024-04-26T20:53:58.517727Z warn ca ca request failed, starting attempt 3 in 426.415363ms 2024-04-26T20:53:58.944303Z warn ca ca request failed, starting attempt 4 in 876.857186ms 2024-04-26T20:53:59.821363Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T20:54:08.696622Z warn ca ca request failed, starting attempt 1 in 108.576993ms 2024-04-26T20:54:08.805986Z warn ca ca request failed, starting attempt 2 in 203.833794ms 2024-04-26T20:54:09.010416Z warn ca ca request failed, starting attempt 3 in 436.854419ms 2024-04-26T20:54:09.448071Z warn ca ca request failed, starting attempt 4 in 780.615065ms 2024-04-26T20:54:10.229853Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")"