2024-04-26T22:42:54.389513Z info FLAG: --concurrency="0" 2024-04-26T22:42:54.389643Z info FLAG: --domain="tetrate-istio-system.svc.cluster.local" 2024-04-26T22:42:54.389680Z info FLAG: --help="false" 2024-04-26T22:42:54.389698Z info FLAG: --log_as_json="false" 2024-04-26T22:42:54.389716Z info FLAG: --log_caller="" 2024-04-26T22:42:54.389733Z info FLAG: --log_output_level="default:info" 2024-04-26T22:42:54.389750Z info FLAG: --log_rotate="" 2024-04-26T22:42:54.389768Z info FLAG: --log_rotate_max_age="30" 2024-04-26T22:42:54.389785Z info FLAG: --log_rotate_max_backups="1000" 2024-04-26T22:42:54.389802Z info FLAG: --log_rotate_max_size="104857600" 2024-04-26T22:42:54.389819Z info FLAG: --log_stacktrace_level="default:none" 2024-04-26T22:42:54.389840Z info FLAG: --log_target="[stdout]" 2024-04-26T22:42:54.389858Z info FLAG: --meshConfig="./etc/istio/config/mesh" 2024-04-26T22:42:54.389875Z info FLAG: --outlierLogPath="" 2024-04-26T22:42:54.389893Z info FLAG: --profiling="true" 2024-04-26T22:42:54.389910Z info FLAG: --proxyComponentLogLevel="misc:error" 2024-04-26T22:42:54.389927Z info FLAG: --proxyLogLevel="warning" 2024-04-26T22:42:54.389945Z info FLAG: --s2a_enable_appengine_dialer="false" 2024-04-26T22:42:54.389963Z info FLAG: --s2a_timeout="3s" 2024-04-26T22:42:54.389980Z info FLAG: --serviceCluster="istio-proxy" 2024-04-26T22:42:54.389997Z info FLAG: --stsPort="0" 2024-04-26T22:42:54.390013Z info FLAG: --templateFile="" 2024-04-26T22:42:54.390031Z info FLAG: --tokenManagerPlugin="GoogleTokenExchange" 2024-04-26T22:42:54.390048Z info FLAG: --vklog="0" 2024-04-26T22:42:54.390066Z info Version 1.18.1-tetrate-v0-726d52e068bcbb7eb08c7fc93b1c81a43d33dfbb-Clean 2024-04-26T22:42:54.402775Z info Maximum file descriptors (ulimit -n): 65536 2024-04-26T22:42:54.404194Z info Proxy role ips=[192.168.2.3] type=router id=custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system domain=tetrate-istio-system.svc.cluster.local 2024-04-26T22:42:54.404752Z info Apply proxy config from env {"discoveryAddress":"istiod-1-18-1.tetrate-istio-system.svc:15012","tracing":{"zipkin":{"address":"zipkin.tetrate-istio-system:9411"}}} 2024-04-26T22:42:54.408168Z info cpu limit detected as 4, setting concurrency 2024-04-26T22:42:54.408947Z info Effective config: binaryPath: /usr/local/bin/envoy concurrency: 4 configPath: ./etc/istio/proxy controlPlaneAuthPolicy: MUTUAL_TLS discoveryAddress: istiod-1-18-1.tetrate-istio-system.svc:15012 drainDuration: 45s proxyAdminPort: 15000 serviceCluster: istio-proxy statNameLength: 189 statusPort: 15020 terminationDrainDuration: 5s tracing: zipkin: address: zipkin.tetrate-istio-system:9411 2024-04-26T22:42:54.409466Z info JWT policy is third-party-jwt 2024-04-26T22:42:54.409702Z info using credential fetcher of JWT type in cluster.local trust domain 2024-04-26T22:42:54.513011Z info Workload SDS socket not found. Starting Istio SDS Server 2024-04-26T22:42:54.513424Z info CA Endpoint istiod-1-18-1.tetrate-istio-system.svc:15012, provider Citadel 2024-04-26T22:42:54.513647Z info Using CA istiod-1-18-1.tetrate-istio-system.svc:15012 cert with certs: var/run/secrets/istio/root-cert.pem 2024-04-26T22:42:54.513044Z info Opening status port 15020 2024-04-26T22:42:54.561668Z info ads All caches have been synced up in 176.701435ms, marking server ready 2024-04-26T22:42:54.562364Z info xdsproxy Initializing with upstream address "istiod-1-18-1.tetrate-istio-system.svc:15012" and cluster "Kubernetes" 2024-04-26T22:42:54.562394Z info sds Starting SDS grpc server 2024-04-26T22:42:54.565141Z info Pilot SAN: [istiod-1-18-1.tetrate-istio-system.svc] 2024-04-26T22:42:54.567348Z info starting Http service at 127.0.0.1:15004 2024-04-26T22:42:54.569379Z info Starting proxy agent 2024-04-26T22:42:54.569462Z info starting 2024-04-26T22:42:54.569505Z info Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields --log-format %Y-%m-%dT%T.%fZ %l envoy %n %g:%# %v thread=%t -l warning --component-log-level misc:error --concurrency 4] 2024-04-26T22:42:54.630170Z warn ca ca request failed, starting attempt 1 in 103.831273ms 2024-04-26T22:42:54.734213Z warn ca ca request failed, starting attempt 2 in 206.073578ms 2024-04-26T22:42:54.940856Z warn ca ca request failed, starting attempt 3 in 368.557238ms 2024-04-26T22:42:55.316264Z warn ca ca request failed, starting attempt 4 in 799.451196ms 2024-04-26T22:42:56.127794Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T22:42:56.580070Z info xdsproxy connected to upstream XDS server: istiod-1-18-1.tetrate-istio-system.svc:15012 2024-04-26T22:42:56.592423Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-1 2024-04-26T22:42:56.597314Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-2 2024-04-26T22:42:56.739518Z warn ca ca request failed, starting attempt 1 in 95.329933ms 2024-04-26T22:42:56.836360Z warn ca ca request failed, starting attempt 2 in 215.189716ms 2024-04-26T22:42:57.052730Z warn ca ca request failed, starting attempt 3 in 408.800067ms 2024-04-26T22:42:57.462677Z warn ca ca request failed, starting attempt 4 in 877.259242ms 2024-04-26T22:42:58.352875Z warn cache slow generate secret lock: 1.755358037s 2024-04-26T22:42:58.352894Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-1 terminated 2024-04-26T22:42:58.353259Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:42:58.518773Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-3 2024-04-26T22:42:58.566759Z warn ca ca request failed, starting attempt 1 in 102.933418ms 2024-04-26T22:42:58.670457Z warn ca ca request failed, starting attempt 2 in 190.676996ms 2024-04-26T22:42:58.861496Z warn ca ca request failed, starting attempt 3 in 361.893022ms 2024-04-26T22:42:59.224243Z warn ca ca request failed, starting attempt 4 in 843.439469ms 2024-04-26T22:43:00.068311Z warn cache slow generate secret lock: 3.410406934s 2024-04-26T22:43:00.068809Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-2 terminated 2024-04-26T22:43:00.070662Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:43:00.303104Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-4 2024-04-26T22:43:00.508719Z warn ca ca request failed, starting attempt 1 in 92.608619ms 2024-04-26T22:43:00.613537Z warn ca ca request failed, starting attempt 2 in 190.511399ms 2024-04-26T22:43:00.805251Z warn ca ca request failed, starting attempt 3 in 388.472266ms 2024-04-26T22:43:01.193945Z warn ca ca request failed, starting attempt 4 in 877.222968ms 2024-04-26T22:43:02.072220Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T22:43:02.072281Z warn cache slow generate secret lock: 3.553249953s 2024-04-26T22:43:02.210782Z warn ca ca request failed, starting attempt 1 in 108.434005ms 2024-04-26T22:43:02.320212Z warn ca ca request failed, starting attempt 2 in 219.440002ms 2024-04-26T22:43:02.540788Z warn ca ca request failed, starting attempt 3 in 362.874179ms 2024-04-26T22:43:02.904553Z warn ca ca request failed, starting attempt 4 in 810.204ms 2024-04-26T22:43:03.715528Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-3 terminated 2024-04-26T22:43:03.715568Z warn cache slow generate secret lock: 3.412133193s 2024-04-26T22:43:03.716004Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:43:04.037465Z warn ca ca request failed, starting attempt 1 in 98.683045ms 2024-04-26T22:43:04.136541Z warn ca ca request failed, starting attempt 2 in 193.65664ms 2024-04-26T22:43:04.331547Z warn ca ca request failed, starting attempt 3 in 364.268458ms 2024-04-26T22:43:04.519016Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-5 2024-04-26T22:43:04.696410Z warn ca ca request failed, starting attempt 4 in 872.026113ms 2024-04-26T22:43:05.632549Z warn cache slow generate secret lock: 2.691731157s 2024-04-26T22:43:05.668988Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-4 terminated 2024-04-26T22:43:05.671022Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:43:05.760705Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-6 2024-04-26T22:43:06.012434Z warn ca ca request failed, starting attempt 1 in 99.226569ms 2024-04-26T22:43:06.112862Z warn ca ca request failed, starting attempt 2 in 209.768863ms 2024-04-26T22:43:06.323532Z warn ca ca request failed, starting attempt 3 in 436.141197ms 2024-04-26T22:43:06.760200Z warn ca ca request failed, starting attempt 4 in 828.250613ms 2024-04-26T22:43:07.588772Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T22:43:07.588810Z warn cache slow generate secret lock: 3.06963814s 2024-04-26T22:43:07.704173Z warn ca ca request failed, starting attempt 1 in 109.880667ms 2024-04-26T22:43:07.818602Z warn ca ca request failed, starting attempt 2 in 191.14399ms 2024-04-26T22:43:08.010602Z warn ca ca request failed, starting attempt 3 in 422.115966ms 2024-04-26T22:43:08.432993Z warn ca ca request failed, starting attempt 4 in 770.08333ms 2024-04-26T22:43:09.203431Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-5 terminated 2024-04-26T22:43:09.203434Z warn cache slow generate secret lock: 3.442416658s 2024-04-26T22:43:09.203815Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:43:09.550441Z warn ca ca request failed, starting attempt 1 in 108.075166ms 2024-04-26T22:43:09.658953Z warn ca ca request failed, starting attempt 2 in 207.173515ms 2024-04-26T22:43:09.866343Z warn ca ca request failed, starting attempt 3 in 439.063636ms 2024-04-26T22:43:10.306172Z warn ca ca request failed, starting attempt 4 in 818.542943ms 2024-04-26T22:43:10.684903Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-7 2024-04-26T22:43:11.125670Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-6 terminated 2024-04-26T22:43:11.125685Z warn cache slow generate secret lock: 2.287768496s 2024-04-26T22:43:11.125864Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:43:11.226627Z warn ca ca request failed, starting attempt 1 in 94.970891ms 2024-04-26T22:43:11.322273Z warn ca ca request failed, starting attempt 2 in 204.272594ms 2024-04-26T22:43:11.365450Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-8 2024-04-26T22:43:11.526782Z warn ca ca request failed, starting attempt 3 in 364.907215ms 2024-04-26T22:43:11.904031Z warn ca ca request failed, starting attempt 4 in 740.001647ms 2024-04-26T22:43:12.645184Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T22:43:12.645223Z warn cache slow generate secret lock: 1.960167051s 2024-04-26T22:43:12.886553Z warn ca ca request failed, starting attempt 1 in 98.183112ms 2024-04-26T22:43:12.984907Z warn ca ca request failed, starting attempt 2 in 187.781436ms 2024-04-26T22:43:13.173514Z warn ca ca request failed, starting attempt 3 in 374.824247ms 2024-04-26T22:43:13.549491Z warn ca ca request failed, starting attempt 4 in 849.117775ms 2024-04-26T22:43:14.399539Z warn cache slow generate secret lock: 3.033924739s 2024-04-26T22:43:14.399852Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-7 terminated 2024-04-26T22:43:14.400610Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:43:14.592466Z warn ca ca request failed, starting attempt 1 in 98.024701ms 2024-04-26T22:43:14.690867Z warn ca ca request failed, starting attempt 2 in 219.097321ms 2024-04-26T22:43:14.910335Z warn ca ca request failed, starting attempt 3 in 412.992375ms 2024-04-26T22:43:15.324053Z warn ca ca request failed, starting attempt 4 in 812.949238ms 2024-04-26T22:43:16.138270Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-8 terminated 2024-04-26T22:43:16.138278Z warn cache slow generate secret lock: 1.981955192s 2024-04-26T22:43:16.138533Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:43:16.333033Z warn ca ca request failed, starting attempt 1 in 105.289466ms 2024-04-26T22:43:16.439502Z warn ca ca request failed, starting attempt 2 in 185.127453ms 2024-04-26T22:43:16.624887Z warn ca ca request failed, starting attempt 3 in 387.89046ms 2024-04-26T22:43:17.013536Z warn ca ca request failed, starting attempt 4 in 811.03946ms 2024-04-26T22:43:17.824801Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" 2024-04-26T22:43:18.268900Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-9 2024-04-26T22:43:18.343964Z warn ca ca request failed, starting attempt 1 in 90.850689ms 2024-04-26T22:43:18.435572Z warn ca ca request failed, starting attempt 2 in 213.54608ms 2024-04-26T22:43:18.616611Z info ads ADS: new connection for node:custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-10 2024-04-26T22:43:18.650266Z warn ca ca request failed, starting attempt 3 in 392.281765ms 2024-04-26T22:43:19.042963Z warn ca ca request failed, starting attempt 4 in 766.016398ms 2024-04-26T22:43:19.810255Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-9 terminated 2024-04-26T22:43:19.810255Z warn cache slow generate secret lock: 1.193474271s 2024-04-26T22:43:19.810566Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for default: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:43:20.062995Z warn ca ca request failed, starting attempt 1 in 104.658817ms 2024-04-26T22:43:20.168304Z warn ca ca request failed, starting attempt 2 in 215.904119ms 2024-04-26T22:43:20.384861Z warn ca ca request failed, starting attempt 3 in 382.350286ms 2024-04-26T22:43:20.767481Z warn ca ca request failed, starting attempt 4 in 775.751564ms 2024-04-26T22:43:21.555601Z warn cache slow generate secret lock: 1.777632681s 2024-04-26T22:43:21.556067Z warning envoy config external/envoy/source/common/config/grpc_stream.h:153 StreamSecrets gRPC config stream to sds-grpc closed: 2, failed to generate secret for ROOTCA: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")" thread=16 2024-04-26T22:43:21.555660Z info ads ADS: "@" custom-ingressgateway-74f9b8578c-2lh2c.tetrate-istio-system-10 terminated 2024-04-26T22:43:21.825471Z warn ca ca request failed, starting attempt 1 in 108.643621ms 2024-04-26T22:43:21.934911Z warn ca ca request failed, starting attempt 2 in 197.796941ms 2024-04-26T22:43:22.133556Z warn ca ca request failed, starting attempt 3 in 404.59315ms 2024-04-26T22:43:22.539281Z warn ca ca request failed, starting attempt 4 in 808.33692ms 2024-04-26T22:43:23.347904Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of \"crypto/rsa: verification error\" while trying to verify candidate authority certificate \"cluster.local\")"