Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] Handle Spark response in Array format #2367

Closed
penghuo opened this issue Oct 25, 2023 · 3 comments
Closed

[BUG] Handle Spark response in Array format #2367

penghuo opened this issue Oct 25, 2023 · 3 comments
Assignees
@penghuo
Labels
bug Something isn't working Flint v2.12.0 Issues targeting release v2.12.0

Comments

@penghuo
Copy link
Collaborator

penghuo commented Oct 25, 2023 

{
          "result": [
            "{'resourceSpans':[{'resource':{'attributes':[{'key':'telemetry.sdk.language','value':{'stringValue':'python'}},{'key':'telemetry.sdk.name','value':{'stringValue':'opentelemetry'}},{'key':'telemetry.sdk.version','value':{'stringValue':'1.19.0'}},{'key':'service.namespace','value':{'stringValue':'opentelemetry-demo'}},{'key':'service.name','value':{'stringValue':'recommendationservice'}},{'key':'telemetry.auto.version','value':{'stringValue':'0.40b0'}}]},'scopeSpans':[{'scope':{'name':'opentelemetry.instrumentation.grpc','version':'0.40b0'},'spans':[{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134631800128753','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'62946fc9762c6a4b','spanId':'f4305b589e138b43','startTimeUnixNano':'1698134631797942068','traceId':'76eef38f2731b429a72b9bed575563ae'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134631801172401','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'62946fc9762c6a4b','spanId':'042cc9e9e5244f8a','startTimeUnixNano':'1698134631800249329','traceId':'76eef38f2731b429a72b9bed575563ae'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134631801517145','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'b3df0e87f82cac39','spanId':'35c3376ad8702d2f','startTimeUnixNano':'1698134631797773158','traceId':'76eef38f2731b429a72b9bed575563ae'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134632138450318','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'b63fb509b48b438f','spanId':'85d2a20603328f03','startTimeUnixNano':'1698134632136137537','traceId':'1d468cfb14c8a5c28acc9b06ffc386c3'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134632139613291','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'b63fb509b48b438f','spanId':'d0c0df185574767f','startTimeUnixNano':'1698134632138565046','traceId':'1d468cfb14c8a5c28acc9b06ffc386c3'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134632139972842','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'e7d349252a440664','spanId':'180a186a96207890','startTimeUnixNano':'1698134632135973735','traceId':'1d468cfb14c8a5c28acc9b06ffc386c3'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134632293510572','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'5a2b01b8534336f8','spanId':'2ef1342f4a30b326','startTimeUnixNano':'1698134632291085221','traceId':'dd58c98ae9b64a2796c48f44a88cc5af'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134632294697557','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'5a2b01b8534336f8','spanId':'0046209ad22e7033','startTimeUnixNano':'1698134632293631973','traceId':'dd58c98ae9b64a2796c48f44a88cc5af'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134632295053736','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'d8579dcc00296ae1','spanId':'072fda52dfddff3e','startTimeUnixNano':'1698134632290913383','traceId':'dd58c98ae9b64a2796c48f44a88cc5af'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134632486978334','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'de9ba749b5142975','spanId':'d3614b849bab6161','startTimeUnixNano':'1698134632484605245','traceId':'5998dc923e515114077ae2c1584a86c2'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134632488145599','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'de9ba749b5142975','spanId':'5193b11e3729914f','startTimeUnixNano':'1698134632487107225','traceId':'5998dc923e515114077ae2c1584a86c2'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134632488494949','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'d0780d1e0c795d08','spanId':'3b131264323cb09e','startTimeUnixNano':'1698134632484436939','traceId':'5998dc923e515114077ae2c1584a86c2'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134632813716198','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'629acaf487091b5e','spanId':'f0c1fcc2bd889675','startTimeUnixNano':'1698134632811340036','traceId':'51c66b42eea674461985714d57615b40'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134632814765149','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'629acaf487091b5e','spanId':'66f6279ded0a0255','startTimeUnixNano':'1698134632813832714','traceId':'51c66b42eea674461985714d57615b40'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134632815126020','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'12e128b8d4e8fb22','spanId':'a1ddde3894149e8b','startTimeUnixNano':'1698134632811154198','traceId':'51c66b42eea674461985714d57615b40'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134633687805041','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'85deb52661894483','spanId':'1d9c59cd6efee84e','startTimeUnixNano':'1698134633685385165','traceId':'2681a4d39fd6e023674067bc95193adb'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134633688979037','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'85deb52661894483','spanId':'08e8044e55d0e4d1','startTimeUnixNano':'1698134633687944658','traceId':'2681a4d39fd6e023674067bc95193adb'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134633689334740','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'df1ea65ad573833e','spanId':'cda6ad2c3170c9a0','startTimeUnixNano':'1698134633685216180','traceId':'2681a4d39fd6e023674067bc95193adb'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134634561183727','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'7a65928769607491','spanId':'e3a8a6851ecb66cf','startTimeUnixNano':'1698134634558854070','traceId':'23980dd6fc6b6af62c792fe74efdb13d'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134634562358217','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'7a65928769607491','spanId':'844e0cb4d297d08b','startTimeUnixNano':'1698134634561307493','traceId':'23980dd6fc6b6af62c792fe74efdb13d'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134634562694854','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'4726404b6a166cc7','spanId':'d2cfd9f49e2c25c3','startTimeUnixNano':'1698134634558687269','traceId':'23980dd6fc6b6af62c792fe74efdb13d'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134635638360760','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'75890c4dc03a2cab','spanId':'7fb4301bb18e25ba','startTimeUnixNano':'1698134635636168464','traceId':'482d65c2240d95ba3c5fbaa07fd240cf'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134635639461651','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'75890c4dc03a2cab','spanId':'eac56e9752fdaa91','startTimeUnixNano':'1698134635638477343','traceId':'482d65c2240d95ba3c5fbaa07fd240cf'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134635639811322','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'d1ebe12404a2e355','spanId':'3d8210ecdf22f030','startTimeUnixNano':'1698134635635999837','traceId':'482d65c2240d95ba3c5fbaa07fd240cf'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134635928336255','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'e259dfc09e2f7034','spanId':'a6f906a377c2bf38','startTimeUnixNano':'1698134635925784542','traceId':'48e2593c3b59f7409e977dda2062dd0b'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134635929538227','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'e259dfc09e2f7034','spanId':'27d47371f49371fa','startTimeUnixNano':'1698134635928483873','traceId':'48e2593c3b59f7409e977dda2062dd0b'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134635929888690','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'5162e3dfffcf131d','spanId':'c8d6df17bd6b571d','startTimeUnixNano':'1698134635925614466','traceId':'48e2593c3b59f7409e977dda2062dd0b'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134636044712971','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'f7366282abd05c67','spanId':'2fe5c865451955b3','startTimeUnixNano':'1698134636042274955','traceId':'17b3a45f29084a71ff7317838c02a8b4'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134636045869704','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'f7366282abd05c67','spanId':'ab9a5948a8070e66','startTimeUnixNano':'1698134636044840479','traceId':'17b3a45f29084a71ff7317838c02a8b4'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134636046216000','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'654f802c743c7d3a','spanId':'bf29d412527ba54a','startTimeUnixNano':'1698134636042104096','traceId':'17b3a45f29084a71ff7317838c02a8b4'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134636053417434','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'ee272d7115eb7eef','spanId':'3e2a7a78358d7df1','startTimeUnixNano':'1698134636051295494','traceId':'066041dac17690ca3a324bfaf0504816'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134636054590823','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'ee272d7115eb7eef','spanId':'68e7e53bb17ee184','startTimeUnixNano':'1698134636053532091','traceId':'066041dac17690ca3a324bfaf0504816'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134636054899533','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'fec080699a471529','spanId':'86ce2e74c849a37b','startTimeUnixNano':'1698134636051151582','traceId':'066041dac17690ca3a324bfaf0504816'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'GetFlag'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.FeatureFlagService'}}],'endTimeUnixNano':'1698134636134044857','kind':3,'name':'/oteldemo.FeatureFlagService/GetFlag','parentSpanId':'15a50c3896b63a19','spanId':'0acdec3429664777','startTimeUnixNano':'1698134636131822114','traceId':'8db5a1e10ef1dcc84ca1cf9fad6cddde'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListProducts'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.ProductCatalogService'}}],'endTimeUnixNano':'1698134636135238995','kind':3,'name':'/oteldemo.ProductCatalogService/ListProducts','parentSpanId':'15a50c3896b63a19','spanId':'d1bc1d67577b549c','startTimeUnixNano':'1698134636134156611','traceId':'8db5a1e10ef1dcc84ca1cf9fad6cddde'},{'attributes':[{'key':'rpc.system','value':{'stringValue':'grpc'}},{'key':'rpc.grpc.status_code','value':{'intValue':'0'}},{'key':'rpc.method','value':{'stringValue':'ListRecommendations'}},{'key':'rpc.service','value':{'stringValue':'oteldemo.RecommendationService'}},{'key':'rpc.user_agent','value':{'stringValue':'grpc-node-js/1.9.0'}},{'key':'net.peer.ip','value':{'stringValue':'172.19.0.23'}},{'key':'net.peer.port','value':{'stringValue':'35360'}},{'key':'app.products_recommended.count','value':{'intValue':'5'}}],'endTimeUnixNano':'1698134636135580410','kind':2,'name':'/oteldemo.RecommendationService/ListRecommendations','parentSpanId':'a3b84684b1fb597e','spanId':'a99934c16f5b28cb','startTimeUnixNano':'1698134636131654450','traceId':'8db5a1e10ef1dcc84ca1cf9fad6cddde'}]},{'scope':{'name':'recommendationservice'},'spans':[{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134631801249909','kind':1,'name':'get_product_list','parentSpanId':'35c3376ad8702d2f','spanId':'62946fc9762c6a4b','startTimeUnixNano':'1698134631797833667','traceId':'76eef38f2731b429a72b9bed575563ae'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134632139698349','kind':1,'name':'get_product_list','parentSpanId':'180a186a96207890','spanId':'b63fb509b48b438f','startTimeUnixNano':'1698134632136033133','traceId':'1d468cfb14c8a5c28acc9b06ffc386c3'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134632294780239','kind':1,'name':'get_product_list','parentSpanId':'072fda52dfddff3e','spanId':'5a2b01b8534336f8','startTimeUnixNano':'1698134632290976820','traceId':'dd58c98ae9b64a2796c48f44a88cc5af'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134632488223744','kind':1,'name':'get_product_list','parentSpanId':'3b131264323cb09e','spanId':'de9ba749b5142975','startTimeUnixNano':'1698134632484497269','traceId':'5998dc923e515114077ae2c1584a86c2'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134632814845183','kind':1,'name':'get_product_list','parentSpanId':'a1ddde3894149e8b','spanId':'629acaf487091b5e','startTimeUnixNano':'1698134632811218962','traceId':'51c66b42eea674461985714d57615b40'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134633689066917','kind':1,'name':'get_product_list','parentSpanId':'cda6ad2c3170c9a0','spanId':'85deb52661894483','startTimeUnixNano':'1698134633685279617','traceId':'2681a4d39fd6e023674067bc95193adb'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134634562437500','kind':1,'name':'get_product_list','parentSpanId':'d2cfd9f49e2c25c3','spanId':'7a65928769607491','startTimeUnixNano':'1698134634558754327','traceId':'23980dd6fc6b6af62c792fe74efdb13d'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134635639543915','kind':1,'name':'get_product_list','parentSpanId':'3d8210ecdf22f030','spanId':'75890c4dc03a2cab','startTimeUnixNano':'1698134635636062611','traceId':'482d65c2240d95ba3c5fbaa07fd240cf'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134635929618984','kind':1,'name':'get_product_list','parentSpanId':'c8d6df17bd6b571d','spanId':'e259dfc09e2f7034','startTimeUnixNano':'1698134635925679041','traceId':'48e2593c3b59f7409e977dda2062dd0b'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134636045950714','kind':1,'name':'get_product_list','parentSpanId':'bf29d412527ba54a','spanId':'f7366282abd05c67','startTimeUnixNano':'1698134636042167094','traceId':'17b3a45f29084a71ff7317838c02a8b4'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134636054668087','kind':1,'name':'get_product_list','parentSpanId':'86ce2e74c849a37b','spanId':'ee272d7115eb7eef','startTimeUnixNano':'1698134636051204657','traceId':'066041dac17690ca3a324bfaf0504816'},{'attributes':[{'key':'app.recommendation.cache_enabled','value':{}},{'key':'app.products.count','value':{'intValue':'10'}},{'key':'app.filtered_products.count','value':{'intValue':'9'}},{'key':'app.filtered_products.list','value':{}}],'endTimeUnixNano':'1698134636135317275','kind':1,'name':'get_product_list','parentSpanId':'a99934c16f5b28cb','spanId':'15a50c3896b63a19','startTimeUnixNano':'1698134636131715947','traceId':'8db5a1e10ef1dcc84ca1cf9fad6cddde'}]}]}]}"
          ],
          "schema": [
            "{'column_name':'resourceSpans','data_type':'array'}"
          ],
          "jobRunId": "00fe8ac6ln7ppg0q",
          "applicationId": "00fd777k3k3ls20p",
          "dataSourceName": "mys3",
          "status": "SUCCESS",
          "error": "",
          "queryId": "Rnd4WXZyOEpEdG15czM=",
          "queryText": "select * from mys3.default.otel_traces limit 1",
          "sessionId": "RGhWWGU2UUFVd215czM=",
          "updateTime": 1698265029107,
          "queryRunTime": 20389
        }
      }
    ]
  }
@penghuo penghuo added bug Something isn't working untriaged v2.11.1 Issues targeting release v2.11.1 and removed untriaged labels Oct 25, 2023
@penghuo
Copy link
Collaborator Author

penghuo commented Oct 25, 2023

The response parse can't handle array type.

https://github.com/opensearch-project/sql/blob/main/spark/src/main/java/org/opensearch/sql/spark/functions/response/DefaultSparkSqlFunctionResponseHandle.java#L63

@YANG-DB YANG-DB self-assigned this Oct 25, 2023
@YANG-DB YANG-DB removed the untriaged label Oct 25, 2023
@YANG-DB YANG-DB mentioned this issue Oct 26, 2023
Closed
6 tasks
@penghuo penghuo added v2.12.0 Issues targeting release v2.12.0 and removed v2.11.1 Issues targeting release v2.11.1 labels Nov 6, 2023
@seankao-az
Copy link
Collaborator

Cannot handle response with missing fields. In the below example, some rows had srcPort = null. When spark writes back to query execution result index, the field with null value is omitted. opensearch-project/opensearch-spark#139

{
    "_index": "query_execution_result_mycw",
    "_id": "-BgYq4sBhRhnt4SbZiGq",
    "_score": 1,
    "_source": {
      "result": [
        "{'srcPort':20641}",
        "{'srcPort':20641}",
        "{}",
        "{}",
        "{'srcPort':20641}",
        "{'srcPort':20641}",
        "{}",
        "{}",
        "{'srcPort':20641}",
        "{'srcPort':20641}"
      ],
      "schema": [
        "{'column_name':'srcPort','data_type':'long'}"
      ],
      "jobRunId": "00feineujeco7g0a",
      "applicationId": "00feggo87noppi09",
      "dataSourceName": "mycw",
      "status": "SUCCESS",
      "error": "",
      "queryId": "cEZnRDVhVlVjWG15Y3c=",
      "queryText": "select srcPort from mycw.default.FLE_GLE_VPC_LoadGen limit 10",
      "sessionId": "dThKam9xTU5sVW15Y3c=",
      "updateTime": 1699382583256,
      "queryRunTime": 11805
    }
}

And we get the following error:

[2023-11-05T02:51:09,749][ERROR][o.o.s.s.r.RestAsyncQueryManagementAction] [5e49ec3a1c321488631cf49c2e2315c1] Error happened during request handling
org.json.JSONException: JSONObject["srcPort"] not found.
        at org.json.JSONObject.get(JSONObject.java:568)
        at org.json.JSONObject.getLong(JSONObject.java:796)
        at org.opensearch.sql.spark.functions.response.DefaultSparkSqlFunctionResponseHandle.extractRow(DefaultSparkSqlFunctionResponseHandle.java:72)
        at org.opensearch.sql.spark.functions.response.DefaultSparkSqlFunctionResponseHandle.constructIteratorAndSchema(DefaultSparkSqlFunctionResponseHandle.java:55)
        at org.opensearch.sql.spark.functions.response.DefaultSparkSqlFunctionResponseHandle.<init>(DefaultSparkSqlFunctionResponseHandle.java:44)
        at org.opensearch.sql.spark.asyncquery.AsyncQueryExecutorServiceImpl.getAsyncQueryResults(AsyncQueryExecutorServiceImpl.java:96)
        at org.opensearch.sql.spark.transport.TransportGetAsyncQueryResultAction.doExecute(TransportGetAsyncQueryResultAction.java:55)
        at org.opensearch.sql.spark.transport.TransportGetAsyncQueryResultAction.doExecute(TransportGetAsyncQueryResultAction.java:28)
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:218)
        at org.opensearch.indexmanagement.rollup.actionfilter.FieldCapsFilter.apply(FieldCapsFilter.kt:118)
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
        at org.opensearch.indexmanagement.controlcenter.notification.filter.IndexOperationActionFilter.apply(IndexOperationActionFilter.kt:39)
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
        at org.opensearch.performanceanalyzer.action.PerformanceAnalyzerActionFilter.apply(PerformanceAnalyzerActionFilter.java:78)
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
        at org.opensearch.security.filter.SecurityFilter.apply0(SecurityFilter.java:395)
        at org.opensearch.security.filter.SecurityFilter.apply(SecurityFilter.java:165)
        at org.opensearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:216)
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:188)
        at org.opensearch.action.support.TransportAction.execute(TransportAction.java:107)
        at org.opensearch.client.node.NodeClient.executeLocally(NodeClient.java:110)
        at org.opensearch.client.node.NodeClient.doExecute(NodeClient.java:97)
        at org.opensearch.client.support.AbstractClient.execute(AbstractClient.java:476)
        at org.opensearch.sql.spark.rest.RestAsyncQueryManagementAction.lambda$executeGetAsyncQueryResultRequest$3(RestAsyncQueryManagementAction.java:147)
        at org.opensearch.sql.datasources.utils.Scheduler.lambda$withCurrentContext$0(Scheduler.java:30)
        at org.opensearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:858)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)

@penghuo penghuo added the Flint label Nov 29, 2023
@penghuo penghuo assigned penghuo and unassigned YANG-DB Jan 29, 2024
@penghuo penghuo mentioned this issue Jan 31, 2024
Merged
6 tasks
@penghuo
Copy link
Collaborator Author

penghuo commented Feb 2, 2024

Fixed in 2.12.

@penghuo penghuo closed this as completed Feb 2, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@penghuo penghuo

Labels
bug Something isn't working Flint v2.12.0 Issues targeting release v2.12.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@penghuo @YANG-DB @seankao-az