Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Traces do not get correctly exported to tempo when mysql-router is related to grafana-agent #195

Closed
shayancanonical opened this issue Oct 9, 2024 · 3 comments
Closed

Traces do not get correctly exported to tempo when mysql-router is related to grafana-agent #195

shayancanonical opened this issue Oct 9, 2024 · 3 comments
Labels
Status: Triage Type: Bug

Comments

@shayancanonical
Copy link

Bug Description

Since mysql-router is a subordinate charm, we first relate grafana-agent with the principal charm (on the juju-info interface). Then, we relate grafana-agent:cos-agent mysql-router. When we do so, I can see loki logs trickling through to grafana/loki. However, I am unable to see traces from mysql-router showing up in tempo.

Upon further inspection, I noticed an error trace in the debug-logs, which I have included below

To Reproduce

  1. checkout branch feature/tempo_ha in mysql-router (https://github.com/canonical/mysql-router-operator/tree/feature/tempo_ha)
  2. charmcraft pack
  3. juju add-model database (lxd controller)
  4. juju deploy mysql --channel 8.0/edge
  5. juju deploy ./<packed_mysql_router_from_step_2>
  6. juju deploy mysql-test-app
  7. juju deploy grafana-agent
  8. juju relate mysql mysql-router
  9. juju relate mysql-router mysql-test-app
  10. juju relate mysql-test-app:juju-info grafana-agent
  11. juju relate mysql-router:cos-agent grafana-agent
  12. <consume grafana, loki, prometheus and tracing offers from the cos model>
  13. juju relate grafana-agent grafana
  14. juju relate grafana-agent prometheus
  15. juju relate grafana-agent loki
  16. juju relate grafana-agent tempo

Environment

Juju: 3.5.4
LXD: 6.1

Model     Controller  Cloud/Region         Version  SLA          Timestamp
database  lxd-3-5-4   localhost/localhost  3.5.4    unsupported  19:22:17Z

SAAS        Status  Store       URL
grafana     active  uk8s-3-5-4  admin/cos.grafana
loki        active  uk8s-3-5-4  admin/cos.loki
prometheus  active  uk8s-3-5-4  admin/cos.prometheus
tempo       active  uk8s-3-5-4  admin/cos.tempo

App             Version          Status  Scale  Charm           Channel        Rev  Exposed  Message
grafana-agent                    active      1  grafana-agent   latest/stable  216  no       
mysql           8.0.37-0ubun...  active      1  mysql           8.0/edge       301  no       
mysql-router    8.0.37-0ubun...  active      1  mysql-router                     0  no       
mysql-test-app  0.0.2            active      1  mysql-test-app  latest/stable   51  no       Last written value=213191

Unit                Workload  Agent  Machine  Public address  Ports           Message
mysql-test-app/0*   active    idle   1        10.205.193.72                   Last written value=213191
  grafana-agent/0*  active    idle            10.205.193.72                   
  mysql-router/0*   active    idle            10.205.193.72                   
mysql/0*            active    idle   0        10.205.193.151  3306,33060/tcp  Primary

Machine  State    Address         Inst id        Base          AZ  Message
0        started  10.205.193.151  juju-4a96ba-0  [email protected]      Running
1        started  10.205.193.72   juju-4a96ba-1  [email protected]      Running

Integration provider                           Requirer                                       Interface                    Type         Message
grafana-agent:grafana-dashboards-provider      grafana:grafana-dashboard                      grafana_dashboard            regular      
grafana-agent:peers                            grafana-agent:peers                            grafana_agent_replica        peer         
loki:logging                                   grafana-agent:logging-consumer                 loki_push_api                regular      
mysql-router:cos                               mysql-router:cos                               cos                          peer         
mysql-router:cos-agent                         grafana-agent:cos-agent                        cos_agent                    subordinate  
mysql-router:database                          mysql-test-app:database                        mysql_client                 subordinate  
mysql-router:deprecated-shared-db-credentials  mysql-router:deprecated-shared-db-credentials  _deprecated_shared_db_peers  peer         
mysql-router:tls                               mysql-router:tls                               tls                          peer         
mysql-router:upgrade-version-a                 mysql-router:upgrade-version-a                 upgrade                      peer         
mysql-test-app:application-peers               mysql-test-app:application-peers               application-peers            peer         
mysql-test-app:juju-info                       grafana-agent:juju-info                        juju-info                    subordinate  
mysql:database                                 mysql-router:backend-database                  mysql_client                 regular      
mysql:database-peers                           mysql:database-peers                           mysql_peers                  peer         
mysql:restart                                  mysql:restart                                  rolling_op                   peer         
mysql:upgrade                                  mysql:upgrade                                  upgrade                      peer         
prometheus:receive-remote-write                grafana-agent:send-remote-write                prometheus_remote_write      regular      
tempo:tracing                                  grafana-agent:tracing                          tracing                      regular      

Storage Unit  Storage ID  Type        Pool    Mountpoint                      Size     Status    Message
mysql/0       database/0  filesystem  rootfs  /var/snap/charmed-mysql/common  1.8 TiB  attached

Relevant log output

unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log ops 2.14.0 up and running.
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log Setting up span exporter to endpoint: http://10.0.0.44:4318/v1/traces
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log Starting root trace with id='a1fe6a71459db2c991922d4c84bc0341'.                                          
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log Emitting Juju event update_status.                                                                                                              
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log Starting new HTTP connection (1): 10.0.0.44:4318
unit-grafana-agent-0: 19:04:26 DEBUG unit.grafana-agent/0.juju-log http://10.0.0.44:4318 "POST /v1/traces HTTP/11" 200 2
unit-mysql-router-0: 19:04:27 DEBUG unit.mysql-router/0.juju-log ops 2.9.0 up and running.                                                                                                                         
unit-mysql-router-0: 19:04:27 DEBUG unit.mysql-router/0.juju-log data={}                                                                                                                                           
unit-mysql-router-0: 19:04:27 DEBUG unit.mysql-router/0.juju-log failed to validate databag: {'egress-subnets': '10.205.193.72/32', 'ingress-address': '10.205.193.72', 'private-address': '10.205.193.72'}        
Traceback (most recent call last):                                                                                                                                                                                 
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/lib/charms/grafana_agent/v0/cos_agent.py", line 369, in load                                                                                    
    return cls.parse_raw(json.dumps(data))  # type: ignore                                                                                                                                                         
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/pydantic/main.py", line 549, in parse_raw                                                                                                              
    return cls.parse_obj(obj)                                                                                                                                                                                      
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/pydantic/main.py", line 526, in parse_obj                                                                                                              
    return cls(**obj)                                                                                    
  File "/var/lib/juju/agents/unit-mysql-router-0/charm/venv/pydantic/main.py", line 341, in __init__                                                                                                               
    raise validation_error                                                                               
pydantic.error_wrappers.ValidationError: 1 validation error for CosAgentRequirerUnitData                 
receivers                                                                                                
  field required (type=value_error.missing)                                                              
unit-mysql-router-0: 19:04:27 INFO unit.mysql-router/0.juju-log failed validating relation data for <ops.model.Relation cos-agent:16>                                                                  
unit-mysql-router-0: 19:04:27 DEBUG unit.mysql-router/0.juju-log Emitting Juju event update_status.

Additional context

No response
@shayancanonical
Copy link
Author

Confirmed that the issue exists with latest/edge of grafana-agent too:

unit-grafana-agent-1: 20:03:56 DEBUG unit.grafana-agent/1.juju-log tracing:46: failed to validate databag: {}
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/lib/charms/tempo_k8s/v2/tracing.py", line 209, in load
    return cls.parse_raw(json.dumps(data))  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 549, in parse_raw
    return cls.parse_obj(obj)
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 526, in parse_obj
    return cls(**obj)
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 341, in __init__
    raise validation_error
pydantic.error_wrappers.ValidationError: 1 validation error for TracingProviderAppData
receivers
  field required (type=value_error.missing)
unit-grafana-agent-1: 20:03:56 INFO unit.grafana-agent/1.juju-log tracing:46: failed validating relation data for <ops.model.Relation tracing:46>
unit-grafana-agent-1: 20:03:56 DEBUG unit.grafana-agent/1.juju-log tracing:46: Emitting Juju event tracing_relation_created.
unit-grafana-agent-1: 20:03:57 WARNING unit.grafana-agent/1.tracing-relation-joined /var/lib/juju/agents/unit-grafana-agent-1/charm/./src/charm.py:689: DeprecationWarning: Calling `ops.main.main()` is deprecated, call `ops.main()` instead
unit-grafana-agent-1: 20:03:57 WARNING unit.grafana-agent/1.tracing-relation-joined   main(GrafanaAgentMachineCharm)
unit-grafana-agent-1: 20:03:57 DEBUG unit.grafana-agent/1.juju-log tracing:46: ops 2.17.0 up and running.
unit-grafana-agent-1: 20:03:57 DEBUG unit.grafana-agent/1.juju-log tracing:46: failed to validate databag: {}
Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/lib/charms/tempo_k8s/v2/tracing.py", line 209, in load
    return cls.parse_raw(json.dumps(data))  # type: ignore
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 549, in parse_raw
    return cls.parse_obj(obj)
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 526, in parse_obj
    return cls(**obj)
  File "/var/lib/juju/agents/unit-grafana-agent-1/charm/venv/pydantic/main.py", line 341, in __init__
    raise validation_error
pydantic.error_wrappers.ValidationError: 1 validation error for TracingProviderAppData
receivers
  field required (type=value_error.missing)
unit-grafana-agent-1: 20:03:57 INFO unit.grafana-agent/1.juju-log tracing:46: failed validating relation data for <ops.model.Relation tracing:46>

This was referenced Oct 9, 2024
Merged
Merged
shayancanonical added a commit to canonical/mysql-router-operator that referenced this issue Oct 10, 2024
@shayancanonical
[DPE-5312] Pull tracing charm libs from tempo_coordinator_k8s instead (…
60ad054 
…#180)

## Issue
We are unable to relay traces through grafana-agent due to
canonical/grafana-agent-operator#195. However,
in the meantime, we would like to update the charm tracing libs as they
now reside in `tempo_coordinator_k8s`

## Solution
Update location of charm tracing libs
@mmkay
Copy link
Contributor

mmkay commented Oct 21, 2024 •
edited
Loading

Important to note that mysql-router is a subordinate charm. I just confirmed that in the same model, if you relate mysql charm to grafana-agent via cos-agent, mysql charm's traces go through so this isn't necessarily an issue with grafana-agent itself (but might be with cos-agent lib).

@mmkay
Copy link
Contributor

mmkay commented Oct 21, 2024

It looks like tracing endpoint wasn't returned correctly in the tempo_ha branch of mysql-router. I've passed the required fixes to @shayancanonical, after the fixes traces get exported to tempo:

Screenshot from 2024-10-21 13-02-42

@mmkay mmkay closed this as completed Oct 21, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Status: Triage Type: Bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mmkay @shayancanonical