Component: Tracing

[StachuDotNet]

This issue is to organize work not in-progress, related to Tracing.

"Tracing" in Darklang is a pluggable, continuous thing that you can hook into and leverage. It supports debuggability, refactoring, and more. Our goal is to keep this a holistic part of Dark.

Work here isn't quite accessible at the moment. There are large refactors pending in this space, and tracing hasn't been reintegrated into the new editing environment yet.

core

• generally refactor, given migration from "just cloud" to multi-runtime (i.e. support distributed / cross-runtime tracing)
• add trace id to bwd headers (The traceID shows up to users as traces. It would be helpful if their users could report traceID, so maybe add x-darklang-trace-id)
• Users should be able to delete a trace (see old Allow deleting a trace #2417)
• Make traces work well with loops (see traces don't work well with loops #3480)
• Persist elapsed time with requests/traces (see Persist elapsed time with requests/traces, and display in Editor #2741)
• revisit claim: "traces don't work well in functions" (see traces don't work well in functions #3484)
• revisit: "Add traces to package manager functions" (see Add traces to package manager functions #2707)
• http handler responses should (optionally?) include a header noting the traceId
• Dark needs a mechanism & UI for profiling code
• consider distributed tracing (see https://microservices.io/patterns/observability/distributed-tracing.html) concepts - ask Stachu for thoughts here
• Ensure telemetry events from ProdExec make it to tracing (see Ensure telemetry events from ProdExec make it to tracing #3459)
• traces permanently store all functions called while they're active (see traces permanently store all functions called while they're active #3482)
• avail functions like Tracing.log that loop into our (dist) tracing system, where users can explicitly trace 'extra' data

mostly relevant to editor

• You should be able to copy parts of traces by selecting it and pressing Ctrl-C (or whatever on your OS).
• copy trace as cURL
• animate the arrival of a new trace
• show both absolute and relative timestamps when viewing a trace (see old Show both relative and absolute timestamps in traces #2464)
• ensure they're nice and readable (see Make traces more readable #3080)
• display trace runtime info (see Persist elapsed time with requests/traces, and display in Editor #2741)

---

personal notes to process later - I'll know what these mean

• trace: start/enter CLI
• trace: specific input
  • fn call
  • http
• connect traces across wire by including traceId

---

More notes to process:

# Dark Tracing Notes and Plans

## Relevant Stuff
- code
  - LibExe/AnalysisTypes.fs
  - LibExe/Interpreter.fs
  - LibExe/RuntimeTypes.fs
  - LibCloud/TraceCloudStorage.fs
  - LibCloud/Tracing.fs
  - Tests/StorageTraces.Tests.fs
  - module Tests.StorageTraces
  - Server.fs (BwdDanger and Bwd)
  - DangerExecution.fs
  - BuiltinDarkInternals/F404s.fs
  - CronChecker
- the `traces_v0` postgres table
    ```
    CREATE TABLE IF NOT EXISTS
    traces_v0
    ( id UUID PRIMARY KEY DEFAULT gen_random_uuid()
    , canvas_id UUID NOT NULL
    -- the handler's (or for a function's default trace, the function's) TLID (used to
    -- store the trace data in Cloud Storage)
    , root_tlid BIGINT NOT NULL
    , trace_id UUID NOT NULL
    , callgraph_tlids BIGINT[] NOT NULL -- functions called during the trace
    );
    ```

is dotnet-trace relevant at all?

## Plans
- [ ] fill this all out

## Questions & TODOs:
- [ ] Q: how do I get this accessible to Dark users to play with?
  - we need some analysis endpoints I guess
  - and a REPL...

additional notes that need processing

• traces across runtimes
• run a cli and get a trace id back
• see that trace data in an endpoint of dark-editor
• dark-editor is a runtime
• traceInternals: bool
  • i.e. package lookup
• include processor info
• research distributed tracing
• probably worth studying other microservice and general architectural patterns
• lay out the idea of an LSP-managed context of editing,
• and insert lil' trace things at relevant points in the editor
• (i.e. play button)