Control http-request transactions of APM RUM agent manually

[mshustov]

Performance WG relies on the APM agent to capture traffic between the Kibana browser app and the Kibana server. During the APM transaction tree analysis, we noticed that some AJAX requests or static assets loading might be marked as a child of other AJAX requests.

@vigneshshanmugam explained this as the normal behavior of the RUM agent and suggested to use manual instrumentation instead of the automatic one.

if there are multiple API calls, in this example /baearch and /data, RUM agent creates a transaction with the first API call and groups them together since they happen without much delay.
The automatic transaction tries to capture most outgoing network requests within a single transaction, if we need to control this, then manual instrumenting is required.

@devcorpio suggested to disable automatic instrumentation for AJAX requests with disableInstrumentations config option.

disableInstrumentations: ['xmlhttprequest', 'fetch']

To have the correct transaction tree we need:

• to opt-out of automatic instrumentation for AJAX requests
• use manual instrumentation for the AJAX requests. @vigneshshanmugam @devcorpio Do you have any examples for this?

[elasticmachine]

Pinging @elastic/kibana-core (Team:Core)

[devcorpio]

Hi @mshustov,

In the part of the code where you perform the http requests you can do something like:

function fetchSomethingSomething(somethingId) {
const transaction = apm.startTransaction('fetchSomethingSomething', 'http-request')

const httpSpan = transaction.startSpan('GET(for instance) **yourendpoint**', 'external.http')
 
  return api.Fetch(
    `**yourendpoint**`
  ) .then((resp) => {
   
    httpSpan.end() // end span hare
    transaction.end() // end transaction here
  }).catch(err => {
      apm.captureError(err)
     throw err;
});
}

In this link there is another example

By the way, If a http request causes the browser to load a new html portion (which includes resources such as css, etc) and you want the agent to include timing information and spans about it in the transaction, you should create the transaction with the flag managed set to true and the agent will handle it automatically. We call this "custom managed transaction", more info here

Thanks,
Alberto

[lizozom]

If we did this, would we still be able to measure page-load and app-load times? Or would static assets be completely excluded?

[mshustov]

Or would static assets be completely excluded?

Assets loaded by js (for example, bundle chunks loaded by webpack) will be excluded. We will have to control page-load and app-load manually to keep the relationships.

[lizozom]

Actually, we already control the page-load transaction ourselves.
I have to admit that it's a bit odd that we have to disable the entire automatic instrumentation at this point.
Maybe we're trying to use apm in a way that we shouldn't be? Maybe the agent does need to be fine tuned in some way? 🤔

[vigneshshanmugam]

If the intention here is to separate API calls happening between client and server in to its own transactions, then the manual transaction is the only way to do it as the RUM agent tries to group all of them together if it can to a specific user activity.

Or would static assets be completely excluded?

We can capture all the network requests associated with the transaction even when we are doing the manual instrumentation. The instrumentation would give us the ability to start/end transaction at arbitrary intervals but also provide the option to capture the underlying network requests.

const transaction = apm.startTransaction('custom managed', 'custom', { managed: true })

// later 
transaction.end()

Let us know if this works.

[vigneshshanmugam]

Had discussion with Liza and posting back from slack thread - https://elastic.slack.com/archives/C017DFNCV5H/p1649692286349239?thread_ts=1649167476.656889&cid=C017DFNCV5H

Are we looking for separating the request  transactions even on the client side? What does that gives us? Because most of the applications prefer to group everything as we can tie up everything the users are doing in a particular session/journey.
If this is a use case that we have missed, would like to solve it on the agent side.

Before we jump in to solutions, Let's understand the intention, because we might be overthinking on how transactions in the browser and Kibana server works. In the RUM agent, everything is tied to a specific user activity. Transactions is just a APM term that is used to group them as we can get away without building a dedicated UI specific to RUM. In the backend agents these are different entities.

There are multiple phases in the frontend

1. Users loads the web page and waits for all the static resources till page load is fired - This is grouped as page-load transaction till
2. Post the page-load, if there are no application switch (routing change in terms of kibana) - if there is a network activity based on user click events or API call then the agent creates transactions such as click or http-request.
3. When the user switches from one app to another (ex - /discover to /apm) - Grouped as route-change and all the underlying network activity will be added as well.

Lets take this example trace - https://kibana-ops-e2e-perf.kb.us-central1.gcp.cloud.es.io:9243/app/apm/services/kibana-frontend/transactions/view?kuery=c6dde9b70e26c84f&rangeFrom=now-7d%2Fd&rangeTo=now&environment=ENVIRONMENT_ALL&transactionName=security_login&transactionType=app-change&comparisonEnabled=true&comparisonType=week&latencyAggregationType=avg&transactionId=c6dde9b70e26c84f&traceId=2e092905071d280d3355258f2be2cb85&flyoutDetailTab=&waterfallItemId=

We have the grouping done at the app-change and the associated calls to two API's /api/licensing/info and /app/banners/info are added as children as they belong to a single user activity that triggered as a result of landing on the login page.

If the intention is to separate all the outgoing API calls as individual transactions, then we are using the RUM agent in a wrong way for the particular use-case.

TLDR: In the client, We need to consider everything as a User Acvitiy/Session instead of transactions or trace.

[lizozom]

@mshustov can we close this issue?
I think we can work with @vigneshshanmugam and the ops team to define what we want to measure better and find a way and build the benchmarks around the data we have.
What do you think?

[lizozom]

Closing. We discussed using an approach combining APM traces with #121992 to get the benchmarking we need.