Add APIs to read stats of segment HTTP requests

[Sergears]

Context
For the integration with the open source client-side CDN load balancer, we need to observe and modify HTTP requsts. The key role of the load balancer is to rewrite the URLs of the segment and manifest requests, for which the existing RequestModifier should work fine. However, we also need the additional capabilities described below.

What is needed

1. For successful segment requests we need to read:
       - the timestamp when the HTTP request was made
       - the timestamp when the HTTP response was received
       - the byte size of the received payload
       - (optional) the time to first byte to measure latency
   The currently existing SegmentResponseModifier does not have access to this info as it only receives an internally fabricated chunk object. For example, in shaka player, this info is accessible with ResponseFilter that gives access the the whole response object.
2. For timeouts and errors:
       - we need to know which URL resulted in the timeout / error
   This could be possible if there is some kind of event listener for onError and onTimeout that gives access to the event object containing the URL. Or, alternatively, an onRetry listener triggered when the player retries the segment request. 

Alternative solution
All of these can be probably be implemented by heavily intervening and extending the low level XHRloader and FetchLoader functions, and wrapping the methods httpRequest.onload, httpRequest.onerror, httpRequest.ontimeout of the passed httpRequest. However this is a brute force solution. Instead, we would want to have the official methods to achieve the needed functionality. They would then be potentially useful for other needs beyond the CDN load balancer. 

P.S. We would eventually also want to integrate with the content steering functionality of dash.js, for which additional APIs may be needed, but this topic deserves a separate issue.

[bbert]

Hi @Sergears and all,

I think you can achieve all you need using current available API, by using the RequestModifier and by getting metrics from the player (player.getDashMetrics().getCurrentHttpRequest() or player.getDashMetrics().getHttpRequests()) or by listening new metrics being generated (player.on('metricAdded')).

However, I think this is the opportunity to define like a common "standardized" API to enable any applications to:

• catch the requests in order to:
  • modify the url
  • add/remove headers
  • ...
• catch the responses (before being processed by the player) in order to:
  • get request metrics (length, ttfb, duration etc)
  • modify/patch the received data
  • ...

We could define an API as following, for requests event:

player.onRequestSend(e: RequestSendEvent)

type RequestSendEvent = {
  // HTTP request related properties
  url: string,
  method: string,
  range?: string,
  headers?: { [key: string]: string },
  credentials?: boolean,
  timeout?: number,

  // Media request related properties that could be useful
  requestType: RequestType, // "manifest", "initializationSegment", "mediaSegment" etc
  mediaType?: MediaType, // "video", "audio" etc
  bitrate?: number,
  segmentDuration?: number
  ...
}

and for responses event:

player.onRequestReceived(e: ResponseReceivedEvent)

type ResponseReceivedEvent = {
  // HTTP request related properties
  url: string,
  redirectedUrl?: string,
  responseCode: number,
  method: string,
  range?: string,
  headers?: { [key: string]: string },
  credentials?: boolean,

  // Media request related properties
  requestType: RequestType,
  mediaType?: MediaType,
  bitrate?: number,
  segmentDuration?: number

  // Response data
  length: number,
  data: any,

  // HTTP metrics
  tcpid?: number,
  tSend: number,
  tFirstByte: number,
  tEnd: number,
  traces?: RequestTrace[]
}

The properties lists are not exhaustive and can be completed, and the objective would be to design an API that could fit to every player type. But starting with dash.js is a good option to prove the concept.

This same API could also be used for license requests.

Also, to go further, beyond CDN load balancer topic, we can consider any other potential uses case for which one would like to execute the requests instead of the player itself.
For this purpose we could extend this API in order to:

• enable the application to indicate to the player to avoid executing the request
• enable the application to signal to the player any progress, error etc and end of request loading end

This would avoid to implement force solutions redefining some parts of each player.
For example:

type RequestSendEvent = {
  ...
  // Request processing related properties
  cte?: boolean, // support for chunked transfer encoding
  catched: boolean, // if set to true by the application, then the player shall not execute the request 
  onprogress?: (e: ProgressEvent) => void, // callback for the player 
  onload?: (e: ProgressEvent) => void,
  onloadend?: (e: ProgressEvent) => void,
  onabort?: (e: ProgressEvent) => void,
  ontimeout?: (e: ProgressEvent) => void,
  onerror?: (e: ErrorEvent) => void,
}

[Sergears]

@bbert thanks for the response, I would definitely support the idea of a standard API that could fit every player, and especially the most powerful version of it where the application could execute the requests instead of the player itself.

[dsilhavy]

Thank you for starting the discussion. I summarized below what we have in place and what we might want to add based on your feedback.

What we have

Modifying segment/manifest requests

The RequestModifier can be used to define synchronous and asynchronous functions to manipulate request urls, headers and the request itself. See https://reference.dashif.org/dash.js/nightly/samples/advanced/extend.html for an example.

Modifying segment/manifest responses

The SegmentResponseModifier can be used to manipulate segments/chunks before they are appended to the MSE SourceBuffers. See https://reference.dashif.org/dash.js/nightly/samples/advanced/modify-segment-response.html for an example.

Modifying license requests

A license request filter can be defined as shown here: https://reference.dashif.org/dash.js/nightly/samples/drm/license-wrapping.html

Modifying license responses

A license response filter can be defined as shown here: https://reference.dashif.org/dash.js/nightly/samples/drm/license-wrapping.html

Metadata Events for the application

dash.js dispatches multiple events related to downloading segments and manifests, see http://cdn.dashjs.org/latest/jsdoc/MediaPlayerEvents.html for a complete list. As an example, the FRAGMENT_LOADING_COMPLETED event contains information about the starttime of a request, the endtime and the loaded bytes. Moreover, there is an error field in case of a download error.

{
    "action": "download",
    "mediaType": "video",
    "type": "InitializationSegment",
    "range": null,
    "url": "https://dash.akamaized.net/akamai/bbb_30fps/bbb_30fps_3840x2160_12000k/bbb_30fps_3840x2160_12000k_0.m4v",
    "serviceLocation": "https://dash.akamaized.net/akamai/bbb_30fps/",
    "requestStartDate": "2023-04-05T06:15:15.524Z",
    "firstByteDate": "2023-04-05T06:15:15.524Z",
    "requestEndDate": "2023-04-05T06:15:15.548Z",
    "quality": 9,
    "index": null,
    "availabilityStartTime": "2023-04-05T06:15:15.501Z",
    "availabilityEndTime": null,
    "wallStartTime": null,
    "bytesLoaded": 707,
    "bytesTotal": 707,
    "delayLoadingTime": null,
    "responseType": "arraybuffer",
    "representationId": "bbb_30fps_3840x2160_12000k",
    "fileLoaderType": "xhr_loader"
}

What is missing at this point is an event to signal a request timeout.

Potential improvements

From my point of view the described functionality to modify requests and responses can be unified to make it easier for applications to modify request and response data.

What I would suggest is the following (probably similar to what you mean in your description above): We allow the app to define requestFilter and responseFilter. Similar to what we are doing for license requests today. Filters need to return a promise. Consequently, we would get rid of the RequestModifier and SegmentResponseModifier as well as the licenseRequestFilter and licenseResponseFilter. It would be up to the application to choose manipulate the available data based on the type of the request or the response, e.g. "type": "InitializationSegment" or type: "MediaSegment".

(What we could consider is keeping the licenseRequestFilter and licenseResponseFilter as separate filters.)

In general, the input for the filters would look like @bbert explained above.

Application based request execution

Let's discuss this in a different issue.

[Sergears]

Thank you @dsilhavy for the proposal. For the load balancer integration, the solution with requestFilter and responseFilter returning promises would work well, and in addition we still need to handle timeouts with something like a onTimeout event.

Regarding the removal of some of the existing APIs, possible retro-compatibility issues need to be looked at.

The last SVTA meeting there was a discussion about us submitting the PR for this. If we go this way, we first need the specs for the new APIs to be agreed upon by the dash.js team.

[dsilhavy]

@Sergears The requestFilter and responseFilter functionality and the event to dispatch information about onTimeout events is something I can address in one of the next sprints. The questions is what would be the timeline on your end? I will probably not be able to provide this before end of May.

Otherwise, I am obviously happy to review your PR if you want to implement this. Although these are breaking changes, we could do it as part of 4.8.0, no need to wait for 5.0.0. We need to adjust the samples in the sample section accordingly and announce these changes in the release notes.

[Sergears]

@dsilhavy I appreciate your feedback, and nice to see that we have converged on the functionality to implement. The discussed ambitious timing to implement this for MHV conference is probably not feasible as we would not be able to start working on this until May, either. We can discuss the timing later, depending on next sprint priorities.

[littlespex]

Thinking about this from the multi-player perspective, it would be great to add something like "Common Media Request" and "Common Media Response" interfaces/types to the Common Media Library. I'm wondering if it would be possible to reuse existing, standardized, APIs for this, or at least borrow the terminology. For example, requestType, mediaType, bitrate and segmentDuration are all defined in CMCD. For the response, most of the timing properties are laid out in the Resource Timing performance API. Something like:

import { CMCD } from '@svta.org/common-media-library';

interface CommonMediaRequest {
  url: string;
  method?: string;
  headers?: Record<string, string>;
  credentials?: RequestCredentials;
  cmcd?: CMCD;
}

// ≈ Partial<PerformanceResourceTiming>
interface ResourceTiming {
  // Returns the timestamp for the time a resource fetch started.
  startTime: number;

  // Returns a timestamp that is the difference between the responseEnd and the startTime properties.
  duration: number;

  // A number representing the size (in octets) of the fetched resource. The size includes the response header fields plus the response payload body.
  transferSize: number;
  
  // A timestamp immediately after the browser receives the first byte of the response from the server.
  responseStart?: number;

  // A timestamp immediately after the browser receives the last byte of the resource or immediately before the transport connection is closed, whichever comes first.
  responseEnd?: number;
}

interface CommonMediaResponse {
  url: string;
  redirected: boolean;
  status: number;
  data: any;
  request: CommonRequest;
  resourceTiming: ResourceTiming;
}

As for timeouts, or other errors for that matter, could those just be another set of properties on the response?

interface CommonResponse {
  ...
  ok: boolean;
  error?: string; // could be 'timeout' or something similar
}

[bbert]

Good point @littlespex , I do agree with your proposal.
If no one has started yet working on it, I can initiate something.
My proposal:

1. create a new branch on Common Media Library repo to add theses Common Media Request/Response
2. create a new branch on dash.js that depends on this Common Media Library's branch and that relies on the Common Media Request/Response for adding request/response filters/plugins

If all agree, for point 1 how can we proceed @littlespex ? Pull request ?

[littlespex]

We are finalizing the contribution guidelines/workflow in the next few days, but for now make a branch and PR. We can also publish a dev/pre-release package.

[bbert]

Hi @littlespex, please let me know when workflow is ready for Common Media Library (for the moment we still can't fork the repo).
I have a 1st working version I can propose.

[littlespex]

@bbert I've added you as a maintainer. You should be able to check out the repo directly and branch from there.

[dsilhavy]

@littlespex @bbert I like the idea, I was also looking to implement the Resource Timing API as part of the ABR/Throughput estimation refactoring.

How are we dealing with player-specific attributes? There might be some attributes that are player-specific and should be part of either the request or the response object. Do you consider this to be a separate object or should there be some kind of generic object like auxilaryInformation included in CommonResponse or CommonRequest that can hold arbitrary values?

[bbert]

Hi @dsilhavy
The basic idea was to enable adding player-specific properties in CommonMediaRequest like proposed here: https://github.com/streaming-video-technology-alliance/common-media-library/blob/f9c2c05eee5ee3ec2eaef1133cf82efaa398b1b5/lib/src/request/CommonMediaRequest.ts#L54

It can of course be renamed and I guess we can consider adding this kind of property in the CommonMediaResponse also.

Maybe we should continue this discussion on svta repo (streaming-video-technology-alliance/common-media-library#18).

[dsilhavy]

This was addressed in #4299.

@bbert Can we close this issue?