Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Media Session API: Add artwork #162

Merged
merged 9 commits into from
Oct 5, 2021
20 changes: 20 additions & 0 deletions src/api/basics/01-single-video.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,26 @@ video-sources:
- src: https://storage.googleapis.com/kino-assets/single-video/video.mp4
type: video/mp4; codecs="avc1.640032,mp4a.40.2"
thumbnail: https://storage.googleapis.com/kino-assets/single-video/thumbnail.png
media-session-artwork:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we already have linter warnings so that we don't forget to add media session artwork to future .md files?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can see how that would be useful, but I don't believe I've ever linted front-matter blocks in markdown files. What tool would you recommend for the task?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding a markdown linter is a great idea. However, it feels a bit out of scope for this PR.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Noted. Do you mind filing a feature request to keep track of it?

- sizes: 96x96
src: https://storage.googleapis.com/kino-assets/single-video/artwork-96x96.png
type: image/png
- sizes: 128x128
src: https://storage.googleapis.com/kino-assets/single-video/artwork-128x128.png
type: image/png
- sizes: 192x192
src: https://storage.googleapis.com/kino-assets/single-video/artwork-192x192.png
type: image/png
- sizes: 256x256
src: https://storage.googleapis.com/kino-assets/single-video/artwork-256x256.png
type: image/png
- sizes: 384x384
src: https://storage.googleapis.com/kino-assets/single-video/artwork-384x384.png
type: image/png
- sizes: 512x512
src: https://storage.googleapis.com/kino-assets/single-video/artwork-512x512.png
type: image/png

---

## Introduction
Expand Down
19 changes: 19 additions & 0 deletions src/api/basics/02-multiple-sources.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,6 +13,25 @@ video-sources:
- src: https://storage.googleapis.com/kino-assets/multiple-sources/vp9.webm
type: video/webm
thumbnail: https://storage.googleapis.com/kino-assets/multiple-sources/thumbnail.png
media-session-artwork:
- sizes: 96x96
src: https://storage.googleapis.com/kino-assets/multiple-sources/artwork-96x96.png
type: image/png
- sizes: 128x128
src: https://storage.googleapis.com/kino-assets/multiple-sources/artwork-128x128.png
type: image/png
- sizes: 192x192
src: https://storage.googleapis.com/kino-assets/multiple-sources/artwork-192x192.png
type: image/png
- sizes: 256x256
src: https://storage.googleapis.com/kino-assets/multiple-sources/artwork-256x256.png
type: image/png
- sizes: 384x384
src: https://storage.googleapis.com/kino-assets/multiple-sources/artwork-384x384.png
type: image/png
- sizes: 512x512
src: https://storage.googleapis.com/kino-assets/multiple-sources/artwork-512x512.png
type: image/png
---

## Introduction
Expand Down
19 changes: 19 additions & 0 deletions src/api/basics/03-using-webvtt.md
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,25 @@ video-subtitles:
src: https://storage.googleapis.com/kino-assets/using-webvtt/cap-cz.vtt
srclang: cz
thumbnail: https://storage.googleapis.com/kino-assets/using-webvtt/thumbnail.png
media-session-artwork:
- sizes: 96x96
src: https://storage.googleapis.com/kino-assets/using-webvtt/artwork-96x96.png
type: image/png
- sizes: 128x128
src: https://storage.googleapis.com/kino-assets/using-webvtt/artwork-128x128.png
type: image/png
- sizes: 192x192
src: https://storage.googleapis.com/kino-assets/using-webvtt/artwork-192x192.png
type: image/png
- sizes: 256x256
src: https://storage.googleapis.com/kino-assets/using-webvtt/artwork-256x256.png
type: image/png
- sizes: 384x384
src: https://storage.googleapis.com/kino-assets/using-webvtt/artwork-384x384.png
type: image/png
- sizes: 512x512
src: https://storage.googleapis.com/kino-assets/using-webvtt/artwork-512x512.png
type: image/png
---

## Introduction
Expand Down
19 changes: 19 additions & 0 deletions src/api/streaming/04-streaming-basics.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,25 @@ video-subtitles:
src: https://storage.googleapis.com/kino-assets/streaming-basics/cap-cz.vtt
srclang: cz
thumbnail: https://storage.googleapis.com/kino-assets/streaming-basics/thumbnail.png
media-session-artwork:
- sizes: 96x96
src: https://storage.googleapis.com/kino-assets/streaming-basics/artwork-96x96.png
type: image/png
- sizes: 128x128
src: https://storage.googleapis.com/kino-assets/streaming-basics/artwork-128x128.png
type: image/png
- sizes: 192x192
src: https://storage.googleapis.com/kino-assets/streaming-basics/artwork-192x192.png
type: image/png
- sizes: 256x256
src: https://storage.googleapis.com/kino-assets/streaming-basics/artwork-256x256.png
type: image/png
- sizes: 384x384
src: https://storage.googleapis.com/kino-assets/streaming-basics/artwork-384x384.png
type: image/png
- sizes: 512x512
src: https://storage.googleapis.com/kino-assets/streaming-basics/artwork-512x512.png
type: image/png
---

## Introduction
Expand Down
19 changes: 19 additions & 0 deletions src/api/streaming/05-efficient-formats.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,25 @@ video-subtitles:
src: https://storage.googleapis.com/kino-assets/efficient-formats/cap-cz.vtt
srclang: cz
thumbnail: https://storage.googleapis.com/kino-assets/efficient-formats/thumbnail.png
media-session-artwork:
- sizes: 96x96
src: https://storage.googleapis.com/kino-assets/efficient-formats/artwork-96x96.png
type: image/png
- sizes: 128x128
src: https://storage.googleapis.com/kino-assets/efficient-formats/artwork-128x128.png
type: image/png
- sizes: 192x192
src: https://storage.googleapis.com/kino-assets/efficient-formats/artwork-192x192.png
type: image/png
- sizes: 256x256
src: https://storage.googleapis.com/kino-assets/efficient-formats/artwork-256x256.png
type: image/png
- sizes: 384x384
src: https://storage.googleapis.com/kino-assets/efficient-formats/artwork-384x384.png
type: image/png
- sizes: 512x512
src: https://storage.googleapis.com/kino-assets/efficient-formats/artwork-512x512.png
type: image/png
---

## Introduction
Expand Down
19 changes: 19 additions & 0 deletions src/api/streaming/06-adaptive-streaming.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,25 @@ video-subtitles:
src: https://storage.googleapis.com/kino-assets/adaptive-streaming/cap-cz.vtt
srclang: cz
thumbnail: https://storage.googleapis.com/kino-assets/adaptive-streaming/thumbnail.png
media-session-artwork:
- sizes: 96x96
src: https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-96x96.png
type: image/png
- sizes: 128x128
src: https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-128x128.png
type: image/png
- sizes: 192x192
src: https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-192x192.png
type: image/png
- sizes: 256x256
src: https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-256x256.png
type: image/png
- sizes: 384x384
src: https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-384x384.png
type: image/png
- sizes: 512x512
src: https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-512x512.png
type: image/png
---

## Introduction
Expand Down
36 changes: 36 additions & 0 deletions src/js/sw/cache.js
Original file line number Diff line number Diff line change
Expand Up @@ -38,9 +38,45 @@ export default [
'/index.html',
'/styles.css',
'https://storage.googleapis.com/kino-assets/single-video/thumbnail.png',
'https://storage.googleapis.com/kino-assets/single-video/artwork-96x96.png',
'https://storage.googleapis.com/kino-assets/single-video/artwork-128x128.png',
'https://storage.googleapis.com/kino-assets/single-video/artwork-192x192.png',
'https://storage.googleapis.com/kino-assets/single-video/artwork-256x256.png',
'https://storage.googleapis.com/kino-assets/single-video/artwork-384x384.png',
'https://storage.googleapis.com/kino-assets/single-video/artwork-512x512.png',
'https://storage.googleapis.com/kino-assets/multiple-sources/thumbnail.png',
'https://storage.googleapis.com/kino-assets/multiple-sources/artwork-96x96.png',
'https://storage.googleapis.com/kino-assets/multiple-sources/artwork-128x128.png',
'https://storage.googleapis.com/kino-assets/multiple-sources/artwork-192x192.png',
'https://storage.googleapis.com/kino-assets/multiple-sources/artwork-256x256.png',
'https://storage.googleapis.com/kino-assets/multiple-sources/artwork-384x384.png',
'https://storage.googleapis.com/kino-assets/multiple-sources/artwork-512x512.png',
'https://storage.googleapis.com/kino-assets/using-webvtt/thumbnail.png',
'https://storage.googleapis.com/kino-assets/using-webvtt/artwork-96x96.png',
'https://storage.googleapis.com/kino-assets/using-webvtt/artwork-128x128.png',
'https://storage.googleapis.com/kino-assets/using-webvtt/artwork-192x192.png',
'https://storage.googleapis.com/kino-assets/using-webvtt/artwork-256x256.png',
'https://storage.googleapis.com/kino-assets/using-webvtt/artwork-384x384.png',
'https://storage.googleapis.com/kino-assets/using-webvtt/artwork-512x512.png',
'https://storage.googleapis.com/kino-assets/streaming-basics/thumbnail.png',
'https://storage.googleapis.com/kino-assets/streaming-basics/artwork-96x96.png',
'https://storage.googleapis.com/kino-assets/streaming-basics/artwork-128x128.png',
'https://storage.googleapis.com/kino-assets/streaming-basics/artwork-192x192.png',
'https://storage.googleapis.com/kino-assets/streaming-basics/artwork-256x256.png',
'https://storage.googleapis.com/kino-assets/streaming-basics/artwork-384x384.png',
'https://storage.googleapis.com/kino-assets/streaming-basics/artwork-512x512.png',
'https://storage.googleapis.com/kino-assets/efficient-formats/thumbnail.png',
'https://storage.googleapis.com/kino-assets/efficient-formats/artwork-96x96.png',
'https://storage.googleapis.com/kino-assets/efficient-formats/artwork-128x128.png',
'https://storage.googleapis.com/kino-assets/efficient-formats/artwork-192x192.png',
'https://storage.googleapis.com/kino-assets/efficient-formats/artwork-256x256.png',
'https://storage.googleapis.com/kino-assets/efficient-formats/artwork-384x384.png',
'https://storage.googleapis.com/kino-assets/efficient-formats/artwork-512x512.png',
'https://storage.googleapis.com/kino-assets/adaptive-streaming/thumbnail.png',
'https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-96x96.png',
'https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-128x128.png',
'https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-192x192.png',
'https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-256x256.png',
'https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-384x384.png',
'https://storage.googleapis.com/kino-assets/adaptive-streaming/artwork-512x512.png',
];
3 changes: 3 additions & 0 deletions src/js/utils/generateCache.js
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,9 @@ export default function generateCache() {
} else if (Object.prototype.toString.call(video.thumbnail) === '[object String]') {
assetsToCache.push(video.thumbnail);
}
if (Array.isArray(video['media-session-artwork'])) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this needed to cache media session artwork images if the Media Session API is not supported?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@beaufortfrancois I was wondering if we even want to cache those files. It's quite a lot of data – 400 kB currently – and the benefit of the artwork being available in the offline mode is marginal IMO. We can always fall back to the default artwork in those cases.

And if you're asking whether we should exclude browsers with no Media Session API support, then I'd say perhaps. With Safari 15 on iOS out the door, there is currently no mainstream evergreen browser that would not support it.

image

What is your opinion on the caching?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What is your opinion on the caching?

Is it possible to cache them only when media session metadata are set?
In other words, only when the media starts playing.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think adding logic to the SW to cache this artwork when it's played, or dowloaded for offline playback, would be a better user experience instead of caching all the artwork during initialization.

Copy link
Collaborator Author

@dero dero Oct 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good idea, guys, thank you. I've implemented the changes in d7ac820

The artwork is now cached when a video is played or downloaded.

@beaufortfrancois @derekherman Can you please take another look?

Copy link
Member

@beaufortfrancois beaufortfrancois Oct 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

QQ: Is this caching assets every time video is played and/or downloaded or only once?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@beaufortfrancois Aha, good point, I did not realize cache.addAll would overwrite the cache entries. Updated the saveToCache implementation in 64ca172. Could you please take a look?

I believe it is sufficient to match the URL, because we're versioning cache. LMK if you think this needs to be more robust. Thanks!

video['media-session-artwork'].forEach((artworkObject) => assetsToCache.push(artworkObject.src));
}
});

const data = `/*
Expand Down
2 changes: 1 addition & 1 deletion src/js/web-components/video-player/VideoPlayer.js
Original file line number Diff line number Diff line change
Expand Up @@ -159,7 +159,7 @@ export default class extends HTMLElement {
title: this.internal.videoData.title || '',
artist: this.internal.videoData.artist || '',
album: this.internal.videoData.album || '',
artwork: MEDIA_SESSION_DEFAULT_ARTWORK,
artwork: this.internal.videoData['media-session-artwork'] || MEDIA_SESSION_DEFAULT_ARTWORK,
});

/**
Expand Down