refactor!: cleanup platform support

[Wykerd]

This brings a cleaner solution to supporting multiple platforms via PlatformShim.

Internal breaking changes:

• Node library is now ESModule by default. CommonJS is still provided via a bundle. This is transparent to users doing require('youtubei.js'), as configured in exports of the package.json, but will break users importing specific paths using require instead of import

Dependency changes:

• Moved to a new protobuf compiler, pbkit, which outputs code with zero runtime dependecies, making the code even more platform agnostic.
• linkedom is no longer required in browser runtimes already providing a DOMParser implementation.
• Added a more secure JS runtime for node platform using isolated-vm. This may be reverted if deemed unnecessary.

Deno support is fully realised with types and all via a new build script build:deno which outputs a transformed/patched version of the src/ directory to a new deno/ directory. Also deno.ts entrypoint is provided at the top level of the library.

The final step for Deno would be to publish to https://deno.land/x. For this I propose:

1. We write a GitHub action to build and push the deno/ directory to a new branch called deno (or similar) upon each release.
2. Configure this branch to be used on https://deno.land/x

I do not have any experience with GH actions, so help on this front would be appreciated.

[absidue]

Instead of making all of these significant changes in a single gigantic pull request, it might be better to do them as separate ones or at the very least split up the changes into different commits. I'm not Luan, so thankfully I don't have to review this pull request, but I know that I would hate to have to review 494 changed files in one go. As everything is in one massive commit there is no easy way to tell which code changes relate to which of the many listed changes in the pull request description.

TL;DR in order to keep Luan mentally sane, I suggest splitting this pull request up into many smaller ones or at the very least into smaller, easier to review commits.

[absidue]

Took a quick look and src/platform/web.ts seems to be a copy of the deno one, including saying it's a sever.

[Wykerd]

@absidue Can't split this up into multiple PRs. Every single file was changed to support ESM on node unfortunately. These are basically just changing the imports to include a .js extension.

Only files with real changes are:

• .gitignore: added additional build artifacts to list
• .eslintignore: added generated protobuf code to list
• README.md: replaced mentions of UniversalCache with Platform.shim.Cache
• package.json: added exports and removed old dependecies
• both files in scripts/: updated to output valid nodenext typescript modules
• Replaced platform dependent utils with Platform.shim in files:
  • src/core/OAuth.ts
  • src/core/Player.ts
  • src/core/Session.ts
  • src/core/Studio.ts
  • src/utils/HTTPClient.ts
  • src/utils/Utils.ts
  • src/utils/FormatUtils.ts
• Cleaned up test/main.test.ts
• Updated to nodenext modules in tsconfig.json
• Added types for platform shims to src/types/
• New protobuf generator outputs in src/proto/generated
• Implemented platform shims in src/platform/ directory for deno, node and browser.
• eval function for untrusted javascript in src/platform/jsruntime

I will make sure to break these up into separate commits in the future, but do hope these notes help @LuanRT during the review.

[absidue]

The reason I'm interested in this pull request is that theoretically according to your description, this should make it easier for FreeTube to use YouTube.js. As it's an Electron app and uses webpack, we've had to do a bit more customisation than just adding it to the package.json and importing it.

[Wykerd]

You should be able to provide support for any platform you want following this:
https://github.com/LuanRT/YouTube.js/blob/Wykerd-build-cleanup/src/platform/README.md

[absidue]

I'll list our issues with the previous setup and how we got around them.

1. webpack didn't like the default web/browser entry point as it was prebundled and the webpack didn't understand how it's imports worked. As the node version overwrites the fetch function, we couldn't use that either as it makes the devtools useless because any fetch call inside the FreeTube code, YouTube.js or other dependencies would get redirected through node, so we use the unbundled web entry point: FreeTube _scripts/webpack.renderer.config.js lines 140-143
2. webpack doesn't do optional imports so the canvas import inside linkedom was causing errors and as YouTube.js doesn't use linkedom in a way that needs the canvas dependency, we initially decided to force webpack to ignore the canvas import entirely. We later discovered that linkedom has a web and worker compatible build that doesn't have any of the canvas or node specific stuff in it: FreeTube _scripts/webpack.renderer.config.js lines 137-138
3. The UniversalCache implementation that YouTube.js has uses Reflect.get to import the fs and path modules, which is fine in a purely node environment but doesn't work in Electron, so we have our own cache implementation and pass that to InnerTube.create: FreeTube src/renderer/helpers/api/PlayerCache.js and FreeTube src/renderer/helpers/api/local.js line L34
   As we have filesystem access in Electron we prefer to have the cache on disk instead of IndexDB.

The most important part for us is number 1, that we'll still be able to import a compiled (ts -> js) but not bundled web build.
This refactor looks like it would resolve number 2 for us as it removes the need for linkedom in the web build, which should hopefully also help our bundle size go down. As for 3, as long as we are still able to pass in our own cache object, which looks to be the case according to the README changes, we'll be fine on that front too.

[absidue]

Switching to a more secure vm sounds like a good idea, although preferably one that isn't a native dependency.
Adding a native dependency means every user of YouTube.js either has to go through the hassle of setting up a compiler on their machine (that of course only works if the dependency doesn't fail to build) or tell their package manager to ignore optional dependencies every time they interact with it. If your project uses electron-builder, like FreeTube, it's entirely impossible to avoid having to build that isolated-vm dependency. (I was going to try these changes to see if I could get them working with FreeTube, but seeing as that native dependency is failing to build on my machine, I'll wait until there has been more discussion on whether that dependency is really necessary.)

[LuanRT]

Switching to a more secure vm sounds like a good idea, although preferably one that isn't a native dependency. Adding a native dependency means every user of YouTube.js either has to go through the hassle of setting up a compiler on their machine or tell their package manager to ignore optional dependencies every time they interact with it. If your project uses electron-builder, like FreeTube, it's entirely impossible to avoid having to build that isolated-vm dependency. (I was going to try these changes to see if I could get them working with FreeTube, but seeing as that native dependency is failing to build on my machine, I'll wait until there has been more discussion on whether that dependency is really necessary.)

I haven't reviewed this PR completely yet but I'm not sure if we'll be needing that additional VM. Jinter is pretty much bare bones and only supports what we really need (that being switch statements, simple logical operations, variables and functions). Although I do plan to make it more complete, I don't think security is a problem as there's no way the code being interpreted could leave the interpreter.

I must admit, however, that isolated-vm is a great fallback in case YouTube changes the nsig/sig code to something more complex. So instead of completely removing it, I suggest we find an alternative if this one is causing issues.

[absidue]

vm2 seems to be quite popular and builds on top of node's built-in vm module. https://www.npmjs.com/package/vm2
As with isolate-vm Jinter will still be needed for the web and deno environments.

[LuanRT]

So I managed to build it, did npm pack and tried importing the library from an ES module and it works, but unfortunately there's no code completion:

This PR:

Main branch:

---

Also, TypeScript projects with the default config will fail to import the library (unless you find the correct path manually):

I understand that there are a few breaking changes, but could these be fixed or are they merely a side effect of the way things are being exported? My concern is that it creates some inconsistency and confusion, especially as we have import .. from 'youtubei.js'; in most of the documentation & examples.

[LuanRT]

vm2 seems to be quite popular and builds on top of node's built-in vm module. https://www.npmjs.com/package/vm2

@absidue
Seems solid.

Edit:
Could also use JS-Interpreter as it works similarly to Jinter, but unfortunately it seems that it cannot run YouTube's nsig JavaScript due to inaccurate implementation of some nodes.

[Wykerd]

I understand that there are a few breaking changes, but could these be fixed or are they merely a side effect of the way things are being exported? My concern is that it creates some inconsistency and confusion, especially as we have import .. from 'youtubei.js'; in most of the documentation & examples.

This was an issue within the package.json not specifying the location of the new types. I have fixed this in the last commit.

[absidue]

Classes like ClientType, Text and EmojiRun can't be imported anymore, as the package.json now has an exports field and they aren't listed in there or exported anywhere else.

[absidue]

It doesn't look like bundle/browser.js is listed anywhere in the package.json anymore, exports.default points to the unbundled web one, so adding that back would definitely be a good idea.

For FreeTube we would want to use the file that exports.default currently points to, so having a way to access that directly would be great, as there currently doesn't seem to be a way to access it unless the automatic selection picks it.
My guess is that what webpack is currently doing is checking for a browser one first, as it doesn't find it, it picks the node one as electron is a mashup of chromium and node, so it never even considers the default one, as that's the lowest priority.

[absidue]

Thanks for all your changes!!! (hopefully I wasn't too annoying 🙈)
Using youtubei.js/web and a few changes to where we import the classes from, it works. It even shaved 0.35MB off FreeTube's javascript bundle!

[absidue]

Uhh okay so the bundle worked but the toDash function is broken. XMLSerializer is probably a better choice than DOMParser for the web build.

[absidue]

Another breaking change that should be added to the changelog is that various functions now return Promises like Format.decipher and toDash().

[Wykerd]

I can probably revert the format.decipher returning a promise since I removed isolated-vm. Will fix the toDash now.

[absidue]

toDash seems to be broken in a different way now, it doesn't throw and error but it now returns [object XMLDocument]. By the looks of it you added the serializeDOM functions but don't make use of them.

[Wykerd]

Oops! Fixed now.

[LuanRT]

Awesome, just tested again and everything is working as expected. Thanks @Wykerd!

And regarding the deno module, this workflow should work:

name: deno-build

on:
  push:
    branches: [main]
 
jobs:
    deploy:
        name: Build YouTube.js for deno
        runs-on: ubuntu-latest

        steps:
            - uses: actions/checkout@v2
            - uses: actions/setup-node@v1
              with:
                  node-version: "16.x"
                  registry-url: "https://registry.npmjs.org"

            - name: npm install and run build:deno
              run: |
                  npm install
                  npm run build:deno
            - name: Move files
              run: |
                  mkdir build
                  mv deno build/deno
                  mv deno.ts build/deno.ts
                  cp {LICENSE,README.md} build
            - name: Push to the deno branch 
              uses: s0/git-publish-subdir-action@develop
              env:
                  REPO: self
                  BRANCH: deno
                  FOLDER: ./build
                  GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
                  SKIP_EMPTY_COMMITS: true
                  MESSAGE: "{msg} ({sha})"

There's one thing I can't seem to figure out though, how do you get https://deno.land/x to use the deno branch? Pardon me if I'm missing something obvious.

[Wykerd]

I am not sure, but I know this something yargs does.
https://github.com/yargs/yargs

[Wykerd]

Seems like you need to create a release tag alongside the normal release which you can then tell deno.land/x to use?
https://github.com/yargs/yargs/tags

[absidue]

@Wykerd @LuanRT here are the official docs on how to do it: https://deno.land/add_module

https://deno.land/[email protected]/advanced/publishing

[LuanRT]

Thank you @absidue, I did check that and added the webhook to the repo. The problem is that deno.land doesn't have any other additional options, only Subdirectory. And after the webhook is added to the repository you can't change anything.

@Wykerd
So we'll probably have to automate releases with something like release-please (see https://github.com/yargs/yargs/blob/main/.github/workflows/release-please.yml and https://github.com/googleapis/release-please#release-please), otherwise it's quite impossible to create the deno tag first and then the main one + release notes.

[Wykerd]

An alternative might be https://nest.land/ which seems to have a CLI to publish packages instead of relying on GitHub webhooks. But setting up deno.land/x will probably be easier in the long run since it wont require manual publishing?

[LuanRT]

Should be good now.

[LuanRT]

I'll be merging this today, unless there's more to be done.

CC @Wykerd