Sandbox: Support making customizations to monaco-typescript

[orta]

Context:

• https://twitter.com/rickbutton/status/1219661139367419905
• https://twitter.com/orta/status/1219679733027426306

When I talked with the TC39 folks, a lot of people were interested in being able to make REPLs for proposals, and one just came out: https://rickbutton.github.io/record-tuple-playground/

To pull it off they needed to fork monaco-typescript and:

• edit the default dts files
• edit the language's syntax used by monarch
• Add a custom transformer function which lives in the TS workers

Given that we distribute our own copy of monaco - I wonder if we can expose an API for these by extending our monaco-editor with a patch to add APIs.

Biggest source of unknown for me is that I don't know how we could pass a function to workers, to my knowledge everything is serialized.

/cc @rickbutton

[orta]

Alternative take: support running ttypescript in monaco

[rickbutton]

@orta what do you imagine that these transform functions will look like? Currently, with the R&T proposal, it is easy to transform the new syntax into objects (which can be parsed by TS) because we can simply replace the unrecognized characters with spaces, but more complex additions to syntax will not be possible (because monaco's knowledge of where the tokens are in the source will be incorrect, so you will get weird problems with syntax highlighting and tooltips/types).

I'm imagining a transform step that includes a source map, so that monaco is able to compensate for more complex source transforms, but I'm not sure what that would look like. Thoughts?

[orta]

Yeah, that's why I've been mentally moving towards using the ttypescript method of allowing Transformers to be applied but on the web also. This means that the TS AST wouldn't need to "lie" because you could be replacing the AST nodes with something TS can understand. Transformers are already a thing in TS world, so if they ever become first class citizen 🤞🏽 then it is forward compat (plus you can test locally via typescript)

People would be able to deploy an AMD version of a plugin like ts-transform-graphql-tag which could be picked up by both monaco-typescript (and the url of transformers can get sent to workers, so no need to pass functions around)

[ark120202]

Not sure if that would fit these use cases, but I have found that it's pretty easy to extend ts.worker in ESM build without forking monaco-typescript. For example, that's how custom transformers could be added:

// ts.worker.ts
import * as ts from "typescript";
import { TypeScriptWorker } from "monaco-editor/esm/vs/language/typescript/tsWorker";
// @ts-ignore
import * as worker from "monaco-editor/esm/vs/editor/editor.worker";

export class CustomTypeScriptWorker extends TypeScriptWorker {
    // Any LanguageServiceHost extensions should work
    public getCustomTransformers(): ts.CustomTransformers {
        return { 
            before: [...]
        }
    }
}

globalThis.onmessage = () => {
    worker.initialize((context: any, createData: any) => new CustomTypeScriptWorker(context, createData));
};

// webpack.config.js
/** @type {import("webpack").Configuration} */
module.exports = {
    entry: {
        "ts.worker": "src/playground/ts.worker.ts",
    },
    plugins: [
        // Replace vendored monaco-typescript's services build with full TypeScript API
        // to remove duplicated typescript versions in bundle (coming from transformers)
        // and make `ts.worker` builds with different ts versions.
        new webpack.NormalModuleReplacementPlugin(/typescriptServices/, require.resolve("typescript")),
    ],
}

[rickbutton]

Interesting!

I've been doing some digging as well, with a locally forked version of monaco-typescript. My concern at the moment is: it looks like transformers can only be used during emit calls, not during the rest of the things that a tsserver can provide. For example, there doesn't seem to be a way to apply a transform to a getSyntacticDiagnostics

[orta]

Yeah, I've dug a bit deeper, transformers as they are built today still have to support the same AST as TypeScript - which means the parser can't change (thus you could never really test a new language feature via a transformer)

For example:

const thing = #{ a: "string" }
thing.a

thing.b

Can create a legit AST, but tooling breaks down because it's semantically incorrect. The # is never in the tree, so it can't get removed at AST level.

I'm not sure if this can work without having a custom TS worker, which has a custom build of TypeScript TBH

[rickbutton]

Yeah that is my theory too. Maybe the solution then is to provide a super easy way to use a custom version of typescript with monaco?

[ark120202]

Yeah, transformers are quite limited right now, but in a worker you have the full access to the language service and it's host. For example, it allows you to easily override getScriptSnapshot (isn't that the only LS-related change in proposal-record-tuple?). Also, since in the constructor you can access this._languageService, it should be possible to use any language service plugin

Maybe the solution then is to provide a super easy way to use a custom version of typescript with monaco?

That's actually what new webpack.NormalModuleReplacementPlugin(/typescriptServices/, require.resolve("typescript")) allows to do, as long as custom typescript version doesn't change the public API

[orta]

Coming back to this, because it's been a while and because of microsoft/TypeScript#37973 - I wonder if I can lower the barrier to forks of TypeScript having a playground.

Right now all of orta/make-monaco-builds is optimized to our s typescript deploys, but maybe I can make it trivial for us to run any of your forks of typescript.

e.g. I build a way to accept PRs which trigger a deploy of someone else's npm package build of typescript - it could be just a json file like:

{ "deploys": {
  "@orta/typescript": "3.5.6"
}}

Anyone can send a PR to add their versions to ^ to trigger a deploy

On CI changes to this are detected and it deploys to @typescript-deploys/monaco-typescript with a tag like _orta_typescript_3_5_6.

This would allow you to write: [playground_url]?ts=_orta_typescript_3_5_6

---

Downsides: this still requires making a working fork of typescript to see any of these changes though

[orta]

Coming back to this again, I've recommend for something like the pipeline operator microsoft/TypeScript#38305

To have a fork of TypeScript, which can then be playgrounded: https://www.typescriptlang.org/play/index.html?ts=4.0.0-pr-38305-12

and it also ends up on npm at https://www.npmjs.com/package/@typescript-deploys/monaco-editor/v/4.0.0-pr-38305-12

Which is an option, if a little technically involved to just get a prototype out

[orta]

Another potential route for this is via the custom webworker API I introduced in microsoft/monaco-typescript#65 which can intercept all TypeScript work in the background (in the same way a language service plugin can)

I'll try experiment with custom syntax using this in a plugin at some time to see if that's explorable

[akutruff]

@orta I'm about to plunge into adding some syntactic sugar on top of TypeScript as well and am also reluctant to fork. Had any new thoughts on how to approach language extension?

Side note: I am amazed by how much you and the TS team are striving to make language and tooling as accessible as possible. Prior to looking into Monaco, I would have never even considered toying with a language. =D Thanks so much!

[orta]

It's not something I've tried but the I think the custom webworker API above could be a reasonable answer which avoids making a version of TypeScript which understands how to skip particular bits of syntax.

Perhaps overriding _getScriptText on the subclass and removing code before it reaches TS - you'd have to re-align error message positioning etc also further down the line but that could be enough to do a few simple explorations

[orta]

Hey folks - I have a fully working solution for this now, and that is to use microsoft/monaco-typescript#65 which lets you customize the monaco-ts <-> tsserver worker. As a complex example, #2063 shows taking a single TypeScript model in monaco and having that split into many TypeScript vfs files and re-directing LSP-ish requests to the right place behind the scenes.

It's not hard to imagine a similar setup where you swap out the custom JS changes before the file is given to tsserver. This technique works with vanilla Monaco since sometime last year.

[rickbutton]

awesome! I'll give this a shot when I have some time and post an example when I have it.

[typescript-bot]

Hello! As per #2804, we are automatically closing all open issues. Please see #2804 for a description of what issues and PRs can be accepted going forward.