Alternate Proposal: Reversable Type Annotations

[matthew-dean]

The goals of the original proposal are stated like this:

This proposal aims to enable developers to add type annotations to their JavaScript code, allowing those annotations to be checked by a type checker that is external to JavaScript. At runtime, a JavaScript engine ignores them, treating the types as comments.

The aim of this proposal is to enable developers to run programs written in TypeScript, Flow, and other static typing supersets of JavaScript without any need for transpilation...

The proposal goes on to articulate essentially these premises:

• Build steps add another layer of concerns to writing code.
• The current alternative to TypeScript w/ a build step is JSDoc
• JSDoc only supports a subset of TypeScript

The conclusion drawn by the authors from these premises is that the only way to achieve these goals (supporting types without a build step) is to add all of these type annotations to the JavaScript language, to be treated as a number of other "comments" other than /* */ and //.

The authors may have a point that these comments need to be added to JavaScript if there are no viable alternatives on the tooling side. The truth is, all of the above premises / statements of the problem could be solved today, without adding any syntax to JavaScript, without disrupting the JavaScript ecosystem, by tooling alone. And it could be solved for TypeScript and Flow, and anything else that is a "superset" of JavaScript.

Here's what I mean.

Creating a single JavaScript file with types

A user opens up VSCode. At the top of their file, they write something like this:

/*# TS */

After that, they write TypeScript. Let's say they're writing something to run on Node.js and output to the console.

/*# TS */

const statement: string = 'Hello world.'
console.log(statement)

They save their file as hello.js. Next they go to their command line, and type:

> node hello.js

It outputs:

> node hello.js
Hello world.

What Magic Is This???

Here's what Node saw:

/*# TS */

const statement/*:: string*/ = 'Hello world.'
console.log(statement)

What happened is that the IDE, in this case, VSCode, recognizes the initial comment block in .js files.

/*# TS */

This tells the IDE (or a linter, or any other tool) to use a designated manager for reading / writing files, in this case: TypeScript. TypeScript itself would use this comment block to translate .js files for the purposes of project-wide type-checking.

Similarly, this could of course look like:

/*# FLOW */

...or whatever else the JS ecosystem supports. These file managers would essentially transpile-on-save, except they would transpile to a comment form that is reversable. Everything that is "erasable" for the runtime would be wrapped in special comment blocks /*: */. (For obvious reasons, if an author writes /*: before transpilation, the file manager should throw an error.)

When the file is loaded by the IDE (or other tools, like linters), it's passed through the file manager to create it's presentable form, which in this case is a simple replace of /*: */ blocks with the content inside of them.

There are a few caveats, of course!

1. This should only allow "erasables" -- i.e. type annotations and syntax that are removed at runtime. For similar reasons as the proposal, something like an enum should probably not be supported by TypeScript, because it transpiles into a completely different form. (TypeScript already offers a {} as const alternative to enums.)
2. Because it's only erasing syntax, the resulting JavaScript needs to be executable in the native environment. So this wouldn't support something like transpiling to ES5, for example, as this gets harder to track in tools like git. There should essentially be a 1:1 line-for-line representation of syntax, just with comments wrapping erasable syntax.

In other words, if an author writes:

/*# TS */

interface User {
  name: string;
  id: number;
}

const user: User = {
  name: "Hayes",
  id: 0,
};

The file that is saved looks like:

/*# TS */

/*:interface User {
  name: string;
  id: number;
}*/

const user/*:: User*/ = {
  name: "Hayes",
  id: 0,
};

It's perfectly runnable JavaScript! And still readable, if you need to look at raw bytes! And nothing needed to be added to the JavaScript language at all.

And when a user loads the saved file in their IDE, they would again see:

/*# TS */

interface User {
  name: string;
  id: number;
}

const user: User = {
  name: "Hayes",
  id: 0,
};

Support For This Syntax

In order for this syntax, or something like it, to work in practice, you need some adjustments on the tooling side:

1. Support in IDEs for file managers of files with these comment headers.
2. Support at the TypeScript / Flow side for transpiling to / from this comment syntax. (Also, eventually, support by third-party transpilers of TypeScript, like SWC.)

That's not nothing! But IMO this is far easier / less contentious than:

1. Injecting a variety of new comment types into JavaScript
2. Upgrading every JavaScript tool to treat all these new forms of type statements / additional syntax as comments.

In this case, there's only one point of change to support types/TypeScript in JavaScript, and it could be done by Microsoft today, as they (and the community) support both the TypeScript compiler and a popular IDE. This is far, far less of a lift than that of the original proposal, which requires points of change at anything and everything that touches JavaScript. (And IMO the original proposal makes a "JavaScript with types" file far, far harder to reason about when looking at the raw bytes, as to what is / isn't an ignorable comment.)

What's also nice is that you can still have a build step that minifies / bundles your .js files, and the nice part is: pretty much any minifier used today would support this format, because comments are typically one of the first things to be removed in a minification step. So there's a whole ecosystem of JavaScript tools that are instantly compatible with this approach.

Other Caveats

I should also note that there is an adjustment for the TypeScript / JavaScript author who is used to bytes-on-disk = bytes in my IDE, and there are probably some side-effects for this adjustment. i.e. I make no claims that this a perfect solution.

The good part is, if an author was insistent that bytes-on-disk be represented exactly as bytes in the IDE, there should be nothing stopping them from wrapping TypeScript / Flow syntax in this comment form by hand (or being able to switch between the two, should the IDE allow), getting type safety / auto-complete support regardless.

Also, as stated, there would still be a need by TypeScript / Flow teams to define what subset of those type systems would be supported by the JS-comment method. However, it should be a greater subset of, say, TypeScript syntax than what is in the original proposal, giving TypeScript authors actually more of what they want without a separate build step, and without separate .js / .ts files.

/*# TS */ and /*: */ are just suggestions, inspired by other similar comments on other Github issues here. Obviously the same idea could work with tweaked syntax.

Lastly — one major caveat to is that parsing from JS-to-TS needs to be fully reversable. This means that a TS parser could not just parse JS into an abstract syntax tree and fudge the details, like indents and line breaks, and back again. It would need a concrete syntax tree representation, including things like semi-colons, and determine which erasable pieces can be wrapped. I’m not sure the existing parser supports this.

Conclusion

I think, if it can be proven that the aims of the original proposal can be solved with tooling alone, (and I think I've demonstrated that it can?) the authors should articulate why a tooling solution would not be sufficient and new syntax must be injected into JavaScript & the JavaScript ecosystem.

Comments welcome!

[matthew-dean]

Just to quickly demonstrate a benefit to this approach, it becomes trivial to support parts of TypeScript which would otherwise be ambiguous and hence unsupportable in the original proposal, such as:

/*# TS */

function foo(x: number): number;
function foo(x: string): string;
function foo(x: string | number): string | number {
    if (typeof x === number) {
          return x + 1
    }
    else {
        return x + "!"
    }
}

This would be output as:

/*# TS */

/*:function foo(x: number): number;
function foo(x: string): string;*/
function foo(x/*:: string | number*/)/*:: string | number*/ {
    if (typeof x === number) {
          return x + 1
    }
    else {
        return x + "!"
    }
}

It's not the most readable on disk, but the point is that it's executable, natively, and still type-checkable on the tooling side. And it's easily reversable to present to the code author.

[matthew-dean]

One of the things an IDE might want to do is support the concept of “folded” comments, such that an author can view the actual source comment (of a TS block) at any time.

In fact, one might want the line/column position to display the actual source position, and show the “fold” in between, but I feel like those are minor implementation details for others to hash out.

In other words, I could see a use case for toggling source / folded representation for a single JS file, just so an author can see the source on disk.

[lucacasonato]

I strongly oppose with this kind of IDE level magic tooling.

What this is is that you are effectively moving the transpilation into the code authoring stage. Instead of having it between the source code, and the distributable, you are putting it between the developer, and the source code.

This has all the same problems as transpilation between source code and distribution. It requires extensive effort tooling to make work well. Instead of a half dozen JS engines needing to support the transpilation, you now require hundreds of IDEs, syntax highlighters, JS runtimes, editors, etc to do this transpilation. I think this may be even worse than the current state of affairs, because the amount of tooling that needs to do the transpilation (both ways) gets larger by a magnitude or two.

Some concrete concerns:

This is confusing to users of text editors. Many folks use text editors like nano, Notepad, or Notepad++ to edit code. They will find some code with these type annotations on Stack Overflow, copy it into their text editor, and then it doesn't run because the type comments were inserted as literal text. This is especially painful for folks just starting out with JavaScript (who from own experience often start by typing some code into a text editor). They will not immediately realize that what their IDE is showing them is not actually what the file contents are.

This might force folks to use an IDE for JavaScript development, even if they don't want to. The experience for folks not using IDEs will be pretty terrible. They will have to type at least 5 extra chars for each type annotation. Even text editor masochists deserve a better experience 😉.

This syntax would require a lot of effort from vary many people to making work well. The tooling that would have to "understand" this new syntax and strip out the /*: and */ for visualization is not limited to just IDEs. Think Chrome Dev Tools, syntax highlighters, the GitHub Gist editor, etc etc etc.

REPLs would need to support this syntax. REPLs are also editors for JS, so they would also need to do this transpilation on the fly.

This is an INSANE amount of effort to make work well consistently. To be able to do this well, next to the JS lexer, a tool now requires an HTML parser to be able to extract <script> blocks out of HTML so it can send them through transpilation. It requires the same for code blocks in md files, or any other space in any other file format where one would want to type JavaScript.

How does copy paste work? If I copy type Foo = "a" | "b"; in my editor, should that copy /*:type Foo = "a" | "b";*/ or type Foo = "a" | "b";?

---

Most importantly however, this does not resolve the problem statement for this proposal:

The strong demand for ergonomic type annotation syntax has led to forks of JavaScript with custom syntax.

This has introduced developer friction and means widely-used JS forks have trouble coordinating with TC39 and must risk syntax conflicts.

The syntax fork is still there. It just has moved from the code on disk, further into the tooling (and much more of it). The tool still needs to have a list of grammar productions for the type annotation syntax to be able to figure out when to add the /*: and */. Risks of syntax issues are still there, as the syntax is not managed in TC39. Additionally, because the syntax definitions are spread across much more tooling, updates to it will propagate much slower.

[matthew-dean]

@lucacasonato

Thanks for your feedback.

There's probably some confusion from things I didn't articulate clearly.

Let's start with some a few clarifications.

1. Regardless of how you slice it, as many people have pointed out on different issues here, the original type annotations proposal (the OP) sets up a strong strawperson in the form of TypeScript vs. JSDoc. JSDoc is a poor avenue for a type system, but it's not the only comment / annotation form one could use.
2. IMO The problem statements of the OP are not solved by the OP. The proposal says, "...the goal of this proposal is to allow a very large subset of TypeScript syntax to appear as-is in JavaScript source files, interpreted as comments."

Maybe it's poorly worded, but the OP reads as not attempting to solve the problem of syntax forks of JavaScript to provide typing across different type systems. i.e. it's not clear that this proposal could / would be used by Flow, for example. IMO if it were a collaboration of different type system authors, it would make a little more sense. But that's not how it reads right now.

More importantly, there are really two things I'm conflating here.

1. Defining types / type annotations as (actual) comments.
2. IDE support for "folding" comments / syntax highlighting. In terms of IDEs, code-coloring is always supported on an individual basis in IDE by IDE, and this is more a code-coloring kind of task, although I was describing it in terms of "hiding" the comment form.

Really, the proposal is for TypeScript to recognize this as TypeScript:

/*# TS */

/*:interface User {
  name: string;
  id: number;
}*/

In terms of the IDE, you're probably right that too much magic to hide /*: */ might be confusing. It's probably sufficient to "dim" /*: */ and syntax highlight the inner block as TypeScript.

That being the case, there would be total and complete support for "text editors like nano, Notepad, or Notepad++ to edit code".

This syntax would require a lot of effort from vary many people to making work well. The tooling that would have to "understand" this new syntax and strip out the /*: and */ for visualization is not limited to just IDEs. Think Chrome Dev Tools, syntax highlighters, the GitHub Gist editor, etc etc etc.

Not really. I didn't articulate it / explain it well, and I think I gave too strong of an impression to IDE support being necessary for this process. The actual proposal is just: put types in comments, in a form that's reversable. The IDE magic I was describing was a way to manage that in a way that's easy for developer input. But there's nothing requiring any IDE magic at all. As I wrote, it's still very readable. It could be made more readable, at the IDE level, but it's not necessary.

REPLs would need to support this syntax. REPLs are also editors for JS, so they would also need to do this transpilation on the fly.

Nah. Why? You mean, because they might produce invalid TypeScript? I mean, the same is true if you edit a TypeScript file in a REPL or nano or Gist, etc.

This might force folks to use an IDE for JavaScript development, even if they don't want to. The experience for folks not using IDEs will be pretty terrible. They will have to type at least 5 extra chars for each type annotation.

I mean, if you've chosen to not use an IDE, you've chosen to not have any assistance when writing code. You've opted to essentially do more work. If you want to maintain code in this way, sure, but then, you gotta ask yourself, why would someone use TypeScript at all? There are much fewer benefits, so you're just writing a verbose superset of JavaScript. I guess, type checking? Is someone using Notepad++ and then constantly rebuilding their TS project? That's already an amazingly terrible development workflow. I'm not sure how 5 extra characters makes a difference there. 🤷‍♂️

This is an INSANE amount of effort to make work well consistently. To be able to do this well, next to the JS lexer, a tool now requires an HTML parser to be able to extract <script> blocks out of HTML so it can send them through transpilation.

I have no idea what this means, but that wouldn't be the case. Again, do you mean to support someone writing TypeScript in a <script> tag? When is this happening, and why do you need TypeScript in your <script> tags? In any case, you could annotate your types with comments, and TypeScript should still function as an external tool, if it's somehow integrated for type completion into, I guess, a <script> tag? But... TypeScript doesn't work like that now.

How does copy paste work?

That's an interesting question. I guess I would lean more at this point towards the characters always being visible / present, but maybe dimmed with syntax coloring, if available. So it would be /*:type Foo = "a" | "b";*/

The syntax fork is still there. It just has moved from the code on disk, further into the tooling (and much more of it). The tool still needs to have a list of grammar productions for the type annotation syntax to be able to figure out when to add the /*: and */. Risks of syntax issues are still there, as the syntax is not managed in TC39. Additionally, because the syntax definitions are spread across much more tooling, updates to it will propagate much slower.

This is an interesting perspective, because I see the OP as having a dangerous amount of syntax fork to being with. It reads very much as "please add all this TypeScript syntax as ignorable comments to JavaScript". It's not clear what Flow or Closure Compiler are supposed to do with this proposal. So it feels like an entrenchment of syntax forking, built right into JavaScript. At least with TS in actual comments, TypeScript can iterate on whatever valid TS syntax they want to support from this form, and it would leave JS intact (free from syntax forking).

[matthew-dean]

@lucacasonato

If it helps, I'm generally just saying TypeScript should do what Flow does: https://flow.org/en/docs/types/comments/. Having magic IDE support on top of that would just be nice, but it's not necessary.

Fascinatingly, I didn't see / know Flow had this and used this same syntax.

[matthew-dean]

I’ll rewrite this to remove the IDE magic

[matthew-dean]

I've separated this proposal out to just the comment syntax proposal, which has a lot of prior art! microsoft/TypeScript#48650