Add support for 4.x compiled scripts

[nikitalita]

Resource Type

No response

Describe the problem or limitation you are having

4.x just added binary tokenization back:
godotengine/godot#87634

Describe the feature / enhancement and how it helps to overcome the problem or limitation

Need to figure out if the new scheme is compatible with our current tokenization scheme and if it would be easy to add the defines to the JSON

[VendorAttestation]

@nikitalita any ETA?

[nikitalita]

I'm going to be pretty busy the next couple of months, so probably not for a while. Pull requests are always welcome, though.

[EoF-1141]

From what I can tell it should be possible to implement this mostly within it's own bytecode_* file.
Since Variants have changed, it will require compat for it.

The zstd decompression would be in the bytecode script regardless.
There are Newline, Indent and Dedent Tokens, but they seem completely unused and instead the info seems to be stored on token_lines and token_columns

I wrote a Python implementation, which seems to work okay so far.
Might try myself at contributing here, the main annoyance is that tokens seem to be handled slightly differently and require more "space handling"

[NiceLookinGuy]

I wrote a Python implementation, which seems to work okay so far. Might try myself at contributing here, the main annoyance is that tokens seem to be handled slightly differently and require more "space handling"

Is it possible for you to share python implementation? It could compensate for lacking 4.x bytecode support in gdsdecomp for the time it's not implemented.

[EoF-1141]

Is it possible for you to share python implementation? It could compensate for lacking 4.x bytecode support in gdsdecomp for the time it's not implemented.
It's more of a PoC than a full implementation. Doesn't have all Variants implemented yet (mainly just the few that I came across), and the formatting isn't cleaned.

So I'd just start on the c++ code for gdsdecomp, copying some of the original Godot code and then have people clean up the formatting.
I think ultimately it might actually be better to have a post processor, that cleans up and will always be run after the bytecode_*

[nikitalita]

I actually have a WIP implementation going, but I am not going to have time to finish for a while. If you would like, I can push this to a branch on here and you can start from that.

[EoF-1141]

I actually have a WIP implementation going, but I am not going to have time to finish for a while. If you would like, I can push this to a branch on here and you can start from that.

Sorry for the late reply. Been pretty busy.
If you want I can take a look and see what I can do there

[nikitalita]

https://github.com/nikitalita/gdsdecomp/tree/4.3-scripts