Planning report

[sumneko]

This article was translated by ChatGPT and I have not reviewed it carefully.

I am writing this article to share with you my recent thoughts and plans for this project, which are a Turing-complete type system and a complete plugin system .

You may have noticed that I haven't been very active lately and have left a lot of issues unresolved. This is partly due to my busy work schedule, but more importantly, I have become somewhat tired of dealing with various minor issues. I believe that this language server has already solved 90% of the problems in Lua programming, but the remaining 10% is very tedious and requires a lot of effort. I believe there are two main factors here:

1. Lua is highly customizable, and many people encounter very specific needs in their customized environments, which can be very difficult to meet one by one.
2. The code of this project is relatively obscure and difficult to understand, so few people can participate in its development. However, please allow me to argue on this point: I was still a programming novice when I started this project, and developed it with the attitude of learning Lua while struggling with the lack of a good IDE. Over the years, it has undergone several partial refactors, but the entire framework is still quite outdated and patched. Although plugins are provided, users find it difficult to customize the functionality due to the difficulty in understanding the core implementation.

Therefore, I plan to carry out a complete overhaul, carefully planning and designing each module. My goal is still to solve 90% of the problems, and let users implement the remaining 10% themselves through annotations and plugins.

The following is my vision:

Annotation system

The annotation system is the current extension, where you can write a piece of code after the type, which will be executed every time the value is evaluated.

---@param a string
---@param b string
---@return string => a .. b
function Concat(a, b)
end

local x = Concat('123', '456') --> `x` is `"123456"`

---------------------------------

---[[@@@
KeysOf = function (t)
    local result = cats.new()
    for field in ipairs(t.fields) do
        result = result | field.key
    end
    return result
end
]]

---@class Point
---@field x number
---@field y number
---@field z number

---@alias keys => KeysOf(Point) --> `keys` is `"x"|"y"|"z"`

---------------------------------

---[[@@@
---@param t table
---@return string[] => function ()
        local result = cats.tuple()
        for field in ipairs(args[1].fields) do
            result:add(field.key)
        end
        return result
    end
]]
function GetKeys(t)
end

local v = GetKeys({a = 1, b = 2, c = 3}) --> `v` is `["a", "b", "c"]`

Plugin system

The plugin system is a Lua file placed in your own project, and the server will load this file as a "Mod" if it is trusted.

Note that the APIs used below are made up by me and will most likely change in the future.

ls.cats.newClass('UInt8', function (class)
    class:extends('integer')
    class.contravariance = true
end)

ls.diagnostic.register('uint8-out-of-range', function (file, throw)
    for _, value in ipairs(file.semantic.typeMatch['UInt8']) do
        for _, int in ipairs(value.asInteger.views) do
            if int < 0 or int > 255 then
                throw {
                    message = 'Out of range',
                    range   = value,
                    level   = 'Warning',
                }
            end
        end
    end
end)

I have completed the refactoring of the basic syntax parser. The entire project refactoring will take more than six months, and I will occasionally share the progress during this period.

The implementation details mentioned above will be repeatedly modified, but the overall idea will not change. Suggestions and ideas from everyone are also welcome.

Original Text (in Chinese)

我写这篇文章想给大家展示一下我最近的想法，聊一聊此项目之后的规划，那就是：图灵完备的类型系统和完备的插件体系 。

你可能注意到我最近一段时间不是很活跃，遗留了大量的issue没有处理。这自然有我忙于工作的原因，但更重要的是我有点厌倦了处理各种细枝末节的问题。我相信这个语言服务器已经解决了Lua编程中90%的问题，但是剩下的10%非常枯燥并且需要消耗大量的精力。我自认为这里有2个主要因素：

1. Lua的定制化程度非常高，很多人会在自己的定制环境中遇到非常具体的需求，一一满足它们很麻烦。
2. 这个项目的代码比较晦涩难懂，很少有人能参与到开发中。不过这方面请容我狡辩一下：我在启动这个项目时还是个编程新手，在苦于缺乏好用的IDE时抱着学习Lua的心态开发的此项目。这么多年过来经过了几次部分重构，但整个框架还是比较老旧且打满了补丁。虽然提供了插件，但是由于难以理解核心实现，用户很难定制功能。

因此我打算进行一次完全的重构，仔细规划设计各个模块。我的目标依然是解决90%的问题，剩下的10%则由用户自己通过注解与插件实现。

以下是我的设想：

注解系统

注解系统是目前的扩展，你可以在类型后面写一段代码，每次求值时会运算这段代码。

---@param a string
---@param b string
---@return string => a .. b
function Concat(a, b)
end

local x = Concat('123', '456') --> `x` is `"123456"`

---------------------------------

---[[@@@
KeysOf = function (t)
    local result = cats.new()
    for field in ipairs(t.fields) do
        result = result | field.key
    end
    return result
end
]]

---@class Point
---@field x number
---@field y number
---@field z number

---@alias keys => KeysOf(Point) --> `keys` is `"x"|"y"|"z"`

---------------------------------

---[[@@@
---@param t table
---@return string[] => function ()
        local result = cats.tuple()
        for field in ipairs(args[1].fields) do
            result:add(field.key)
        end
        return result
    end
]]
function GetKeys(t)
end

local v = GetKeys({a = 1, b = 2, c = 3}) --> `v` is `["a", "b", "c"]`

插件系统

插件系统是放置在你自己工程中的Lua文件，在信任的情况下服务器会加载此文件当做“Mod”。

注意，下面用到的API都是我现编的，之后99.99%会改

ls.cats.newClass('UInt8', function (class)
    class:extends('integer')
    class.contravariance = true
end)

ls.diagnostic.register('uint8-out-of-range', function (file, throw)
    for _, value in ipairs(file.semantic.typeMatch['UInt8']) do
        for _, int in ipairs(value.asInteger.views) do
            if int < 0 or int > 255 then
                throw {
                    message = 'Out of range',
                    range   = value,
                    level   = 'Warning',
                }
            end
        end
    end
end)

目前我已经完成了基础语法解析器的重构，整个项目的重构需要半年以上的时间，在此期间我会偶尔分享进度。

上述的实现细节还会反复修改，但总体思路不会变，也欢迎大家提出自己的奇思妙想。

[carsakiller]

Hey Sumneko!

I believe that this language server has already solved 90% of the problems in Lua programming

I agree, the language server is already capable of solving the most common occurrences in Lua programming. However, generics and better type narrowing are two of the biggest things I frequently run in to. Right now, if I write an Array class, I either mark it as an Array and get completion for its functions, or pretend it's a table and get support for its values. The narrowing issue has prevented me from properly documenting CC:Tweaked's event system. I would like to just redefine os.pullEvent in a definition file, one for each event input and return the output, but the types aren't narrowed. Event systems have always been hard to document because of this.

the remaining 10% is very tedious and requires a lot of effort. I believe there are two main factors here [the reasons...]

I agree. Because Lua is an embedded language, it can be found pretty much anywhere in almost any configuration. This makes it incredibly hard to make a language server for it. It is not just expected in the browser or Node.js, Lua could be running in a game, in a webpage, in a mobile app, or on a homemade arcade machine. We would need at least 10 Sumneko's to personally customize the language server for all of these use cases 😄. Covering the general usage and making the language server modular/extensible is likely the only realistic way to try to support as many people as possible.

I can attest that the source code for the language server is a challenge to understand. I am not very knowledgeable when it comes to running Lua as a standalone application, but taking a look through the code is daunting. The language server is very impressive, especially considering you started with little knowledge of Lua, but there are sections of its code that are noticeably in need of a refactor. This is potentially where the biggest improvement could be made. If the code and implementation is easy to understand, hopefully there will be more contributions from the community to assist you in maintaining. It will also potentially allow people to modify the entire language server to their specific use-case, if need be.

Annotation System

Allowing users to run code in their annotations would definitely help them cover many use cases. We could also provide some common functions (sort of like TypeScript utility types), like Keys, Values, Exclude, etc. However, this does also open the door to malicious code, which could be especially concerning when it comes to sharing addons/definitions. The other thing to consider is error handling; what will happen when someone's annotation code throws an error?

Plugin System

Sounds like a solid improvement over the current system. Developing a solid API for users to interact with is crucial to improving the customization of the language server. What will make a plugin “trusted”?

Looks like there will be lots to document, count me in! 🙂

---

Let me know where/when I can help! I would love to help where I am capable to make this refactor worth the effort and make the language server the best it can be! This may also be a good time to rename some things 😉.

[carsakiller]

Another improvement could be making it as easy as possible for people to get their dev environment set up to help contribute. Lua can be a pain to set up on Windows, so maybe creating a dev container would help?

[CppCXY]

First of all, I think this is a syntax error for standard lua:

---[[@@@@

]]

In addition, I think we should separate the core implementation from the language server, separating the code analysis core from the language server appearance. If we can use the code analysis core as a third-party library for lua, then people who need it can use it directly and build their own language server on top of it.

Also, allowing trusted programs to automatically load third-party code is often seen as a vulnerability that requires careful thought.

[sumneko]

In addition, I think we should separate the core implementation from the language server, separating the code analysis core from the language server appearance. If we can use the code analysis core as a third-party library for lua, then people who need it can use it directly and build their own language server on top of it.

I am planning to put the core analyzer in another thread, and have the main thread act as an intermediary communicating with the client. The question is how should the communication protocol be designed? Should I use my own protocol or use LSP?

[tmillr]

Definitely LSP. It's already defined and portable, and using it means less work and not breaking compatibility with clients/editors.

[fesily]

I don't think it's a good idea to write code inside comments, simply make it possible for users to add metafunctions in the plugin system instead of writing metafunctions in comments.
It's a formal difference, but it's a blow to code readability in what is already a multi-paradigm mess of a language, in adding more functions to the comment system.
Is dismissal-proof work-style programming, -. -
Of course there are more important reasons, the goal of a language server is to provide better help. Any more code added to the comments will just make it more confusing for those who come along later, right? What the hell is this thing, the bar has been raised even higher!

---

我觉得注释里面写代码不是一个好主意，干脆在插件系统里让用户可以添加元函数，而不是在注释里写元函数。
虽然这只是一个形式上的差别，但是对于代码可读性是一个打击，本来就是一个多范式的混乱语言，在往注释系统里添加更多的功能。
防开除式编程么，-.-
当然还有更主要的原因，语言服务器的目标本来就是提供更好的帮助。注释中加的怎么多的代码，只会让后来者更困扰是吧，这玩意到底是个什么，门槛更高了

[fesily]

安全问题是无解的，沙盒逃逸在 5.4 版本也是存在的。写在注释里可能会意外引入的第三方库里给你加点料进去，写在插件里也是应该要求主动开启，而不是默认启用。

[sumneko]

可以直接复制代码不需要新增文件修改配置

[sumneko]

写在注释里不能引入第三方库。

[fesily]

我表述的不准确，我的意思是代码里可以引入第三方库，这个库代码是没问题的，但是恶意攻击者可以针对注释里运行代码这个功能，特定攻击使用咱们语言服务器的用户。
沙盒逃逸只要提供默认的lua 功能就能逃逸，并不需要运行 require 之类的敏感函数

[sumneko]

注释中不会提供完整的Lua功能，debug库等函数都会禁用。另外语言服务本来就不会去执行用户的代码，dostring 类的函数注释中也不会提供。唯一有可能出问题的情况是，插件中向注释提供了不安全的函数。

这里再明确一下：
插件的代码拥有完整的权限，但是在加载插件前需要用户确认（默认策略下VSCode的可信任工作区中也要二次确认）
注释只能使用受限的Lua基础函数，以及语言服务和插件显性申明提供给注释的函数

[emmericp]

The AST-based plugin API that you added recently already solves 90% of my remaining 10% which is amazing, thank you!

Custom diagnostics also sound amazing and will probably solve a lot more problems

[mkster]

Will this rewrite also address slow loading times with larger projects like the one from tencent does? https://github.com/Tencent/LuaHelper

[fesily]

I don't think that's possible.

[tmillr]

If I'm not mistaken, the reason loading times aren't great has to do with the runtime and is because this project is written in Lua (and it does not support/target LuaJIT either). If the LSP ran on LuaJIT, or was built from a pre-compiled language, it'd be much faster.

[MillhioreBT]

I love the idea that this annotation plugin is improved with the possibility of executing code and thus being able to increase its customization...

By the way, I think it would be very good if the code blocks were written like this:

---@do
--[[
	-- CODE HERE
]] --
---@end

This way we can start a block with the annotation ---@Do and close the block with the annotation ---@EnD
This block should contain a multiline comment where the code to be executed will be.

we can add a special argument to the annotations to make evaluations of functions encapsulated in the environment of the annotations
per example:

---@do
--[[
	function getType(luaObj)
		return cats.type('enum')
	end

	function getSuggest(luaObj)
		if type(luaObj) ~= "table" then return end

		local result = cats.suggest()
		for key, value in pairs(luaObj) do
			results[key] = value
		end

		return result
	end
]] --
---@end

---@eval [type: getType, suggest: getSuggest]
MagicEffectClasses = {
	FIRST_MAGIC_EFFECT = 1,
	SECOND_MAGIC_EFFECT = 2,
	THIRD_MAGIC_EFFECT = 4,
	FOURTH_MAGIC_EFFECT = 8
}

This pseudo code would make the MagicEffectClasses table of type enum and would also modify the autocomplete code suggestions to our liking.

Actual enum:

With the new evaluation method to modify the suggestion:

I can think of many more great ideas to customize annotations, I hope you continue with the plugin and I hope one day I can contribute more than just suggestions <3