Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Basic Lua scripting support #550

Merged
merged 49 commits into from
Aug 16, 2024
Merged

Basic Lua scripting support #550

merged 49 commits into from
Aug 16, 2024

Conversation

badrishc
Copy link
Contributor

@badrishc badrishc commented Aug 2, 2024

This PR adds very basic support for Lua scripts in Garnet (EVAL, EVALSHA, SCRIPT commands) using the NLua library. We hope for contributions from the community to extend as per their needs. Consider this a research prototype. Note that performance is not expected to be as good as C# custom procedures and transactions, due to the overheads of Lua scripts and the conversions from C# to Lua and back.

Tasks:

  • Basic tests passing for GET, SET, ZADD
  • BDN tests to see performance limits when using NLua
  • Optimize script invocation code path for end-to-end performance
  • Add support for all API calls in Lua scripts (by formatting into RESP command)
  • Error handling, support more types, math
  • Add switch (--lua and EnableLua) to enable Lua script on server (turned off by default)
  • Enable transactions (opt in at server-level) via --lua-transaction-mode and LuaTransactionMode boolean option
  • Support ACL

Future work:

  • Better script cache management (currently all scripts are cached with no upper bound)
  • Support LuaTransactionMode on a per-script basis
  • Support shared vs exclusive key locks (currently we pessimistically lock all keys as exclusive)
  • Add cluster slot verification
  • Support read-only scripts (EVAL_RO)

Fix #509

@badrishc badrishc marked this pull request as draft August 8, 2024 00:11
@badrishc badrishc changed the title [WIP] Basic Lua scripting support Basic Lua scripting support Aug 8, 2024
@badrishc
Copy link
Contributor Author

badrishc commented Aug 9, 2024

End-to-End Performance

Increasing number of client sessions, 8-byte keys and values, using Resp.benchmark with batch size 4096, no transactions.

  1. Script: return KEYS[1]

How to run: Resp.benchmark --op SCRIPTRETKEY -t 1,2,4,8,16 --dbsize 1024

  1. Script: return redis.call('get', KEYS[1])

How to run: Resp.benchmark --op SCRIPTGET -t 1,2,4,8,16 --dbsize 1024

@TalZaccai TalZaccai self-requested a review August 13, 2024 20:22
libs/server/Lua/LuaRunner.cs Show resolved Hide resolved
libs/server/Lua/LuaRunner.cs Show resolved Hide resolved
libs/server/Lua/LuaRunner.cs Outdated Show resolved Hide resolved
libs/server/AOF/AofProcessor.cs Outdated Show resolved Hide resolved
libs/server/Resp/Parser/RespCommand.cs Show resolved Hide resolved
libs/server/Lua/LuaCommands.cs Outdated Show resolved Hide resolved
@badrishc badrishc requested a review from TedHartMS August 14, 2024 00:33
libs/server/ArgSlice/ScratchBufferManager.cs Show resolved Hide resolved
libs/server/Garnet.server.csproj Show resolved Hide resolved
libs/server/Lua/LuaCommands.cs Outdated Show resolved Hide resolved
libs/server/Lua/LuaCommands.cs Show resolved Hide resolved
libs/server/Lua/LuaCommands.cs Outdated Show resolved Hide resolved
@badrishc badrishc requested a review from TalZaccai August 16, 2024 21:54
@badrishc badrishc merged commit d7b640a into main Aug 16, 2024
26 checks passed
@badrishc badrishc deleted the badrishc/lua-script-v0 branch August 16, 2024 23:37
@github-actions github-actions bot locked and limited conversation to collaborators Oct 16, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Lua support - EVAL and friends
4 participants