Cmd + K AI helper in live editor

[jonastemplestein]

Screen.Recording.2023-09-27.at.19.52.37_compressed_rotated.mp4

As discussed here, here's my attempt to bring functionality similar

A few notes

• Massive kudos to the livebook team for making livebook so easy to run locally in dev mode. It worked straight away without any fiddling
• Even if this doesn't get merged, I'd love any and all feedback on the many questions raised in the code. I haven't coded much in years and haven't done javascript in anger in 10 years, so I'm feeling pretty rusty. And I was often unsure whether I was following conventions correctly
• You can switch between GPT-4 and anthropic in session_live.ex:1085 - lmk if you need API keys to test

TODO

Required before this is useful / can be shipped

• Share context of previous cells with the AI agent - if the code doesn't all fit into the context, we could at least share bindings and defined modules or something
• Multi-message conversations (question about whether the messages state should live on server or client - probably client)
• Tests...
• Rejecting changes
• Handling many different error or loading states and edge cases - especially when the user (or another user!) are using the editor
• Handle UI edge cases (e.g. mobile users, changing geometry, etc)
• Tune the prompt. For example, it spends way too much time outputting its plan right now

Ideas for some time later

• Try to get the LLM to output diffs instead of whole code blocks to speed things up
• Create separate AI UI that lives outside an individual editor so you can ask it to modify or create entire livebooks (maybe a side panel?)
• Create open-interpreter-like loop that solves entire problems end-to-end in livebook
• Share relevant documentation with AI helper (especially with claude, which has a huge context window)

[josevalim]

Latest LiveView (v0.20) supports assign_async for uses cases precisely like these, so once we update it, you should be able to use it just fine!

[josevalim]

Hi @jonastemplestein, the demo looks impressive!

Unfortunately, it also shows the problem with AI: the code it generates for fizzbuzz is actually inefficient because it appends to the list on every operation, which has to copy the left side. A best approach is to prepend and then reverse. :D

Asking it to write doctests was fantastic though. I think the refactoring/debugging/explanation features may bring more value than code generation.

The code overall looks good although I did not look at the JS bits. Perhaps one suggestion is to change the Finch streaming to, instead of it streaming the tokens, have it already stream only the parts that you care about. It will definitely be less data to send around processes. You may also be able to provide better APIs depending if you are expecting a diff/code or a chat conversation? Perhaps you could ask the model to not bother sending text when all you want is code.

Multi-message conversations (question about whether the messages state should live on server or client - probably client)

It depends, do you want the AI state to be shared across all users or not?

[jonastemplestein]

Thanks for having a look! I really appreciate it 🙏 I'll keep plugging away at this, as I'm learning a lot and keen to have such a feature in livebook.

If this is something you'd be interested in launching in livebook, I'm happy to help with that effort or share what I've learned so far.

I agree the code generation isn't replacing human engineers at the moment, but it can give them more leverage - especially when working in a new stack, as lots of people coming to elixir via livebook are. And then, in a year or two when there's a more powerful model, you can just plug that in.

Perhaps one suggestion is to change the Finch streaming to, instead of it streaming the tokens, have it already stream only the parts that you care about

Thanks, will have a look - we can strip out the various bits of JSON before it hits a process boundary, but we do need to stream non-code tokens as we need them for back-and-forth conversations

Multi-message conversations (question about whether the messages state should live on server or client - probably client)

It depends, do you want the AI state to be shared across all users or not?

Oh, I don't think it makes sense for other users to see these inline "conversations". Although for a code-interpreter like outer loop that lives in a separate panel and creates multiple cells, maybe.

The "server" solution I was thinking of was just to keep the fully agent responses in the liveview process. This would save having to send the full response back to the client (who only needs to show the code)

I think it makes most sense to send everything to the client

[jonastemplestein]

Actually, one more question if I may...

The biggest design decision I go back and forth on is whether it would be better to implement this as a live component as I had originally done. It'd make it faster to build the UI and add basic interactions, loading states, etc without hand-writing a bunch of fiddly javascript - especially once I add more complex UI states for back-and-forth conversations

But the downside of the live component approach was that there was overall more code in more different places, as I still need a somewhat beefy JS class to deal with the monaco API and the token streaming (which was hard to do efficiently via live component assigns)

What's your view on this? Or maybe it doesn't really matter

[josevalim]

I don't think the difference between LV and LC is going to matter on how much JS you need to write. LC can still be useful to decouple large parts of the code and logic... but at this point I would stick more with "it doesn't really matter". Once you need to pass the notebook as context, then maybe you will have more reasons to push towards one direction or the other.

Btw, did you have any feedback about training ChatGPT or Claude on the Elixir docs?

[jonastemplestein]

Btw, did you have any feedback about training ChatGPT or Claude on the Elixir docs?

I did, yes! I just got home yesterday and was going to write it up. What's the best way to share with interested parties in a less public forum?

[josevalim]

Email me on my GitHub email and I can share with the Livebook team if that's ok. :) Thank you!