Draft: support faster inference methods

[pszemraj]

support faster and advanced inference methods

• torch.compile
• optimum onnx
• use fire for the CLI

[pszemraj]

after further thought I am going to update the UI/app in a later round of improvements bc I don't have time for it these days

[pszemraj]

changes look fine from me self-reviewing what I did.. will clone into a colab and test basics to make sure

[lefnire]

This would be awesome! Does it handle the conversion of a huggingface model to optimized onnx pretty automatically? Or do you need to convert manually first, and it supports loading in an onnx model?

[pszemraj]

thanks for your interest ☺️ I will try and have it merged this weekend latest

Does it handle the conversion of a huggingface model to optimized onnx pretty automatically?

yep it's pretty automatic (for ONNX runtime at least). Basically the same as ONNX models for inference.

One caveat though - this PR "enables" ONNX support, but doesn't mean that I have gone through and validated that ONNX inference itself is fine vs. standard, so any issues with the 'base' conversation/support in ONNX would still show up here. My take is that there may be some issues that pop up and need to be resolved at the source ONNX code, simply because ONNX inference with long-context models probably wasn't happening much in the past. For example: LongT5 is supported officially by ONNX, but when testing with long-t5-base I find that the last batch of tokens I run inference with results in like .. 20% of the decoded letters being the letter "d" for seemingly no reason?? Earlier batches are fine, so.. YMMV and test/validate before using.

I'll try and add a note somewhere about that

[pszemraj]

#9 will now be handled by this

[pszemraj]

oook finally happy with it