HEIM

[teetone]

New Scenarios, Models and Metrics for HEIM: https://crfm.stanford.edu/heim/latest/

[yifanmai]

Partial review only. My main objections:

Blocking:

• Dependency weight - It would be good if we could reduce the required dependency weight further (i.e. remove the google dependencies, which are unneeded for text)
• CI weight - increasing test time by 3x is not acceptable.
• Type checking - there's no type checking to prevent image requests from being sent to text models or vice versa. There's also no type checking to prevent RunSpecs from being composed in nonsensical ways (text metrics on image models) though admittedly this is an existing issue.
• JSON schema extension - The JSON schemas are modified in backward incompatible ways. Additionally, these causes breaks round-trip serialization for certain JSON objects.
• Adapter / request / response extensibility - I would like a proposal that allows extending to model with different type signatures i.e. text / image input, text / image output (or better, N inputs and N outputs, where each of N can be a different type, and there can be multiple input / outputs of the same type). At the bare minimum, this should support CLIP-style models.

Non-blocking:

• This forks the frontend. We want to reconcile it eventually.
• Documentation - there is zero documentation. Additionally, this overrides HELM documentation in undesirable ways (e.g. overriding README)
• This embeds the images as base64 strings, in the JSON, which is maybe not great (I need to think about this more)

[yifanmai]

This causes Request to not round-trip to JSON - see the comments under TextToImageAdapterSpec.

[teetone]

Closing in favor of the fork: https://github.com/stanford-crfm/heim.

[yifanmai]

Thanks Tony!

More rationale for fork discussed offline, besides the blockers already discussed above:

• Little cross interaction between the HEIM code and the HELM code - HEIM users will rarely touch the HELM models and vice versa.
• You cannot pip install HEIM from PyPI and have it work out of the box, because some of its dependences are installed from GitHub rather than PyPI. PyPI packages are only allowed to have PyPI dependencies. So a user would have to manually install the HEIM dependencies anyway.