Here is what you will learn as part of this chapter:
- Building a training set from a feature table
- Baselining with AutoML
- Tracking experiments with MLflow
- Classifying beyond the basic
- Integrating innovation
- Applying our learning
Here are the technical requirements needed to complete the hands-on examples in this chapter:
- In order to use the OpenAI API, you need to set up a paid account.
- For our LLM model, we will integrate with the ChatGPT model from OpenAI. You will need an API Key and install the OpenAI Python library
- We use the SQLAlchemy Dialect for Databricks workspace and sql analytics clusters using the officially supported databricks-sql-connector dbapi.
In the chapter
- MLFlow Tracking
- MLFlow Model flavors
- Introducing the Spark PyTorch Distributor
- Data & AI Summit 2023: Generative AI at Scale Using GAN and Stable Diffusion
- New Expert-Led Large Language Models (LLMs) Courses on edX
- OpenAI
Further Reading
- Introducing AI Functions: Integrating Large Language Models with Databricks SQL
- Deploy Your LLM Chatbot With Retrieval Augmented Generation (RAG), llama2-70B (MosaicML inferences) and Vector Search
- Best Practices for LLM Evaluation of RAG Applications
- Unifying Your Data Ecosystem with Delta Lake Integration
- Reading and Writing from and to Delta Lake from non-Databricks platforms
- Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM
- Ray 2.3 release (PyPI)
- Ray on Spark Databricks docs
- Announcing Ray support on Databricks and Apache Spark Clusters Blog post
- Ray docs
- Databricks Blog: Best Practices for LLM Evaluation of RAG Applications