Skip to content

Latest commit

 

History

History
141 lines (81 loc) · 5.47 KB

homework.md

File metadata and controls

141 lines (81 loc) · 5.47 KB
Raw

Homework

The goal of this homework is to get familiar with MLflow, the tool for experiment tracking and model management.

Q1. Install MLflow

To get started with MLflow you'll need to install the MLflow Python package.

For this we recommend creating a separate Python environment, for example, you can use conda environments, and then install the package there with pip or conda.

Once you installed the package, run the command mlflow --version and check the output.

What's the version that you have?

Q2. Download and preprocess the data

We'll use the Green Taxi Trip Records dataset to predict the duration of each trip.

Download the data for January, February and March 2023 in parquet format from here.

Use the script preprocess_data.py located in the folder homework to preprocess the data.

The script will:

  • load the data from the folder <TAXI_DATA_FOLDER> (the folder where you have downloaded the data),
  • fit a DictVectorizer on the training set (January 2023 data),
  • save the preprocessed datasets and the DictVectorizer to disk.

Your task is to download the datasets and then execute this command:

python preprocess_data.py --raw_data_path <TAXI_DATA_FOLDER> --dest_path ./output

Tip: go to 02-experiment-tracking/homework/ folder before executing the command and change the value of <TAXI_DATA_FOLDER> to the location where you saved the data.

How many files were saved to OUTPUT_FOLDER?

  • 1
  • 3
  • 4
  • 7

Q3. Train a model with autolog

We will train a RandomForestRegressor (from Scikit-Learn) on the taxi dataset.

We have prepared the training script train.py for this exercise, which can be also found in the folder homework.

The script will:

  • load the datasets produced by the previous step,
  • train the model on the training set,
  • calculate the RMSE score on the validation set.

Your task is to modify the script to enable autologging with MLflow, execute the script and then launch the MLflow UI to check that the experiment run was properly tracked.

Tip 1: don't forget to wrap the training code with a with mlflow.start_run(): statement as we showed in the videos.

Tip 2: don't modify the hyperparameters of the model to make sure that the training will finish quickly.

What is the value of the min_samples_split parameter:

  • 2
  • 4
  • 8
  • 10

Q4. Launch the tracking server locally

Now we want to manage the entire lifecycle of our ML model. In this step, you'll need to launch a tracking server. This way we will also have access to the model registry.

Your task is to:

  • launch the tracking server on your local machine,
  • select a SQLite db for the backend store and a folder called artifacts for the artifacts store.

You should keep the tracking server running to work on the next two exercises that use the server.

In addition to backend-store-uri, what else do you need to pass to properly configure the server?

  • default-artifact-root
  • serve-artifacts
  • artifacts-only
  • artifacts-destination

Q5. Tune model hyperparameters

Now let's try to reduce the validation error by tuning the hyperparameters of the RandomForestRegressor using hyperopt. We have prepared the script hpo.py for this exercise.

Your task is to modify the script hpo.py and make sure that the validation RMSE is logged to the tracking server for each run of the hyperparameter optimization (you will need to add a few lines of code to the objective function) and run the script without passing any parameters.

After that, open UI and explore the runs from the experiment called random-forest-hyperopt to answer the question below.

Note: Don't use autologging for this exercise.

The idea is to just log the information that you need to answer the question below, including:

  • the list of hyperparameters that are passed to the objective function during the optimization,
  • the RMSE obtained on the validation set (February 2023 data).

What's the best validation RMSE that you got?

  • 4.817
  • 5.335
  • 5.818
  • 6.336

Q6. Promote the best model to the model registry

The results from the hyperparameter optimization are quite good. So, we can assume that we are ready to test some of these models in production. In this exercise, you'll promote the best model to the model registry. We have prepared a script called register_model.py, which will check the results from the previous step and select the top 5 runs. After that, it will calculate the RMSE of those models on the test set (March 2023 data) and save the results to a new experiment called random-forest-best-models.

Your task is to update the script register_model.py so that it selects the model with the lowest RMSE on the test set and registers it to the model registry.

Tip 1: you can use the method search_runs from the MlflowClient to get the model with the lowest RMSE,

Tip 2: to register the model you can use the method mlflow.register_model and you will need to pass the right model_uri in the form of a string that looks like this: "runs:/<RUN_ID>/model", and the name of the model (make sure to choose a good one!).

What is the test RMSE of the best model?

  • 5.060
  • 5.567
  • 6.061
  • 6.568

Submit the results