Fix Monorepo Setup & Separate by Role

[arpitjasa-db]

Changes

This PR makes multiple changes in line with the Separate by Role Project Proposal to separate CICD generation from project generation to allow monorepos to work and making the division more intuitive:

• Add a new parameter input_setup_cicd_and_project that will allow three parameters: CICD_and_Project (the current default flow), Project_Only which sets up only project files, and CICD_Only which only sets up the root directory and provides a workflow to generate CI/CD files for each project in the directory.
• Update CI/CD workflows pull requests to only trigger if files within respective project folder are modified in a PR.
• Update docs to clarify how to set this up.

Fixes #122

Also includes various quality-of-life improvements and minor bug fixes such as constant config versions, clean-up of unused variables, and proper file generation.

Tests

Project Generation Workflow:

CICD Zip Generation Workflow

[mingyu89]

The approach of copying the CICD files of the first project will not work well because it assumes that all the projects have the same CICD platform and feature store selections, which may not be the case.

[arpitjasa-db]

Removing these links since var.git_repo_url has been broken for a while, and linking to these jobs does not merit providing databricks_prod_workspace_host as a variable.

[niall-turbitt]

testing this out and have taken the following workflow:

• create a project with cicd plus project (intending to make this a monorepo)
• add second project, using project_only option

All works fine at this stage. I have a mono repo with 2 projects and CI/CD for the first project. I now want to add the CI/CD pipelines for the second project, and run through initialization again, selecting CICD_only, and setting the input_root_dir to the monorepo project root dir

• I get an error at this point, that there is already a .gitignore file
  
• I try deleting this gitignore file and try the same setup, but this time get an error that the readme exists
  

What should be the intended workflow to add the CI/CD pipelines for a second project in a monorepo?

[niall-turbitt]

I'm getting this fs_param.json file generated for projects, when feature store is not selected:

Can we remove this?

[arpitjasa-db]

I'm getting this fs_param.json file generated for projects, when feature store is not selected:

Can we remove this?

Yeah good question, we could remove it and check for the presence/absence of the file instead in our workflows (but that does complicate the code a bit). Or we can rename is to project_params.json. What would be better?

[niall-turbitt]

in the generated top level README, it looks like we're generated an additional ( under the Setting up CI/CD section

[mingyu89]

I did a first pass and didn't see any major issue. This is indeed a complex PR.

The preview comments has been addressed

[mingyu89]

This is really a huge improvement of the CUJs.

[niall-turbitt]

I'm getting this fs_param.json file generated for projects, when feature store is not selected:
Can we remove this?

Yeah good question, we could remove it and check for the presence/absence of the file instead in our workflows (but that does complicate the code a bit). Or we can rename is to project_params.json. What would be better?

project_params.json seems like a good option

[niall-turbitt]

couple of minor things that need to be fixed in order to get the deploy cicd pipeline running for Azure DevOps. Aside from those, looks good. Thanks for this @arpitjasa-db , appreciate this was a tricky one to work through!

Set up a mono repo to test here https://dev.azure.com/niallturbitt/_git/monorepo-mlops-stacks - have added you as an admin Arpit

[niall-turbitt]

just catching this. For the second project I generated there were no staging or prod profiles defined in the databricks.yaml file. Understand the reasoning for this; a data scientist adding a second project may not know up front the staging/prod workspace URLs.

What this means though, is that when the CI/CD pipelines are triggered for the second project, there is not staging/prod profiles defined for the bundles CI/CD pipeline to use

We either need to somehow automate the addition of staging/test profiles to the databricks.yaml file of the new project whenever we are running the deploy cicd pipeline for the new project. Or direct the user to do so somewhere in documentation

[arpitjasa-db]

just catching this. For the second project I generated there were no staging or prod profiles defined in the databricks.yaml file. Understand the reasoning for this; a data scientist adding a second project may not know up front the staging/prod workspace URLs.

What this means though, is that when the CI/CD pipelines are triggered for the second project, there is not staging/prod profiles defined for the bundles CI/CD pipeline to use

We either need to somehow automate the addition of staging/test profiles to the databricks.yaml file of the new project whenever we are running the deploy cicd pipeline for the new project. Or direct the user to do so somewhere in documentation

@niall-turbitt I actually added the profile generation as part of the deploy CICD pipeline. Did it not work?

[arpitjasa-db]

@niall-turbitt I actually added the profile generation as part of the deploy CICD pipeline. Did it not work?

Ah actually I see we forgot to include that file when we did git add. It should be fixed now! Also made the other changes you had in the monorepo I think. Would you mind taking a final pass @niall-turbitt and see if all the issues you found were fixed? Thank you!

[niall-turbitt]

this is going to be annoying with Azure DevOps...

In order to run the Push CICD Bundle to a New Branch step, we will need to enable the project Build Service to be able to contribute and create a branch for the project. i.e when the deploy CI/CD pipeline is triggered, the build service that runs this pipeline needs the necessary permissions to be able to push

To do this, you need to do the following: go to Project Settings -> Repositories -> Security -> Select <project_name> Build Service under users and set “Contribute”, “Create Branch”, and "Contribute to pull requests" to “Allow”

@arpitjasa-db this will only be applicable to the Monorepo case where you have an instantiate project, have added a second project and then want to set up CI/CD for this second project. I would suggest we add this to the Configure CI/CD - Azure DevOps docs in mlops-setup.md

[niall-turbitt]

I was also hitting issues on this with the push.

Resolved this with : git push --force origin add-cicd-for-${{ parameters.ProjectName }}

[arpitjasa-db]

Bit concerned about adding --force on the off-chance someone has the same branch or has force pushes disabled on their repository. Seems a bit weird to need to do add this flag

[arpitjasa-db]

Added --force-with-lease to prevent the overwriting possibility, and someone can just update the pipeline to remove this if they have force disabled on the whole repo/org

[niall-turbitt]

@arpitjasa-db the above comments I had added but didn't properly publish, so don't think you caught them. I see the refs/heads/feature-branch-pr branch issue has already been resolved.

The only other thing I think needs addressing is this point around documenting the permissions for the build service to be able to run the deploy CICD pipeline successfully

[niall-turbitt]

one other minor nit that I think could be confusing for users is the following in a monorepo setup:

• I am adding a second project to a monorepo, and I specify the project name as project-2 during initialization
• This is added under a folder project_2 in the monorepo
• When running the deploy CICD pipeline to set up the CICD pipelines for project 2 we have to set the name of the project via a param:

parameters:
  - name: ProjectName
    displayName: Name of the project to deploy CI/CD for.
    default: mono-mlops-stacks
    type: string

From the default mono-mlops-stacks I would assume this param should be project-2 to set up for the new project. However this param has to be project_2 to map to the folder

[arpitjasa-db]

one other minor nit that I think could be confusing for users is the following in a monorepo setup:

• I am adding a second project to a monorepo, and I specify the project name as project-2 during initialization
• This is added under a folder project_2 in the monorepo
• When running the deploy CICD pipeline to set up the CICD pipelines for project 2 we have to set the name of the project via a param:

parameters:
  - name: ProjectName
    displayName: Name of the project to deploy CI/CD for.
    default: mono-mlops-stacks
    type: string

From the default mono-mlops-stacks I would assume this param should be project-2 to set up for the new project. However this param has to be project_2 to map to the folder

@niall-turbitt good catch! Added a step in GH and ADO to create a underscore version of the project name and use that for the directory instead so they can continue using the normal name