Enhance Kedro Deployment

[DimedS]

Overview

This parent issue tracks our ongoing efforts to improve Kedro deployment. Based on user research, we aim to address key challenges by enhancing plugins, refining documentation, and developing new features to better support our community's deployment needs.

Research Initiatives

We began our research in October 2024 with a user survey and follow-up interviews:

• Survey: We received 43 responses (https://www.surveys.online/jfe/form/SV_8pGfndbdtbrfbaS).
• User Interviews: We conducted 11 in-depth interviews, recorded and analysed using Dovetail.
• Synthesis and Findings: All insights and synthesis are consolidated on the Kedro Miro Board. Research Playback: Deployment: UX Research Playback #4325.

Key Insights and Challenges

1. Plugin Compatibility: Users relying on Kedro's connection plugins for third-party platforms face outdated or compatibility issues, making the conversion of Kedro nodes into platform components challenging and leading them to seek alternative solutions. Improve Third-Party Deployment Plugins Reliability and Compatibility #4318
2. Node Grouping Functionality: Users value merging multiple nodes into a single task on the deployment platform for clarity and efficiency, but current plugins provides limited functionality. Improve Node Grouping in Kedro Deployment #4319
3. Kedro-Databricks Integration: Users deploy Kedro projects on Databricks in two ways: longer methods that generate a .whl file on DBFS, and quicker methods that make project code directly accessible in Databricks repo with the options of running in notebooks.
4. Support for Online Inference: Users are increasingly seeking to deploy online inference pipelines (such as LLM calls) in isolated environments for real-time predictions; however, Kedro offers limited support for this functionality.
5. Container Deployment Efficiency: Users often deploy with Docker images, but for larger projects, using a single container for the entire project can be inefficient.

Next Steps

We will continue to address these insights through targeted improvements and new feature development. This issue will track the progress of all related tasks and discussions, with updates and deliverables shared as they are completed.

Feel free to contribute, discuss, or raise additional concerns related to Kedro deployment in the comments below.