Allow relative paths outside of the garden.yaml root

[kvokka]

Feature request

Facing the #1190 issue, I do not have dirty option. Lack of this ability block the entire integration. In the future a developer may find some other situations, when it is required.

All usage of ../ in path is prohibited, even when i know, what and why i'm doing it.

I think some project-wide flag for this may be very handy. Of course, you will write that it is a bad practice and developer should avoid it, but it may keep "show go on".

[edvald]

Understood. The reasons for not allowing paths outside the project are indeed partly to avoid "bad practices" but are also technical. We need to be able to assert that any referenced paths are within git repos, and in most cases in the same repo as the project. This becomes much more difficult when we allow arbitrary paths. For modules, staging builds also becomes difficult if one can reference arbitrary paths.

Which field(s) were you trying to set this way?

[kvokka]

I understand, that allowing this is bad practice from the very beginning. But in other hand, developer need some last resort tool.

Given the project root is dev/garden.yaml
And service1 root is dev/core/garden.yaml
When dev/support/core/garden.yaml with:

---
kind: Module
description: Image for core
type: container
name: poker-core
dockerfile: ../../docker/development/Dockerfile
---
kind: Module
description: Chart for core
type: helm
name: core
chartPath: ../../charts/core
...

I tried to use some external configuration path, which is actually impossible.

Please, understand me right, I'm totally agree that it is a bad practice. But uglifying is better than blocking.

Maybe switch to #1201 ? This should allow to add some configs to the root. Not very pretty, but should work for a while.

[edvald]

I can look into both options. There are several edge-cases we need to consider, especially as they relate to in-cluster building, but either or both should be feasible.

[ox-michaelradionov]

I am not sure if I have the same issue, but I've just started trying out Garden and have some issues with relative paths. I have a set of microservices, each in it's own repository, and a separate repository with Kubernetes manifests. All of them are fetched to the same directory resulting in the following dir structure:

- manifests/
- microservice1/
- microservice2/

If I am following Garden requirements correctly, I must have a Project which can then reference Modules. I've created a "garden.yml" with Project in "manifests/" folder, as it used to be my root for deploying the application, and also created "garden.yml" for each microservice of "Module" kind.

- manifests/
   - garden.yml             # kind: Project
- microservice1/
   - garden.yml             # kind: Module
- microservice2/
   - garden.yml             # kind: Module

In order for Project to find Modules, I've added modules.include to "manifests/garden.yml"

kind: Project
name: my-app
modules:
  include:
    - ../microservice1
    - ../microservice2

When running $ garden scan I am getting the following error

Error validating project (/garden.yml): key .modules.include[0] must be a relative sub-path (may not contain '..' segments or be an absolute path)

I know that my project folder structure is not perfect because it relies on all repos being in the same folder and having exact names. Maybe you could advise me how to restructure it? I don't really want to move all my microservices to a monorepo; creating a root repo with Git submodules for each microservice seems a bit strange too.

Currently I use Skaffold to deploy my app, it works with "../" relative paths, but I liked some Garden features and wanted to try it out as a possible replacement.

[kvokka]

The option which worked for me, was garden link for the sub projects.
The folders structure does not matter then.

[ox-michaelradionov]

Thanks, I skipped this feature because it seems like linking is not reflected in configuration and therefore can't be shared, if I am not mistaken we will have to run these commands for each of the microservices manually to be able to freely work on the project.

While re-reading linking now I also noticed a section above it where I can setup a source to point to a local folder instead of a remote repository, but it has path limitations too.

When I use

kind: Project
name: my-app
sources:
  - name: microservice1
    repositoryUrl: file:../microservice1#master

and run $ garden scan, git clone fails, because file: protocol does not support relative path.

Downloading remote project failed with error:

Error: Command failed with exit code 128 (Unknown system error -128)
git clone --recursive --depth=1 --branch=master file:../microservice1 /var/www/xxx/yyy/manifests/.garden/sources/project/microservice1--c06e288ea7

When I remove the protocol and only keep a relative path, which should now work with Git, then Garden rejects it, because it wants a path to start with a valid protocol:

kind: Project
name: my-app
sources:
  - name: microservice1
    repositoryUrl: ../microservice1#master

Error validating project (/garden.yml): key .sources[0][repositoryUrl] must be a valid Git repository URL, key .sources
[0][repositoryUrl] must be a valid uri with a scheme matching the file pattern

[kvokka]

I've been there. And I had the very similar issue with almost the same problems (it was about 1 week ago).

I also do not like idea of linking as it is implemented- it pushing to create something external, which will describe the folder structure and link everything. There are options how it can be improved, but anyway, now it does not looks convenient.

It was not possible to use it with relative path, since it was outside of the repo or with remote path.
So the repository owner suggests me to use link mechanism, and described, that at the moment it is an only way to manage other repos (and it does not matter whether it is git-submodules or just other repos).

After a lot of tries with Garden had to switch to Tilt (anyway the project which should be moved to k8s requires some common adjustments with any tool, so it is simpler to stick with more straightforward solution for now). Will try with Garden later.

[ox-michaelradionov]

Yeah, seems like this area of tools for simplifying development workflows with Kubernetes is still under active development, there are plenty of them, but all of them have different features and slightly different approach and priorities. Skaffold is missing multi-module setup and dependency resolution, it only works with single configuration file, which is why I was looking at Garden, to be able to move k8s manifests closer to each microservice. There is a similar project Devspace, but it is also missing some features that I require, like renaming images to append dynamic Docker registry, which is available in Skaffold and Garden. Tilt looks interesting, I've never seen it, thanks for suggestion, but unfortunately it does not have dependency management and it's not worth switching without getting any new benefits in comparison to my current setup.

And I just caught myself thinking that maybe I should design my application the way that it won't depend on these tools that hard and use them to simply speedup the development workflows, but not to build on top of them like it's a must-have. So if I have some dependencies between services, I maybe should figure them out in K8s instead of having this logic defined in a developer tool config.

[kvokka]

Anyway, when you develop something in different services you'll have to deal with some integration problems. Does not matter how you design all services - they must be able to communicate with each other. And exactly this step was the point to improve for me, because not it is a pain to debug some integration issues or even test it. As a bonus- simpler setup and reinstall of the entire stack.

Tilt does not manage th dependencies, but it run everything idempotently. So, you can push a tilt config to each service, and then include all this configs to 1 global. this will provide the simple order.
You have to use init containers anyway, for dev and for prod, so let dockerize do its job.
i tried it out with a few services already, and it still looks promising.

[edvald]

This is an interesting design challenge that I'd say we're still working on, essentially trying to make working with multiple repos more intuitive and smooth. We have some ideas that I think could be helpful in that regard, I can post or link them here once we've fleshed them out a bit.

The challenge is in making something portable and consistent, so that the Garden configuration isn't dependent on some particular external folder structure, but to still have it friendly and intuitive to use. Our current implementation works quite well for deploying outside code, but isn't as smooth as we'd like when you're working on all those repositories yourself.

And to your particular setup @ox-michaelradionov, you're right, garden link is inherently not codified in configuration. The notion is that the referenced git repo is the canonical reference, and then each developer can have whichever folder structure that suits them. The downside is that there are imperative steps to get the whole thing set up for each user.

All that said, there may be a simple thing we can add to configs, which is simply to allow overriding the project/module root (which is currently always the location of the config file itself). Allowing relative references outside of project and module roots would be much more complicated, because it'd break assumptions and logic in a number of places, but I don't see any issues with detaching the root location of both projects and modules from the location of the config file. We'll dig into this and post back soon.

[kvokka]

@edvald Maybe just add some paths file, which, will like everything garden needs before execution and unlink after?
Do this optional, so when the developer want to leave folder structure flexible, it means that he/she should be link everything manually, or use some special file (or even special settings).

It looks like it can require minimal changes. What do you think about it?

[edvald]

We had something very similar in mind, basically adding an optional localPath field when you specify remote sources. If that's specified, we automatically do the linking for you, no extra steps involved. So basically the same idea, except we add an option to existing configs instead of configuring it in separate files. Not too hard to implement, and would really help the UX I think.

[kvokka]

Great! It can really help.

[ox-michaelradionov]

It reminds me of Devspace I've tried recently, where they allow to set a path to local repository using path option, instead of remote repo ( https://devspace.cloud/docs/cli/configuration/reference#dependencies ). Dependency in their terminology is a Module in your terminology. And there is a warning that you can't use both local path and remote repo URL at the same time.

[edvald]

That's a good reference, thank you! We'll likely approach it a bit differently and allow localPath as an optional addition to the remote URL, which is a little more complex to implement, but I think is a suitably complete solution. That way the build should always work without any additional input, which is important for CI, but still automate away some of that garden link hassle when you're developing.

[PavelvKalinin]

any updates?

[edvald]

Hey @PavelvKalinin, sorry no updates at the moment, except that we've been doing more of a deep dive into the broader issue #1477. We're still deciding what exactly to start on, but this is a somewhat tricky subject so we're being a little cautious.

[stale]

This issue has been automatically marked as stale because it hasn't had any activity in 60 days. It will be closed in 14 days if no further activity occurs (e.g. changing labels, comments, commits, etc.). Please feel free to remove the label if you think it doesn't apply. Thank you for submitting this issue and helping make Garden a better product!