Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Introduce additional folder layer under org folder #42

Open
seymen opened this issue Aug 23, 2018 · 14 comments
Open

Introduce additional folder layer under org folder #42

seymen opened this issue Aug 23, 2018 · 14 comments
Assignees

Comments

@seymen
Copy link

seymen commented Aug 23, 2018

Problem: In a multiple-org scenario (e.g. non-prod and prod orgs), it is not possible to manage org level entities separately per org.

Current folder structure:

org
|_ apiproducts.json

Suggestion is to introduce an additional folder layer under org:

org
|_ foo-non-prod
        |_ apiproducts.json
|_ foo-prod
        |_ apiproducts.json
@ssvaidyanathan
Copy link
Collaborator

@msadasivam - This will have impact to the structure for org under resources, should we release this as 1.3 ?

@msadasivam
Copy link
Member

The plugin doesn't assume "resources/edge" as a default for config.dir.

This was the original thinking. Will this work?

non-prod
-Dapigee.config.dir=resources/myenterprise-nonprod
prod
-Dapigee.config.dir=resources/myenterprise

@ssvaidyanathan
Copy link
Collaborator

Then this is creating duplicate directory structure with all the nested directories

@msadasivam
Copy link
Member

The git structure would reflect the org structure.
resources/myenterprise/prod
resources/myenterprise/staging
resources/myenterprise-nonprod/dev
resources/myenterprise-nonprod/uat
resources/myenterprise-nonprod/test

This structure will hold the config for each env organized by orgs. The APIs will not live in this structure .

@ssvaidyanathan
Copy link
Collaborator

Then in that case, I cannot use the same maven command to install org and env, api configurations. I will have to run it separately and also maintain separate directories

@msadasivam
Copy link
Member

Yes, APIs and config will be 2 different mvn commands. The "Further organization for even larger config files" section in this page has this structure documented.
https://github.com/apigee/apigee-config-maven-plugin/wiki/Organize-config-in-a-large-project
Expect this to be already in a fairly complex situation.

@ssvaidyanathan ssvaidyanathan self-assigned this Sep 21, 2018
@ssvaidyanathan
Copy link
Collaborator

should we go ahead with this change or go ahead with the wiki link provided ?

@gsjurseth
Copy link

Why not have something like an org-overrides-folder property wherein you drop those files that override anything that may otherwise be picked up... So for apiProducts you could have:

config/org/apiProducts.json

For the default but then

org-overrides/myFancyShmancyOrg/apiProducts.json

For an override if we're working on the org named: myFancyShmancyOrg.

This way we don't have to duplicate the entire structure and create a place where we're managing the same data more than once.

@msadasivam
Copy link
Member

@gsjurseth interesting idea, can we use this one. This is probably worth a separate thread. #50

@msadasivam
Copy link
Member

@seymen what has been your experience so far on the current design and alternative ? do we have enough to attempt a revamp.

@karsten-wagner
Copy link

All, I am not sure if this is still open or closed by one of the contributions since this issue opened...
I feel like we are successfully using a multi-file config setup that covers all organizations and all environments we have.

The folder structure typically looks like this:

repository-root
├── proxy-1                             <-- This allows multiple proxies/sharedflows in a single repository when needed
│   ├── apiproxy                        <-- This is the part for the apigee-edge-maven-plugin
│   └── config                          <-- This is the root folder for all configuration
│       ├── org-1                       <-- Ideally name the same way as your organization technical name
│       │   ├── org
│       │   │   ├── apiproducts.json
│       │   │   ├── developers.json
│       │   │   └── apps.json
│       │   └── env
│       │       ├── env-1               <-- Ideally name the same way as your environment technical name
│       │       │   └── kvms.json
│       │       └── env-2
│       │           └── kvms.json
│       └── org-2
│           ├── org
│           │   └── apiproducts.json
│           └── env
│               └── env-3
│                   └── kvms.json
└── proxy-2
    └── ...

Our shared POM covers the case like this:

<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    ...
    <build>
        <plugins>
            <plugin><artifactId>maven-resources-plugin</artifactId> ... <!-- Plugin config --></plugin>
            <plugin><artifactId>apigee-edge-maven-plugin</artifactId> ... <!-- Plugin config as per examples in this repo --></plugin>
            <plugin><artifactId>apigee-config-maven-plugin</artifactId> ... <!-- Plugin config as per examples in this repo --></plugin>
        </plugins>
    </build>
    <!-- Environment specific properties -->
    <profiles>
        <profile>
            <id>env-1</id> <!-- We have defined [profile id = environment name] to ease up command following below -->
            <properties>
                <apigee.org>org-1</apigee.org> <!-- org-1 from above folder structure -->
                <apigee.env>env-1</apigee.env> <!-- env-1 from above folder structure -->
                <apigee.profile>${apigee.env}</apigee.profile> <!-- Not clear where this comes from, but is included in all examples - works well like this... -->
                <apigee.config.options>none</apigee.config.options> <!-- Default no-op setting, usually overwritten as command line parameter -->
                <apigee.config.dir>${project.basedir}/config/${apigee.org}</apigee.config.dir> <!-- Automatically derived from above described folder structure by naming convention, can be overwritten as command line parameter -->
                ... <!-- Some other parameters for the deploy plugin and OAuth based token generation -->
            </properties>
        </profile>
        ... <!-- Other environments, usually copy & paste from above profile with only different org/env parameters -->
    </profiles>
</project>

To deploy we execute below command (a shell script with some auto-replacement parameters):

mvn clean install -P env-1 \
  -D "apigee.config.options=create" -D "username=$(git config user.email)" -D "bearer=$(get_token)"

The first line is the bare minimum part of the command, the second line shows how to overwrite some of the parameters defined in the shared POM on the command line.
For convenience we are reading user name from git config and use the "get_token" script to login - this can be replaced by username/password, but with above way the shell script is reusable and we do not have passwords in shell history...

As a final hint - we are uploading the shared POM to our companies Maven repository/artifactory and use it as a dependency in the local proxie's POM.

@ssvaidyanathan
Copy link
Collaborator

ssvaidyanathan commented Jul 28, 2020

Thanks @karsten-wagner - Yes, that's been our general recommendation as well. The issue is still opened as there were some conversations/discussion on what to do. But in general, what you suggested is what I have recommended to others as well.

Just curious - you have multiple proxies in the same repo. If only proxy2 was modified, how is your pipeline tracking that deploying that alone? That has been challenging (looking at the git log and then triggering the build), so wanted to check with you. The easiest option is to deploy all the proxies in that repo but thats not a scalable approach.

@karsten-wagner
Copy link

@ssvaidyanathan, you are right, during development stage we deploy multiple proxies on every commit. Having multiple in the same repository is usually the case when we have a very close link between them.
As an example we have a shared flow with a re-usable feature and a second shared flow which does the error handling for the feature. Usually there is a strong indicator both need to change at the same time.

Once we tag the changes as a ready "release", we are packaging the repository as an artifact to allow later roll back to a previous version if needed. That package is then "deployed" to the target organization/environment via a separate job in CI/CD. During that second phase we can select to deploy only code changes of parts of that package (e.g. single proxy/flow, or just overwrite the products if desired).

@ssvaidyanathan
Copy link
Collaborator

Sounds good.. Hence I recommend one repo per proxy.. so you get the flexibility.. Thanks for getting back...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants