Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FISH-788 Support sub-directories for MPCONFIG SecretDirConfigSource. #5006 #5007

Merged
merged 23 commits into from
Jan 28, 2021

Conversation

poikilotherm
Copy link
Contributor

@poikilotherm poikilotherm commented Nov 25, 2020

Description

Closes #5006

This adds support for mounting your secrets (or configmapped values) in subdirectories to create scopes.

The relative path is used as the property name. A secret file SECRET_DIR/foo/bar/secret
will be available as MPCONFIG value foo.bar.secret.

You can still put a file foo.bar.secret in SECRET_DIR and have the same value available.
Any combination of dots and directories is supported, also the most specific (read: most subdirs and least dots in file and dir names) will win. Updates during runtime are respected and evaluated with longest match, too. An example: SECRET_DIR/foo/bar.secret or SECRET_DIR/foo.bar/secret` is fine, too.

Important Info

Blockers

None.

Testing

New tests

Included.

Testing Performed

Run tests in SecretsDirConfigSourceTest

Testing Environment

  • AdoptOpenJDK (build 11.0.9+11)
  • Linux, Fedora 33
  • Maven 3.6.3

Documentation

See payara/Payara-Community-Documentation#114

Notes for Reviewers

None.

This adds support for mounting your secrets in subdirectories to create scopes.
The relative path is used as the property name. A secret file "SECRET_DIR/foo/bar/secret"
will be available as MPCONFIG value "foo.bar.secret".

You can still put a file "foo.bar.secret" in "SECRET_DIR" and have the same value available.
If both are present, the topmost file will be picked.

Mix'n'match isn't supported. Retrieving "foo.bar.secret" from "SECRET_DIR/foo/bar.secret"
will not work reliably when added at runtime.
@AlanRoth AlanRoth changed the title Support sub-directories for MPCONFIG SecretDirConfigSource. #5006 FISH-788 Support sub-directories for MPCONFIG SecretDirConfigSource. #5006 Nov 25, 2020
@AlanRoth AlanRoth added PR: Awaiting CLA Contributor does not have a CLA or has submitted an unconfirmed CLA. Type: Community Contribution labels Nov 25, 2020
@poikilotherm
Copy link
Contributor Author

poikilotherm commented Nov 25, 2020

@AlanRoth IIRC I signed the CLA already when I contributed to the former https://github.com/payara/docker-payaraserver-full repo.
Although I would love more of that Payara swag 😸

@AlanRoth
Copy link

Ah, indeed you have. My apologies.

@AlanRoth AlanRoth added PR: CLA CLA submitted on PR by the contributor and removed PR: Awaiting CLA Contributor does not have a CLA or has submitted an unconfirmed CLA. labels Nov 25, 2020
@AlanRoth
Copy link

jenkins test please

@AlanRoth AlanRoth requested a review from jbee November 26, 2020 13:21
Copy link
Contributor

@jbee jbee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @poikilotherm, I think this is a good initiative that needs to mature a little bit more.

As it is I find it confusing to reason about since the moving parts are checked and manipulated in many places. Many of the flaws I pointed out did not start with your changes but making the essential logic more complex really highlights that this class needs some more attention.

First of all what consistency do we really want to support? Are files allowed to be updated at runtime? If so what is the observable effect? What about misses and removed entries? And finally collisions.

As far as I understood you want to handle collisions as first (on time axis) found wins until eventually that very source would no longer hold the property which would open for other files to supply it? What about if two files from the very start contain same property? Which one wins? Do we make sure we process the files in the correct order for this effect?

@AlanRoth
Copy link

jenkins test please

@poikilotherm
Copy link
Contributor Author

poikilotherm commented Nov 27, 2020

Thanks @poikilotherm, I think this is a good initiative that needs to mature a little bit more.

Hi @jbee,
thank you for your extensive review 👍

As it is I find it confusing to reason about since the moving parts are checked and manipulated in many places. Many of the flaws I pointed out did not start with your changes but making the essential logic more complex really highlights that this class needs some more attention.

Absolutely. For starters: should we rename this class to ConfigDirSource to be consistent with docs? In 5.184 the asadmin command changed to what it is now, docs weren't updated and this class sticked with the old name. Maybe it's time for a change...

First of all what consistency do we really want to support? Are files allowed to be updated at runtime? If so what is the observable effect?

Files updated at runtime definitely is a must-have.
See https://kubernetes.io/docs/concepts/configuration/secret/#mounted-secrets-are-updated-automatically

What about misses and removed entries?

Removed entries definitely is a problem. As the code is working now, removed entries are still cached (and served). On second thought, that might be undesireable in case someone actually tries to "move" a config value (it's not necessarily a secret, also this is the primary use case) to another source with lower ordinal.

And finally collisions.
As far as I understood you want to handle collisions as first (on time axis) found wins until eventually that very source would no longer hold the property which would open for other files to supply it? What about if two files from the very start contain same property? Which one wins? Do we make sure we process the files in the correct order for this effect?

Collisions can happen in one scenario only: someone creates a file CONFIG_DIR/foo.bar.test and a folder structure with CONFIG_DIR/foo/bar/test. This case is handled in

// --> the list of possible paths is used "first match, first serve".
List<Path> paths = Arrays.asList(Paths.get(secretsDir.toString(), property),
Paths.get(secretsDir.toString(), property.replace('.', File.separatorChar)));

First match wins, so someone using the source as available since v5.183 experiences the same behaviour, maintaining backward compatibility. I'd be happy to remove this and allow for the new subdir style only (so no more files with dots in flat file hierarchy).
See also my added doc bits in payara/Payara-Community-Documentation#114

One edge case I didn't look at so far: it might be necessary to add support for ignoring file extensions. Someone might mount a file with a Docker volume as "property.txt", which could not be found. Let me know if you think this is a thing.

There is a small chance for this slippery edge case to make unexpected behaviour happen: someone adding a file to CONFIG_DIR/foo/bar.test on application server start would make that file available as foo.bar.test, as the file already contains a dot, which is not removed. This cannot happen when added after server start, trying to read the property file on cache miss, as the property name is transformed with all dots replaced to a file path.

@poikilotherm
Copy link
Contributor Author

poikilotherm commented Nov 27, 2020

Small addition: we might also want to check that the file we are reading is not bigger than 512KB (which would already be quite huge for text configs) to prevent loading binary files etc by accident.

@jbee
Copy link
Contributor

jbee commented Nov 27, 2020

@poikilotherm Good points. After getting more familiar with the case I think we should opt for the file watcher to update the map "in the background" while lookups are simply served from the map with no file IO checks. Map has entries with 3 fields. If it is in the map it is returned, if not we have no such property. When file watcher calls back for file changes we check all possible variations of the file for existence (dots or path variants) and always apply the same rule which one wins. This gives us "immediate update" while also always reflecting the same logic for what file wins even in case a new file is added at runtime that wins over a file existing prior to it. Does that make sense?

As a side benefit we do get much clearer code where some methods only read the map and the file watcher code only updates the map (apart from the initial initialisation).

@poikilotherm
Copy link
Contributor Author

poikilotherm commented Nov 30, 2020

@jbee I fully agree with you. Will update my PR accordingly.

@jbee @AlanRoth as we need a background runner for the file watcher, is there any preferred way of creating one in Payara?

Otherwise I would just go with ExecutorService.

Found PayaraExecutorService, I'll use that.

Looks like PayaraFileWatcher it is. Reverting. Has been added in Micro only.

@poikilotherm
Copy link
Contributor Author

@jbee @AlanRoth I completed a first draft of the rewrite. Please review 😄

I didn't yet refactor the tests, please ignore 'em.

@jbee
Copy link
Contributor

jbee commented Dec 1, 2020

Doing a review but it will take 1-2 more days before I get it done...

Copy link
Contributor

@jbee jbee left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall this is what I was thinking about, looks good. Don't be discouraged by the number of detail suggestions I made. Its mostly missing modifiers. I hope you can improve the initialisation of the source a bit. Maybe follow my suggestion to add an "init" method to add the watcher part so you can use the class with or without?

Ping again if you feel we can do a final review including tests.

@poikilotherm
Copy link
Contributor Author

Thanks @jbee for your extensive review. Glad you like what you see... Will address next week, so this might make it into 5.2020.7 😄

@AlanRoth
Copy link

AlanRoth commented Dec 4, 2020

so this might make it into 5.2020.7 😄

@poikilotherm, 5.2020.8 definitely 😉 5.2020.7 is ready and releasing very soon. Sorry to be a bearer of bad news, but you can take your time!

@poikilotherm
Copy link
Contributor Author

@jbee one more question. There are a lot of occurrences of "secret dir" in class names, functions etc. Should I clean this up in the same go or is this beyond scope?

@AlanRoth
Copy link

AlanRoth commented Dec 4, 2020

If you are able to refactor all occurrences then sure, but I'm not too sure what you mean and what the changes would be?

@pdudits
Copy link
Contributor

pdudits commented Jan 27, 2021

I don't think we should be cutting away any extensions then (oh, I overlooked that part of code!). Windows users will manage without it, and it will be less surprising.

…rConfigSource

As requested by @pdudits, do not cut off file extensions silently.
People on Windows can deal with text files without a file extension,
better stay consistent.

Relates to payara#5006
@poikilotherm
Copy link
Contributor Author

poikilotherm commented Jan 27, 2021

Completely reverted ignoring selective files in ce30d87.
Removed the file extension cutoff in 45d7a33.

@pdudits
Copy link
Contributor

pdudits commented Jan 27, 2021

Jenkins test please

@poikilotherm
Copy link
Contributor Author

So looks like this is getting somewhere.
For whatever reason GitHub didn't let me re-request a review from @jbee

What's the way forward from here? I assume I should create another PR to the docs repo. Anything else I should try or do?

@pdudits
Copy link
Contributor

pdudits commented Jan 27, 2021

Yes, updating the docs is the way to go next. I'll take care of merging this one in.

@AlanRoth
Copy link

Hi @poikilotherm,

Congratulations on getting the PR through! 96 Comments! Our second most commented PR.

Head over to our documentation, here and submit a PR. Let me know if you require any assistance.

Thank you,
Alan

@AlanRoth AlanRoth merged commit ef826d7 into payara:master Jan 28, 2021
@poikilotherm
Copy link
Contributor Author

I love you folks! 🎉
Thank you very much for your support and pushing me to do my best. Learned a lot from this.

party

@poikilotherm poikilotherm deleted the 5006-enhance-secretdir branch January 28, 2021 09:47
poikilotherm added a commit to poikilotherm/Payara-Community-Documentation that referenced this pull request Jan 28, 2021
…nality.

This doc change aligns the docs with the new implementation of the directory
config source for MicroProfile Config API.

- See PR payara/Payara#5007
- Internal ticket FISH-788
@lprimak
Copy link
Contributor

lprimak commented Jan 28, 2021

Another comment (sorry) - does not pass QA:
Causes this error on default installation with every app deploy:

[#|2021-01-28T14:21:49.724-0600|SEVERE|Payara 5.2020.8-SNAPSHOT|fish.payara.nucleus.microprofile.config.source.DirConfigSource|_ThreadID=90;_ThreadName=admin-thread-pool::admin-listener(1);_TimeMillis=1611865309724;_LevelValue=1000;|
  MPCONFIG DirConfigSource: error during setup.
java.io.IOException: Given MPCONFIG directory 'secrets' is no directory, cannot be read or has a leading dot.
	at fish.payara.nucleus.microprofile.config.source.DirConfigSource.findDir(DirConfigSource.java:287)
	at fish.payara.nucleus.microprofile.config.source.DirConfigSource.<init>(DirConfigSource.java:219)
	at fish.payara.nucleus.microprofile.config.spi.ConfigProviderResolverImpl.getDefaultSources(ConfigProviderResolverImpl.java:334)
	at fish.payara.nucleus.microprofile.config.spi.ConfigProviderResolverImpl.getDefaultSources(ConfigProviderResolverImpl.java:370)
	at fish.payara.nucleus.microprofile.config.spi.ConfigProviderResolverImpl.getConfig(ConfigProviderResolverImpl.java:283)
	at fish.payara.nucleus.microprofile.config.spi.ConfigProviderResolverImpl.getConfig(ConfigProviderResolverImpl.java:297)
	at fish.payara.nucleus.microprofile.config.spi.ConfigProviderResolverImpl.getConfig(ConfigProviderResolverImpl.java:206)
	at org.eclipse.microprofile.config.ConfigProvider.getConfig(ConfigProvider.java:91)
	at fish.payara.microprofile.config.activation.ConfigApplicationContainer.start(ConfigApplicationContainer.java:57)
	at org.glassfish.internal.data.EngineRef.start(EngineRef.java:123)
	at org.glassfish.internal.data.ModuleInfo.start(ModuleInfo.java:293)
	at org.glassfish.internal.data.ApplicationInfo.start(ApplicationInfo.java:364)
	at com.sun.enterprise.v3.server.ApplicationLifecycle.initialize(ApplicationLifecycle.java:623)
	at org.glassfish.deployment.admin.DeployCommand.execute(DeployCommand.java:583)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:556)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2$1.run(CommandRunnerImpl.java:552)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$2.execute(CommandRunnerImpl.java:551)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:582)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$3.run(CommandRunnerImpl.java:574)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:360)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:573)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.doCommand(CommandRunnerImpl.java:1497)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl.access$1300(CommandRunnerImpl.java:120)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1879)
	at com.sun.enterprise.v3.admin.CommandRunnerImpl$ExecutionContext.execute(CommandRunnerImpl.java:1755)
	at org.glassfish.admin.rest.utils.ResourceUtil.runCommand(ResourceUtil.java:272)
	at org.glassfish.admin.rest.utils.ResourceUtil.runCommand(ResourceUtil.java:240)
	at org.glassfish.admin.rest.utils.ResourceUtil.runCommand(ResourceUtil.java:294)
	at org.glassfish.admin.rest.resources.TemplateListOfResource.createResource(TemplateListOfResource.java:332)
	at org.glassfish.admin.rest.resources.TemplateListOfResource.post(TemplateListOfResource.java:166)
	at sun.reflect.GeneratedMethodAccessor323.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory.lambda$static$0(ResourceMethodInvocationHandlerFactory.java:52)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:124)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:167)
	at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:176)
	at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:79)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:469)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:391)
	at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:80)
	at org.glassfish.jersey.server.ServerRuntime$1.run(ServerRuntime.java:253)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:248)
	at org.glassfish.jersey.internal.Errors$1.call(Errors.java:244)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:292)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:274)
	at org.glassfish.jersey.internal.Errors.process(Errors.java:244)
	at org.glassfish.jersey.process.internal.RequestScope.runInScope(RequestScope.java:265)
	at org.glassfish.jersey.server.ServerRuntime.process(ServerRuntime.java:232)
	at org.glassfish.jersey.server.ApplicationHandler.handle(ApplicationHandler.java:680)
	at org.glassfish.jersey.grizzly2.httpserver.GrizzlyHttpContainer.service(GrizzlyHttpContainer.java:356)
	at org.glassfish.admin.rest.adapter.RestAdapter$2.service(RestAdapter.java:335)
	at org.glassfish.admin.rest.adapter.RestAdapter.service(RestAdapter.java:189)
	at com.sun.enterprise.v3.services.impl.ContainerMapper$HttpHandlerCallable.call(ContainerMapper.java:520)
	at com.sun.enterprise.v3.services.impl.ContainerMapper.service(ContainerMapper.java:217)
	at org.glassfish.grizzly.http.server.HttpHandler.runService(HttpHandler.java:182)
	at org.glassfish.grizzly.http.server.HttpHandler.doHandle(HttpHandler.java:156)
	at org.glassfish.grizzly.http.server.HttpServerFilter.handleRead(HttpServerFilter.java:218)
	at org.glassfish.grizzly.filterchain.ExecutorResolver$9.execute(ExecutorResolver.java:95)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeFilter(DefaultFilterChain.java:260)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.executeChainPart(DefaultFilterChain.java:177)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.execute(DefaultFilterChain.java:109)
	at org.glassfish.grizzly.filterchain.DefaultFilterChain.process(DefaultFilterChain.java:88)
	at org.glassfish.grizzly.ProcessorExecutor.execute(ProcessorExecutor.java:53)
	at org.glassfish.grizzly.nio.transport.TCPNIOTransport.fireIOEvent(TCPNIOTransport.java:524)
	at org.glassfish.grizzly.strategies.AbstractIOStrategy.fireIOEvent(AbstractIOStrategy.java:89)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.run0(WorkerThreadIOStrategy.java:94)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy.access$100(WorkerThreadIOStrategy.java:33)
	at org.glassfish.grizzly.strategies.WorkerThreadIOStrategy$WorkerThreadRunnable.run(WorkerThreadIOStrategy.java:114)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.doWork(AbstractThreadPool.java:569)
	at org.glassfish.grizzly.threadpool.AbstractThreadPool$Worker.run(AbstractThreadPool.java:549)
	at java.lang.Thread.run(Thread.java:748)
|#]

@poikilotherm
Copy link
Contributor Author

Educated guess: forgot that every config source is initialized, no matter if enabled or not. So I have to add a null handler to the constructor or findDir(), to silently ignore this.

What happens on configuration change? Would a user expect the source to start working when configuring via asadmin without a restart of the server?

@lprimak
Copy link
Contributor

lprimak commented Jan 29, 2021

What happens on configuration change? Would a user expect the source to start working when configuring via asadmin without a restart of the server?

yes absolutely

@pdudits
Copy link
Contributor

pdudits commented Jan 29, 2021

What happens on configuration change? Would a user expect the source to start working when configuring via asadmin without a restart of the server?

yes absolutely

But that would be feature of its own, as entire Config Provider does not update its "meta-configuration" at runtime, rather at app deployment time.

@poikilotherm
Copy link
Contributor Author

OK I'll open a new PR, as this one is merged and closed.

I'll only work on the "not-configured" case, leaving the "hot config reload" for later.

@pdudits
Copy link
Contributor

pdudits commented Jan 29, 2021

I do have one in baking actually, need to test it yet.

@poikilotherm
Copy link
Contributor Author

poikilotherm commented Jan 29, 2021

I do have one in baking actually, need to test it yet.

Shall I prepare sth. for supporting it in this source? (Control flow, ...)

poikilotherm added a commit to poikilotherm/Payara that referenced this pull request Jan 29, 2021
During QA after merging payara#5007, @lprimak stumbled over a SEVERE
log message during deploys, triggered by the non-existing directory
of DirConfigSource. The directory config value has had a default
before, also the docs say otherwise.

The level issue has been fixed by checking only default config value
being used, resulting in an INFO level message. The logic has been
refactored not to use an exception, but Optional API.

- Docs will be fixed to include the formerly hidden default value
- Part of payara#5006
@poikilotherm
Copy link
Contributor Author

@pdudits just created #5104, plz review 😄

@pdudits
Copy link
Contributor

pdudits commented Jan 29, 2021

image

Oh, you beaten me by two minutes! And yours is a bit nicer. But I need to retest it now :)

@poikilotherm
Copy link
Contributor Author

I misunderstood that part then. I read you worked on the hot reload stuff, but you worked on the source itself... Sorry 😐

lprimak pushed a commit that referenced this pull request Jan 29, 2021
…5104)

* fix(mpconfig): make DirConfigSource less disrupting if not configured

During QA after merging #5007, @lprimak stumbled over a SEVERE
log message during deploys, triggered by the non-existing directory
of DirConfigSource. The directory config value has had a default
before, also the docs say otherwise.

The level issue has been fixed by checking only default config value
being used, resulting in an INFO level message. The logic has been
refactored not to use an exception, but Optional API.

- Docs will be fixed to include the formerly hidden default value
- Part of #5006

* style(mpconfig): reducing log level in DirConfigSource.findDir()

As requested by @pdudits, reducing the log level to FINE
if the directory targeted by default value (secrets) is not present.
lprimak pushed a commit that referenced this pull request Feb 5, 2021
FISH-788 Support sub-directories for MPCONFIG SecretDirConfigSource. #5006
lprimak pushed a commit that referenced this pull request Feb 5, 2021
…5104)

* fix(mpconfig): make DirConfigSource less disrupting if not configured

During QA after merging #5007, @lprimak stumbled over a SEVERE
log message during deploys, triggered by the non-existing directory
of DirConfigSource. The directory config value has had a default
before, also the docs say otherwise.

The level issue has been fixed by checking only default config value
being used, resulting in an INFO level message. The logic has been
refactored not to use an exception, but Optional API.

- Docs will be fixed to include the formerly hidden default value
- Part of #5006

* style(mpconfig): reducing log level in DirConfigSource.findDir()

As requested by @pdudits, reducing the log level to FINE
if the directory targeted by default value (secrets) is not present.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
PR: CLA CLA submitted on PR by the contributor
Projects
None yet
Development

Successfully merging this pull request may close these issues.

MPCONFIG Secret Dir: support secrets in subdirectories / FISH-788
6 participants