Resumable plugins rebased

[skriss]

This is a rebase & some cleanup to #495. Will squash commits down before merge.

I'd like to do some more manual testing before merge, but my main feedback points have been addressed and we need to get this in now-ish, so any further issues that come up can be addressed in subsequent PRs.

FYI @nrb @carlisia. Don't expect full reviews since it's a lot of code.

[skriss]

Recording a couple of potential action items:

• this includes breaking changes to the protobuf for plugins. We should reach out to all of our known plugin developers to make sure they're not surprised by this.
• backup/restore controller - this PR logs the per-backup & restore logs to both a file in object storage AND to stdout, so there's info available if the obj storage upload fails. Do we want to log to stdout in all cases? Can we do this only if the upload fails?
• eliminating the backup service and moving to pkg-level functions resulted in some annoying wrapper structs being created in order to enable mocking in unit tests. Can we use a monkey-patching-like approach here to simplify?
• the sample plugins repo will need to be updated

[ncdc]

backup/restore controller - this PR logs the per-backup & restore logs to both a file in object storage AND to stdout, so there's info available if the obj storage upload fails. Do we want to log to stdout in all cases? Can we do this only if the upload fails?

I can't think of an easy way to do that. Is there any harm in always uploading to the ark server log too?

eliminating the backup service and moving to pkg-level functions resulted in some annoying wrapper structs being created in order to enable mocking in unit tests. Can we use a monkey-patching-like approach here to simplify?

We can bring it back

[skriss]

Re: backup service, interested in opinions on the last commit that I just pushed as an approach that allows us not to have the backup service and also doesn't require a bunch of interface/struct definitions for testing purposes. I am a fan of it.

cc @nrb, appreciate input when you get a chance. if you're ok with it i'll do the same for the other functions.

[skriss]

@ncdc re: logs, I was just concerned about the server logs getting super-verbose. Couldn't we just copy the contents of the log temp file into the server log if/only if the call to upload it to obj storage fails?

[ncdc]

I guess it's worth a try

[ncdc]

Or you could revert that part of the PR

[skriss]

eh, i'm gonna leave the logs as-is. we can always change if it's an issue down the road.

[nrb]

Generally +1 on the approach. For the rebase, I'd suggest at least keeping the message from the first commit.

A question - does this require any changes to the https://github.com/heptio/ark-plugin-example/ repo for 3rd party contributors?

[ncdc]

A question - does this require any changes to the https://github.com/heptio/ark-plugin-example/ repo for 3rd party contributors?

Yes, and we should start tagging and/or releasing versions of the example repo to match tagged/released versions of ark (and master=master)

[nrb]

@skriss Specifically addressing the backupDownloader replacement, +1 there. Makes sense to me.

[skriss]

@nrb did the rest of the refactoring work, commit per type. at this point I think i'm ready to merge (once I squash everything down)

[nrb]

The refactored work LGTM. Looks like the rebase work isn't terrible, either.

[skriss]

FYI pushed one more commit on this. @carlisia @nrb if you have free time today, would be awesome to do some testing. I'll plan to squash later today and then we can get it merged.

[nrb]

Testing looked good to me. The plugin process restarted and I never saw more than one.

[skriss]

squashed and ready to merge.