add integration tests to make sure we can update deps

[escapewindow]

I tend to try to bump all the scriptworker deps in puppet when I bump the scriptworker major version, to keep us up to date. In https://bugzilla.mozilla.org/show_bug.cgi?id=1328719#c37 , @JohanLorenzo was concerned about updating deps without testing.

Let's either

• have integration tests that give us more confidence in making sure things work end to end, or
• use our -dev or -dep pool to test dep updates. Ideally this is an easy task or set of tasks to trigger that show that things work, without affecting any prod products.

[JohanLorenzo]

Thanks for filling this issue, Aki!

I'm not sure how we can tackle this particular issue. We do have an integration test suite, but neither the integration tests, nor the -dep pool talks to Google Play. I know pyopenssl is used by both scriptworker and google-api-python-client. The latter is used to authenticate to Google Play.

I don't feel comfortable letting an integration test suite to authenticate to Google Play. This means we should store some credentials somewhere (probably public). Then our account may be stolen and used to push visible stuff on Google Play. We can mitigate the risk by creating a dummy account that isn't linked to the mozilla one. However, I'm afraid we may end up blacklisted if someone abuses this account.

On the other hand, dep instances are meant to not talk to Google Play. This ensures we keep testing our code, without erroring out at the first request GP rejects.

We can test version bump right before a nightly gets published (by 10:30am UTC each day). We would notice things break rapidly before another release comes up.

What do you think @escapewindow ?

[escapewindow]

• what if we had a commandline test we could run from pushapk scriptworker? SSH in, ideally to dep/dev pushapk scriptworker, update pushapkscript+deps, and run the command.
• I'm ok having an integration test suite authenticate to Google Play if a) it doesn't write anything to production, and b) the creds aren't available where they shouldn't be available. It could use that dummy account.

The above depends entirely on what we'd be testing. If we can find issues in updating cryptography etc by just querying the existing GP strings, then that's a pretty simple test. If we can upload an old apk and rely on the buildid error and that tests everything, that's an ok test. If we have to build, sign, and upload a new apk every time we want to test, that's much more complicated.

What broke last time when we updated those modules?

[JohanLorenzo]

Sorry, my tab had been open for a while, but I haven't responded :/

what if we had a commandline test we could run from pushapk scriptworker? SSH in, ideally to dep/dev pushapk scriptworker, update pushapkscript+deps, and run the command.

We can script this, indeed. One sad thing about the current dep scriptworker (there is no dev instance): it doesn't make any request to Google Play. Thus, updating GP-related deps (like cryptography) can't be tested on such instances. We would need a different kind of instances which reply to both points (a and b) you raised.

If we can find issues in updating cryptography etc by just querying the existing GP strings, then that's a pretty simple test.

Excellent point! I think that's doable. I believe the permission model allows us to query string without being able to modify them. @sylvestre, can you confirm the following: can a Google Play account have a read-only access on listings, and nothing else? If so, let's create such an account on a dummy project. Then, we can use Travis' encryption to store creds of this dummy account and have this integration test run in CI.

I couldn't find logs of what failed last time, but I'm positive your suggested test should work, @escapewindow! I'm not a fan of reuploading existing APK, especially if failures appear on Treeherder (i.e.: we reran a passed task to make it fail). I did a few weeks ago and it confused sheriffs.