Run specs on CircleCi with shared configuration

[kennyadsl]

A new way to run specs on extensions

The problem

Everytime a new Solidus version is released or goes EOL we need to refresh the CI configuration of all the extensions adding or removing a version. 😢
This is not maintainable with the current core team, and even if we had more development work this is not worth the time, everyone can find better things to do than repeat the same commit on 30 extensions.

Requirements

Before presenting the solution I'd like to list our requirements: what we need in order to be able to move from TravisCI to another solution?

• Centralized configuration: we need an external container for the environment configuration where specs run, at least for the supported solidus versions, which is the thing that changes more often
• Ability to change the centralized configuration and have this change reflected into all the extensions that are using it without the need to make a commit for that
• Setup a recurring scheduled build against master: this ensures that if we change something in core that brakes extensions we are notified before releasing Solidus.
• Being able to access build status data via API to populate extensions.solidus.io, which is currently done in this repo: https://github.com/solidusio/extensions.solidus.io

Proposed Solution

Using CircleCI Orbs we can define a shared configuration that every extension will use at its latest version.

Lately, we experimented a lot with CircleCI Orbs at Nebulab. Our goal there is having a shared CircleCI configuration that we can use on several projects since we use the same tasks. In the past it has been a pain when we needed to update the CircleCI configuration and spread the change on every project that we manage. Working on that I thought that we could use this same approach for this extensions problem and I started experimenting with this PR.

This is the orb used in this PR. This orb is packed and released at every commit on the master branch of this solidusio/circleci-orbs-extensions repository.

It basically defines shared tasks that our extensions will use.

With @volatile

orbs:
  solidusio_extensions: solidusio/extensions@volatile

we ensure that the extension will always build against the last orb code, which will contain the list of currently supported Solidus versions. The run-tests command is the only file that we need to change on the orb.

The orb also defines two executors (for mysql and postgres), which allow us to configure the environment where we want tests to run.

The configuration of each extension will only have the responsibility to call these shared commands and executors from the orb.

Extensions Matrix website

I've also created the integration with CircleCI to generate the matrix. Here's the Draft PR and the resulting preview.

We can either use it and dismiss the old Travis matrix, or move to a simpler solution (like a simple badge in solidus.io/extensions).

What do you think?

Your feedback is important here. We are interested in help improving this solution if any of you have experience with CircleCI and Orbs and also if you have any feedback about if it would be acceptable to dismiss the extensions matrix website, in favor of a simpler solution.

Let me know!

---

TODO:

• create the CircleCI orb configuration so that we can change supported versions in a single point
• always use latest version of the orb so we can easily propagate the change without any commit
• scheduled weekly builds on all extensions
• update extensions matrix (move to solidus.io/extensions with a badge?)

[tvdeyen]

Wow! This is awesome. 👏

Should we remove the Travis build from mandatory checks for this repo then? The CircleCI builds pass, but the travis ones fail for obvious reasons.

[ericsaupe]

Awesome work!

[kennyadsl]

@tvdeyen I think they can also live together for a while, failures are related to Solidus v2.8, which needed some backport fixes, done with solidusio/solidus#3194. I've restarted the failing builds, now we should have both green.

My only concern is with the extension matrix website, but I'm working on supporting both systems in there and gradually switch.