Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Avoid running integration tests for pre-release PRs #309

Closed
Strift opened this issue Dec 19, 2024 · 2 comments
Closed

Avoid running integration tests for pre-release PRs #309

Strift opened this issue Dec 19, 2024 · 2 comments

Comments

@Strift
Copy link
Collaborator

Strift commented Dec 19, 2024

Description

The GitHub integration currently includes two workflows:

  • Integration Tests: These run tests against the latest stable release (latest Docker tag)
  • Pre-release Integration Tests: These run tests against the latest version of Meilisearch (e.g., meilisearch:v1.x.y.rc-z Docker tag)

The Integration Tests often fail on bump-* branches because these branches include updates that depend on the target version (the new release), which isn’t yet compatible with the stable version used in these tests.

Proposed Solution

Update the CI configuration to ensure that only Pre-release Integration Tests are executed for bump-* branches. This adjustment will align the tests with the appropriate Meilisearch version and prevent unnecessary failures.

@curquiza
Copy link
Member

curquiza commented Dec 24, 2024

Hello @Strift

Hum, I have difficulty to fully agree on this one.

Seeing if the tests are failing with the current stable release is a way to see potential errors that are not necessarily expected and that could help during pre-release
I allow us to see the changes we are doing are breaking from one version of meilisearch to another and ensure we have the appropriate communication in the changelog. It can also prevent from bug in the engine, i.e. you update your code base, you did not expect the changes to break the previous version, but it did, so you report it to the engine team.

These situations (tests failing with the current stable release) are not supposed to happen often. Indeed, once the vector store will be stabilized, the "breaking" changes we have are kind of rare. We had one with the highlight recently, but I have to say this is the first one in more than a year that is not related to vector store.

So, for me, we should keep these tests, and not remove them from running on bump-meilisearch-vXXX. I understand it makes the things confuse the first time, but these red tests are important to watch and acknowledge.

If we see we have way more often breaking changes than I said, this could be something to reconsider.

But I would love to know the @brunoocasali opinion on it as well!

PS: Since this problem is not only a problem on meilisearch-js, and could happen in every SDK, you can open it on integration guide instead, so I move it 😉

@curquiza curquiza transferred this issue from meilisearch/meilisearch-js Dec 24, 2024
@brunoocasali
Copy link
Member

I agree with @curquiza, @Strift her answer was self-explanatory, I don't see what I can add to the discussion 😅
I will close this issue since I see your thumbs-up reaction in her answer as well :D

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants