Update test suite files

[behrtam]

Maybe I'm wrong but if a test suite file is changed the only way to get this update is by manually removing it and exercism fetch; providing he or she knows about the change.

I think it would be better if it's updated just with exercism fetch or at least that the user gets a message after fetch if there are not updated test suites.

[kytrinyx]

I'd like to hear more opinions on this one. I can think of reasons to do it either way.

[behrtam]

Do we have a help description anywhere that explains users how to update changed exercises manually?

[kytrinyx]

I don't think we do.

One thing we could do is put a .version file in each exercise directory and then the CLI could check the existing version with one that comes back and either tell the user that there's a newer version available, or perhaps say something like "rerun with --force in order to replace the test file with the new one".

Replacing the test file will put back any skips or pending etc, which people will need to remove manually (depending on the language).

[lcowell]

I'm thinking - if a user fetches without specifying an exercise fetch or fetch go, the default behaviour should be to preserve the existing files and require a force flag to overwrite. If a user specifies an exercise eg fetch go leap can we assume they want to pull down the latest version?

I'm wondering if we can avoid the .version file and just use the file timestamps. If the file is newer than the released_at of the exercise (assuming something like that can exist) we suggest the user could rerun with --force

Have we ever seen a new version of ruby or elixir etc break old tests? That seems a good motivation to add this.

[kytrinyx]

New versions of the languages have broken tests in elixir, rust, and there's also some version sheer in python, but I don't think this would fix that.

I'm not sure if the timestamps thing would work... it might. I'll experiment a little bit.

I'd rather not assume anything about what the user wants. Maybe we could add a --force flag that would always overwrite the tests with the most recent version if they pass it, and then warn them about new versions (and tell them to use --force if they want it) when they fetch. Maybe also that could be a part of the status command (There's a new version of the hamming test suite in Go)

[Tonkpils]

I'd rather not assume anything about what the user wants. Maybe we could add a --force flag that would always overwrite the tests with the most recent version if they pass it, and then warn them about new versions (and tell them to use --force if they want it) when they fetch. Maybe also that could be a part of the status command (There's a new version of the hamming test suite in Go)

I like that. I don't see why people would go back to fetch an exercise that they've already done, so the status command would be a great place to have the message about the fetch --force

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[englishm]

I recently mis-reported a long-ago fixed issue with the Python hello-world exercise due in part to the behavior this issue describes.

See: exercism/python#1353

I like @kytrinyx's idea about adding a --force option, and warning if updates are available.

[kytrinyx]

The nextercism client will be a bit smarter about this, if I recall correctly 👍

[kytrinyx]

Ok, I've had a look at this and I believe it's safe to close, since we're dealing with this quite differently in v2 (we pin the version that you're working against).