travis: Build all stubs, and test all stubs except for untestable ones #464

petertseng · 2017-01-08T07:51:33Z

Closes #421

Now all stubs work with stack build in Travis, and all stubs except for a few exceptions work with stack test --no-run-tests, which compiles with the tests.

We use the .meta directory of exercism/DEPRECATED.trackler#4 to store a DONT-TEST-STUB marker. This entire directory is not served to students.

This PR makes the assumption: We want all the stubs to build.

We can relax this assumption, and not even run stack build on selected stubs, by adding a .meta/DONT-BUILD-STUB file, if we desire. If you believe any change in this PR was made to make the stub build but should not have been made, please state the case, and we could add .meta/DONT-BUILD-STUB.

This PR makes the acknowledgement that it would be ideal if we can get all the stubs to run the first test and then fail. It then also acknowleges that we couldn't get all stubs to meet this goal at once (in fact, there are some stubs for which we might never want this goal to be met?!), but lets us track how close/far we are from this goal by inspecting how many DONT-TEST-STUB files we have.

petertseng · 2017-01-08T07:52:19Z

exercises/allergies/HINTS.md

@@ -6,5 +6,6 @@ with `Eq` and `Show` instances, and implement the following functions:
 - `allergies`
 - `isAllergicTo`

-Your can use the provided signatures if you are unsure about the types, but
-don't let them restrict your creativity.
+You will find a dummy data declaration and type signatures already in place,


If this is too different from the original ("Don't let them restrict your creativity"), please help me to combine the two in a sensible manner.

It is great the way it is. 👍

petertseng · 2017-01-08T07:52:35Z

exercises/clock/HINTS.md

@@ -12,5 +12,6 @@ The function `fromInteger`, from `Num`, must convert minutes
 to 24 hour clock time. It is not necessary to have a sensible
 implementation of `abs` or `signum`.

-Your can use the provided signatures if you are unsure about the types, but
-don't let them restrict your creativity.
+You will find a dummy data declaration and type signatures already in place,


If this is too different from the original ("Don't let them restrict your creativity"), please help me to combine the two in a sensible manner.

petertseng · 2017-01-08T07:53:05Z

exercises/grade-school/.meta/DONT-TEST-STUB

@@ -0,0 +1,2 @@
+No type signatures on functions (TODO: Should we have them?),


if you believe we should have them, I will gladly add them and remove this DONT-TEST-STUB

I see no reason to not have them. 👍
But we can also fix that later.

petertseng · 2017-01-08T07:57:09Z

exercises/sublist/src/Sublist.hs

@@ -1,3 +1,6 @@
 module Sublist (Sublist(..), sublist) where

+data Sublist = Equal | Sublist | Superlist | Unequal deriving (Eq, Show)


Potentially controversial. If we don't do this though, we have to DONT-TEST-STUB Sublist. I think it's reasonable to do it because it's not the student really has a choice on how to do this, they'll just read the test file and do it this way.

compare to:

go-counting where we define data Color = Black | White

connect where we define data Mark = Cross | Nought

bowling where we define data BowlingError = IncompleteGame | InvalidRoll { rollIndex :: Int, rollValue :: Int }

robot-simulator where we define data Bearing = North | East | South | West

Agreed!

I'm usually in favor of anything to avoid the students having to read the test file.

Haskell is already hard enough to learn, and it is better to allow the students to see a lot of data declarations until they get used to it than let them stuck reading the test suites.

As part of the work in #421.

Otherwise, there are ambiguous type constraints regarding the output of `largestProduct` and `Eq`. As part of the work in #421.

As part of the work in #421.

In #421 we express a desire that all our stubs should compile. Not all of them can successfully run their tests, so we will allow a file that says not to run the tests. This file will be stored in the .meta directory. Support for this was added in Trackler in exercism/DEPRECATED.trackler#4 - in particular, it will not be served to students. Regardless, all stubs should build, ensuring that we have not made a syntax error! If we desire, we can evaluate from time to time whether it is legitimate for the exercises declaring DONT-TEST-STUB to remain that way. We don't use `--pedantic` since there are many cases where, say, a `Dummy` constructor is not used or a function argument is not used. Closes #421

petertseng · 2017-01-08T08:03:22Z

.travis.yml

@@ -53,6 +53,27 @@ script:
        exercisename=$(basename "$exercise")
        pushd ${exercise}

+        examplename="stub"


these lines (stub-testing code) have very high duplication with the example-testing code below. I'm not sure if it's desirable that we deduplicate that.

Seems fine for me. If we change our mind we can rewrite it later. 👍

petertseng · 2017-01-08T08:05:58Z

Highly likely to increase our build times. Reviewers should check to make sure the increased time is worth the increased assurance.

rbasso

When we added hlint to Travis, the build times and cache size where:

    Building time (without cache)
        nightly-2016-07-17 :  21m03s
        nightly            :  21m09s

    Rebuild time (with cache)
        nightly-2016-07-17 :   1m41s
        nightly            :   2m03s

Now we have:

Build: 25m52s / 25m12s
Rebuild: 2m21s / 3m3s
Cache:  1651.95MB

So I think we are fine! 😄

Thanks for all the work, @petertseng.

rbasso · 2017-01-08T10:00:36Z

exercises/grade-school/.meta/DONT-TEST-STUB

@@ -0,0 +1,2 @@
+No type signatures on functions (TODO: Should we have them?),


I see no reason to not have them. 👍
But we can also fix that later.

rbasso · 2017-01-08T10:09:46Z

exercises/sublist/src/Sublist.hs

@@ -1,3 +1,6 @@
 module Sublist (Sublist(..), sublist) where

+data Sublist = Equal | Sublist | Superlist | Unequal deriving (Eq, Show)


Agreed!

I'm usually in favor of anything to avoid the students having to read the test file.

Haskell is already hard enough to learn, and it is better to allow the students to see a lot of data declarations until they get used to it than let them stuck reading the test suites.

rbasso · 2017-01-08T10:13:57Z

.travis.yml

@@ -53,6 +53,27 @@ script:
        exercisename=$(basename "$exercise")
        pushd ${exercise}

+        examplename="stub"


Seems fine for me. If we change our mind we can rewrite it later. 👍

rbasso · 2017-01-08T10:15:25Z

exercises/allergies/HINTS.md

@@ -6,5 +6,6 @@ with `Eq` and `Show` instances, and implement the following functions:
 - `allergies`
 - `isAllergicTo`

-Your can use the provided signatures if you are unsure about the types, but
-don't let them restrict your creativity.
+You will find a dummy data declaration and type signatures already in place,


It is great the way it is. 👍

petertseng · 2017-01-08T10:59:13Z

Looking over it one last time, and if I don't see anything I'm merging it.

@petertseng

Clock is the last remaining exercise that has the .meta/DONT-TEST-STUB file because the type in the src/Clock.hs stub is not compatible with the test suite. Other examples of exercises that were revised so that testing the stub became possible are listed in #466. The difference is that those all had `data` definitions, whereas Clock is ideally solved with a 'newtype' and an appropriate alias. The DONT-TEST-STUB mechanism was added by @petertseng in #464 and in particular the commits - 1d91ceb (Travis code) - 342b932 (Clock-specific) The change made in this commit argues that since the example solution has `deriving Eq`, adding this to the stub *will* make the exercise a bit easier, but it still won't give away the central piece. The motivation behind removing DONT-TEST-STUB in Clock is that with this single effort, we can say that all stubs can be compiled and tested. To motivate further, I will cite @rbasso in #466: > I think that this is the way to go. We have to sacrifice a little > flexibility in designing exercises, but we gain a more coherent > user-experience and automatic testing of stubs. 👍 This bumps the exercise's version from 2.4.0.9 to 2.4.0.10.

petertseng commented Jan 8, 2017

View reviewed changes

petertseng added 14 commits January 7, 2017 23:57

allergies: Add dummy data to stub so that it compiles

73f42a3

As part of the work in #421.

back-account: Add dummy data to stub so that it compiles

4c12735

As part of the work in #421.

clock: Add dummy data to stub so that it compiles

342b932

As part of the work in #421.

grade-school: Add dummy data to stub so that it compiles

4a9b488

As part of the work in #421.

largest-series-product: Add signature to stub so that it runs tests

5c17705

Otherwise, there are ambiguous type constraints regarding the output of `largestProduct` and `Eq`. As part of the work in #421.

linked-list: Add dummy data to stub so that it compiles

053164e

As part of the work in #421.

matrix: Add dummy data to stub so that it compiles

f684559

As part of the work in #421.

meetup-day: Add dummy data to stub so that it compiles

7d76947

As part of the work in #421.

robot-name: Add dummy data to stub so that it compiles

2299b7c

As part of the work in #421.

robot-simulator: Add dummy data to stub so that it compiles

34d4039

As part of the work in #421.

simple-linked-list: Add dummy data to stub so that it compiles

97624b6

As part of the work in #421.

sublist: Add full data type and signature to stub so that it compiles

60e23bf

As part of the work in #421.

zipper: Add dummy data to stub so that it compiles

3622396

As part of the work in #421.

petertseng commented Jan 8, 2017

View reviewed changes

rbasso approved these changes Jan 8, 2017

View reviewed changes

petertseng merged commit d3c8a49 into exercism:master Jan 8, 2017

petertseng deleted the travis-stubs branch January 8, 2017 11:06

This was referenced Jan 8, 2017

grade-school: Add signatures to stub so it runs tests #465

Merged

data and constructors required by tests: Should we provide them in stubs? #466

Closed

sshine mentioned this pull request Jan 20, 2020

Clock: Add deriving Eq to stub to make it testable #891

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

travis: Build all stubs, and test all stubs except for untestable ones #464

travis: Build all stubs, and test all stubs except for untestable ones #464

petertseng commented Jan 8, 2017 •

edited

Loading

petertseng Jan 8, 2017

rbasso Jan 8, 2017

petertseng Jan 8, 2017

petertseng Jan 8, 2017

rbasso Jan 8, 2017

petertseng Jan 8, 2017

rbasso Jan 8, 2017

petertseng Jan 8, 2017 •

edited

Loading

rbasso Jan 8, 2017

petertseng commented Jan 8, 2017

rbasso left a comment

rbasso Jan 8, 2017

rbasso Jan 8, 2017

rbasso Jan 8, 2017

rbasso Jan 8, 2017

petertseng commented Jan 8, 2017

		@@ -0,0 +1,2 @@
		No type signatures on functions (TODO: Should we have them?),

		@@ -1,3 +1,6 @@
		module Sublist (Sublist(..), sublist) where

		data Sublist = Equal \| Sublist \| Superlist \| Unequal deriving (Eq, Show)

travis: Build all stubs, and test all stubs except for untestable ones #464

travis: Build all stubs, and test all stubs except for untestable ones #464

Conversation

petertseng commented Jan 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petertseng Jan 8, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petertseng commented Jan 8, 2017

rbasso left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

petertseng commented Jan 8, 2017

petertseng commented Jan 8, 2017 •

edited

Loading

petertseng Jan 8, 2017 •

edited

Loading