Strange Travis failures #5120

angerman · 2018-02-06T14:01:36Z

Travis seems to fail with:

cabal/dist-newstyle/src/pretty-show-1.6.16/dist: renamePath:rename: unsatisfied constraints (Directory not empty)Configuring library for pretty-show-1.6.16..

lately a lot.

So I've basically run git bisect with git clean -xfd && cabal new-build cabal-install on the tree.

That ended up with:

84ab693b0162c372c5127e0a27a52bf902beea55 is the first bad commit
commit 84ab693b0162c372c5127e0a27a52bf902beea55
Author: Mikhail Glushenkov <[email protected]>
Date:   Thu Jan 4 10:35:30 2018 +0000

    Unbreak Haddock doc generation.
    
    + a whitespace fix.

:040000 040000 375f64d216a741dba303b827e5da2513bd47f490 93da2a12bae5c288aecccff300d636296751937f M	cabal-install
bisect run success

log is

git bisect start
# good: [abc18aa7a2677a549f2ff57729a0dee026a7f69c] Capitalisation.
git bisect good abc18aa7a2677a549f2ff57729a0dee026a7f69c
# bad: [18317ce96afbdc08e317b41d10802132266b2ec3] Add buildDir includes as needed.
git bisect bad 18317ce96afbdc08e317b41d10802132266b2ec3
# good: [ab585e864929a8d19ed844d82ff2e9d002ae6313] Merge remote-tracking branch 'upstream/dcoutts/project-config-explicit-includes'
git bisect good ab585e864929a8d19ed844d82ff2e9d002ae6313
# bad: [67276f870a55e6a913b98ef7b2ca22a8def21a6e] Merge pull request #4861 from fgaz/new-run/allow-running-tests-benchs
git bisect bad 67276f870a55e6a913b98ef7b2ca22a8def21a6e
# bad: [ce775dbbfe7d04fb9376e2562002a3bbcb0337ed] Add duplicate field negative tests.
git bisect bad ce775dbbfe7d04fb9376e2562002a3bbcb0337ed
# bad: [ec9ce1c82dcff66608a7b1875e11c220828ec2ee] cabal-testsuite: test cabal new-test with coverage
git bisect bad ec9ce1c82dcff66608a7b1875e11c220828ec2ee
# bad: [552ae22bdc7a265bd57b131f3749e239476bd7ca] Rename rewriteFileV to rewriteFileEx.
git bisect bad 552ae22bdc7a265bd57b131f3749e239476bd7ca
# good: [6e42aa2502ff8e318607d8614693c712f5aac709] Merge pull request #4991 from ezyang/pr/T4986
git bisect good 6e42aa2502ff8e318607d8614693c712f5aac709
# good: [a3e1833890afe970da5eeae3b9d5cac8473734f1] Restore BC compatibility for rewriteFile, fixes #4981
git bisect good a3e1833890afe970da5eeae3b9d5cac8473734f1
# bad: [587bc30577759a93615f391ec42afeaecef2f964] Merge pull request #4996 from ezyang/pr/rewrite-file-bc
git bisect bad 587bc30577759a93615f391ec42afeaecef2f964
# bad: [84ab693b0162c372c5127e0a27a52bf902beea55] Unbreak Haddock doc generation.
git bisect bad 84ab693b0162c372c5127e0a27a52bf902beea55
# first bad commit: [84ab693b0162c372c5127e0a27a52bf902beea55] Unbreak Haddock doc generation.

hope that's of any use...

The text was updated successfully, but these errors were encountered:

phadej · 2018-02-06T19:49:51Z

ogre@bigtest:~/cabal$ /opt/cabal/2.0/bin/cabal new-build cabal-install -j10
In order, the following will be built (use -v for more details):
 - Cabal-2.1.0.0 (lib) (first run)
 - happy-1.19.9 (exe:happy) (requires build)
 - hackage-security-0.5.2.2 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)

2.0 does something weird?

The quick'n'dirty solution is to not parellise builds, but that would hurt in time.

phadej · 2018-02-06T21:21:28Z

oh, we had bad commits there otherwise too:

Distribution/Simple/Register.hs:423:62: error:
    • Couldn't match type ‘Distribution.ModuleName.ModuleName’
                     with ‘IPI.ExposedModule’

EDIT: not really proposing, but maybe we should squash more often.

angerman · 2018-02-07T01:11:16Z

A second run, which basically ran:

#!/bin/bash
(git clean -xfd && cabal new-build cabal-install -j4) && (git clean -xfd && cabal new-build cabal-install -j4) && (git clean -xfd && cabal new-build cabal-install -j4)

yielded

912f2236875a2c364a703dd3600aacdd1674abb6 is the first bad commit
commit 912f2236875a2c364a703dd3600aacdd1674abb6
Author: Mikhail Glushenkov <[email protected]>
Date:   Tue Dec 19 08:27:48 2017 +0000

    Remove the constraint on pretty-show.
    
    The issue is fixed now.

:100644 100644 8947e15203e54a549198ca7618d98f2201436be0 65aa59ba590b11c4d90ddf3703c590f1e9edab6b M	cabal.project
bisect run success

log is

git bisect start
# good: [abc18aa7a2677a549f2ff57729a0dee026a7f69c] Capitalisation.
git bisect good abc18aa7a2677a549f2ff57729a0dee026a7f69c
# bad: [18317ce96afbdc08e317b41d10802132266b2ec3] Add buildDir includes as needed.
git bisect bad 18317ce96afbdc08e317b41d10802132266b2ec3
# bad: [ab585e864929a8d19ed844d82ff2e9d002ae6313] Merge remote-tracking branch 'upstream/dcoutts/project-config-explicit-includes'
git bisect bad ab585e864929a8d19ed844d82ff2e9d002ae6313
# good: [f0d688a9814d3316c5a1267733c4c8a211500de8] Formatting.
git bisect good f0d688a9814d3316c5a1267733c4c8a211500de8
# bad: [a3163408c570e3edd876c5d3280f08a5e8e64235] Add INLINE pragmas for funAppMon methods of ParsecParser
git bisect bad a3163408c570e3edd876c5d3280f08a5e8e64235
# bad: [7a6559af48d4668087387e03d11ef038ac4f9e6e] Unbreak the build with directory < 1.2.3.0.
git bisect bad 7a6559af48d4668087387e03d11ef038ac4f9e6e
# good: [b06aac9553887cb88645a049379afa4de52ae4f8] Merge pull request #4952 from gwils/issue-4443
git bisect good b06aac9553887cb88645a049379afa4de52ae4f8
# bad: [86af280bcd7e3c16f929d92a17a1239be6d9d96a] Merge pull request #4929 from grayjay/duplicate-flag-message
git bisect bad 86af280bcd7e3c16f929d92a17a1239be6d9d96a
# good: [5270950fdf41f387450b04f48c3d01a600bb2bd8] Call simplifyVersionRange in solver convenience functions (.||.) and (.&&.).
git bisect good 5270950fdf41f387450b04f48c3d01a600bb2bd8
# good: [4f7ac10dfd93afed1c6810a19df146d8adbe1fb3] Solver: Combine dependencies on the same package in the same 'build-depends'.
git bisect good 4f7ac10dfd93afed1c6810a19df146d8adbe1fb3
# bad: [912f2236875a2c364a703dd3600aacdd1674abb6] Remove the constraint on pretty-show.
git bisect bad 912f2236875a2c364a703dd3600aacdd1674abb6
# first bad commit: [912f2236875a2c364a703dd3600aacdd1674abb6] Remove the constraint on pretty-show.

angerman · 2018-02-07T01:30:16Z

Something's fishy with that commit:

cabal @ 912f22368 $ git clean -xfd && cabal new-build cabal-install -j4 --dry-run                                                                                                                                                             Removing dist-newstyle/
Resolving dependencies...
In order, the following would be built (use -v for more details):
 - Cabal-2.1.0.0 (lib) (first run)
 - happy-1.19.9 (exe:happy) (requires build)
 - hackage-security-0.5.2.2 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)

cabal @ b06aac955 $ git clean -xfd && cabal new-build cabal-install -j4 --dry-run                                                                                                                                                             Removing dist-newstyle/
Resolving dependencies...
In order, the following would be built (use -v for more details):
 - Cabal-2.1.0.0 (lib) (first run)
 - happy-1.19.9 (exe:happy) (requires build)
 - hackage-security-0.5.2.2 (lib) (requires build)
 - pretty-show-1.6.13 (lib) (requires build)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)

angerman · 2018-02-07T01:32:57Z

912f223 is

From 912f2236875a2c364a703dd3600aacdd1674abb6 Mon Sep 17 00:00:00 2001
From: Mikhail Glushenkov <[email protected]>
Date: Tue, 19 Dec 2017 08:27:48 +0000
Subject: [PATCH] Remove the constraint on pretty-show.

The issue is fixed now.
---
 cabal.project | 1 -
 1 file changed, 1 deletion(-)

diff --git a/cabal.project b/cabal.project
index 8947e1520..65aa59ba5 100644
--- a/cabal.project
+++ b/cabal.project
@@ -1,6 +1,5 @@
 packages: Cabal/ cabal-testsuite/ cabal-install/ solver-benchmarks/
 constraints: unix >= 2.7.1.0,
-             pretty-show == 1.6.13,
              cabal-install +lib +monolithic
 
 -- Uncomment to allow picking up extra local unpacked deps:
-- 
2.16.0

I don't yet see why that change would start producing multiple entries...

angerman · 2018-02-07T01:40:22Z

So, with the constraint, we pick pretty-show once, without the constraint we pick it twice? Anyone got any idea where that might come from?

angerman · 2018-02-07T03:39:23Z

Found it!
In pretty-show > 1.6.13, the cabal file gained the following line

  build-tool-depends: happy:happy

I don't know why, but that line causes cabal

$ cabal --version
cabal-install version 2.0.0.1
compiled using version 2.0.1.0 of the Cabal library

to start adding pretty-show twice, and cabal-install twice to the build plan.
As a work around I would suggest, we remove that line from pretty-show for the time being?

@phadej, @23Skidoo any better idea?

gbaz · 2018-02-07T04:39:37Z

this rabbithole seems relevant?

#4009
haskell/happy#113

(and tangentially: #130)

Note the release of a new happy 3 days ago?

angerman · 2018-02-07T05:21:47Z

The really confusing part is, why the solver starts to think it needs to build cabal-install and pretty-show twice once show-prettys cabal file lists the build-tool-depends. As @phadej noted on IRC, this doesn't happen with cabal HEAD apparently (I haven't verified this).

The bootstrapping cabal on travis however does seem to exhibit this bug.

phadej · 2018-02-07T09:57:06Z

FWIW, My run of git bisect found the same commit

912f2236875a2c364a703dd3600aacdd1674abb6 is the first bad commit
commit 912f2236875a2c364a703dd3600aacdd1674abb6
Author: Mikhail Glushenkov <[email protected]>
Date:   Tue Dec 19 08:27:48 2017 +0000

    Remove the constraint on pretty-show.
    
    The issue is fixed now.

:100644 100644 8947e15203e54a549198ca7618d98f2201436be0 65aa59ba590b11c4d90ddf3703c590f1e9edab6b M      cabal.project
bisect run success

phadej · 2018-02-07T10:40:28Z

@23Skidoo

% git show e80e1a11
commit e80e1a11fc6863de91a689ef87b24c1401659e41
Author: Mikhail Glushenkov <[email protected]>
Date:   Tue Dec 12 10:57:00 2017 +0000

    Temporarily pin pretty-show to 1.6.13.
    
    Should unbork the test suite.

diff --git a/cabal.project b/cabal.project
index 65aa59b..8947e15 100644
--- a/cabal.project
+++ b/cabal.project
@@ -1,5 +1,6 @@
 packages: Cabal/ cabal-testsuite/ cabal-install/ solver-benchmarks/
 constraints: unix >= 2.7.1.0,
+             pretty-show == 1.6.13,
              cabal-install +lib +monolithic
 
 -- Uncomment to allow picking up extra local unpacked deps:

phadej · 2018-02-07T10:40:47Z

Then you removed it, but seems that test-suite is still b0rked.

phadej · 2018-02-07T10:45:44Z

FWIW, I see in master with cabal-install-head:

Configuring library for pretty-show-1.6.16..
cabal: The program 'happy' is required but it could not be found.

cabal: Failed to build pretty-show-1.6.16 (which is required by
cabal-install-2.1.0.0). The failure occurred during the configure step.

though happy is built above in the log!

23Skidoo · 2018-02-07T11:30:49Z

Does reverting that commit help?

phadej · 2018-02-07T11:43:29Z

% /opt/cabal/2.0/bin/cabal new-build --dry cabal-install
Warning: Parsing the index cache failed (Unknown encoding for constructor).
Trying to regenerate the index cache...
Resolving dependencies...
In order, the following would be built (use -v for more details):
 - Cabal-2.1.0.0 (lib) (cannot read state cache)
 - HTTP-4000.3.9 (lib) (requires build)
 - happy-1.19.9 (exe:happy) (cannot read state cache)
 - hackage-security-0.5.2.2 (lib) (cannot read state cache)
 - pretty-show-1.6.16 (lib) (cannot read state cache)
 - pretty-show-1.6.16 (lib) (cannot read state cache)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (cannot read state cache)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (cannot read state cache)

yet plan.json contains only single pretty-show:lib component! So cabal-install-2.0 did something weird while executing the plan (or its output is broken)

phadej · 2018-02-07T11:56:37Z

Resolving dependencies...
In order, the following would be built (use -v for more details):
 - Cabal-2.1.0.0 (lib) (first run)
 - network-2.6.3.3 (lib) (requires build)
 - resolv-0.1.1.1 (lib) (requires build)
 - happy-1.19.9 (exe:happy) (requires build)
 - hackage-security-0.5.2.2 (lib) (requires build)
 - HTTP-4000.3.9 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - pretty-show-1.6.16 (lib) (requires build)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (first run)
Assertion failed
CallStack (from HasCallStack):
  assert, called at ./Distribution/Client/InstallPlan.hs:619:5 in main:Distribution.Client.InstallPlan

610 -- | Given a package in the processing state, mark the package as completed
611 -- and return any packages that are newly in the processing state (ie ready to
612 -- process), along with the updated 'Processing' state.
613 --
614 completed :: (IsUnit ipkg, IsUnit srcpkg)
615           => GenericInstallPlan ipkg srcpkg
616           -> Processing -> UnitId
617           -> ([GenericReadyPackage srcpkg], Processing)
618 completed plan (Processing processingSet completedSet failedSet) pkgid =
619     assert (pkgid `Set.member` processingSet) $
620     assert (processingInvariant plan processing') $

phadej · 2018-02-07T11:57:39Z

So: This is failure in cabal-install-2.0, and it seems to be fixed in master. I don't see other solution than to pinpoint pretty-show (or happy) version!

phadej · 2018-02-07T12:00:14Z

except we cannot do either:

pretty-show

% /opt/cabal/2.0/bin/cabal new-build --dry Cabal cabal-install --disable-tests --constraint=pretty-show==1.6.13
Resolving dependencies...
cabal: Could not resolve dependencies:
rejecting: cabal-install-2.1.0.0:-monolithic (constraint from project config
TODO requires opposite flag selection)
trying: cabal-install-2.1.0.0:+monolithic
next goal: pretty-show (dependency of cabal-install-2.1.0.0:+monolithic)
rejecting: pretty-show-1.6.16, pretty-show-1.6.15, pretty-show-1.6.14
(constraint from command line flag requires ==1.6.13)
rejecting: pretty-show-1.6.13 (conflict: cabal-install-2.1.0.0:monolithic =>
pretty-show>=1.6.15)
rejecting: pretty-show-1.6.12, pretty-show-1.6.11, pretty-show-1.6.10,
pretty-show-1.6.9, pretty-show-1.6.8.2, pretty-show-1.6.8.1,
pretty-show-1.6.8, pretty-show-1.6.7, pretty-show-1.6.6, pretty-show-1.6.5,
pretty-show-1.6.3, pretty-show-1.6.2, pretty-show-1.6.1, pretty-show-1.6,
pretty-show-1.5, pretty-show-1.4.1, pretty-show-1.3.2, pretty-show-1.3.1,
pretty-show-1.3, pretty-show-1.2, pretty-show-1.1.1, pretty-show-1.1,
pretty-show-1 (constraint from command line flag requires ==1.6.13)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: cabal-install, pretty-show,
cabal-install-2.1.0.0:flag

happy

EDIT we have to use any.happy

% /opt/cabal/2.0/bin/cabal new-build --dry Cabal cabal-install --disable-tests --constraint='any.happy<1.19.9'
Warning: Parsing the index cache failed (Unknown encoding for constructor).
Trying to regenerate the index cache...
Resolving dependencies...
In order, the following would be built (use -v for more details):
 - Cabal-2.1.0.0 (lib) (cannot read state cache)
 - HTTP-4000.3.9 (lib) (requires build)
 - hackage-security-0.5.2.2 (lib) (cannot read state cache)
 - cabal-install-2.1.0.0 (lib:cabal-install, exe:cabal) +monolithic +lib (cannot read state cache)

See haskell#5120

phadej · 2018-02-07T16:27:35Z

master is fixed by:

commit 9b9414c69121fe4954ee8585477e5179467e486b
Author: Edward Z. Yang <[email protected]>
Date:   Mon Jan 1 16:09:28 2018 -0500

    Nub exe_deps, fixes #4986
    
    Previously, if you had:
    
      build-tool-depends: happy:happy
      build-tools: happy
    
    This would cause an executable dependency to show up twice in exe_deps.
    It turns out that this made its way all the way to cabal-install's
    InstallPlan, and then broke an invariant in the install plan execution
    engine.  This is the "correct" fix but the whole arrangement is a little
    brittle.
    
    Signed-off-by: Edward Z. Yang <[email protected]>

which make complete sense!

phadej · 2018-02-07T16:30:43Z

We could try to convince Iavor to make a pretty-show release with cabal-version: 2.0 and only build-tool-depends, i.e. no build-tools

https://github.com/yav/pretty-show/blob/9888ffa7eac9cd581caaceadbc3dfc609916c861/pretty-show.cabal#L57-L58

phadej · 2018-02-07T16:33:43Z

Note: cabal knows that build-tools: happy have to be translated into build-tool-depends: happy:happy. So either one would be enough.

build-tool-depends is however more correct long-term, but requires cabal-version: 2.0. Not strictly, for older cabal-install (and stacks) build will succeed if happy is somehow otherwise in $PATH

23Skidoo · 2018-02-07T23:42:42Z

/cc @yav

23Skidoo · 2018-02-07T23:45:07Z

@phadej

master is fixed by:

commit 9b9414c69121fe4954ee8585477e5179467e486b

Thanks for investigating this, cherry-picked into 2.0. Note: that commit is part of PR #4991. See also #4986.

angerman · 2018-02-08T01:15:07Z

What we can do is:

in the travis build scripts:

cabal get --pristine pretty-show; how would be specify we want 1.6.16?
Add pretty-show-1.6.16/ to the cabal.project file.
Patch pretty-show-1.6.16/pretty-show.cabal.

while that is hacky:

it is local to our travis installation only.
doesn't require outside intervention.
should work around the mentioned bug.

23Skidoo · 2018-02-08T02:05:56Z

If this is fixed in master, can't we just temporarily switch to cabal-install-head from @hvr's PPA?

angerman · 2018-02-08T02:30:34Z

@23Skidoo do try out?

23Skidoo · 2018-02-08T02:33:39Z

See #5126.

angerman · 2018-02-10T13:02:45Z

This seems to be resolved.

angerman assigned 23Skidoo and phadej Feb 6, 2018

angerman changed the title ~~Strance travis failures~~ Strange Travis failures Feb 6, 2018

angerman mentioned this issue Feb 7, 2018

basedir #4874

Merged

4 tasks

phadej added a commit to phadej/cabal that referenced this issue Feb 7, 2018

Workaround Travis failures (cabal-install-2.0 is broken)

2b5c7f7

See haskell#5120

This was referenced Feb 7, 2018

Workaround Travis failures (cabal-install-2.0 is broken) #5123

Closed

"In-place"d build-tools not made available via $PATH #5104

Closed

23Skidoo added a commit to 23Skidoo/cabal that referenced this issue Feb 8, 2018

Travis: work around haskell#5120 by moving to cabal-install-head.

ec30d09

23Skidoo added a commit to 23Skidoo/cabal that referenced this issue Feb 8, 2018

Travis: work around haskell#5120 by moving to cabal-install-head.

5e0ee6a

23Skidoo added type: bug continuous-integration labels Feb 8, 2018

23Skidoo added a commit to 23Skidoo/cabal that referenced this issue Feb 8, 2018

Travis: work around haskell#5120 by moving to cabal-install-head.

801e168

23Skidoo added a commit to 23Skidoo/cabal that referenced this issue Feb 8, 2018

Travis: work around haskell#5120 by moving to cabal-install-head.

9523d48

angerman mentioned this issue Feb 9, 2018

Temp travis fix #5132

Closed

4 tasks

grayjay mentioned this issue Feb 10, 2018

Don't ignore goal order parameter when generating summarized solver log. #5133

Merged

4 tasks

angerman closed this as completed Feb 10, 2018

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Strange Travis failures #5120

Strange Travis failures #5120

angerman commented Feb 6, 2018

phadej commented Feb 6, 2018 •

edited

Loading

phadej commented Feb 6, 2018 •

edited

Loading

angerman commented Feb 7, 2018

angerman commented Feb 7, 2018 •

edited

Loading

angerman commented Feb 7, 2018

angerman commented Feb 7, 2018

angerman commented Feb 7, 2018

gbaz commented Feb 7, 2018 •

edited

Loading

angerman commented Feb 7, 2018

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018

23Skidoo commented Feb 7, 2018

phadej commented Feb 7, 2018 •

edited

Loading

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018 •

edited

Loading

phadej commented Feb 7, 2018 •

edited

Loading

phadej commented Feb 7, 2018 •

edited

Loading

phadej commented Feb 7, 2018

23Skidoo commented Feb 7, 2018

23Skidoo commented Feb 7, 2018 •

edited

Loading

angerman commented Feb 8, 2018

23Skidoo commented Feb 8, 2018

angerman commented Feb 8, 2018

23Skidoo commented Feb 8, 2018

angerman commented Feb 10, 2018

Strange Travis failures #5120

Strange Travis failures #5120

Comments

angerman commented Feb 6, 2018

phadej commented Feb 6, 2018 • edited Loading

phadej commented Feb 6, 2018 • edited Loading

angerman commented Feb 7, 2018

angerman commented Feb 7, 2018 • edited Loading

angerman commented Feb 7, 2018

angerman commented Feb 7, 2018

angerman commented Feb 7, 2018

gbaz commented Feb 7, 2018 • edited Loading

angerman commented Feb 7, 2018

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018

23Skidoo commented Feb 7, 2018

phadej commented Feb 7, 2018 • edited Loading

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018

phadej commented Feb 7, 2018 • edited Loading

pretty-show

happy

phadej commented Feb 7, 2018 • edited Loading

phadej commented Feb 7, 2018 • edited Loading

phadej commented Feb 7, 2018

23Skidoo commented Feb 7, 2018

23Skidoo commented Feb 7, 2018 • edited Loading

angerman commented Feb 8, 2018

23Skidoo commented Feb 8, 2018

angerman commented Feb 8, 2018

23Skidoo commented Feb 8, 2018

angerman commented Feb 10, 2018

phadej commented Feb 6, 2018 •

edited

Loading

phadej commented Feb 6, 2018 •

edited

Loading

angerman commented Feb 7, 2018 •

edited

Loading

gbaz commented Feb 7, 2018 •

edited

Loading

phadej commented Feb 7, 2018 •

edited

Loading

phadej commented Feb 7, 2018 •

edited

Loading

phadej commented Feb 7, 2018 •

edited

Loading

phadej commented Feb 7, 2018 •

edited

Loading

23Skidoo commented Feb 7, 2018 •

edited

Loading