Cython cache diff compare

[nparley]

As talked about in #13425 with @gfyoung and @jreback this PR does cython caching by comparing the pyx files and not relying on the git history.

[nparley]

At the moment this code does the follow:

1. Caches all the pyx files at the end of a run under $HOME/.cache/pyxfiles keeping the directory structure incase two cython files have the same name but in different directories.
2. Caches all the corresponding compiled files for those pyx files.

then at the start of a run:

1. Finds all the current pyx files in the checkout git repo on travis
2. For each file compares it to the file that is in the cache
3. If the file does not exist in cache a new cython file has been added => Do not use cython cache and rebuild
4. If the file is different from the cython file in cache => Do not use cython cache and rebuild
5. If the number of pyx files are different then a file has been deleted (a rename will be picked up by 3) => Do not use cython cache and rebuild
6. If all files are the same => Use cython cache, i.e. extract and touch the compiled pyx files.

The thing this does not do is the case where a file exists on the cache but has been deleted from the repo. Do you think this should be caught as well or should it not matter? I guess a compiled file would be copied over but then just not referenced or used in the build.

[codecov-io]

Current coverage is 84.42%

Merging #13526 into master will increase coverage by 0.08%

@@             master     #13526   diff @@
==========================================
  Files           138        138          
  Lines         51111      51349   +238   
  Methods           0          0          
  Messages          0          0          
  Branches          0          0          
==========================================
+ Hits          43107      43350   +243   
+ Misses         8004       7999     -5   
  Partials          0          0          

Powered by Codecov. Last updated by e8e0aae...8b4ad0b

[nparley]

Tests to perform:

• If pyx files have not changed then cached cython cache is reused
• If a pyx file is changed the cython cache is not used
• If a pyx file is added the cython cache is not used
• If a pyx file is deleted the cython cache is not used
• Check a pyx file change when rewritting git history i.e. cache is not used

[jreback]

this will not work. you would need to compare .pyx file sizes I think to make this reliable. Just adding / removing a file is not good enough at all. Most often we have a change in actual contents.

[nparley]

The actually checking is done below. diff -u $i $PYX_CACHE_DIR${i} for each pyx file . This code is only here to check no pyx files have been deleted in full.

[jreback]

oh i c.

[jreback]

are you meaning to leave in these comments?

[nparley]

I was thinking about it; as a quick way of turning off caching of cython files in the future if something goes wrong. But happy to take them out also.

[jreback]

ok that's fine then.

[jreback]

@nparley ok let's try this. ping when green / ready.

[nparley]

There is an interesting gocha with the way travis checks out commits. On a branch travis actually checks out the hash. However, for a PR travis just checks out the latest merge. Therefore if you push lots of changes quickly before travis has started (like I just did early), then builds might not actually reflect the push. (They might all be testing the last push).

[nparley]

Travis is having problems with Mac OSx runs at the moment. I will not do any more until they fix it.

[nparley]

@jreback OK this is ready to go.

[jorisvandenbossche]

Looks good to me!

[jreback]

thanks @nparley

let's give this a watch over next few days to make sure its behaving.