Error with io.sort.mb = 1

[sritchie]

Hey guys, can one of you check out this problem? I'd been seeing this with an older version of Cascalog... would be good to see if this still fucks up.

  (in-ns forma.source.hdf-test)

  ;; Cascalog is losing data during its tests, for some reason. What is
   ;; it about my particular tuples that are screwing things up here?
  (let [src       (io/hfs-wholefile hdf-path)
        not-uniqued (<- [?dataset ?chunkid !chunk]
                        (src ?filename ?hdf)
                        (unpack-modis [[:ndvi]] ?hdf :> ?dataset ?freetile)
                        (raster-chunks [24000] ?freetile :> ?chunkid !chunk)
                        (:distinct false))
        uniqued     (<- [?dataset ?chunkid !chunk]
                        (src ?filename ?hdf)
                        (unpack-modis [[:ndvi]] ?hdf :> ?dataset ?freetile)
                        (raster-chunks [24000] ?freetile :> ?chunkid !chunk))]
    (with-job-conf {"io.sort.mb" 100}
      ;; Everything works as expected with 100MB:
      (fact (??<- [?count]
                  (uniqued ?dataset ?chunkid !chunk)
                  (c/count ?count))
        => [[60]]))

    (with-job-conf {"io.sort.mb" 1}
      ;; The same job gets weird with 1MB...
      (fact (??<- [?count]
                  (uniqued ?dataset ?chunkid !chunk)
                  (c/count ?count))
        => [[25]])

      ;; Unless we don't distinct the tuples.
      (fact (??<- [?count]
                  (not-uniqued ?dataset ?chunkid !chunk)
                  (c/count ?count))
        => [[60]])))