Probing hashmap

[jelmervdl]

Attempt to reduce the memory hunger of docalign since it is turning into an issue in HPLT processing.

Aah, crap, peak memory footprint isn't lower :(

This version (mostly focus on memory, runtime is on my desktop while doing other things so not realistic)

Calculated DF from 2464504 documents
DF queue performance:
  underflow: 4823
   overflow: 0
Pruned 157562017 (81.1585%) entries from DF
Very frequent ngram set is now 227054 long.
Read 307 documents into memory
Load queue performance:
  underflow: 16
   overflow: 0
Read queue performance (Note: blocks when score queue fills up):
  underflow: 4822
   overflow: 0
Score queue performance:
  underflow: 4828
   overflow: 0
  6m25.74s real   42m5.71s user   1m46.59s sys
         18629947392  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
            10793957  page reclaims
                   5  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                  23  voluntary context switches
             1391819  involuntary context switches
       8201683541841  instructions retired
       6036882163532  cycles elapsed
         18185654272  peak memory footprint
      55

Current version in main repo:

Calculated DF from 2464504 documents
DF queue performance:
  underflow: 4338
   overflow: 0
Pruned 157562017 (81.1585%) entries from DF
Very frequent ngram set is now 227054 long.
Read 307 documents into memory
Load queue performance:
  underflow: 16
   overflow: 0
Read queue performance (Note: blocks when score queue fills up):
  underflow: 4818
   overflow: 0
Score queue performance:
  underflow: 4828
   overflow: 0
  13m0.76s real   1h25m24.56s user    1m43.82s sys
         18356985856  maximum resident set size
                   0  average shared memory size
                   0  average unshared data size
                   0  average unshared stack size
             6078042  page reclaims
                   5  page faults
                   0  swaps
                   0  block input operations
                   0  block output operations
                   0  messages sent
                   0  messages received
                   0  signals received
                  37  voluntary context switches
             2789048  involuntary context switches
       9243225569447  instructions retired
      11959176496660  cycles elapsed
         15795527680  peak memory footprint
      55

[jelmervdl]

Okay, so just replacing the table isn't doing the job. Next attempt: don't let the thread-specific hash maps become so big? easy to test this by just running 1 thread and seeing what happens with memory footprint.