Better sizehint for primes

[mschauer]

Better sizehint for primes(lo,hi).

Before it failed for 1 << lo < hi as the sizehint! actually consumed hi / log(hi) memory independent of lo, but the needed size is better approximated by hi / log(hi) - lo/log(lo). Add a test which test primes and friends on short intervals with 1 << lo.

[tkelman]

2^40 overflows on 32 bit

[mschauer]

Oh, yes, and I swear I did not just try to make a point here - #5573

[pabloferz]

I think you don't need max here except for max(lo,0) and max(lo,2) as hi will be greater or equal than 7 at this point.

[pabloferz]

I guess we can clean this up a bit by writing lo = max(2, lo) after the lo ≤ hi check. And then just write hi / log(hi) - lo / log(lo) for the sizehint

[mschauer]

Thanks for the feedback. Not a big surprise, but for example now the batched infinite prime iterator
primeiter = Base.flatten(primes(1000000*(i-1)+1,1000000*i) for i in countfrom(1))
actually works

julia-before> @time first(drop(primeiter, 10^9))
 93.870623 seconds (2.00 G allocations: 81.113 TB, 10.60% gc time)
22801763513
julia-after> @time first(drop(primeiter, 10^9))
 88.758689 seconds (2.00 G allocations: 87.218 GB, 3.50% gc time)
22801763513

For comparison,

julia> @time primes(22801763513)[end];
114.720337 seconds (10 allocations: 13.415 GB, 0.15% gc time)

takes a bit longer.

[pabloferz]

We could use better bound than n/log(n), but if the increase in time for big numbers is not that bad this PR might be a good compromise.

[mschauer]

It's only slightly too big (with relative error of 5% for n=10^9 for example), which is good. That is better than choosing something too small but much closer (Li(n) for example).

[pabloferz]

A much better bound for n>3 is n/(log(n)-1.12). Se here https://projecteuclid.org/download/pdf_1/euclid.rmjm/1181070157

[pabloferz]

But that will only work for n > 8 as it is a decreasing function before that point.

[pabloferz]

Ok, as we have

x / log(x) < π(x) < x / (log(x) - 1.12)

then

hi / log(hi) - lo / (log(lo) - 1.12) < π_hi - π_lo < hi / (log(hi) - 1.12) - lo / log(lo)

so it would probably be better to have (though it would use more memory than you have now):

sizehint!(list, floor(Int, hi / (log(hi) - 1.12) - lo / log(lo))

I guess is just a matter of experimenting a bit.

[mschauer]

Do we have π(x) < x / (log(x) - 1.12) ?

[pabloferz]

See equation 10 from the reference a pasted above

[mschauer]

hi / (log(hi) - 1.12) - lo / log(lo) is much too big if hi-lo is relatively small. I'll take
hi / (log(hi) - 1.12) - lo / (log(lo) - 1.12) to make it a tighter upper bound in the standard setting primes(n). That also shaves another 10 percent of the memory consumption in the example.

[pabloferz]

Just beware of lo / (log(lo) - 1.12) when lo < 4 you should use lo / (log(lo) - (lo > 3 ? 1.12 : 0) or something like that

[mschauer]

I think we can take that one. That should work fine in practice, as sizehint!() rounds to the next multiple of a resizing factor and does not need much precision, but only good overall estimates which give enough space to accommodate the array.

[pabloferz]

LGTM.

[mschauer]

@ararslan Do you want to have a look? I think this is good to go.

[ararslan]

@mschauer Funny you should ask me, I'm just some guy. 😜

Looks good to me (and it appears the AppVeyor failure was just a timeout). 👍

[ararslan]

Want to try closing and reopening the PR to see if we can get AppVeyor to cooperate?

[mschauer]

Thanks to the unknown helping hand triggering AppVeyor.

[mschauer]

#16357