fix: remove default memory cap for index training

[wjones127]

Removed hard-coded 4GB limit, as sometimes that produces errors like:

OSError: LanceError(IO): Resources exhausted: Failed to allocate additional 5618883160 bytes for ExternalSorter[0] with 0 bytes already allocated - maximum available is 4284481536, /rustc/79e9716c980570bfd1f666e3b16ac583f0168962/library/core/src/convert/mod.rs:716:9

Instead users can set this as an environment variable:

export LANCE_MEMORY_LIMIT=$((4 * 1024 * 1024 * 1024))

I did an environment variable for now because the indexing parameters are a mess and I don't want to refactor them yet.

Because we don't know how much memory the client machine may have, we set it to unbounded by default.

Closes #1701

[westonpace]

I think this is fine for a tentative / experimental feature. It's not documented though, and if we were to document it, then it would be misleading since we haven't capped memory usage in all places in Lance.

Maybe just change the title of the commit to "remove default memory cap for index training" so people reading the release notes don't think we actually have a memory limit feature.

[Padge91]

Is this environment variable respected by the lancedb python API? I'm using version 0.9.0, and it appears to have no effect when running the following:

export LANCE_MEMORY_LIMIT=$((4 * 1024 * 1024 * 1024))
...
>>> await table.create_index("vector", config=index.IvfPq(distance_type="cosine", num_partitions=10000), replace=True)

The process gets killed, and using vmstat to monitor memory while it's running shows the available memory on the machine (32GB) dropping down into the MB range.

[Padge91]

Is this environment variable respected by the lancedb python API? I'm using version 0.9.0, and it appears to have no effect when running the following:

export LANCE_MEMORY_LIMIT=$((4 * 1024 * 1024 * 1024))
...
>>> await table.create_index("vector", config=index.IvfPq(distance_type="cosine", num_partitions=10000), replace=True)

The process gets killed, and using vmstat to monitor memory while it's running shows the available memory on the machine (32GB) dropping down into the MB range.

Looks like it may be less of this setting, and more likely related to this thread: lancedb/lancedb#767

[westonpace]

The LANCE_MEMORY_LIMIT environment variable no longer exists. The shuffling process it used to apply to has changed considerably (and will be changing more in the nearish future).

I have been working recently on splitting the index training process into steps (#2553 #2562 #2556). I still have some ways to go here but hope to finish in the next week or two. Once that is done I hope to be able to explore and tune the individual steps so that we can better communicate how much memory is required for each step and find ways to improve.

[Padge91]

Great, thanks for that info. I appreciate your help!