Environment.set_mapsize() safety

[dw]

Ensure that cached read-only transactions do not break when set_mapsize() is invoked.

[dw]

Ensure write transactions are synchronized, see #97.

class Environment:
    env_change_in_progress = False
    env_change_complete_event = threading.Event()

    def begin():
        if env_change_in_progress:
            env_change_complete_event.wait()

    def set_mapsize(..., timeout=30.0):
        env_change_in_progress = True
        while True:
            if time() > timeout:
                 raise TimeoutError
            if not any_deps_are_active_txns():
                 break
            sleep(0.500)
         resize_map()
         env_change_in_progress = False
         env_change_complete_event.notify()

[jnwatson]

I'm going to fix this. I have a solid crashing unit test for txn.get/set_mapsize race, a txn.put/set_mapsize race, and a env.close/txn.get race. I like the general idea of keeping the overhead low, but there's still a place after the begin checks for env_change_in_progress and before the active txns list is updated where a transaction user isn't locked out.

The problem is a bit worse on the close side. The close function needs to wait for all other transactions and cursors to complete their function.

How would do you propose to implement the any_deps_are_active_txns() function?

[jnwatson]

Upon further thought, I think the right approach is to increment a in_use count almost every method on entrance and decrement every method on exit, and have every method start with a env_change_in_progress check. set_mapsize and close would loop waiting for in_use to drop to 0. The GIL would protect any races with the increment/change check.

[dw]

The in-use count sounds good!

…

On 13 May 2018 at 22:37, Nic Watson ***@***.***> wrote: Upon further thought, I think the right approach is to increment a in_use count almost every method on entrance and decrement every method on exit, and have every method start with a env_change_in_progress check. set_mapsize and close would loop waiting for in_use to drop to 0. The GIL would protect any races with the increment/change check. — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub <#96 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AAAJC3BPM1SAFLXkptXw_9X0UZLi7-Bdks5tyKe0gaJpZM4FpImA> .

[jnwatson]

Well, I fixed it in a branch, but it added a bunch of code and overhead for every call. (It is not unlike the many unfruitful attempts to remove the GIL from cpython).

Unfortunately, I think the correct approach for now is to document the behavior and kick the can to the user.