Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix race condition in PyStringMap keys method #156

Closed
wants to merge 1 commit into from
Closed

Fix race condition in PyStringMap keys method #156

wants to merge 1 commit into from

Conversation

tpoliaw
Copy link
Contributor

@tpoliaw tpoliaw commented Oct 25, 2019

If the map was modified during a call to keys(), the keyset would change
size and cause an ArrayIndexOutOfBoundsException. This creates a copy of
the keyset and uses that throughout the method.

Sample to show bug in previous version

from threading import Thread

def update_globals():
    for i in range(1000):
        globals()['thing%d'%i] = i

t = Thread(target=update_globals)
t.start()
for i in range(1000):
    dir()

Would result in (with varying index each time)

Traceback (most recent call last):
  File "/tmp/demo.py", line 10, in <module>
    dir()
java.lang.ArrayIndexOutOfBoundsException: 22
	at org.python.core.PyStringMap.stringmap_keys(PyStringMap.java:643)
	at org.python.core.PyStringMap$stringmap_keys_exposer.__call__(Unknown Source)
	at org.python.core.PyObject.invoke(PyObject.java:3634)
	at org.python.core.__builtin__.dir(__builtin__.java:457)
	at org.python.core.BuiltinFunctions.__call__(__builtin__.java:40)
	at org.python.core.PyObject.__call__(PyObject.java:450)
	at org.python.pycode._pyx0.f$0(/tmp/demo.py:9)
	at org.python.pycode._pyx0.call_function(/tmp/demo.py)
	at org.python.core.PyTableCode.call(PyTableCode.java:173)
	at org.python.core.PyCode.call(PyCode.java:18)
	at org.python.core.Py.runCode(Py.java:1687)
	at org.python.util.PythonInterpreter.execfile(PythonInterpreter.java:299)
	at org.python.util.jython.runSimpleFile(jython.java:363)
	at org.python.util.jython.runStream(jython.java:395)
	at org.python.util.jython.runFile(jython.java:420)
	at org.python.util.jython.main(jython.java:615)
java.lang.ArrayIndexOutOfBoundsException: java.lang.ArrayIndexOutOfBoundsException: 22

I can add this as a regression test but wasn't sure what the best approach was for tests that might have different results each time it's run.

If the map was modified during a call to keys(), the keyset would change
size and cause an ArrayIndexOutOfBoundsException. This creates a copy of
the keyset and uses that throughout the method.
@jeff5
Copy link
Member

jeff5 commented Oct 31, 2019

Thanks for looking into this and a potential solution.

The thread synchronisation implicit in some operations in CPython, where they happen to be implemented in C, sets up expectations I sometimes think unreasonable.

This seems simple enough that I wouldn't press for a regression test. I think it needs a short comment that we have to take a stable local copy of the keys because they may change while iterating (which is not an error).

Looking deeper, the use of ConcurrentHashMap's iterator inside toArray()) is supplying that tolerance for concurrent change.

@jeff5
Copy link
Member

jeff5 commented Feb 1, 2020

@tpoliaw : I'd be happy to incorporate this if you sign the CLA. It involves creating an account at bugs.python.org (a chore), and telling it your GitHub name, but on the plus side, then you're set up to contribute to CPython too.

@tpoliaw
Copy link
Contributor Author

tpoliaw commented Feb 3, 2020

I think I signed the CLA a while ago with this account's email but will add my github name to it.

@jeff5
Copy link
Member

jeff5 commented Feb 3, 2020

That checks out now, thanks. Sorry I didn't recognise that you'd contributed before.

@jeff5
Copy link
Member

jeff5 commented Feb 23, 2020

Now in at https://hg.python.org/jython/rev/57e47c817d26 . Thanks Peter.

@jeff5 jeff5 closed this Feb 23, 2020
darjus pushed a commit that referenced this pull request Feb 23, 2020
If the map was modified during a call to keys(), the keyset would change
size and cause an ArrayIndexOutOfBoundsException. This creates a copy of
the keyset and uses that throughout the method.

--HG--
extra : amend_source : 025dd98595b444af0c4ffb67ae4bd4252fb752b4
@jeff5
Copy link
Member

jeff5 commented Jun 4, 2020

Raised as jythontools/jython pull 156.

@tpoliaw tpoliaw deleted the map_race_condition branch June 4, 2020 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants