Enable parallel formula evaluation in the Python script session.

[jmao-denver]

As a DH Python user, I want to take advantage of the query engine's parallel processing capability as much as possible by utilizing the modified behavior of JPY releasing and reacquiring of the GIL around calling a Java method from Python.

[jmao-denver]

test_parallelization.py

from deephaven import parquet
import time, random

#t = parquet.read("/data/transactions.parquet")
t = parquet.read("/data/transactions.parquet").tail(655360)

def closing_price() -> float:
    return random.randrange(24000, 24100)

class Closing:
    ...

closing = Closing()
closing.price = 0

def run_udpates():
    start = time.perf_counter()
    for _ in range(1):
        t1 = t.update("PL = AMOUNT - (long)closing.price")
    end = time.perf_counter()
    return (end - start)

exec_times = []
for _ in range(5):
    exec_times.append(run_udpates())
print(exec_times)

test_numba.py

import time, random
from numba import vectorize, int64, jit
from deephaven import parquet
# t = parquet.read("/data/transactions.parquet")
t = parquet.read("/data/transactions.parquet").tail(655360)

@vectorize([int64(int64, int64)])
@jit(nogil=True)
def vectorized_func(x, y):
    return x % 3 + y

def run_udpates():
    start = time.perf_counter()
    t1 = t.update(f"PL = (long)vectorized_func(i, ii)")
    end = time.perf_counter()
    return (end - start)

exec_times = []
for _ in range(5):
    exec_times.append(run_udpates())
print(exec_times)

[jmao-denver]

Parallel, forced stateless

- JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Xmx6g  -Ddeephaven.console.type=${DEEPHAVEN_CONSOLE_TYPE} -Ddeephaven.application.dir=${DEEPHAVEN_APPLICATION_DIR} -DOperationInitializationThreadPool.threads=8 -DUpdateGraphProcessor.updateThreads=8 -DQueryTable.minimumParallelSelectRows=4096

• test_parallelization.py: [30.742132300001686, 27.54671560000861, 27.565786299994215, 26.4070253999962, 28.149148799988325]

• test_numb.py: [1.8152265999960946, 0.08204210000985768, 0.06094409999786876, 0.05909200001042336, 0.05965729999297764]

Serial:

- JAVA_TOOL_OPTIONS=-agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=*:5005 -Xmx6g  -Ddeephaven.console.type=${DEEPHAVEN_CONSOLE_TYPE} -Ddeephaven.application.dir=${DEEPHAVEN_APPLICATION_DIR}

• test_parallelization.py: [2.8941321000020253, 0.5896039999934146, 0.6469336000009207, 0.616867499993532, 0.5658154999982798]

• test_numba.py: [1.9125722000026144, 0.07256509999569971, 0.06443029998627026, 0.060122700000647455, 0.057902000000467524]