Dev/gfql endpoint

[lmeyerov]

Some very exciting things happening here -- maybe time to call pygraphistry 1.x.x ??

Remote dataset binding

Bind to remote (lazy):

g1 = graphistry.bind(dataset_id="abc123")

Decouple upload from plot(), and bind the upload:

g_uploaded = g_unuploaded.upload()
print(g_uploaded._dataset_id)
print(g_uploaded._nodes_file_id)
print(g_uploaded._edges_file_id)
print(g_uploaded._url)

Remote GFQL

# + auto-uploads if not already
g2 = g1.chain_remote([....], engine='gpu')

# no need to download results if you just care whether matched vs not
df_meta = g1.chain_remote_shape([...])

Remote Python too!

json_obj = g1.python_remote("""
from graphistry import Plottable

def task(g: Plottable):
  return {'status': True}
""")

---

Notes

Changes

• Now tracks dataset_id, nodes_file_id, edges_file_id, and url, including clearing them out whenever the nodes/edges gets re-bound with new data

• plot(render= expanded from bool to Union[bool, Literal["auto", "g", "ipython", "databricks", "browser"]] to make more predictable

[mj3cheun]

comment references gfql here

[mj3cheun]

also could clarify that if a dataset_id exists on the plottable it will be reused, as opposed to reuploading every time python_remote is called with it omitted

[lmeyerov]

so this is a bit problematic in that this would trigger re-upload, and not sure if that's 'right':

g2 = g1....

# auto-upload 1
res1 = g2.python_remote(script1)

# auto-upload 2
res2 = g2.python_remote(script2)

A few ideas:

• inplace update of g2 , breaking purity of v3 = g2.xyz()
• Some sort of global memoizable trick, so even if g2._dataset_id is not mutated on first upload, we can detect g2 did have a recent upload? Ex: some sort of global weak reference lookup table of last n g objects?
• Don't worry about it, and encourage users to do an explicit g2 = g1.upload()...

[mj3cheun]

didnt consider this... this cant happen in python_remote because that never modifies the plottable, but chain_remote definitely does.

my opinion is that perhaps we should avoid self mutation of plottable and instead return a "clone" (with identical metadata etc) of the original plottable with the updated node and edge lists? this should also better mirror the behaviour of non-remote chain anyway?

i dont have strong opinions on autouploading the clone

[lmeyerov]

Does the python endpoint return a Plottable, JSON, or any? How would I know ahead of time, or how would the python client sniff?

[mj3cheun]

for now the python endpoint returns either a string or a json. which one it returns is completely dependent on the return type of the function defined (by the user) in execute. the python endpoint code will detect the type of the returned value and use the appropriate response format.

you can read the MIME type of the response to get which

[lmeyerov]

hmm, it might not be so bad to extend the python endpoint to reuse the gfql endpoint's machinery here, i'll take a look

[lmeyerov]

(mimetype is cool, that helps!)

[lmeyerov]

I agree on avoiding mutation on the base obj, maybe we do some variants like:

# generic helper
_ : Any = g1.remote_python("...", output_mode=...)

# explicit mypy-friendly
g2 = g1.remote_python_g("...")
g2, g1_bound = g1.remote_python_g2("...")
o = g1.remote_python_json("...")
o, g1_bound = g1.remote_python_json2("...")

I'm tempted to not do the <xyz>2 variants above, and if someone wants to reuse uploads, they have to be explicit:

g1 = g0.upload(...)
# or g1 = graphistry.bind(dataset_id="...")

g2 = g1.remote_python_g(...)