[FEA] Expose UMAP embedding graph

[lmeyerov]

Is your feature request related to a problem? Please describe.
We would like access to umap's computed graph for downstream tasks like visualization and other ML methods

See additional use cases discussed in #4228

Describe the solution you'd like

When computing an embedding, have an option to also expose the weighted graph / cover tree, such as via a numpy sparse matrix (how umap_learn does it) or a cugraph weighted graph

Ex:

embedding = umap.UMAP(n_components=2, emit_graph=True).fit(df)
coo = embedding.graph_.tocoo()
edges_df = pd.DataFrame({'src': coo.row, 'dst': coo.col, 'weight': coo.data})

Describe alternatives you've considered

umap_learn has umap.UMAP(transform_mode='graph', ...), except using that might mean having to call umap() twice. An explicit flag to expose the graph as part of the output may be more in line with expected use.

Currently, when using umap_learn, we do the above. When using cuML, we manually run knn to try to infer the graph from the embedding, but that's awkward and less accurate.

• @cjnolet @AjayThorve @trxcllnt

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[lmeyerov]

FWIW, this is of continued interest for use cases around security, fraud, genomics, and visualizing embeddings in general. We are discussing w/ the umap team on explainable AI approaches that build on this. Meanwhile, we're doing a k-nn edge-recovery dance to work around for RAPIDS flavors.

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[lmeyerov]

This is of increasing relevance fwiw :)

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[lmeyerov]

We are still interested in this. We're adding some autoumap bits to pygraphistry, and have to workaround when users switch from umap_learn to cuml.umap...

[github-actions]

This issue has been labeled inactive-30d due to no recent activity in the past 30 days. Please close this issue if no further response or action is needed. Otherwise, please respond with a comment indicating any updates or changes to the original issue and/or confirm this issue still needs to be addressed. This issue will be labeled inactive-90d if there is no activity in the next 60 days.

[lmeyerov]

Still of interest :) We are getting ready for the cuml engine variant of our new integrated auto-umap branch (graphistry/pygraphistry#305), and if helpful, I can point to some Nvidia+RAPIDS partners who will be using

cc @taureandyernv

[cjnolet]

@lmeyerov this shouldn't be too hard to do. Just to clarify- what you want is the fuzzy simplicial set graph here?

[lmeyerov]

Yep -- with priority for the one in the original space, not the embedding, and with the normalized/undirected weights.

Our intuition is the original is, for explainability, the original space's weighted 1-simplexes are already interpretable and more precise. Likewise, enables graph layout with the same initial seed. Getting the embedding's simplex is interesting too, mostly for enabling us to highlight which 1-simplexes were added vs lost.. but that's priority 2.

[lmeyerov]

ping :) we're about to release graphistry.nodes(cudf.read_csv('genomes.csv')).umap(engine='umap_learn').plot(), and would love to do graphistry.nodes(cudf.read_csv('genomes.csv')).umap(engine='cuml').plot() as well :)

[taureandyernv]

@dantegd @cjnolet will we be able to get this into 22.06?

[cjnolet]

@taureandyernv we are going to try and aim for 22.06. There's a PR open (#4711) to expose the simplicial set functions, which required the need to expose sparse objects in Python which were populated by the c++ layer (e.g. the number of nonzeros isn't known ahead of time) which should make exposing the connectivites graph from a trained model even easier.

[cjnolet]

@lmeyerov, https://github.com/rapidsai/cuml/pull/4756/files added the attribute model.graph_ so I'm going to close this as done. Please feel free to re-open if this still doesn't satisfy the requirement.

[lmeyerov]

Excellent, is this for the 22.06 release (and in nightly's already)?

[cjnolet]

@lmeyerov, yep, the feature made it into 22.06 (and the nightlies).