From 169d1625fd93d99c481051f78047464e43fdee02 Mon Sep 17 00:00:00 2001 From: Don Acosta <97529984+acostadon@users.noreply.github.com> Date: Wed, 29 May 2024 08:43:47 -0400 Subject: [PATCH] adding notebook to demo nx_cugraph (#4366) This notebook will be used to demontstrate how to use nx-cugraph and show the speed-up. Authors: - Don Acosta (https://github.com/acostadon) Approvers: - Brad Rees (https://github.com/BradReesWork) URL: https://github.com/rapidsai/cugraph/pull/4366 --- docs/cugraph/source/nx_cugraph/nx_cugraph.md | 159 +++--------- .../nx_cugraph_codeless_switching.ipynb | 244 ++++++++++++++++++ 2 files changed, 275 insertions(+), 128 deletions(-) create mode 100644 notebooks/cugraph_benchmarks/nx_cugraph_codeless_switching.ipynb diff --git a/docs/cugraph/source/nx_cugraph/nx_cugraph.md b/docs/cugraph/source/nx_cugraph/nx_cugraph.md index 92fbf90a43b..ff2fc3d1da8 100644 --- a/docs/cugraph/source/nx_cugraph/nx_cugraph.md +++ b/docs/cugraph/source/nx_cugraph/nx_cugraph.md @@ -24,142 +24,45 @@ Each chart has three measurements. ![Single Source Shortest Path](../images/sssp.png) ![Weakly Connected Components](../images/wcc.png) +### Command line example +Open bc_demo.ipy and paste the code below. -The following algorithms are supported and automatically dispatched to nx-cuGraph for acceleration. +``` +import pandas as pd +import networkx as nx + +url = "https://data.rapids.ai/cugraph/datasets/cit-Patents.csv" +df = pd.read_csv(url, sep=" ", names=["src", "dst"], dtype="int32") +G = nx.from_pandas_edgelist(df, source="src", target="dst") -#### Algorithms +%time result = nx.betweenness_centrality(G, k=10) +``` +Run the command: ``` -bipartite - ├─ basic - │ └─ is_bipartite - └─ generators - └─ complete_bipartite_graph -centrality - ├─ betweenness - │ ├─ betweenness_centrality - │ └─ edge_betweenness_centrality - ├─ degree_alg - │ ├─ degree_centrality - │ ├─ in_degree_centrality - │ └─ out_degree_centrality - ├─ eigenvector - │ └─ eigenvector_centrality - └─ katz - └─ katz_centrality -cluster - ├─ average_clustering - ├─ clustering - ├─ transitivity - └─ triangles -community - └─ louvain - └─ louvain_communities -components - ├─ connected - │ ├─ connected_components - │ ├─ is_connected - │ ├─ node_connected_component - │ └─ number_connected_components - └─ weakly_connected - ├─ is_weakly_connected - ├─ number_weakly_connected_components - └─ weakly_connected_components -core - ├─ core_number - └─ k_truss -dag - ├─ ancestors - └─ descendants -isolate - ├─ is_isolate - ├─ isolates - └─ number_of_isolates -link_analysis - ├─ hits_alg - │ └─ hits - └─ pagerank_alg - └─ pagerank -operators - └─ unary - ├─ complement - └─ reverse -reciprocity - ├─ overall_reciprocity - └─ reciprocity -shortest_paths - └─ unweighted - ├─ single_source_shortest_path_length - └─ single_target_shortest_path_length -traversal - └─ breadth_first_search - ├─ bfs_edges - ├─ bfs_layers - ├─ bfs_predecessors - ├─ bfs_successors - ├─ bfs_tree - ├─ descendants_at_distance - └─ generic_bfs_edges -tree - └─ recognition - ├─ is_arborescence - ├─ is_branching - ├─ is_forest - └─ is_tree +user@machine:/# ipython bc_demo.ipy ``` -#### Generators +You will observe a run time of approximately 7 minutes...more or less depending on your cpu. + +Run the command again, this time specifiying cugraph as the NetworkX backend of choice. +``` +user@machine:/# NETWORKX_BACKEND_PRIORITY=cugraph ipython bc_demo.ipy ``` -classic - ├─ barbell_graph - ├─ circular_ladder_graph - ├─ complete_graph - ├─ complete_multipartite_graph - ├─ cycle_graph - ├─ empty_graph - ├─ ladder_graph - ├─ lollipop_graph - ├─ null_graph - ├─ path_graph - ├─ star_graph - ├─ tadpole_graph - ├─ trivial_graph - ├─ turan_graph - └─ wheel_graph -community - └─ caveman_graph -small - ├─ bull_graph - ├─ chvatal_graph - ├─ cubical_graph - ├─ desargues_graph - ├─ diamond_graph - ├─ dodecahedral_graph - ├─ frucht_graph - ├─ heawood_graph - ├─ house_graph - ├─ house_x_graph - ├─ icosahedral_graph - ├─ krackhardt_kite_graph - ├─ moebius_kantor_graph - ├─ octahedral_graph - ├─ pappus_graph - ├─ petersen_graph - ├─ sedgewick_maze_graph - ├─ tetrahedral_graph - ├─ truncated_cube_graph - ├─ truncated_tetrahedron_graph - └─ tutte_graph -social - ├─ davis_southern_women_graph - ├─ florentine_families_graph - ├─ karate_club_graph - └─ les_miserables_graph +This run will be much faster, typically around 20 seconds depending on your GPU. +``` +user@machine:/# NETWORKX_BACKEND_PRIORITY=cugraph ipython bc_demo.ipy +``` +There is also an option to add caching. This will dramatically help performance when running multiple algorithms on the same graph. +``` +NETWORKX_BACKEND_PRIORITY=cugraph CACHE_CONVERTED_GRAPH=True ipython bc_demo.ipy ``` -#### Other +When running Python interactively, cugraph backend can be specified as an argument in the algorithm call. +For example: ``` -convert_matrix - ├─ from_pandas_edgelist - └─ from_scipy_sparse_array +nx.betweenness_centrality(cit_patents_graph, k=k, backend="cugraph") ``` + + +The latest list of algorithms that can be dispatched to nx-cuGraph for acceleration is found [here](https://github.com/rapidsai/cugraph/blob/main/python/nx-cugraph/README.md#algorithms). diff --git a/notebooks/cugraph_benchmarks/nx_cugraph_codeless_switching.ipynb b/notebooks/cugraph_benchmarks/nx_cugraph_codeless_switching.ipynb new file mode 100644 index 00000000000..e05544448b1 --- /dev/null +++ b/notebooks/cugraph_benchmarks/nx_cugraph_codeless_switching.ipynb @@ -0,0 +1,244 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "# Benchmarking Performance of NetworkX with Rapids GPU-based nx_cugraph backend vs on cpu\n", + "# Skip notebook test\n", + "This notebook demonstrates compares the performance of nx_cugraph as a dispatcher for NetworkX algorithms. \n", + "\n", + "We do this by executing Betweenness Centrality, Breadth First Search and Louvain Community Detection, collecting run times with and without nx_cugraph backend and graph caching enabled. nx_cugraph is a registered NetworkX backend. Using it is a zero code change solution.\n", + "\n", + "In the notebook switching to the nx-cugraph backend is done via variables set using the [NetworkX config package](https://networkx.org/documentation/stable/reference/backends.html#networkx.utils.configs.NetworkXConfig) **which requires networkX 3.3 or later !!**\n", + "\n", + "\n", + "They can be set at the command line as well.\n", + "\n", + "### See this example from GTC Spring 2024\n", + "\n", + "\n", + "\n", + "Here is a sample minimal script to demonstrate No-code-change GPU acceleration using nx-cugraph.\n", + "\n", + "----\n", + "bc_demo.ipy:\n", + "\n", + "```\n", + "import pandas as pd\n", + "import networkx as nx\n", + "\n", + "url = \"https://data.rapids.ai/cugraph/datasets/cit-Patents.csv\"\n", + "df = pd.read_csv(url, sep=\" \", names=[\"src\", \"dst\"], dtype=\"int32\")\n", + "G = nx.from_pandas_edgelist(df, source=\"src\", target=\"dst\")\n", + "\n", + "%time result = nx.betweenness_centrality(G, k=10)\n", + "```\n", + "----\n", + "Running it with the nx-cugraph backend looks like this:\n", + "```\n", + "user@machine:/# ipython bc_demo.ipy\n", + "CPU times: user 7min 38s, sys: 5.6 s, total: 7min 44s\n", + "Wall time: 7min 44s\n", + "\n", + "user@machine:/# NETWORKX_BACKEND_PRIORITY=cugraph ipython bc_demo.ipy\n", + "CPU times: user 18.4 s, sys: 1.44 s, total: 19.9 s\n", + "Wall time: 20 s\n", + "```\n", + "----\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "First import the needed packages" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "import pandas as pd\n", + "import networkx as nx\n", + "import time\n", + "import os" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This installs the NetworkX cuGraph dispatcher if not already present." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "try: \n", + " import nx_cugraph\n", + "except ModuleNotFoundError:\n", + " os.system('conda install -c rapidsai -c conda-forge -c nvidia nx-cugraph')" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "This is boiler plate NetworkX code to run:\n", + "* betweenness Centrality\n", + "* Bredth first Search\n", + "* Louvain community detection\n", + "\n", + "and report times. it is completely unaware of cugraph or GPU-based tools.\n", + "[NetworkX configurations](https://networkx.org/documentation/stable/reference/utils.html#backends) can determine how they are run." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "def run_algos(G):\n", + " runtime = time.time()\n", + " result = nx.betweenness_centrality(G, k=10)\n", + " print (\"Betweenness Centrality time: \" + str(round(time.time() - runtime))+ \" seconds\")\n", + " runtime = time.time()\n", + " result = nx.bfs_tree(G,source=1)\n", + " print (\"Breadth First Search time: \" + str(round(time.time() - runtime))+ \" seconds\")\n", + " runtime = time.time()\n", + " result = nx.community.louvain_communities(G,threshold=1e-04)\n", + " print (\"Louvain time: \" + str(round(time.time() - runtime))+ \" seconds\")\n", + " return" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Downloads a patent citation dataset containing 3774768 nodes and 16518948 edges and loads it into a NetworkX graph." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "filepath = \"./data/cit-Patents.csv\"\n", + "\n", + "if os.path.exists(filepath):\n", + " print(\"File found\")\n", + " url = filepath\n", + "else:\n", + " url = \"https://data.rapids.ai/cugraph/datasets/cit-Patents.csv\"\n", + "df = pd.read_csv(url, sep=\" \", names=[\"src\", \"dst\"], dtype=\"int32\")\n", + "G = nx.from_pandas_edgelist(df, source=\"src\", target=\"dst\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Setting the NetworkX dispatcher with an environment variable or in code using NetworkX config package which is new to [NetworkX 3.3 config](https://networkx.org/documentation/stable/reference/backends.html#networkx.utils.configs.NetworkXConfig).\n", + "\n", + "These convenience settinge allow turning off caching and cugraph dispatching if you want to see how long cpu-only takes.\n", + "This example using an AMD Ryzen Threadripper PRO 3975WX 32-Cores cpu completed in slightly over 40 minutes." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "use_cugraph = True\n", + "cache_graph = True" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "if use_cugraph:\n", + " nx.config[\"backend_priority\"]=['cugraph']\n", + "else:\n", + " # Use this setting to turn off the cugraph dispatcher running in legacy cpu mode.\n", + " nx.config[\"backend_priority\"]=[]\n", + "if cache_graph:\n", + " nx.config[\"cache_converted_graphs\"]= True\n", + "else:\n", + " # Use this setting to turn off graph caching which will convertthe NetworkX to a gpu-resident graph each time an algorithm is run.\n", + " nx.config[\"cache_converted_graphs\"]= False\n" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "Run the algorithms on GPU. \n", + "\n", + "**Note the messages NetworkX generates to remind us cached graph shouldn't be modified.**\n", + "\n", + "```\n", + "For the cache to be consistent (i.e., correct), the input graph must not have been manually mutated since the cached graph was created.\n", + "\n", + "Using cached graph for 'cugraph' backend in call to bfs_edges.\n", + "```" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": {}, + "outputs": [], + "source": [ + "%%time\n", + "run_algos(G)\n", + "print (\"Total Algorithm run time\")" + ] + }, + { + "cell_type": "markdown", + "metadata": {}, + "source": [ + "___\n", + "Copyright (c) 2024, NVIDIA CORPORATION.\n", + "\n", + "Licensed under the Apache License, Version 2.0 (the \"License\"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0\n", + "\n", + "Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an \"AS IS\" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.\n", + "___" + ] + } + ], + "metadata": { + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.8" + } + }, + "nbformat": 4, + "nbformat_minor": 4 +}