GDS_Workshop.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Graph Data Science Workshop with Neo4j\n",
    "\n",
    "Click on the link below to open a Colab version of the notebook. You will be able to create your own version."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "id": "JAPoU8Sm5E6e"
   },
   "source": [
    "<table align=\"left\">\n",
    "\n",
    "  <td>\n",
    "    <a href=\"https://colab.research.google.com/github/neo4j-field/graph-summit-apac-2023/blob/main/GDS_Workshop.ipynb\" target=\"_blank\">\n",
    "      <img src=\"https://cloud.google.com/ml-engine/images/colab-logo-32px.png\" alt=\"Colab logo\">Run your own notebook in Colab\n",
    "    </a>\n",
    "  </td>\n",
    "</table>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Target\n",
    "\n",
    "Do fraud analysis on a group of persons and transactions using graphs and data science.  \n",
    "\n",
    "## Context\n",
    "\n",
    "This notebook allows you load a dataset based on an updated version of [Paysim](https://www.sisu.io/posts/paysim/).  \n",
    "PaySim is an approximation using an agent-based model and some anonymized, aggregate transactional data from a real mobile money network operator to create synthetic financial data sets academics and hackers can use for exploring ways to detect fraudulent behavior.  \n",
    "Using this [code](https://github.com/voutilad/paysim), you can generate your own dataset with different caracteristics (size, fraud occurences...).    \n",
    "\n",
    "We're going to leverage [Neo4j Graph Data Science (GDS)](https://neo4j.com/docs/graph-data-science/current/algorithms/) to investigate through the data and find out fraud patterns and fraudsters.  \n",
    "\n",
    "## Dataset\n",
    "\n",
    "The dataset used in this notebook represents money transfers between around 2500 clients, 75 merchants, 5 banks with 175000 transactions across 30 days.  \n",
    "There are 5 types of transactions:  \n",
    "* CashIn: a client moves money into the network via a merchant\n",
    "* CashOut: a client moves money out of the network via a merchant\n",
    "* Debit: a client moves money into a bank\n",
    "* Transfer: a client sends money to another client\n",
    "* Payment: a client exchanges money for something via a merchant\n",
    "\n",
    "We will try to identify Clients which are fraudsters, potentially targeting other users with fake accounts to accept payments for goods that stolen, illegal or even non existent.  \n",
    "We added to the original Paysim model some clients details (Phone, Email, SSN) to help identify clients using fake or stolen credentials to gain access to the system (first party fraud).  \n",
    "\n",
    "---\n",
    "\n",
    "## Let's get a graph database\n",
    "\n",
    "We will use a Neo4j graph database created on the [Neo4j sandbox](https://neo4j.com/sandbox/).  \n",
    "Once connected, on the _Select a project_ page, go to the section _Your own data_ and select the _Blank Sandbox_.  \n",
    "Click on the _Create_ button at the bottom of the page.  \n",
    "After few seconds, you should see the below.  \n",
    "<img src=\"./img/sandbox_start.png\" alt=\"Sandbox Start\" width=\"75%\" title=\"Sandbox Start\">  \n",
    "\n",
    "And once it's up and running, you can access the connection details by clicking on the top right down arrow and picking the *Connection details* tab.  \n",
    "You will need 2 things:\n",
    "* Password  \n",
    "* Bolt URL   \n",
    "\n",
    "<img src=\"./img/sandbox_details.png\" alt=\"Sandbox Details\" width=\"75%\" title=\"Sandbox Details\">  \n",
    "\n",
    "---\n",
    "\n",
    "## Let's code\n",
    "\n",
    "First we will import the [Neo4j GDS python library](https://pypi.org/project/graphdatascience/)  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "D3z2cYxEUpWJ",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Install Neo4j GDS Python Client\n",
    "import sys\n",
    "!{sys.executable} -m pip install graphdatascience\n",
    "\n",
    "# Import our GDS entry point\n",
    "from graphdatascience import GraphDataScience"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Instantiate your GDS Session\n",
    "\n",
    "Use Neo4j/Bolt URI and credentials according to your setup  \n",
    "\n",
    "For local standalone instance Bolt connection without auth    \n",
    "`gds = GraphDataScience(\"bolt://localhost:7687\", auth=None)`  \n",
    "\n",
    "For local standalone instance Bolt connection with auth    \n",
    "`gds = GraphDataScience(\"bolt://localhost:7687\", auth=(\"neo4j\", \"<password>\"))`  \n",
    "\n",
    "For remote cluster Neo4j connection with auth  \n",
    "`gds = GraphDataScience(\"neo4j://<FQDN or IP Address>:7687\", auth=(\"neo4j\", \"<password>\"))`  \n",
    "\n",
    "For remote standalone instance Bolt connection with auth   \n",
    "`gds = GraphDataScience(\"bolt://<FQDN or IP Address>:7687\", auth=(\"neo4j\", \"<password>\"))` "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "xi7xemCCWTwZ",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# >> Update the password and the URL here <<\n",
    "gds = GraphDataScience(\"bolt://xxx.xxx.xxx.xxx:7687\", auth=(\"neo4j\", \"your-database-password\")) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the GDS version installed"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "u3TcuHHHWTzR",
    "outputId": "622f8dc2-61b4-44b7-e028-be995db0dbed",
    "tags": []
   },
   "outputs": [],
   "source": [
    "print(f\"Neo4j GDS Version: {gds.version()}\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Optional - Set database if you're not using the default _neo4j_ database. \n",
    "\n",
    "Not applicable for Neo4j Sandbox as we have only one database named _neo4j_."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jNfTsoQKWpXt",
    "tags": []
   },
   "outputs": [],
   "source": [
    "#gds.set_database(\"my-db\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Cleaning the database or making it ready for a rerun of the notebook.\n",
    "We are starting with a fresh clean database, however if the database was previously loaded, we have the option to clear it out first here. Then we will use it to load the data from CSV files, running [Cypher](https://neo4j.com/developer/cypher/) queries. The RELOAD_DATA flag can be used to skip this step for experimenting with different algorithms later on."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "BZ-Bk77eWpac",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Set flag to control reloading of all data\n",
    "RELOAD_DATA = True\n",
    "\n",
    "\n",
    "if RELOAD_DATA: # Delete all, takes few miniutes on a full database\n",
    "    gds.run_cypher(\n",
    "        \"\"\"\n",
    "        MATCH (n) CALL {\n",
    "          WITH n\n",
    "          DETACH DELETE n\n",
    "        } IN TRANSACTIONS OF 10 ROWS;\n",
    "        \"\"\"\n",
    "    )\n",
    "else: # Reset the GDS properties when we re run the book without erasing all\n",
    "    gds.run_cypher(\n",
    "        \"\"\"\n",
    "        MATCH (c:Client) SET c.fraud_group = null, c.intra_fraud_group = null, c.score = null;\n",
    "        \"\"\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "execution": {
     "iopub.execute_input": "2023-04-12T07:09:48.034862Z",
     "iopub.status.busy": "2023-04-12T07:09:48.034711Z",
     "iopub.status.idle": "2023-04-12T07:09:48.056279Z",
     "shell.execute_reply": "2023-04-12T07:09:48.055279Z",
     "shell.execute_reply.started": "2023-04-12T07:09:48.034862Z"
    }
   },
   "source": [
    "### Test reading some data\n",
    "\n",
    "Using [LOAD CSV](https://neo4j.com/docs/cypher-manual/current/clauses/load-csv/), we are loading csv files into the database, creating the graph on the fly.  \n",
    "The first cell is to test the file access, by reading it and showing only the first 5 rows.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "T_STaebhWpcy",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Checking if we can access the data\n",
    "if RELOAD_DATA:\n",
    "    nodeListCSV = gds.run_cypher(\n",
    "    \"\"\"\n",
    "    LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/neo4j-field/graph-summit-apac-2023/main/data/clients.csv\" AS row\n",
    "    RETURN row.NAME as Name, row.PHONENUMBER as phoneNumber, row.SSN as SSN, row.EMAIL as email LIMIT 5\n",
    "    \"\"\"\n",
    "    )\n",
    "    \n",
    "# The object returned is a Pandas Data Frame, so we can explore using standard Pandas methods\n",
    "nodeListCSV.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating constraints and indexes\n",
    "\n",
    "For data integrity, we will create [constraints](https://neo4j.com/docs/cypher-manual/current/constraints/) to have a robust graph data model. Each constraint enforces uniqueness of an identifier for a given label. An index is also created for the name property on Client nodes, this allows fast lookups when querying clients by name."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "931Nv3tbaOOO",
    "tags": []
   },
   "outputs": [],
   "source": [
    "if RELOAD_DATA:\n",
    "    # First we create index\n",
    "    CONSTRAINTS = [\n",
    "      \"CREATE CONSTRAINT ClientConstraint IF NOT EXISTS FOR (p:Client) REQUIRE p.id IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT EmailConstraint IF NOT EXISTS FOR (p:Email) REQUIRE p.email IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT PhoneConstraint IF NOT EXISTS FOR (p:Phone) REQUIRE p.phoneNumber IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT SSNConstraint IF NOT EXISTS FOR (p:SSN) REQUIRE p.ssn IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT MerchantConstraint IF NOT EXISTS FOR (p:Merchant) REQUIRE p.id IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT BankConstraint IF NOT EXISTS FOR (p:Bank) REQUIRE p.id IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT TransactionConstraint IF NOT EXISTS FOR (p:Transaction) REQUIRE p.globalStep IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT DebitConstraint IF NOT EXISTS FOR (p:Transaction) REQUIRE p.globalStep IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT CashInConstraint IF NOT EXISTS FOR (p:CashIn) REQUIRE p.globalStep IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT CashOutConstraint IF NOT EXISTS FOR (p:CashOut) REQUIRE p.globalStep IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT TransferConstraint IF NOT EXISTS FOR (p:Transfer) REQUIRE p.globalStep IS UNIQUE;\",\n",
    "      \"CREATE CONSTRAINT PaymentConstraint IF NOT EXISTS FOR (p:Payment) REQUIRE p.globalStep IS UNIQUE;\",\n",
    "      \"CREATE INDEX      ClientNameIndex IF NOT EXISTS FOR (n:Client) ON (n.name)\"\n",
    "    ]\n",
    "    for c in CONSTRAINTS:\n",
    "        gds.run_cypher(c)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Loading all the data\n",
    "\n",
    "We will load 7 csv files:  \n",
    "* one for clients   \n",
    "* one for merchants  \n",
    "* five for transactions  \n",
    "\n",
    "We can see how each node is created with a label and at least one property.  \n",
    "We see all the relationships between all the nodes, we represent each transaction as a relationship between its participants.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "9gzLKgHkaboi",
    "tags": []
   },
   "outputs": [],
   "source": [
    "if RELOAD_DATA:\n",
    "    \n",
    "    # Load Clients data\n",
    "    gds.run_cypher(\n",
    "    \"\"\"\n",
    "        LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/neo4j-field/graph-summit-apac-2023/main/data/clients.csv\" AS row\n",
    "        WITH row\n",
    "        MERGE (c:Client { id: row.ID })\n",
    "        SET c.name = row.NAME\n",
    "        MERGE (p:Phone { phoneNumber: row.PHONENUMBER })\n",
    "        MERGE (c)-[:HAS_PHONE]->(p)\n",
    "        MERGE (s:SSN { ssn: row.SSN })\n",
    "        MERGE (c)-[:HAS_SSN]->(s)\n",
    "        MERGE (e:Email { email: row.EMAIL })\n",
    "        MERGE (c)-[:HAS_EMAIL]->(e);\n",
    "    \"\"\"\n",
    "    )\n",
    "    \n",
    "    # Load Merchants data\n",
    "    gds.run_cypher(\n",
    "    \"\"\"\n",
    "        LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/neo4j-field/graph-summit-apac-2023/main/data/merchants.csv\" AS row\n",
    "        WITH row\n",
    "        MERGE (m:Merchant { id: row.ID })\n",
    "        SET m.name = row.NAME, m.highRisk = toBoolean(row.HIGHRISK);\n",
    "    \"\"\"\n",
    "    )\n",
    "    \n",
    "    # Load Debit data\n",
    "    gds.run_cypher(\n",
    "    \"\"\"\n",
    "        LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/neo4j-field/graph-summit-apac-2023/main/data/debit.csv\" AS row\n",
    "        WITH row\n",
    "        MERGE (b:Bank { id: row.IDDEST })\n",
    "        SET b.name = row.NAMEDEST\n",
    "        MERGE (c:Client { id: row.IDORIG })\n",
    "        MERGE (t:Transaction:Debit { globalStep: toInteger(row.GLOBALSTEP) })\n",
    "        SET t.amount = toFloat(row.AMOUNT)\n",
    "        MERGE (t)-[:TO]->(b)\n",
    "        MERGE (c)-[:PERFORMED]->(t);\n",
    "    \"\"\"\n",
    "    )\n",
    "    \n",
    "    # Load CashIn data, largest file then takes few seconds\n",
    "    gds.run_cypher(\n",
    "    \"\"\"\n",
    "        LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/neo4j-field/graph-summit-apac-2023/main/data/cashin.csv\" AS row\n",
    "        CALL {\n",
    "            WITH row\n",
    "            MERGE (m:Merchant { id: row.IDDEST })\n",
    "            SET m.name = row.NAMEDEST\n",
    "            MERGE (c:Client { id: row.IDORIG })\n",
    "            MERGE (t:Transaction:CashIn { globalStep: toInteger(row.GLOBALSTEP) })\n",
    "            SET t.amount = toFloat(row.AMOUNT)\n",
    "            MERGE (t)-[:TO]->(m)\n",
    "            MERGE (c)-[:PERFORMED]->(t)\n",
    "        } IN TRANSACTIONS OF 10 ROWS;\n",
    "    \"\"\"\n",
    "    )\n",
    "    \n",
    "    # Load CashOut data\n",
    "    gds.run_cypher(\n",
    "    \"\"\"\n",
    "        LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/neo4j-field/graph-summit-apac-2023/main/data/cashout.csv\" AS row\n",
    "        CALL {\n",
    "            WITH row\n",
    "            MERGE (m:Merchant { id: row.IDDEST })\n",
    "            SET m.name = row.NAMEDEST\n",
    "            MERGE (c:Client { id: row.IDORIG })\n",
    "            SET c.name = row.NAMEORIG\n",
    "            MERGE (t:Transaction:CashOut { globalStep: toInteger(row.GLOBALSTEP) })\n",
    "            SET t.amount = toFloat(row.AMOUNT)\n",
    "            MERGE (t)-[:TO]->(m)\n",
    "            MERGE (c)-[:PERFORMED]->(t)\n",
    "        } IN TRANSACTIONS OF 10 ROWS;\n",
    "    \"\"\"\n",
    "    )\n",
    "    \n",
    "    # Load Payment data\n",
    "    gds.run_cypher(\n",
    "    \"\"\"\n",
    "        LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/neo4j-field/graph-summit-apac-2023/main/data/payment.csv\" AS row\n",
    "        CALL {\n",
    "            WITH row\n",
    "            MERGE (m:Merchant { id: row.IDDEST })\n",
    "            SET m.name = row.NAMEDEST\n",
    "            MERGE (c:Client { id: row.IDORIG })\n",
    "            SET c.name = row.NAMEORIG\n",
    "            MERGE (t:Transaction:Payment { globalStep: toInteger(row.GLOBALSTEP) })\n",
    "            SET t.amount = toFloat(row.AMOUNT)\n",
    "            MERGE (t)-[:TO]->(m)\n",
    "            MERGE (c)-[:PERFORMED]->(t)\n",
    "        } IN TRANSACTIONS OF 5 ROWS;\n",
    "    \"\"\"\n",
    "    )\n",
    "    \n",
    "    # Load Transfer data\n",
    "    gds.run_cypher(\n",
    "    \"\"\"\n",
    "    LOAD CSV WITH HEADERS FROM \"https://raw.githubusercontent.com/neo4j-field/graph-summit-apac-2023/main/data/transfer.csv\" AS row\n",
    "    CALL {\n",
    "        WITH row\n",
    "        MERGE (cd:Client { id: row.IDDEST })\n",
    "        SET cd.name = row.NAMEDEST\n",
    "        MERGE (co:Client { id: row.IDORIG })\n",
    "        SET co.name = row.NAMEORIG\n",
    "        MERGE (t:Transaction:Transfer { globalStep: toInteger(row.GLOBALSTEP) })\n",
    "        SET t.amount = toFloat(row.AMOUNT)\n",
    "        MERGE (t)-[:TO]->(cd)\n",
    "        MERGE (co)-[:PERFORMED]->(t)\n",
    "    } IN TRANSACTIONS OF 5 ROWS;\n",
    "    \"\"\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We have now taken a series of flat data sources and constructed a rich graph representation of the connections present in the sample dataset. At this point we have the following data model :\n",
    " \n",
    "<img src=\"./img/initial_data_model.png\" alt=\"Initial graph data model\" width=\"75%\"  title=\"Initial Graph Data Model\">  \n",
    "\n",
    "---\n",
    "### Enriching the graph\n",
    "\n",
    "Using the transaction details, we are able to enrich the model by adding the ordering of the transaction using the global step (a synthetic timestamp of sorts)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "4o2iOkxVa-it",
    "tags": []
   },
   "outputs": [],
   "source": [
    "if RELOAD_DATA:\n",
    "    # Update data model with new relationships\n",
    "    gds.run_cypher(\n",
    "    \"\"\"\n",
    "    MATCH (c:Client) with c.id as clientId\n",
    "    CALL {\n",
    "        WITH clientId\n",
    "        MATCH (c:Client {id: clientId})-[:PERFORMED]->(tx:Transaction)\n",
    "        WITH c, tx ORDER BY tx.globalStep\n",
    "        WITH c, collect(tx) AS txs\n",
    "        WITH c, txs, head(txs) AS _start, last(txs) AS _last\n",
    "\n",
    "        MERGE (c)-[:FIRST_TX]->(_start)\n",
    "        MERGE (c)-[:LAST_TX]->(_last)\n",
    "        WITH c, apoc.coll.pairsMin(txs) AS pairs\n",
    "\n",
    "        UNWIND pairs AS pair\n",
    "          WITH pair[0] AS a, pair[1] AS b\n",
    "          MERGE (a)-[n:NEXT]->(b)\n",
    "    } IN TRANSACTIONS OF 10 ROWS;\n",
    "    \"\"\"\n",
    "    )"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "These changes have created a new layer in the graph, where relationships show transactions in chronological order :\n",
    "\n",
    "<img src=\"./img/enhanced_data_model.png\" alt=\"Enhanced graph data model\" width=\"75%\" title=\"Enhanced Graph Data Model\"> \n",
    "\n",
    "This allows us to query and view transaction data in different ways, for example we can show Johns transactions as a group or in their order as below :\n",
    "\n",
    "Performed transactions | Ordered transactions\n",
    "- | - \n",
    "![alt](./img/performed_relationships.png) | ![alt](./img/ordered_relationships.png)\n",
    "\n",
    "\n",
    "\n",
    "\n",
    "### Having a first look at the dataset\n",
    "\n",
    "Neo4j maintains statistics of the various node labels and relationship types found in the active database. We can gain access to this information using a call to apoc.meta.stats with our python client. Here we are simply asking for the stats and then listing the relative frequency of each label (transaction types, clients, merchants etc) that exist in our dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = gds.run_cypher(\n",
    "    \"\"\"\n",
    "    CALL apoc.meta.stats() YIELD nodeCount, labels\n",
    "    UNWIND keys(labels) as label\n",
    "    RETURN label as nodeLabel, \n",
    "        labels[label] as frequency,\n",
    "        round(toFloat(labels[label])/nodeCount, 3) as relativeFrequency\n",
    "    ORDER BY frequency DESC\n",
    "    \"\"\"\n",
    ")\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Let's have a look on how the money is exchanged across entities\n",
    "\n",
    "Here we are using some Cypher aggregations to perform analysis across all transactions in the database. The first phase calculates the total count and value of all transactions. Next we aggregate on each transaction type (label) to calculate the relative value and count percentages of each."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "result = gds.run_cypher(\n",
    "    \"\"\"\n",
    "    MATCH (t:Transaction)\n",
    "    WITH sum(t.amount) AS globalSum, count(t) AS globalCnt \n",
    "    MATCH (t:Transaction)\n",
    "    WITH labels(t)[1] as txType, count(t) as txCnt, sum(t.amount) as txTotal, globalSum, globalCnt\n",
    "    RETURN\n",
    "        txType,\n",
    "        toInteger(round(txTotal/1000000)) + 'M' AS TotalMarketValue,\n",
    "        round(100 * txTotal / globalSum, 1) AS `%MarketValue`,\n",
    "        round(100 * toFloat(txCnt) / globalCnt, 1) AS `%MarketTransactions`,\n",
    "        toInteger(txTotal / txCnt) AS AvgTransactionValue,\n",
    "        txCnt AS NumberOfTransactions\n",
    "    ORDER BY `%MarketTransactions` DESC\n",
    "    \"\"\"\n",
    ")\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "---\n",
    "## Let's do some data Graph Data Science ! \n",
    "\n",
    "Now that our graph is constructed and filled with data, we can use Neo4j Graph Data Science to look for anomolies in the graph that are often associated with fraudulent behaviour. \n",
    "\n",
    "One of the unique features of the Neo4j platform is that GDS can be used on projections generated directly from a live transactional database. Obviously this is critical for quickly identifying fraud as it occurs as opposed to performing batch post analysis on stale data.\n",
    "\n",
    "Our first step is to define a new graph projection. Projections are created in memory from live data and may be used immediately for analysis. Our first graph projection will be used to analyse client identification data provided at signup. All projections must be uniquely named, here we are checking if our 'firstPartyFraud' graph already exists and if so we can drop and recreate it. \n",
    "\n",
    "---"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "OtUtta8EqVEf",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# My first graph project name to use wcc algorithm\n",
    "graphName = 'firstPartyFraud'\n",
    "\n",
    "# Remove existing projection with the same name, in case of a re run of the notebook\n",
    "if gds.graph.exists(graphName).exists:\n",
    "    gds.graph.drop(gds.graph.get(graphName))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Discovering first party fraud\n",
    "\n",
    "By checking connections among clients based on the identity information from their accounts, we can identify potentially fake profiles and shady clients. Our projection only needs to contain the slice of data pertinent to the type of analysis we are performing. Projections may also be created directly from a Cypher query to target even more specific data when required, however in this case we are using a 'native projection' based on the types of nodes and relationships only.\n",
    "\n",
    "We can start with a memory estimate of our projection, this is an optional yet useful step for ensuring the size of our GDS instance is sufficient for the task at hand. No projection is created here, just some data estimating the size of the in memory footprint it would create.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "gds.graph.project.estimate(\n",
    "    ['Client', 'SSN', 'Email', 'Phone'],     # Nodes to be added in the projection\n",
    "    ['HAS_SSN', 'HAS_EMAIL', 'HAS_PHONE'])   # Relationships to be added in the projection"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see from the output that projections are highly optimised and very compact in memory as they contain only the information we request (in this case only selected nodes and their connections, no unnecessary properties).\n",
    "\n",
    "\n",
    "### Next, create the projection to be used\n",
    "\n",
    "Once we are happy with the estimate, we use very similar syntax to create the actual projection. Here we are using our name and receiving a reference as the variable 'projection'. In addition prejectionPandas is returned to provide some statistics of the creation and the projection itself."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "jg8EWiKaqVHO",
    "tags": []
   },
   "outputs": [],
   "source": [
    "projection, projectionPandas = gds.graph.project(\n",
    "    graphName, \n",
    "    ['Client', 'SSN', 'Email', 'Phone'], \n",
    "    ['HAS_SSN', 'HAS_EMAIL', 'HAS_PHONE'])\n",
    "\n",
    "projectionPandas"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "\n",
    "### Selecting our algorithm - Weakly Connected Components\n",
    "\n",
    "One hallmark of first party fraud is re-use of stolen personal data in the creation of multiple fraudulent accounts. Often bad actors purchase the same stolen information and it will therefore present as a number of accounts with various combinations of the same information.  \n",
    "\n",
    "When you look at the information shared by multiple accounts as a graph, groups of fraudulent accounts tend to form a strongly connected subgraph. Legitimate accounts are typically isolated or only reuse some information (maybe a phone number or email in common) whereas large groups of clients with many connections are often associated with stolen information.\n",
    "\n",
    "The [Weakly Connected Components](https://neo4j.com/docs/graph-data-science/current/algorithms/wcc/) algorithm is perfect for this purpose as it identifies these connected groups of users that are weakly connected to the rest of the user graph. If we can find larger groups of connected clients using WCC, there is a strong chance they are related to stolen data reuse and first party fraud.\n",
    "\n",
    "\n",
    "### Running WCC in streaming mode\n",
    "\n",
    "Algorithms can be run in a number of modes depending on the use case. The streaming mode returns the result of an algorithm as a stream, just like the return of a cypher query. Let's try executing the WCC algorithm on our new projection in the streaming mode."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 394
    },
    "id": "j7dls0G-qVMN",
    "outputId": "ea568b15-21d0-404c-a908-0d76bfc25fa9",
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = gds.wcc.stream(projection)\n",
    "result.head(10)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "result.groupby(['componentId']).count().sort_values('nodeId', ascending=False).head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "As we can see WCC streaming mode returns the component (or group) ID for each of the nodes represented in the projection. Just listing it didnt give very useful information however we can count the nodes in each group and find the ID of the largest groups discovered by the WCC algorithm (note these counts include all nodes in the group including the identifying data nodes).\n",
    "\n",
    "\n",
    "### Writing group information directly to the database\n",
    "\n",
    "While streaming mode is useful for identifying and analysing groups directly in the notebook, we may want to actually write group membership information into a property on the node itself to enable further analysis (either in Bloom or using it in a subsequent algorithm execution).\n",
    "\n",
    "We can use the write mode directly on algorithm execution. In this case the WCC algorithm will write the componentId directly to the node into a property of our choice. This works well when we are happy for all nodes to have data written.\n",
    "\n",
    "In the example below however, we *only* want to write a 'fraud_group' property when the node is a member of a group with more than 1 node... that is ignore groups of a single node as they are not of interest to our analysis. In this case we use cypher to process the output of the WCC algorithm, chose only those nodes in a group of size > 1, then manually write each componentId using a SET clause. Note also, this method only matches on Client labels as they are the only member of the groups we are interested in analysing. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "Z2NF86MlzHVR",
    "outputId": "dada8627-55e7-42ab-91ca-26296e4d19b2",
    "tags": []
   },
   "outputs": [],
   "source": [
    "#result_wcc = gds.wcc.write(projection, writeProperty='fraud_group')\n",
    "#result_wcc\n",
    "\n",
    "result_wcc = gds.run_cypher(\"\"\"\n",
    "CALL gds.wcc.stream('\"\"\" + graphName + \"\"\"') YIELD nodeId, componentId\n",
    "WITH componentId, collect(gds.util.asNode(nodeId).id) AS clientIds       // Fetch the Node instance from the db and use its PaySim id\n",
    "WITH *, size(clientIds) AS groupSize WHERE groupSize > 1                 // Note that in this case, clients is a list of paysim ids.\n",
    "UNWIND clientIds AS clientId                                             // Let's unwind the list, MATCH, and tag them individually.\n",
    "    MATCH (c:Client {id:clientId})\n",
    "    SET c.fraud_group = componentId;\n",
    "\"\"\")\n",
    "result_wcc"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Take a closer look at our potential fraud groups\n",
    "\n",
    "Now that we have identified our possible fraud groups and have a property to identify which group (if any) that each client is a member of, we can use this data to start looking closer at the larger identified groups. We  also create an index on our new fraud_group property to make Cypher queries referencing this property even faster."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "7d0hPxuYzHX3",
    "outputId": "9e0895f1-8f65-4b5f-c5ca-0bdf42ff6c45",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create an index on the new property just created by the wcc algorithm on Clients\n",
    "gds.run_cypher(\"CREATE INDEX ClientFraudIndex IF NOT EXISTS FOR (c:Client) on c.fraud_group;\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 363
    },
    "id": "IFvM5X4KzHaJ",
    "outputId": "740dab9c-9124-432f-82d2-4fad1665a524",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Look at the community created by the algorithm\n",
    "# We can see the biggest community has 10 elements\n",
    "result = gds.run_cypher(\"\"\"\n",
    "  MATCH (c:Client) WHERE c.fraud_group IS NOT NULL\n",
    "  WITH c.fraud_group AS groupId, collect(c.id) AS members\n",
    "  WITH groupId, size(members) AS groupSize\n",
    "  WITH collect(groupId) AS groupsOfSize, groupSize\n",
    "  RETURN groupSize, size(groupsOfSize) AS numOfGroups, groupsOfSize as FraudGroupIds\n",
    "  ORDER BY groupSize DESC;\n",
    "\"\"\")\n",
    "result.head(10)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Using Bloom to visualise fraud groups\n",
    "\n",
    "<img src=\"./img/opening_bloom.png\" alt=\"Opening Bloom\" width=\"75%\"  title=\"Opening Bloom\">\n",
    "\n",
    "Lets take a look at some of these communities in Neo4j Bloom.  We will download and import the perspective from the bloom directory of the workshop github repository. \n",
    "\n",
    "<a id=\"raw-url\" href=\"https://raw.githubusercontent.com/neo4j-field/graph-summit-apac-2023/main/bloom/graph_summit_workshop.json\">Click here to download Bloom Perspective</a>\n",
    " \n",
    "Now use the import feature button in bloom to add the perspective to our new Bloom instance.\n",
    "\n",
    "<img src=\"./img/import_perspective.png\" alt=\"Import Bloom Perspective\" width=\"75%\" title=\"Import Bloom Perspective\"> \n",
    "\n",
    "We can now click on the perspective to open it and explore the dataset further, for example using the search bar for \"Find client with name Carson Wynn\" and then using a scene action (right click) on Carson's node to \"Show suspected fraud group\" can provides us with information about this users common data with others in the group.\n",
    "\n",
    "Try the following search phrases using Bloom\n",
    "\n",
    "* Find client with name John Kirby\n",
    "  * Select and right click on John's node to use scene actions\n",
    "* Show largest first party fraud groups\n",
    "* Find client with name Carson Wynn\n",
    "  * Select and right click on Carson to explain fraud group\n",
    "\n",
    "<img src=\"./img/first_party_fraud.png\" alt=\"Fraud group 4162\" width=\"75%\" title=\"Fraud group 4162\"> \n",
    "\n",
    "\n",
    "\n",
    "## Finding interconnections *between* fraud groups\n",
    "\n",
    "While finding, identifying and removing fraudulent accounts is a great use case for graph analytics, the real power comes from being able to dig deeper and further into connections using multiple datasets assembled into a powerful representation of connections in your data. \n",
    "\n",
    "Now that we have suspected fraudulent accounts identified, what can we learn from any transaction activity they have been able to perform. There must be a way for members to profit from these accounts and looking deeper as connections between the groups might lead us to central players in a larger fraud operation.\n",
    "\n",
    "The following Cypher looks at transactional relationships that members of larger fraud groups have with accounts outside of their immediate group. Obviously transfers within the group are expected but looking at how money moves out of the group is a key to finding the central actors in a larger organisation. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 112
    },
    "id": "gan-K2JOdvsr",
    "outputId": "99097d3d-5fb5-432f-ce38-ce5bb7ec9035",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# We will focus on fraud groups above 5 members\n",
    "fraudGroupMinSize = 5\n",
    "\n",
    "result = gds.run_cypher(\"\"\"\n",
    "  MATCH (c:Client) WHERE c.fraud_group IS NOT NULL\n",
    "  WITH c.fraud_group AS groupId, collect(c.id) AS members\n",
    "  WITH groupId, size(members) AS groupSize WHERE groupSize > $gs\n",
    "  MATCH (:Client {fraud_group:groupId})-[]-(txn:Transaction)-[]-(c:Client)      \n",
    "  WHERE c.fraud_group IS NULL\n",
    "  UNWIND labels(txn) AS txnType                                                 \n",
    "  RETURN distinct(txnType), count(txnType);\n",
    "\"\"\", params= {'gs': fraudGroupMinSize} )\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Here we see among the hundreds of thousands of transactions in the dataset, there are a relatively small number of transactions that eminate outward from these groups. We can capture this information as a layer in the graph and use it to further analyse these \"suspicious\" connections.\n",
    "\n",
    "### Let's create a new property to identify suspect clients\n",
    "\n",
    "Let's use these suspicious connections to create a new meta-graph of TRANSACTED_WITH relationships. The Cypher code below identifies these suspects transacting outside of each fraud ring, marks them with a suspect property and connects them together with the new relationship type.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 81
    },
    "id": "vVzy325Hdvu-",
    "outputId": "379bb171-6001-4739-e0e7-bad10308e000",
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = gds.run_cypher(\"\"\"\n",
    "  MATCH (c:Client) WHERE c.fraud_group IS NOT NULL\n",
    "  WITH c.fraud_group AS groupId, collect(c.id) AS members\n",
    "  WITH groupId, size(members) AS groupSize WHERE groupSize > $gs\n",
    "  MATCH (c1:Client {fraud_group:groupId})-[]-(t:Transaction)-[]-(c2:Client)     \n",
    "  WHERE c2.fraud_group IS NULL\n",
    "  SET c1.suspect = true, c2.suspect = true                                      \n",
    "  MERGE (c1)-[r:TRANSACTED_WITH]->(c2)                                          \n",
    "  ON CREATE SET r += t\n",
    "  RETURN count(r);\n",
    "\"\"\", params= {'gs': fraudGroupMinSize})\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### We found some suspect transactions, let's investigate them using again WCC\n",
    "\n",
    "Now we have built a metagraph of transactions between members of our previously identified fraud groups and other clients outside of those groups, we are able to create a projection of these \"intra-group\" connections for further analysis."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "graphName2 = 'intraGroupTransactions'\n",
    "\n",
    "# Remove existing graph with the same name\n",
    "if gds.graph.exists(graphName2).exists:\n",
    "    gds.graph.drop(gds.graph.get(graphName2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Creating a new projection using only suspect clients\n",
    "\n",
    "This time we are using a [cypher projection](https://neo4j.com/docs/graph-data-science/current/management-ops/projections/graph-project-cypher/) that specifically targets only those client nodes marked as suspects and our new TRANSACTED_WITH relationships."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "yzo487KpkRGh",
    "tags": []
   },
   "outputs": [],
   "source": [
    "projection2, projectionPandas2 = gds.graph.project.cypher(graphName2, \n",
    "          'MATCH (c:Client {suspect:true}) RETURN id(c) AS id', \n",
    "          'MATCH (c1:Client {suspect:true})-[r:TRANSACTED_WITH]->(c2:Client) RETURN id(c1) AS source, id(c2) as target')\n",
    "projectionPandas2"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can now run the Weakly Connected Components algorithm across this subgraph projection to identify any communities that exist in this connected web of individual fraud groups. Note this time we are executing the WCC algorithm in 'write' mode, which will directly write the detected group id for every projected node as a property on these nodes in the database."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "__-LyayFkRLD",
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = gds.wcc.write(projection2, writeProperty='intra_fraud_group');"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 49
    },
    "id": "iFgy6whHk0MK",
    "outputId": "1485b417-20b5-4aa2-9f61-9f8942bd3a09",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# Create an index on the new property\n",
    "gds.run_cypher(\"CREATE INDEX IntraGroupIndex IF NOT EXISTS FOR (c:Client) on c.intra_fraud_group;\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/",
     "height": 206
    },
    "id": "uTHK8ZwXk0Pz",
    "outputId": "9978be1c-bae1-4ef2-8a2c-aef4f0388237",
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = gds.run_cypher(\"\"\"\n",
    "MATCH (c:Client) WHERE c.intra_fraud_group IS NOT NULL\n",
    "WITH c.intra_fraud_group AS intraGroupId, collect(c.id) AS members\n",
    "RETURN intraGroupId, size(members) AS groupSize\n",
    "ORDER BY groupSize DESC;\n",
    "\"\"\")\n",
    "result.head(5)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Finding the *really* bad actors using Betweenness Centrality\n",
    "\n",
    "Now we have discovered that there are communities of transaction activity between our original first party fraud groups, what information can we glean from it ? Typically when we have multiple 'cells' of fraudulent activity, there needs to be a process for 'exiting' or profiting from the underlying activity. Often we are looking for a central entity via which most of the fraudulent activity will eventually flow. \n",
    "\n",
    "These accounts may at first have appeared legitimate and have been created with unique credentials that did not flag them as fraudulent, however using the [Betweenness Centrality](https://neo4j.com/docs/graph-data-science/current/algorithms/betweenness-centrality/) Graph Data Science algorithm we can quickly find these central nodes in a wider fraud operation. \n",
    "\n",
    "We will make our final projection of the largest \"intra-fraud\" group discovered in the previous step. This will be the group that connects the most first party fraud \"cells\" together and it is likely to uncover and particularly important or central players in the wider operation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "tags": []
   },
   "outputs": [],
   "source": [
    "graphName3 = 'betweenness'\n",
    "\n",
    "# Remove existing graph with the same name\n",
    "if gds.graph.exists(graphName3).exists:\n",
    "    gds.graph.drop(gds.graph.get(graphName3))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "id": "ICdNnyXylOgQ",
    "tags": []
   },
   "outputs": [],
   "source": [
    "# This projection selects only the largest intra fraud group community\n",
    "projection3, projectionPandas3 = gds.graph.project.cypher(graphName3, \n",
    "    \"\"\"MATCH (c:Client) WHERE c.intra_fraud_group IS NOT NULL WITH c.intra_fraud_group AS secondGroupId, collect(c.id) AS members \n",
    "       WITH secondGroupId, size(members) AS groupSize ORDER BY groupSize DESC LIMIT 1 \n",
    "       MATCH (c:Client {intra_fraud_group:secondGroupId})-[r:TRANSACTED_WITH]-(c2:Client) \n",
    "       RETURN id(c) AS id\n",
    "    \"\"\",\n",
    "    \"\"\"MATCH (c:Client) WHERE c.intra_fraud_group IS NOT NULL WITH c.intra_fraud_group AS secondGroupId, collect(c.id) AS members \n",
    "       WITH secondGroupId, size(members) AS groupSize ORDER BY groupSize DESC LIMIT 1 \n",
    "       MATCH (c1:Client {intra_fraud_group:secondGroupId})-[:TRANSACTED_WITH]-(c2:Client) \n",
    "       RETURN id(c1) AS source, id(c2) AS target\n",
    "    \"\"\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "colab": {
     "base_uri": "https://localhost:8080/"
    },
    "id": "ploAIAQPlizR",
    "outputId": "203c451b-b758-4bde-fd19-047dff602038",
    "tags": []
   },
   "outputs": [],
   "source": [
    "result = gds.betweenness.write(projection3, writeProperty='score')\n",
    "result"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Using Bloom to highlight key fraudsters !\n",
    "\n",
    "Now that we are able to identify the largest intra-fraud group community and have calculated a betweenness centrality score for each of the nodes in this group, we can visualise this data using Bloom. Using a [saved cypher search phase](https://neo4j.com/docs/bloom-user-guide/current/bloom-tutorial/search-phrases-advanced/), Bloom is able to search for and render the largest community.\n",
    "\n",
    "Bloom is also able to use rule based scene rendering to colour and size nodes and relationships based on any of their data properties. In this example we have used the betweenness centralitity score calculated above to highlight the central or important nodes in the suspected fraud community. \n",
    "\n",
    "Try the following search phrase in Bloom\n",
    "\n",
    "* Show intra-group transactions\n",
    "  * Use Bloom rule based styling to highlight centrality results\n",
    "\n",
    "<img src=\"./img/betweeness_analysis.png\" alt=\"Visualising betweeness centrality\" width=\"100%\" height=\"100%\" title=\"Visualising betweeness centrality\"> \n"
   ]
  }
 ],
 "metadata": {
  "colab": {
   "authorship_tag": "ABX9TyOiPOIwDvHXAbwXfUP4II87",
   "include_colab_link": true,
   "provenance": []
  },
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}