SummerSchool2023_IA_Sante.ipynb

{
  "cells": [
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# Traitement des données pour l'intelligence artificielle et la santé\n",
        "**Author:** [LudovicSaintBauzel](https://www.isir.upmc.fr/personnel/saintbauzel/)<br>\n",
        "**Date created:** 2023/06/01<br>\n",
        "**Last modified:** 2023/06/06<br>\n",
        "**Description:** Atelier sur l'apprentissage et le traitement des données pour le \"summer school\" Intelligence Artificielle et santé."
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "colab_type": "text",
        "id": "view-in-github"
      },
      "source": [
        "<a href=\"https://colab.research.google.com/github/LudovicSaintBauzel/AI4Health/blob/main/SummerSchool2023_IA_Sante.ipynb\" target=\"_parent\"><img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/></a>"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "MmiCawmLuazh"
      },
      "source": [
        "Initialisation des librairies pour le travail."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "k4SI4GvZtz26"
      },
      "outputs": [],
      "source": [
        "import pandas as pd\n",
        "import numpy as np\n",
        "import matplotlib.pyplot as plt\n",
        "import tensorflow as tf\n",
        "from sklearn.model_selection import train_test_split\n",
        "from sklearn.preprocessing import StandardScaler, RobustScaler\n",
        "import os\n",
        "import glob"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "9QI4ZeCtuE1H"
      },
      "source": []
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "8M9-NdiOuGm_"
      },
      "source": [
        "# Chargement des données"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "Gz6a8TX5t0cN"
      },
      "source": [
        "Récupérons les données d'une experience d'assistance au levé robotisé. "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "1378KaG9uV-r",
        "outputId": "a0ff7879-cb0d-4837-e29d-2f0a65ce4786"
      },
      "outputs": [],
      "source": [
        "!curl -O https://nuage.isir.upmc.fr/index.php/s/ZiPJ3RFGFaF3HBs/download/records.zip"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "V7l_8Ae0uLq7"
      },
      "source": [
        "# Analyse et travail sur les données"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "DCofHZeFuUX_",
        "outputId": "d02011b4-e97e-4407-aac9-69c78c2980af"
      },
      "outputs": [],
      "source": [
        "!unzip -o records.zip\n",
        "!ls "
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "lVRl5nch-uPj"
      },
      "source": [
        "## Observons l'organisation des données"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "PmXequtj9Afp"
      },
      "source": [
        "Maintenant que les données sont téléchargées, il est possible de connaître l'organisation des données en ouvrant le fichier (/content/records/DESCRIPTIONS_DONNEES.txt). On note que le fichier follow_traj.txt contient toutes les informations utiles pour la classification que nous souhaitons effectuer. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "mmex1RpgxOpZ",
        "outputId": "c50a68b0-aa13-4e0d-9b6d-1a7f0928b6ab"
      },
      "outputs": [],
      "source": [
        "os.chdir('/content/records/')\n",
        "\n",
        "# Liste des noms de fichiers CSV\n",
        "fichiers_csv = [[file for file in glob.glob(\"*/*/*/follow_traj.txt\")]]"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Une vision naïve du chargement des données "
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "donnees = []\n",
        "\n",
        "# Charger chaque fichier CSV et ajouter les données à la liste\n",
        "for fichier_csv in fichiers_csv[0]:\n",
        "    print(fichier_csv)\n",
        "    data = pd.read_csv(fichier_csv, sep='\\s+', header=None)\n",
        "    donnees.append(data)\n",
        "\n",
        "# Concaténer toutes les données en un seul DataFrame\n",
        "donnees_combinees = pd.concat(donnees)\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "En fait il existe une quantité de raison pour que les fichiers soient corrompus. Dans notre cas certains fichiers sont vides. ```os.stat st_size ```\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 287
        },
        "id": "6L3TVf4m8ZGG",
        "outputId": "aeae9d54-57c7-4d15-82a1-434aed2528af"
      },
      "outputs": [],
      "source": [
        "# Liste pour stocker les données de tous les fichiers\n",
        "donnees = []\n",
        "\n",
        "filtering_data = ['']\n",
        "#'records/test1/traj/follow_traj.txt',\n",
        "#                  'records/BO2504/test/follow_traj.txt',\n",
        "#                  'records/fuzzy/tt/follow_traj.txt',\n",
        "#                  'records/test1/fx/follow_traj.txt',\n",
        "#                  # Above Problematic data : Below test data not relevant\n",
        "#                  'records/test1/av/follow_traj.txt',\n",
        "#                  'records/test2/traj/follow_traj.txt',\n",
        "#                  'records/test2/av/follow_traj.txt',\n",
        "#                  'records/traj/test/follow_traj.txt',\n",
        "#                  'records/move/follow_traj.txt'\n",
        "#                  ]\n",
        "\n",
        "# Charger chaque fichier CSV et ajouter les données à la liste\n",
        "for fichier_csv in fichiers_csv[0]:\n",
        "    \n",
        "    ??? \n",
        "    if (???) and (fichier_csv not in filtering_data):\n",
        "        print(fichier_csv)\n",
        "        data = pd.read_csv(fichier_csv, sep='\\s+', header=None)\n",
        "        donnees.append(data)\n",
        "    else:\n",
        "        print(\"File filtered not loaded : \"+str(fichier_csv)+\"\\n\")\n",
        "\n",
        "# Concaténer toutes les données en un seul DataFrame\n",
        "donnees_combinees = pd.concat(donnees)\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Desfois il faut aussi supprimer des lignes aberrantes. Retrait des lignes qui contiennent \"not a number\". \n",
        "\n",
        "``` dropna ```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 134
        },
        "id": "jVYHzwV6GXMP",
        "outputId": "fe63c82f-f23d-44af-cd63-a83761821ed7"
      },
      "outputs": [],
      "source": [
        "\n",
        "donnees_combinees = ???\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "wLz5SoyeLw01"
      },
      "source": [
        "# Apprentissage des données"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "3KWhtgXfL60L"
      },
      "source": [
        "## Construction des données de validation et de test"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Diviser les données en features (X) et labels (y). Par exemple prenons les forces des poignées en entrées (features X)  et la sortie consRob en sortie. Pour cela on doit reprendre les bonnes colonnes grâce au fichier (/content/records/DESCRIPTIONS_DONNEES.txt).\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "5TglDaGiMLhN"
      },
      "outputs": [],
      "source": [
        "X = donnees_combinees.iloc[:, ???:???].values\n",
        "y = donnees_combinees.iloc[:, ???].values"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Diviser les données en ensembles d'entraînement et de test. ```train_test_split ```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.model_selection import train_test_split\n",
        "X_train, X_test, y_train, y_test = ???"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "On peut ensuite construire un scaler sur les données de train et l'appliquer aux données de test. \n",
        "\n",
        "```StandardScaler fit_transform```"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "\n",
        "from sklearn.preprocessing import RobustScaler, StandardScaler, MinMaxScaler\n",
        "# Normaliser les données\n",
        "???\n",
        "X_train = ???\n",
        "X_test = ???\n",
        "\n",
        "# Normaliser les données de sortie\n",
        "min_y_train = min(y_train)\n",
        "range_y_train = max(y_train) - min(y_train)\n",
        "y_train = (y_train)/range_y_train\n",
        "y_test = (y_test)/range_y_train\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {
        "id": "88f27_YxMFcB"
      },
      "source": [
        "# Apprentissage \n",
        "## Création du modèle de réseau de neurones"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "Comme expliqué il est important de comprendre l'enjeux des paramètres de notre modèle. Le point à garder à l'esprit est le nombre de données qui sont embarqué dans le réseau. Et de se garantir du pouvoir généralisant de la méthode. Etant donnée le fonctionnement de l'algorithme de rétropropagation, on peut difficilement envisager d'avoir moins de 10 données pour un lien dans le réseau. \n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "FciRpp5LMNoH"
      },
      "outputs": [],
      "source": [
        "import tensorflow as tf\n",
        "# Créer le modèle du réseau de neurones\n",
        "model = tf.keras.models.Sequential()\n",
        "model.add(tf.keras.layers.Dense(\n",
        "    20, activation='sigmoid', input_shape=(X_train.shape[1],)))\n",
        "# model.add(tf.keras.layers.Dense(4, activation='sigmoid'))\n",
        "\n",
        "#model.add(tf.keras.layers.Dense(12, activation='sigmoid'))\n",
        "model.add(tf.keras.layers.Dense(1, activation='sigmoid'))\n",
        "\n",
        "# Compiler le modèle\n",
        "model.compile(optimizer='adam', loss='binary_crossentropy',\n",
        "              metrics=['accuracy'])"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Apprentissage du modèle"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Entraîner le modèle\n",
        "#\n",
        "epochs = 20\n",
        "callbacks = [\n",
        "    #tf.keras.callbacks.ModelCheckpoint(\"save_at_{epoch}.keras\"),\n",
        "]\n",
        "\n",
        "history = model.fit(X_train, y_train, epochs=epochs,\n",
        "                    batch_size=8, callbacks=callbacks, validation_data=(X_test, y_test))\n"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "## Affichage de l'évolution de l'apprentissage et de la sortie du système"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "import matplotlib.pyplot as plt \n",
        "# plot\n",
        "accuracy = history.history['accuracy']\n",
        "loss = history.history['loss']\n",
        "val_accuracy = history.history['val_accuracy']\n",
        "val_loss = history.history['val_loss']\n",
        "\n",
        "# Tracer la précision\n",
        "plt.plot(accuracy, label='Entraînement')\n",
        "plt.plot(val_accuracy, label='Validation')\n",
        "plt.title('Précision du modèle')\n",
        "plt.xlabel('Époque')\n",
        "plt.ylabel('Précision')\n",
        "plt.legend()\n",
        "plt.show()\n",
        "\n",
        "# Tracer la perte\n",
        "plt.plot(loss, label='Entraînement')\n",
        "plt.plot(val_loss, label='Validation')\n",
        "plt.title('Perte du modèle')\n",
        "plt.xlabel('Époque')\n",
        "plt.ylabel('Perte')\n",
        "plt.legend()\n",
        "plt.show()\n",
        "# Tracer la performance du modèle\n",
        "\n",
        "# Évaluer le modèle\n",
        "yres = model.predict(X_test)\n",
        "plt.plot(yres[0:100], label='Prédiction')\n",
        "plt.plot(y_test[0:100], label='Réel')\n",
        "plt.title('Performance du modèle')\n",
        "plt.xlabel('Échantillon')\n",
        "plt.ylabel('Valeur')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    },
    {
      "attachments": {},
      "cell_type": "markdown",
      "metadata": {},
      "source": [
        "# regression"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "from sklearn.linear_model import ElasticNet\n",
        "# Créer un regressor ElasticNet\n",
        "regressor = ElasticNet(alpha=0.1, l1_ratio=0.5)\n",
        "\n",
        "# Entraîner le modèle\n",
        "#\n",
        "epochs = 20\n",
        "callbacks = [\n",
        "    #tf.keras.callbacks.ModelCheckpoint(\"save_at_{epoch}.keras\"),\n",
        "]\n",
        "    \n",
        "# Entraîner le regressor sur les données d'entraînement\n",
        "res = regressor.fit(X_train, y_train)\n",
        "res\n"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {},
      "outputs": [],
      "source": [
        "# Faire des prédictions sur les données de test\n",
        "predictions = regressor.predict(X_test)\n",
        "\n",
        "plt.plot(predictions[0:100], label='Prédiction')\n",
        "plt.plot(y_test[0:100], label='Réel')\n",
        "plt.title('Performance du modèle')\n",
        "plt.xlabel('Échantillon')  \n",
        "plt.ylabel('Valeur')\n",
        "plt.legend()\n",
        "plt.show()"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "authorship_tag": "ABX9TyPNIx3f9Jfc8M7MGENOqhHU",
      "include_colab_link": true,
      "provenance": [],
      "toc_visible": true
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.11.3"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}