20220915_jquantsapi_uki_predictor.ipynb

{"cells":[{"cell_type":"markdown","metadata":{"collapsed":false,"id":"MLHnF1ObPmHi","pycharm":{"name":"#%% md\n"}},"source":["[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/AlpacaDB/jquants-api-sample/blob/main/20220915_jquantsapi_uki_predictor.ipynb)"]},{"cell_type":"markdown","metadata":{"collapsed":true,"id":"ZNHMp04xPmHk","pycharm":{"name":"#%% md\n"}},"source":["# J-Quants APIを用いた財務情報発表後の高値安値予測モデル\n","\n","本ノートブックでは、J-Quants API から取得したデータを用いて、J-Quantsファンダメンタルズ分析チャレンジで第2位を受賞された[UKIさんのモデル](https://github.com/UKI000/JQuants-Forum/blob/452a4f4bc086ef0a8b087efc707c51abad5ed50e/jquants01_fund_uki_predictor.py) を動かすデモとなります。\n","\n","財務情報発表後の高値安値を予測するモデルとなっております。\n","\n","Google Colab上で動作確認を行っています。\n","\n","---\n","\n","**このノートブックはGoogle Driveを使用します。**\n","\n","- Google Drive の以下のフォルダーにデータを書き込みます。\n","    - `MyDrive/drive_ws/marketdata`\n","    - `MyDrive/drive_ws/models/20220915_uki_model`"]},{"cell_type":"markdown","metadata":{"id":"F6f8tHWoiO7i"},"source":["## J-Quants API 利用準備: リフレッシュトークンの取得\n","J-Quants APIを利用するためには [J-Quants API のメニューページ](https://jpx-jquants.com/dashboard/menu/) から取得できるリフレッシュトークンが必要になります。\n","会員登録とリフレッシュトークンの取得を完了させてから次に進みます。"]},{"cell_type":"markdown","metadata":{"id":"93Xp82bTiSaD"},"source":["## ライブラリとデータ保存先の設定"]},{"cell_type":"markdown","metadata":{"id":"O6b_t3dqiUlG"},"source":["必要なライブラリのinstall/importと設定を行います。\n","\n","本ノートブックでは、J-Quants API のPythonクライアントライブラリである [jquants-api-client-python](https://github.com/J-Quants/jquants-api-client-python) を使用します。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"KNdjpBoYPmHl","pycharm":{"name":"#%%\n"}},"outputs":[],"source":["# 必要なライブラリのインストールとインポート\n","!pip install scikit-learn xgboost pandas numpy jquants-api-client\n","\n","# J-Quants の株式分析チュートリアルからバックテスト用ライブラリを取得します。\n","# !wget -q https://raw.githubusercontent.com/JapanExchangeGroup/J-Quants-Tutorial/main/handson/Chapter03/backtest/backtest.py -O backtest.py\n","\n","import getpass\n","import io\n","import os\n","import pickle\n","from datetime import datetime, timedelta\n","from typing import List\n","\n","import jquantsapi\n","import numpy as np\n","import pandas as pd\n","from matplotlib import pyplot as plt\n","# from backtest import Backtest\n","from dateutil import tz\n","from requests import HTTPError\n","from xgboost.sklearn import XGBRegressor"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iUZsaZ0zRbFk","pycharm":{"name":"#%%\n"}},"outputs":[],"source":["# Googleドライブをマウント\n","from google.colab import drive\n","drive.mount('/content/drive')"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"0pc9_UtqjuAx","pycharm":{"name":"#%%\n"}},"outputs":[],"source":["# pandas の表示制限を調整します\n","pd.set_option(\"display.max_rows\", 1000)\n","pd.set_option(\"display.max_columns\", 1000)\n","pd.set_option(\"display.width\", 2000)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"nQfcFSG2RW0a","pycharm":{"name":"#%%\n"}},"outputs":[],"source":["# 必要なコンフィグの定義\n","# Googleドライブをマウントするディレクトリ\n","GOOGLE_DRIVE_MOUNT_DIR_PATH = \"/content/drive\"\n","# データを保存しているGoogleドライブ上のディレクトリ\n","STORAGE_DIR_PATH = f\"{GOOGLE_DRIVE_MOUNT_DIR_PATH}/MyDrive/drive_ws/marketdata\"\n","# デバッグ中\n","# STORAGE_DIR_PATH = \"/tmp/marketdata\"\n","# モデルを保存しているGoogleドライブ上のディレクトリ\n","MODEL_DIR_PATH = f\"{GOOGLE_DRIVE_MOUNT_DIR_PATH}/MyDrive/drive_ws/models\"\n","\n","# J-Quants API から取得するデータの期間\n","HISTORICAL_DATA_YEARS = 5\n","DAYS_IN_YESR = 365 # FIXME\n","start_dt: datetime = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0) - timedelta(days=(HISTORICAL_DATA_YEARS*365))\n","end_dt: datetime = datetime.now().replace(hour=0, minute=0, second=0, microsecond=0)\n","start_dt_yyyymmdd = start_dt.strftime(\"%Y%m%d\")\n","end_dt_yyyymmdd = end_dt.strftime(\"%Y%m%d\")\n","\n","# 各種CSVデータを保存するファイルパス\n","# 元データ\n","raw_stock_list_csvfile_path = f\"{STORAGE_DIR_PATH}/raw_stock_list_{start_dt_yyyymmdd}_{end_dt_yyyymmdd}.csv.gz\"\n","raw_stock_fins_csvfile_path = f\"{STORAGE_DIR_PATH}/raw_stock_fin_{start_dt_yyyymmdd}_{end_dt_yyyymmdd}.csv.gz\"\n","raw_stock_price_csvfile_path = f\"{STORAGE_DIR_PATH}/raw_stock_price_{start_dt_yyyymmdd}_{end_dt_yyyymmdd}.csv.gz\"\n","# 処理済みデータ\n","stock_list_csvfile_path = f\"{STORAGE_DIR_PATH}/stock_list_uki_model.csv.gz\"\n","stock_fins_csvfile_path = f\"{STORAGE_DIR_PATH}/stock_fin_uki_model.csv.gz\"\n","stock_price_csvfile_path = f\"{STORAGE_DIR_PATH}/stock_price_uki_model.csv.gz\"\n","stock_labels_csvfile_path = f\"{STORAGE_DIR_PATH}/stock_labels_uki_model.csv.gz\"\n","# 生成したモデルを保存するパス\n","model_path = f\"{MODEL_DIR_PATH}/20220915_uki_model\"\n","\n","# トレーニング・検証データ期間\n","PURGING_DAYS = 31\n","TRAIN_DATA_DAYS = (3 * 365) - PURGING_DAYS  # FIXME: データ期間から割合で決定する\n","train_start_dt: datetime = start_dt\n","train_end_dt: datetime = start_dt + timedelta(days=TRAIN_DATA_DAYS)\n","\n","# テスト・評価期間\n","test_start_dt = train_end_dt + timedelta(days=PURGING_DAYS)\n","\n","# デバッグ中は短い期間でチェック・・・\n","# train_start_dt: datetime = datetime(2022, 3, 1)\n","# train_end_dt: datetime = datetime(2022, 6,30)\n","# test_start_dt: datetime = datetime(2022, 7, 1)\n","# test_end_dt: datetime = datetime(2022, 7, 31)\n","# デバッグ用の銘柄\n","example_codes = [7203, 9201, 6758] # toyota, jal, sony (特に銘柄選定に意味はないです)\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"pSU_qUHghgke"},"outputs":[],"source":["# データ保存先のフォルダーを作成しておきます\n","os.makedirs(STORAGE_DIR_PATH, exist_ok=True)\n","# モデル保存先のフォルダーを作成しておきます\n","os.makedirs(model_path, exist_ok=True)"]},{"cell_type":"markdown","metadata":{"id":"DV7RUzQ6hxge"},"source":["## リフレッシュトークンの設定"]},{"cell_type":"markdown","metadata":{"id":"QZLukNwph3l_"},"source":["入力ボックスに https://jpx-jquants.com/dashboard/menu/ から取得したJ-Quants APIのリフレッシュトークンを入力してください。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"xlcGKlpsh0RP"},"outputs":[],"source":["refresh_token = getpass.getpass(prompt=\"J-Quants Refresh Token:\")\n","# リフレッシュトークンを使用できるか検証します。\n","test_cli = jquantsapi.Client(refresh_token=refresh_token)\n","try:\n","    id_token = test_cli.get_id_token()\n","    if len(id_token) > 0:\n","        print(\"refresh_tokenは正常です。次の手順に進んでください。\")\n","except HTTPError:\n","    print(\"refresh_tokenを使用できません。再度値を確認してください。\")"]},{"cell_type":"markdown","metadata":{"id":"UMRxCnmFicpK"},"source":["## データのダウンロードおよび保存\n","\n","銘柄一覧および、上記で設定したデータ取得期間(start_dt から end_dt) の\n","全銘柄の価格と財務情報データを取得します。加えて、取得したデータを保存しておきます。データ処理方法を変えたりしても再度データ取得しなくて良くなります。\n","\n","このステップの実行には環境によって数十分ほど時間がかかります。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"H5k1eJckievN"},"outputs":[],"source":["cli = jquantsapi.Client(refresh_token=refresh_token)\n","# cli = jquantsapi.Client()\n","\n","# 銘柄一覧\n","if not os.path.isfile(raw_stock_list_csvfile_path):\n","    stock_list_load: pd.DataFrame = cli.get_list()\n","    stock_list_load.to_csv(raw_stock_list_csvfile_path, compression=\"gzip\", index=False)\n","stock_list_load: pd.DataFrame = pd.read_csv(raw_stock_list_csvfile_path, dtype=str)\n","\n","# 株価情報\n","if not os.path.isfile(raw_stock_price_csvfile_path):\n","    stock_price_load: pd.DataFrame = cli.get_price_range(start_dt=start_dt, end_dt=end_dt)\n","    stock_price_load.to_csv(raw_stock_price_csvfile_path, compression=\"gzip\", index=False)\n","stock_price_load: pd.DataFrame = pd.read_csv(raw_stock_price_csvfile_path, dtype=str)\n","\n","# 財務情報\n","if not os.path.isfile(raw_stock_fins_csvfile_path):\n","    stock_fin_load: pd.DataFrame = cli.get_statements_range(start_dt=start_dt, end_dt=end_dt)\n","    stock_fin_load.to_csv(raw_stock_fins_csvfile_path, compression=\"gzip\", index=False)\n","stock_fin_load: pd.DataFrame = pd.read_csv(raw_stock_fins_csvfile_path, dtype=str)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ABKT8eWwih1d"},"outputs":[],"source":["# 上記で保存したファイルを削除するためには以下のコマンドを実行します\n","# 以降のセルでエラーが出た際にファイルを削除することをためしてみてください。\n","# os.remove(raw_stock_list_csvfile_path)\n","# os.remove(raw_stock_price_csvfile_path)\n","# os.remove(raw_stock_fins_csvfile_path)"]},{"cell_type":"markdown","metadata":{"id":"hDndMzmWjnSK"},"source":["## データ処理\n","\n","過去にJ-Quants株式分析チュートリアルやデータ分析コンペティションで用いられたデータとの互換性をなるべく取るため、データの処理を行います。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ke6Z1izFIM4y"},"outputs":[],"source":["# 株価情報のobject型をdatetime64[ns]型に変換\n","stock_price_load[\"Date\"] = pd.to_datetime(stock_price_load[\"Date\"])\n","\n","# 株価情報のいくつかがobject型になっているので数値型に変換\n","stock_price_load[\"AdjustmentOpen\"] = stock_price_load[\"AdjustmentOpen\"].astype(np.float64)\n","stock_price_load[\"AdjustmentHigh\"] = stock_price_load[\"AdjustmentHigh\"].astype(np.float64)\n","stock_price_load[\"AdjustmentLow\"] = stock_price_load[\"AdjustmentLow\"].astype(np.float64)\n","stock_price_load[\"AdjustmentClose\"] = stock_price_load[\"AdjustmentClose\"].astype(np.float64)\n","stock_price_load[\"AdjustmentFactor\"] = stock_price_load[\"AdjustmentFactor\"].astype(np.float64)\n","stock_price_load[\"TurnoverValue\"] = stock_price_load[\"TurnoverValue\"].astype(np.float64)\n","\n","# 累積調整係数を作成します\n","def generate_cumulative_adjustment_factor(df):\n","   # 分割併合等の係数を適用日に変更\n","   df.loc[:, \"AdjustmentFactor\"] = df[\"AdjustmentFactor\"].shift(-1).fillna(1.0)\n"," \n","   # 調整係数を作成するために逆順にソートする\n","   df = df.sort_values(\"Date\", ascending=False)\n","   # 累積株価調整係数を作成\n","   df.loc[:, \"EndOfDayQuote CumulativeAdjustmentFactor\"] = 1 / df[\"AdjustmentFactor\"].cumprod()\n","   # ソート順を昇順にする\n","   df = df.sort_values(\"Date\")\n"," \n","   return df\n","# 累積調整係数を追加\n","stock_price_load = stock_price_load.sort_values([\"Code\", \"Date\"])\n","stock_price_load = stock_price_load.groupby(\"Code\", group_keys=False).apply(generate_cumulative_adjustment_factor).reset_index(drop=True)\n","\n","# 普通株 (5桁で末尾が0) の銘柄コードを4桁にします\n","stock_price_load.loc[(stock_price_load[\"Code\"].str.len() == 5) & (stock_price_load[\"Code\"].str[-1] == \"0\"), \"Code\"] = stock_price_load.loc[(stock_price_load[\"Code\"].str.len() == 5) & (stock_price_load[\"Code\"].str[-1] == \"0\"), \"Code\"].str[:-1]\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"vyX_t49EIM4y"},"outputs":[],"source":["# 財務情報のいくつかがobject型になっているので数値型に変換\n","numeric_cols_fin = ['AverageNumberOfShares', 'BookValuePerShare', 'EarningsPerShare','Equity', 'EquityToAssetRatio',\n","                'ForecastDividendPerShare1stQuarter', 'ForecastDividendPerShare2ndQuarter', 'ForecastDividendPerShare3rdQuarter',\n","                'ForecastDividendPerShareAnnual', 'ForecastDividendPerShareFiscalYearEnd', 'ForecastEarningsPerShare', 'ForecastNetSales', 'ForecastOperatingProfit',\n","                'ForecastOrdinaryProfit', 'ForecastProfit', 'NetSales', 'NumberOfIssuedAndOutstandingSharesAtTheEndOfFiscalYearIncludingTreasuryStock',\n","                'OperatingProfit', 'OrdinaryProfit', 'Profit', 'ResultDividendPerShare1stQuarter','ResultDividendPerShare2ndQuarter','ResultDividendPerShare3rdQuarter',\n","                'ResultDividendPerShareAnnual','ResultDividendPerShareFiscalYearEnd','TotalAssets']\n","stock_fin_load[numeric_cols_fin] = stock_fin_load[numeric_cols_fin].apply(pd.to_numeric, errors='coerce', axis=1)\n","\n","# 財務情報のobject型をdatetime64[ns]型に変換\n","stock_fin_load[\"DisclosedDate\"] = pd.to_datetime(stock_fin_load[\"DisclosedDate\"]) #開示\n","stock_fin_load[\"CurrentFiscalYearEndDate\"] = pd.to_datetime(stock_fin_load[\"CurrentFiscalYearEndDate\"])  # 当事業年度終了日\n","stock_fin_load[\"CurrentFiscalYearStartDate\"] = pd.to_datetime(stock_fin_load[\"CurrentFiscalYearStartDate\"])  # 当事業年度開始日\n","stock_fin_load[\"CurrentPeriodEndDate\"] = pd.to_datetime(stock_fin_load[\"CurrentPeriodEndDate\"]) # 当会計期間終了日\n","\n","# 財務情報の値を調整します\n","stock_fin_load[\"Result_FinancialStatement FiscalYear\"] = stock_fin_load[\"CurrentFiscalYearEndDate\"].dt.strftime(\"%Y\")\n","\n","# 財務情報の同一日に複数レコードが存在することに対応します。\n","# ある銘柄について同一日に複数の開示が行われた場合レコードが重複します。\n","# ここでは簡易的に処理するために特定のTypeOfDocumentを削除した後に、開示時間順に並べて一番最後に発表された開示情報を採用しています。\n","stock_fin_load = stock_fin_load.loc[~stock_fin_load[\"TypeOfDocument\"].isin([\"ForecastRevision\", \"NumericalCorrection\", \"ForecastRevision_REIT\"])]\n","stock_fin_load = stock_fin_load.sort_values([\"DisclosedDate\", \"DisclosedTime\"]).drop_duplicates(subset=[\"LocalCode\", \"DisclosedDate\"], keep=\"last\")\n","\n","# 普通株 (5桁で末尾が0) の銘柄コードを4桁にします\n","stock_fin_load.loc[(stock_fin_load[\"LocalCode\"].str.len() == 5) & (stock_fin_load[\"LocalCode\"].str[-1] == \"0\"), \"LocalCode\"] = stock_fin_load.loc[(stock_fin_load[\"LocalCode\"].str.len() == 5) & (stock_fin_load[\"LocalCode\"].str[-1] == \"0\"), \"LocalCode\"].str[:-1]\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"fy821clEjoKO"},"outputs":[],"source":["# 銘柄一覧\n","# 普通株 (5桁で末尾が0) の銘柄コードを4桁にします\n","stock_list_load.loc[(stock_list_load[\"Code\"].str.len() == 5) & (stock_list_load[\"Code\"].str[-1] == \"0\"), \"Code\"] = stock_list_load.loc[(stock_list_load[\"Code\"].str.len() == 5) & (stock_list_load[\"Code\"].str[-1] == \"0\"), \"Code\"].str[:-1]\n","\n","# 銘柄一覧と株価データから予測対象列を作成します\n","# prediction_target は、プライム(111)、スタンダード(112)、グロース(113)の 2022-01-01 以降の売買代金上位2000銘柄とします\n","target_list = stock_list_load.loc[stock_list_load[\"MarketCode\"].isin([\"0111\", \"0112\", \"0113\"]), \"Code\"]\n","pred_targets = (stock_price_load.loc[(stock_price_load[\"Date\"] >= \"2022-01-01\") & (stock_price_load[\"Code\"].isin(target_list)), [\"Code\", \"TurnoverValue\"]].groupby(\"Code\").sum()).sort_values(\"TurnoverValue\", ascending=False).head(2000).index\n","stock_list_load[\"prediction_target\"] = stock_list_load[\"Code\"].isin(pred_targets)\n","# universe_comp2 は、プライム、スタンダード、グロースのデータの最新日付時点の時価総額上位2000銘柄とします\n","latest_date = pd.to_datetime(sorted(stock_price_load[\"Date\"].unique())[-1]).strftime(\"%Y-%m-%d\")\n","df_atp = stock_price_load.loc[stock_price_load[\"Date\"] == str(latest_date), [\"Code\", \"AdjustmentClose\"]]\n","df_atp = df_atp.loc[df_atp[\"Code\"].isin(target_list)]\n","def _nos(df):\n","    df = df.sort_values([\"DisclosedDate\", \"DisclosedTime\"])\n","    df = df.ffill()\n","    df = df.tail(1)\n","    return df\n","df_nos = stock_fin_load.groupby(\"LocalCode\")[[\"DisclosedDate\", \"DisclosedTime\", \"NumberOfIssuedAndOutstandingSharesAtTheEndOfFiscalYearIncludingTreasuryStock\"]].apply(_nos)\n","df_nos = df_nos.reset_index()\n","df_mcap = pd.merge(df_atp, df_nos, left_on=[\"Code\"], right_on=[\"LocalCode\"], how=\"inner\")\n","df_mcap[\"mcap\"] = df_mcap[\"AdjustmentClose\"] * df_mcap[\"NumberOfIssuedAndOutstandingSharesAtTheEndOfFiscalYearIncludingTreasuryStock\"]\n","universe_comp2 = sorted(df_mcap.sort_values(\"mcap\").tail(2000)[\"Code\"])\n","stock_list_load[\"universe_comp2\"] = stock_list_load[\"Code\"].isin(universe_comp2)\n"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"4Cel6PlljxNk"},"outputs":[],"source":["# stock_list: データの互換性のための各種列名変換など\n","stock_list: pd.DataFrame = pd.DataFrame()\n","stock_list[\"Local Code\"] = stock_list_load[\"Code\"]\n","stock_list[\"Name (English)\"] = stock_list_load[\"CompanyName\"]  # 英語名の代わりに日本語名を設定\n","stock_list[\"Section/Products\"] = stock_list_load[\"MarketCodeName\"]\n","stock_list[\"33 Sector(Code)\"] = stock_list_load[\"Sector33Code\"]\n","stock_list[\"33 Sector(Name)\"] = stock_list_load[\"Sector33CodeName\"]\n","stock_list[\"17 Sector(Code)\"] = stock_list_load[\"Sector17Code\"]\n","stock_list[\"17 Sector(Name)\"] = stock_list_load[\"Sector17CodeName\"]\n","stock_list[\"Size (New Index Series)\"] = stock_list_load[\"ScaleCategory\"]\n","stock_list[\"prediction_target\"] = stock_list_load[\"prediction_target\"]\n","stock_list[\"universe_comp2\"] = stock_list_load[\"universe_comp2\"]"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"bHXuFoeBjzkD"},"outputs":[],"source":["# stock_price: データの互換性のための各種列名変換など\n","stock_price: pd.DataFrame = pd.DataFrame()\n","stock_price[\"Local Code\"] = stock_price_load[\"Code\"]\n","#stock_price[\"Date\"] = stock_price_load[\"Date\"]\n","stock_price[\"base_date\"] = stock_price_load[\"Date\"]\n","stock_price['EndOfDayQuote Date'] = stock_price_load[\"Date\"]\n","stock_price[\"EndOfDayQuote Open\"] = stock_price_load[\"AdjustmentOpen\"]\n","stock_price[\"EndOfDayQuote High\"] = stock_price_load[\"AdjustmentHigh\"]\n","stock_price[\"EndOfDayQuote Low\"] = stock_price_load[\"AdjustmentLow\"]\n","stock_price[\"EndOfDayQuote Close\"] = stock_price_load[\"AdjustmentClose\"]\n","stock_price[\"EndOfDayQuote ExchangeOfficialClose\"] = stock_price_load[\"AdjustmentClose\"]\n","stock_price[\"EndOfDayQuote Volume\"] = stock_price_load[\"AdjustmentVolume\"]\n","stock_price[\"EndOfDayQuote CumulativeAdjustmentFactor\"] = stock_price_load[\"EndOfDayQuote CumulativeAdjustmentFactor\"]\n","\n","stock_price.sort_values([\"Local Code\", \"base_date\"], inplace=True)\n","# ExchangeOfficialClose は欠損値がある場合は前日終値とします\n","stock_price[\"EndOfDayQuote ExchangeOfficialClose\"] = stock_price.groupby([\"Local Code\"])[\"EndOfDayQuote ExchangeOfficialClose\"].ffill()\n","# 前日終値の列を終値列から作成\n","stock_price[\"EndOfDayQuote PreviousClose\"] = stock_price.groupby([\"Local Code\"])[\"EndOfDayQuote Close\"].shift(1)\n","stock_price.head()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"1mtq6ejdj1wS"},"outputs":[],"source":["# stock_financial: データの互換性のための各種列名変換など\n","stock_fin: pd.DataFrame = pd.DataFrame()\n","stock_fin[\"Local Code\"] = stock_fin_load[\"LocalCode\"]\n","stock_fin[\"Result_FinancialStatement FiscalPeriodEnd\"] = stock_fin_load[\"CurrentPeriodEndDate\"].dt.strftime(\"%Y/%m\")\n","stock_fin[\"Result_FinancialStatement TotalAssets\"] = stock_fin_load[\"TotalAssets\"] # 総資産\n","stock_fin[\"Result_FinancialStatement NetAssets\"] = stock_fin_load[\"Equity\"] # 純資産\n","stock_fin[\"Result_FinancialStatement NetSales\"] = stock_fin_load[\"NetSales\"] # 純売上高\n","stock_fin[\"Result_FinancialStatement OperatingIncome\"] = stock_fin_load[\"OperatingProfit\"] # 営業利益\n","stock_fin[\"Result_FinancialStatement OrdinaryIncome\"] = stock_fin_load[\"OrdinaryProfit\"]  # 経常利益\n","stock_fin[\"Result_FinancialStatement NetIncome\"] = stock_fin_load[\"Profit\"]  # 当期純利益\n","stock_fin[\"Result_FinancialStatement ReportType\"] = stock_fin_load[\"TypeOfCurrentPeriod\"] # {\"1Q\", \"2Q\", \"3Q\", \"FY\"}\n","stock_fin[\"Result_FinancialStatement FiscalYear\"] = stock_fin_load[\"Result_FinancialStatement FiscalYear\"] \n","stock_fin[\"base_date\"] = stock_fin_load[\"DisclosedDate\"] # 開示日\n","stock_fin[\"TypeOfDocument\"] = stock_fin_load[\"TypeOfDocument\"] # 書類種別\n","stock_fin[\"RetrospectiveRestatement\"] = stock_fin_load[\"RetrospectiveRestatement\"] #修正再表示フラグ\n","stock_fin[\"Forecast_FinancialStatement FiscalPeriodEnd\"] = stock_fin_load[\"CurrentFiscalYearEndDate\"].dt.strftime(\"%Y/%m\")\n","stock_fin.loc[stock_fin[\"Result_FinancialStatement ReportType\"] == \"FY\", \"Forecast_FinancialStatement FiscalPeriodEnd\"] = (\n","    stock_fin_load[\"CurrentFiscalYearEndDate\"] + pd.Timedelta(365, unit=\"D\")\n",").dt.strftime(\"%Y/%m\")  # 本決算の場合は次の年度予想なので1年後の日付にします\n","stock_fin[\"Forecast_FinancialStatement ReportType\"] = \"FY\"  #  予想は通期固定\n","stock_fin[\"Forecast_FinancialStatement NetSales\"] = stock_fin_load[\"ForecastNetSales\"]\n","stock_fin[\"Forecast_FinancialStatement OperatingIncome\"] = stock_fin_load[\"ForecastOperatingProfit\"]\n","stock_fin[\"Forecast_FinancialStatement NetIncome\"] = stock_fin_load[\"ForecastProfit\"]\n","stock_fin[\"Forecast_FinancialStatement OrdinaryIncome\"] = stock_fin_load[\"ForecastOrdinaryProfit\"]\n","\n","stock_fin.head()"]},{"cell_type":"markdown","metadata":{"id":"KV7dx5tpj7-B"},"source":["目的変数を作成します。40分程度かかります。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"l8WCqzmgj5wM"},"outputs":[],"source":["def create_label_high_low(stock_code:int, target_date:datetime, lookaheads:List[int], stock_price:pd.DataFrame):\n","   #memo: 例えばtarget_dateが2022-08-01, lookaheads=[1,5,10]だった場合、stock_priceは2022-08-01より10営業日先のデータまではいってないといけない\n","\n","   #df_price = stock_price.loc[(stock_price[\"Local Code\"] == stock_code) & (stock_price[\"base_date\"] <= target_date)].copy()\n","   df_price = stock_price.loc[stock_price[\"Local Code\"] == stock_code].copy()\n","\n","   output_columns = [\"base_date\", \"Local Code\"]\n","   for lookahead in lookaheads:\n","       output_columns.append(\"label_date_{}\".format(lookahead))\n","       output_columns.append(\"label_high_{}\".format(lookahead))\n","       output_columns.append(\"label_low_{}\".format(lookahead))\n","       t_col = \"label_date_{}\".format(lookahead)\n","       df_price.loc[:, t_col] = df_price.loc[:, \"base_date\"].shift(-lookahead)\n","\n","   if len(df_price) == 0:\n","       return pd.DataFrame(None, columns=output_columns)\n","\n","   for lookahead in lookaheads:\n","       # df_high_high: base_dateからn営業日の間の高値の最大値\n","       df_high_high = df_price.loc[:, \"EndOfDayQuote High\"].rolling(lookahead, min_periods=1).max()\n","       df_high_high = df_high_high.shift(-lookahead)\n","       df_high_high_diff = df_high_high - df_price.loc[:, \"EndOfDayQuote ExchangeOfficialClose\"]\n","       df_price.loc[:, \"label_high_{}\".format(lookahead)] = df_high_high_diff / df_price.loc[:, \"EndOfDayQuote ExchangeOfficialClose\"]\n","\n","       # df_low_low: base_dateからn営業日の間の安値の最小値\n","       df_low_low = df_price.loc[:, \"EndOfDayQuote Low\"].rolling(lookahead, min_periods=1).min()\n","       df_low_low = df_low_low.shift(-lookahead)\n","       df_low_low_diff = df_low_low - df_price.loc[:, \"EndOfDayQuote ExchangeOfficialClose\"]\n","       df_price.loc[:, \"label_low_{}\".format(lookahead)] = df_low_low_diff / df_price.loc[:, \"EndOfDayQuote ExchangeOfficialClose\"]\n","\n","   df_price.replace(np.inf, np.nan, inplace=True)\n","   df_price = df_price[df_price[\"base_date\"] <= target_date]\n","   return df_price.loc[:, output_columns]\n","\n","def create_delivery_label_high_low(stock_codes:List[int], target_date:pd.Timestamp, lookaheads:List[int], stock_price:pd.DataFrame):\n","   buff = []\n","   for stock_code in stock_codes:\n","       df = create_label_high_low(stock_code, target_date, lookaheads, stock_price)\n","       buff.append(df)\n","   df_labels = pd.concat(buff)\n","   return df_labels\n","\n","def output_stock_labels(stock_labels_csvfile_path:str, df_labels:pd.DataFrame, output_start_dt, end_dt:datetime):\n","   df_labels = df_labels.set_index(\"base_date\")\n","   df_labels = df_labels.loc[df_labels.index <= end_dt].copy()\n","   df_labels.index.name = \"base_date\"\n","   df_labels_output = df_labels.loc[(df_labels.index >= output_start_dt) & (df_labels.index <= end_dt)]\n","   label_output_columns = [\n","       \"Local Code\",\n","       \"label_date_5\",\n","       \"label_high_5\",\n","       \"label_low_5\",\n","       \"label_date_10\",\n","       \"label_high_10\",\n","       \"label_low_10\",\n","       \"label_date_20\",\n","       \"label_high_20\",\n","       \"label_low_20\",\n","   ]\n","   df_labels_output.to_csv(stock_labels_csvfile_path, compression=\"gzip\", float_format=\"%.5f\", columns=label_output_columns)\n","\n","stock_codes = sorted(stock_price[\"Local Code\"].unique())\n","\n","lookaheads = [5, 10, 20]\n","stock_labels = create_delivery_label_high_low(stock_codes, end_dt, lookaheads, stock_price)"]},{"cell_type":"markdown","metadata":{"id":"w--tA0WskDxo"},"source":["## GoogleDriveへのCSV保存\n","\n","取得したデータをcsv.gz形式で保存します。\n","stock_priceおよびstock_listが巨大なファイル（非圧縮, 2017-01-01〜2022-07-31の期間だと540MB程度)なので、保存には10分ほどかかることがあります。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"Hbi7GMzUkFY7"},"outputs":[],"source":["stock_list.to_csv(stock_list_csvfile_path, compression=\"gzip\", index=False)\n","stock_price.to_csv(stock_price_csvfile_path, compression=\"gzip\", index=False)\n","stock_fin.to_csv(stock_fins_csvfile_path, compression=\"gzip\", index=False)\n","output_stock_labels(stock_labels_csvfile_path, stock_labels, start_dt, end_dt)"]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"3DVheOgtPmHm","pycharm":{"name":"#%% md\n"}},"source":["## データセットの読み込み\n","\n","事前に生成しておいたcsvデータを読み込みます。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"UNM14YkLPmHm","pycharm":{"name":"#%%\n"}},"outputs":[],"source":["# ノートブックの冪等性を高めるためこれらの変数はこのセル以外でいじらない\n","stock_list:pd.DataFrame = pd.read_csv(stock_list_csvfile_path)\n","stock_price:pd.DataFrame = pd.read_csv(stock_price_csvfile_path)\n","stock_fin:pd.DataFrame = pd.read_csv(stock_fins_csvfile_path)\n","stock_labels:pd.DataFrame = pd.read_csv(stock_labels_csvfile_path)\n","\n","# object型をdatetime64[ns]型に変換\n","stock_price[\"base_date\"] = pd.to_datetime(stock_price[\"base_date\"])\n","stock_price[\"EndOfDayQuote Date\"] = pd.to_datetime(stock_price[\"EndOfDayQuote Date\"])\n","stock_fin[\"base_date\"] = pd.to_datetime(stock_fin[\"base_date\"])\n","stock_fin[\"Result_FinancialStatement FiscalPeriodEnd\"] = pd.to_datetime(stock_fin[\"Result_FinancialStatement FiscalPeriodEnd\"])\n","stock_fin[\"Forecast_FinancialStatement FiscalPeriodEnd\"] = pd.to_datetime(stock_fin[\"Forecast_FinancialStatement FiscalPeriodEnd\"])\n","stock_labels[\"base_date\"] = pd.to_datetime(stock_labels[\"base_date\"])\n","\n","# Unnamedな列を削除しておく\n","stock_price = stock_price.loc[:, ~stock_price.columns.str.contains('^Unnamed')]\n","stock_fin = stock_fin.loc[:, ~stock_fin.columns.str.contains('^Unnamed')]"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"iJMma7oUVgi-"},"outputs":[],"source":["stock_list.tail(2)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"RcrzWo9tVio1"},"outputs":[],"source":["stock_price.tail(2)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"rhj9D46vVkeu"},"outputs":[],"source":["stock_fin.tail(2)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"ZhqU7OP_VpIZ"},"outputs":[],"source":["stock_labels.tail(2)"]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"gow4WEA4PmHp","pycharm":{"name":"#%% md\n"}},"source":["## モデルの作成と保存\n","\n","特徴量と目的変数を生成し、学習を行います。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"jONRK_atPmHp","pycharm":{"name":"#%%\n"}},"outputs":[],"source":["# stock_priceを使ったテクニカル指標を各銘柄ごとに計算\n","def get_technical(stock_price:pd.DataFrame, code:int)->pd.DataFrame:\n","    technical_df = stock_price[stock_price[\"Local Code\"] == code].copy()\n","    # 終値\n","    technical_df[\"close\"] = technical_df[\"EndOfDayQuote Close\"]\n","    # 騰落率\n","    technical_df[\"ror_1\"] = technical_df[\"EndOfDayQuote Close\"].pct_change(1)\n","    technical_df[\"ror_5\"] = technical_df[\"EndOfDayQuote Close\"].pct_change(5) # 5営業日前からの終値の変化率\n","    technical_df[\"ror_10\"] = technical_df[\"EndOfDayQuote Close\"].pct_change(10)\n","    technical_df[\"ror_20\"] = technical_df[\"EndOfDayQuote Close\"].pct_change(20)\n","    technical_df[\"ror_40\"] = technical_df[\"EndOfDayQuote Close\"].pct_change(40)\n","    technical_df[\"ror_60\"] = technical_df[\"EndOfDayQuote Close\"].pct_change(60)\n","    technical_df[\"ror_100\"] = technical_df[\"EndOfDayQuote Close\"].pct_change(100)\n","\n","    # 売買代金\n","    technical_df[\"volume\"] = technical_df[\"EndOfDayQuote Close\"] * technical_df[\"EndOfDayQuote Volume\"]\n","    technical_df = technical_df.replace([np.inf, -np.inf], np.nan)\n","\n","    technical_df[\"vol_1\"] = technical_df[\"volume\"]\n","    technical_df[\"vol_5\"] = technical_df[\"volume\"].rolling(5).mean() # 5日移動平均\n","    technical_df[\"vol_10\"] = technical_df[\"volume\"].rolling(10).mean()\n","    technical_df[\"vol_20\"] = technical_df[\"volume\"].rolling(20).mean()\n","    technical_df[\"vol_40\"] = technical_df[\"volume\"].rolling(40).mean()\n","    technical_df[\"vol_60\"] = technical_df[\"volume\"].rolling(60).mean()\n","    technical_df[\"vol_100\"] = technical_df[\"volume\"].rolling(100).mean()\n","    technical_df[\"d_vol\"] = technical_df[\"volume\"] / technical_df[\"vol_20\"]\n","\n","    # レンジ (前日の終値に対して何%値動きしたか)\n","    technical_df[\"range\"] = (\n","        technical_df[[\"EndOfDayQuote PreviousClose\", \"EndOfDayQuote High\"]].max(axis=1) \n","        - technical_df[[\"EndOfDayQuote PreviousClose\", \"EndOfDayQuote Low\"]].min(axis=1)\n","        ) / technical_df[\"EndOfDayQuote PreviousClose\"]\n","    technical_df = technical_df.replace([np.inf, -np.inf], np.nan)\n","\n","    # レンジの移動平均\n","    technical_df[\"atr_1\"] = technical_df[\"range\"]\n","    technical_df[\"atr_5\"] = technical_df[\"range\"].rolling(5).mean()\n","    technical_df[\"atr_10\"] = technical_df[\"range\"].rolling(10).mean()\n","    technical_df[\"atr_20\"] = technical_df[\"range\"].rolling(20).mean()\n","    technical_df[\"atr_40\"] = technical_df[\"range\"].rolling(40).mean()\n","    technical_df[\"atr_60\"] = technical_df[\"range\"].rolling(60).mean()\n","    technical_df[\"atr_100\"] = technical_df[\"range\"].rolling(100).mean()\n","    technical_df[\"d_atr\"] = technical_df[\"range\"] / technical_df[\"atr_20\"]\n","\n","    # ギャップ (前日の終値よりも当日の始値がどれだけ値動きしたか) のレンジ\n","    technical_df[\"gap_range\"] = (np.abs(technical_df[\"EndOfDayQuote Open\"] - technical_df[\"EndOfDayQuote PreviousClose\"])) / technical_df[\n","        \"EndOfDayQuote PreviousClose\"]\n","    technical_df[\"g_atr_1\"] = technical_df[\"gap_range\"]\n","    technical_df[\"g_atr_5\"] = technical_df[\"gap_range\"].rolling(5).mean()\n","    technical_df[\"g_atr_10\"] = technical_df[\"gap_range\"].rolling(10).mean()\n","    technical_df[\"g_atr_20\"] = technical_df[\"gap_range\"].rolling(20).mean()\n","    technical_df[\"g_atr_40\"] = technical_df[\"gap_range\"].rolling(40).mean()\n","    technical_df[\"g_atr_60\"] = technical_df[\"gap_range\"].rolling(60).mean()\n","    technical_df[\"g_atr_100\"] = technical_df[\"gap_range\"].rolling(100).mean()\n","\n","    # デイレンジ\n","    technical_df[\"day_range\"] = (technical_df[\"EndOfDayQuote High\"] - technical_df[\"EndOfDayQuote Low\"]) / technical_df[\n","        \"EndOfDayQuote PreviousClose\"]\n","    technical_df[\"d_atr_1\"] = technical_df[\"day_range\"]\n","    technical_df[\"d_atr_5\"] = technical_df[\"day_range\"].rolling(5).mean()\n","    technical_df[\"d_atr_10\"] = technical_df[\"day_range\"].rolling(10).mean()\n","    technical_df[\"d_atr_20\"] = technical_df[\"day_range\"].rolling(20).mean()\n","    technical_df[\"d_atr_40\"] = technical_df[\"day_range\"].rolling(40).mean()\n","    technical_df[\"d_atr_60\"] = technical_df[\"day_range\"].rolling(60).mean()\n","    technical_df[\"d_atr_100\"] = technical_df[\"day_range\"].rolling(100).mean()\n","\n","    # ヒゲレンジ: ((高値-低値) - (始値と終値の差))/終値\n","    technical_df[\"hig_range\"] = ((technical_df[\"EndOfDayQuote High\"] - technical_df[\"EndOfDayQuote Low\"]) - np.abs(\n","        technical_df[\"EndOfDayQuote Open\"] - technical_df[\"EndOfDayQuote Close\"])) / technical_df[\"EndOfDayQuote PreviousClose\"]\n","    technical_df[\"h_atr_1\"] = technical_df[\"hig_range\"]\n","    technical_df[\"h_atr_5\"] = technical_df[\"hig_range\"].rolling(5).mean()\n","    technical_df[\"h_atr_10\"] = technical_df[\"hig_range\"].rolling(10).mean()\n","    technical_df[\"h_atr_20\"] = technical_df[\"hig_range\"].rolling(20).mean()\n","    technical_df[\"h_atr_40\"] = technical_df[\"hig_range\"].rolling(40).mean()\n","    technical_df[\"h_atr_60\"] = technical_df[\"hig_range\"].rolling(60).mean()\n","    technical_df[\"h_atr_100\"] = technical_df[\"hig_range\"].rolling(100).mean()\n","\n","    # ボラティリティ\n","    technical_df[\"vola_5\"] = technical_df[\"ror_1\"].rolling(5).std()\n","    technical_df[\"vola_10\"] = technical_df[\"ror_1\"].rolling(10).std()\n","    technical_df[\"vola_20\"] = technical_df[\"ror_1\"].rolling(20).std()\n","    technical_df[\"vola_40\"] = technical_df[\"ror_1\"].rolling(40).std()\n","    technical_df[\"vola_60\"] = technical_df[\"ror_1\"].rolling(60).std()\n","    technical_df[\"vola_100\"] = technical_df[\"ror_1\"].rolling(100).std()\n","\n","    # HL(High-Low)バンド: 直近n日の高値（Hバンド）、安値（Lバンド）\n","    technical_df[\"hl_5\"] = technical_df[\"EndOfDayQuote High\"].rolling(5).max() - technical_df[\"EndOfDayQuote Low\"].rolling(5).min()\n","    technical_df[\"hl_10\"] = technical_df[\"EndOfDayQuote High\"].rolling(10).max() - technical_df[\"EndOfDayQuote Low\"].rolling(10).min()\n","    technical_df[\"hl_20\"] = technical_df[\"EndOfDayQuote High\"].rolling(20).max() - technical_df[\"EndOfDayQuote Low\"].rolling(20).min()\n","    technical_df[\"hl_40\"] = technical_df[\"EndOfDayQuote High\"].rolling(40).max() - technical_df[\"EndOfDayQuote Low\"].rolling(40).min()\n","    technical_df[\"hl_60\"] = technical_df[\"EndOfDayQuote High\"].rolling(60).max() - technical_df[\"EndOfDayQuote Low\"].rolling(60).min()\n","    technical_df[\"hl_100\"] = technical_df[\"EndOfDayQuote High\"].rolling(100).max() - technical_df[\"EndOfDayQuote Low\"].rolling(100).min()\n","\n","    # マーケットインパクト\n","    technical_df[\"mi\"] = technical_df[\"range\"] / (technical_df[\"EndOfDayQuote Volume\"] * technical_df[\"EndOfDayQuote Close\"])\n","    technical_df = technical_df.replace([np.inf, -np.inf], np.nan) # \"EndOfDayQuote Volume\"=0の場合への考慮\n","\n","    technical_df[\"mi_5\"] = technical_df[\"mi\"].rolling(5).mean()\n","    technical_df[\"mi_10\"] = technical_df[\"mi\"].rolling(10).mean()\n","    technical_df[\"mi_20\"] = technical_df[\"mi\"].rolling(20).mean()\n","    technical_df[\"mi_40\"] = technical_df[\"mi\"].rolling(40).mean()\n","    technical_df[\"mi_60\"] = technical_df[\"mi\"].rolling(60).mean()\n","    technical_df[\"mi_100\"] = technical_df[\"mi\"].rolling(100).mean()\n","\n","    feat = [\"EndOfDayQuote Date\", \"Local Code\", \"close\",\n","            \"ror_1\", \"ror_5\", \"ror_10\", \"ror_20\", \"ror_40\", \"ror_60\", \"ror_100\",\n","            \"vol_1\", \"vol_5\", \"vol_10\", \"vol_20\", \"vol_40\", \"vol_60\", \"vol_100\", \"d_vol\",\n","            \"atr_1\", \"atr_5\", \"atr_10\", \"atr_20\", \"atr_40\", \"atr_60\", \"atr_100\", \"d_atr\",\n","            \"g_atr_1\", \"g_atr_5\", \"g_atr_10\", \"g_atr_20\", \"g_atr_40\", \"g_atr_60\", \"g_atr_100\",\n","            \"d_atr_1\", \"d_atr_5\", \"d_atr_10\", \"d_atr_20\", \"d_atr_40\", \"d_atr_60\", \"d_atr_100\",\n","            \"h_atr_1\", \"h_atr_5\", \"h_atr_10\", \"h_atr_20\", \"h_atr_40\", \"h_atr_60\", \"h_atr_100\",\n","            \"vola_5\", \"vola_10\", \"vola_20\", \"vola_40\", \"vola_60\", \"vola_100\",\n","            \"hl_5\", \"hl_10\", \"hl_20\", \"hl_40\", \"hl_60\", \"hl_100\",\n","            \"mi_5\", \"mi_10\", \"mi_20\", \"mi_40\", \"mi_60\", \"mi_100\"]\n","    technical_df = technical_df[feat]\n","    technical_df.columns = [\"datetime\", \"code\", \"close\",\n","                      \"ror_1\", \"ror_5\", \"ror_10\", \"ror_20\", \"ror_40\", \"ror_60\", \"ror_100\",\n","                      \"vol_1\", \"vol_5\", \"vol_10\", \"vol_20\", \"vol_40\", \"vol_60\", \"vol_100\", \"d_vol\",\n","                      \"atr_1\", \"atr_5\", \"atr_10\", \"atr_20\", \"atr_40\", \"atr_60\", \"atr_100\", \"d_atr\",\n","                      \"g_atr_1\", \"g_atr_5\", \"g_atr_10\", \"g_atr_20\", \"g_atr_40\", \"g_atr_60\", \"g_atr_100\",\n","                      \"d_atr_1\", \"d_atr_5\", \"d_atr_10\", \"d_atr_20\", \"d_atr_40\", \"d_atr_60\", \"d_atr_100\",\n","                      \"h_atr_1\", \"h_atr_5\", \"h_atr_10\", \"h_atr_20\", \"h_atr_40\", \"h_atr_60\", \"h_atr_100\",\n","                      \"vola_5\", \"vola_10\", \"vola_20\", \"vola_40\", \"vola_60\", \"vola_100\",\n","                      \"hl_5\", \"hl_10\", \"hl_20\", \"hl_40\", \"hl_60\", \"hl_100\",\n","                      \"mi_5\", \"mi_10\", \"mi_20\", \"mi_40\", \"mi_60\", \"mi_100\"]\n","    technical_df[\"datetime\"] = pd.to_datetime(technical_df[\"datetime\"])\n","    technical_df = technical_df.set_index([\"datetime\", \"code\"])\n","    return technical_df\n","\n","# example\n","# get_technical(stock_price, example_codes[0]).tail()"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"hBQvnNRLPmHq","pycharm":{"name":"#%%\n"}},"outputs":[],"source":["# stock_finを使った指標を各銘柄ごとに計算\n","def get_financial(stock_fin:pd.DataFrame, code:int)->pd.DataFrame:\n","    fin_df = stock_fin[stock_fin[\"Local Code\"] == code].copy()\n","\n","    # TypeOfDocumentの値によってはTotalAssetsなどの値がNaNになっているのでffill\n","    fin_df = fin_df.ffill()\n","\n","    # --- 本決算／中間決算フラグ、修正開示フラグ、事後修正有無フラグ ---\n","    fin_df[\"annual\"] = 0 # 0: 中間決算, 1:本決算\n","    fin_df[\"revision\"] = 0 # 1: 修正再表示\n","    # FYFinancialStatements*** は本決算\n","    fin_df.loc[fin_df[\"TypeOfDocument\"].isin([\"FYFinancialStatements_Consolidated_JP\", \"FYFinancialStatements_Consolidated_US\", \"FYFinancialStatements_Consolidated_IFRS\"]), \"annual\"] = 1\n","    fin_df.loc[fin_df[\"RetrospectiveRestatement\"]==True, \"revision\"] = 1\n","    feat1 = [\"annual\", \"revision\"]\n","\n","    # --- 原系列 ---\n","    # (memo: 前期と当期を比較すると季節的な変動が含まれ、経済的な変動を評価しづらくなる。この季節的な変動を取り除く前のものを原系列と呼ぶ)\n","\n","    # --- r_sales ---\n","    # pre_result_period_end: 前回の情報開示日時点における事業年度終了日\n","    fin_df[\"pre_result_period_end\"] = fin_df[\"Result_FinancialStatement FiscalPeriodEnd\"].shift(1)\n","    fin_df[\"r_sales\"] = np.nan\n","\n","    # 前の行から会計年度が変わったけどReportTypeが1Qでない = 年度の途中からのデータを見ており一番最初のレコードが2Qとか3Qのものだったとき?よくわかってない\n","    fin_df.loc[((fin_df[\"Result_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_result_period_end\"]) & (\n","            fin_df[\"Result_FinancialStatement ReportType\"] != \"1Q\")), \"r_sales\"] = fin_df[\n","        \"Result_FinancialStatement NetSales\"].diff(1)\n","    fin_df.loc[((fin_df[\"Result_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_result_period_end\"]) & (\n","            fin_df[\"Result_FinancialStatement ReportType\"] == \"1Q\")), \"r_sales\"] = fin_df[\n","        \"Result_FinancialStatement NetSales\"]\n","    fin_df[\"r_sales\"] = fin_df[\"r_sales\"].ffill()\n","\n","    # --- r_ope_income ---\n","    fin_df[\"r_ope_income\"] = np.nan\n","    fin_df.loc[((fin_df[\"Result_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_result_period_end\"]) & (\n","            fin_df[\"Result_FinancialStatement ReportType\"] != \"1Q\")), \"r_ope_income\"] = fin_df[\n","        \"Result_FinancialStatement OperatingIncome\"].diff(1)\n","    fin_df.loc[((fin_df[\"Result_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_result_period_end\"]) & (\n","            fin_df[\"Result_FinancialStatement ReportType\"] == \"1Q\")), \"r_ope_income\"] = fin_df[\n","        \"Result_FinancialStatement OperatingIncome\"]\n","    fin_df[\"r_ope_income\"] = fin_df[\"r_ope_income\"].ffill()\n","\n","    # --- r_ord_income ---\n","    fin_df[\"r_ord_income\"] = np.nan\n","    fin_df.loc[((fin_df[\"Result_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_result_period_end\"]) & (\n","            fin_df[\"Result_FinancialStatement ReportType\"] != \"1Q\")), \"r_ord_income\"] = fin_df[\n","        \"Result_FinancialStatement OrdinaryIncome\"].diff(1)\n","    fin_df.loc[((fin_df[\"Result_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_result_period_end\"]) & (\n","            fin_df[\"Result_FinancialStatement ReportType\"] == \"1Q\")), \"r_ord_income\"] = fin_df[\n","        \"Result_FinancialStatement OrdinaryIncome\"]\n","    fin_df[\"r_ord_income\"] = fin_df[\"r_ord_income\"].ffill()\n","\n","    # --- r_net_income ---\n","    fin_df[\"r_net_income\"] = np.nan\n","    fin_df.loc[((fin_df[\"Result_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_result_period_end\"]) & (\n","            fin_df[\"Result_FinancialStatement ReportType\"] != \"1Q\")), \"r_net_income\"] = fin_df[\n","        \"Result_FinancialStatement NetIncome\"].diff(1)\n","    fin_df.loc[((fin_df[\"Result_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_result_period_end\"]) & (\n","            fin_df[\"Result_FinancialStatement ReportType\"] == \"1Q\")), \"r_net_income\"] = fin_df[\n","        \"Result_FinancialStatement NetIncome\"]\n","    fin_df[\"r_net_income\"] = fin_df[\"r_net_income\"].ffill()\n","\n","    # --- pre_forcast_period_end ---\n","    fin_df[\"pre_forecast_period_end\"] = fin_df[\"Forecast_FinancialStatement FiscalPeriodEnd\"].shift(1)\n","\n","    # --- f_sales ---\n","    fin_df[\"f_sales\"] = np.nan\n","    fin_df.loc[((fin_df[\"Forecast_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_forecast_period_end\"]) & (\n","            fin_df[\"Forecast_FinancialStatement ReportType\"] != \"1Q\")), \"f_sales\"] = fin_df[\n","        \"Forecast_FinancialStatement NetSales\"].diff(1)\n","    fin_df.loc[((fin_df[\"Forecast_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_forecast_period_end\"]) & (\n","            fin_df[\"Forecast_FinancialStatement ReportType\"] == \"1Q\")), \"f_sales\"] = fin_df[\n","        \"Forecast_FinancialStatement NetSales\"]\n","    fin_df[\"f_sales\"] = fin_df[\"f_sales\"].ffill()\n","\n","    # --- f_ope_income ---\n","    fin_df[\"f_ope_income\"] = np.nan\n","    fin_df.loc[((fin_df[\"Forecast_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_forecast_period_end\"]) & (\n","            fin_df[\"Forecast_FinancialStatement ReportType\"] != \"1Q\")), \"f_ope_income\"] = fin_df[\n","        \"Forecast_FinancialStatement OperatingIncome\"].diff(1)\n","    fin_df.loc[((fin_df[\"Forecast_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_forecast_period_end\"]) & (\n","            fin_df[\"Forecast_FinancialStatement ReportType\"] == \"1Q\")), \"f_ope_income\"] = fin_df[\n","        \"Forecast_FinancialStatement OperatingIncome\"]\n","    fin_df[\"f_ope_income\"] = fin_df[\"f_ope_income\"].ffill()\n","\n","    # --- f_ord_income ---\n","    fin_df[\"f_ord_income\"] = np.nan\n","    fin_df.loc[((fin_df[\"Forecast_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_forecast_period_end\"]) & (\n","            fin_df[\"Forecast_FinancialStatement ReportType\"] != \"1Q\")), \"f_ord_income\"] = fin_df[\n","        \"Forecast_FinancialStatement OrdinaryIncome\"].diff(1)\n","    fin_df.loc[((fin_df[\"Forecast_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_forecast_period_end\"]) & (\n","            fin_df[\"Forecast_FinancialStatement ReportType\"] == \"1Q\")), \"f_ord_income\"] = fin_df[\n","        \"Forecast_FinancialStatement OrdinaryIncome\"]\n","    fin_df[\"f_ord_income\"] = fin_df[\"f_ord_income\"].ffill()\n","\n","    # --- f_net_income ---\n","    fin_df[\"f_net_income\"] = np.nan\n","    fin_df.loc[((fin_df[\"Forecast_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_forecast_period_end\"]) & (\n","            fin_df[\"Forecast_FinancialStatement ReportType\"] != \"1Q\")), \"f_net_income\"] = fin_df[\n","        \"Forecast_FinancialStatement NetIncome\"].diff(1)\n","    fin_df.loc[((fin_df[\"Forecast_FinancialStatement FiscalPeriodEnd\"] != fin_df[\"pre_forecast_period_end\"]) & (\n","            fin_df[\"Forecast_FinancialStatement ReportType\"] == \"1Q\")), \"f_net_income\"] = fin_df[\n","        \"Forecast_FinancialStatement NetIncome\"]\n","    fin_df[\"f_net_income\"] = fin_df[\"f_net_income\"].ffill()\n","\n","    # --------------------\n","    fin_df[\"r_expense1\"] = fin_df[\"r_sales\"] - fin_df[\"r_ope_income\"]\n","    fin_df[\"r_expense2\"] = fin_df[\"r_ope_income\"] - fin_df[\"r_ord_income\"]\n","    fin_df[\"r_expense3\"] = fin_df[\"r_ord_income\"] - fin_df[\"r_net_income\"]\n","\n","    fin_df[\"f_expense1\"] = fin_df[\"f_sales\"] - fin_df[\"f_ope_income\"]\n","    fin_df[\"f_expense2\"] = fin_df[\"f_ope_income\"] - fin_df[\"f_ord_income\"]\n","    fin_df[\"f_expense3\"] = fin_df[\"f_ord_income\"] - fin_df[\"f_net_income\"]\n","\n","    fin_df[\"r_assets\"] = fin_df[\"Result_FinancialStatement TotalAssets\"]\n","    fin_df[\"r_equity\"] = fin_df[\"Result_FinancialStatement NetAssets\"]\n","\n","    # 現在 J-Quants APIからは取れなさそう\n","    # fin_df[\"operating_cf\"] = fin_df[\"Result_FinancialStatement CashFlowsFromOperatingActivities\"]\n","    # fin_df[\"financial_cf\"] = fin_df[\"Result_FinancialStatement CashFlowsFromFinancingActivities\"]\n","    # fin_df[\"investing_cf\"] = fin_df[\"Result_FinancialStatement CashFlowsFromInvestingActivities\"]\n","\n","    feat2 = [\"r_sales\", \"r_ope_income\", \"r_ord_income\", \"r_net_income\", \"f_sales\", \"f_ope_income\", \"f_ord_income\",\n","            \"f_net_income\",\n","            \"r_expense1\", \"r_expense2\", \"r_expense3\", \"f_expense1\", \"f_expense2\", \"f_expense3\",\n","            \"r_assets\", \"r_equity\",] #\"operating_cf\", \"financial_cf\", \"investing_cf\"]\n","\n","\n","    # --- 複合指標　原系列 ---\n","    # ------ 純利益系 ------\n","    fin_df[\"r_pm1\"] = fin_df[\"Result_FinancialStatement NetIncome\"] / fin_df[\"Result_FinancialStatement NetSales\"]\n","    fin_df[\"r_roe1\"] = fin_df[\"Result_FinancialStatement NetIncome\"] / fin_df[\"Result_FinancialStatement NetAssets\"]\n","    fin_df[\"r_roa1\"] = fin_df[\"Result_FinancialStatement NetIncome\"] / fin_df[\n","        \"Result_FinancialStatement TotalAssets\"]\n","\n","    fin_df[\"f_pm1\"] = fin_df[\"Forecast_FinancialStatement NetIncome\"] / fin_df[\n","        \"Forecast_FinancialStatement NetSales\"]\n","    fin_df[\"f_roe1\"] = fin_df[\"Forecast_FinancialStatement NetIncome\"] / fin_df[\n","        \"Result_FinancialStatement NetAssets\"]\n","    fin_df[\"f_roa1\"] = fin_df[\"Forecast_FinancialStatement NetIncome\"] / fin_df[\n","        \"Result_FinancialStatement TotalAssets\"]\n","\n","    # 経常利益系\n","    fin_df[\"r_pm2\"] = fin_df[\"Result_FinancialStatement OrdinaryIncome\"] / fin_df[\n","        \"Result_FinancialStatement NetSales\"]\n","    fin_df[\"r_roe2\"] = fin_df[\"Result_FinancialStatement OrdinaryIncome\"] / fin_df[\n","        \"Result_FinancialStatement NetAssets\"]\n","    fin_df[\"r_roa2\"] = fin_df[\"Result_FinancialStatement OrdinaryIncome\"] / fin_df[\n","        \"Result_FinancialStatement TotalAssets\"]\n","\n","    fin_df[\"f_pm2\"] = fin_df[\"Forecast_FinancialStatement OrdinaryIncome\"] / fin_df[\n","        \"Forecast_FinancialStatement NetSales\"]\n","    fin_df[\"f_roe2\"] = fin_df[\"Forecast_FinancialStatement OrdinaryIncome\"] / fin_df[\n","        \"Result_FinancialStatement NetAssets\"]\n","    fin_df[\"f_roa2\"] = fin_df[\"Forecast_FinancialStatement OrdinaryIncome\"] / fin_df[\n","        \"Result_FinancialStatement TotalAssets\"]\n","\n","    # 営業利益系\n","    fin_df[\"r_pm3\"] = fin_df[\"Result_FinancialStatement OperatingIncome\"] / fin_df[\n","        \"Result_FinancialStatement NetSales\"]\n","    fin_df[\"r_roe3\"] = fin_df[\"Result_FinancialStatement OperatingIncome\"] / fin_df[\n","        \"Result_FinancialStatement NetAssets\"]\n","    fin_df[\"r_roa3\"] = fin_df[\"Result_FinancialStatement OperatingIncome\"] / fin_df[\n","        \"Result_FinancialStatement TotalAssets\"]\n","\n","    fin_df[\"f_pm3\"] = fin_df[\"Forecast_FinancialStatement OperatingIncome\"] / fin_df[\n","        \"Forecast_FinancialStatement NetSales\"]\n","    fin_df[\"f_roe3\"] = fin_df[\"Forecast_FinancialStatement OperatingIncome\"] / fin_df[\n","        \"Result_FinancialStatement NetAssets\"]\n","    fin_df[\"f_roa3\"] = fin_df[\"Forecast_FinancialStatement OperatingIncome\"] / fin_df[\n","        \"Result_FinancialStatement TotalAssets\"]\n","\n","    # コスト\n","    fin_df[\"r_cost1\"] = ((fin_df[\"Result_FinancialStatement NetSales\"] - fin_df[\n","        \"Result_FinancialStatement OperatingIncome\"]) / fin_df[\"Result_FinancialStatement NetSales\"])\n","    fin_df[\"r_cost2\"] = ((fin_df[\"Result_FinancialStatement OperatingIncome\"] - fin_df[\n","        \"Result_FinancialStatement OrdinaryIncome\"]) / fin_df[\"Result_FinancialStatement NetSales\"])\n","    fin_df[\"r_cost3\"] = ((fin_df[\"Result_FinancialStatement OrdinaryIncome\"] - fin_df[\n","        \"Result_FinancialStatement NetIncome\"]) / fin_df[\"Result_FinancialStatement NetSales\"])\n","\n","    fin_df[\"f_cost1\"] = ((fin_df[\"Forecast_FinancialStatement NetSales\"] - fin_df[\n","        \"Forecast_FinancialStatement OperatingIncome\"]) / fin_df[\"Forecast_FinancialStatement NetSales\"])\n","    fin_df[\"f_cost2\"] = ((fin_df[\"Forecast_FinancialStatement OperatingIncome\"] - fin_df[\n","        \"Forecast_FinancialStatement OrdinaryIncome\"]) / fin_df[\"Forecast_FinancialStatement NetSales\"])\n","    fin_df[\"f_cost3\"] = ((fin_df[\"Forecast_FinancialStatement OrdinaryIncome\"] - fin_df[\n","        \"Forecast_FinancialStatement NetIncome\"]) / fin_df[\"Forecast_FinancialStatement NetSales\"])\n","\n","    # 売上高回転率\n","    fin_df[\"r_turn\"] = fin_df[\"Result_FinancialStatement NetSales\"] / fin_df[\n","        \"Result_FinancialStatement TotalAssets\"]\n","    fin_df[\"f_turn\"] = fin_df[\"Forecast_FinancialStatement NetSales\"] / fin_df[\n","        \"Result_FinancialStatement TotalAssets\"]\n","\n","    # 財務健全性\n","    fin_df[\"equity_ratio\"] = fin_df[\"Result_FinancialStatement NetAssets\"] / fin_df[\n","        \"Result_FinancialStatement TotalAssets\"]\n","\n","    # 総資本キャッシュフロー比率 --- 現在J-Quants APIからは取得できなさそう\n","    # fin_df[\"o_cf_ratio\"] = (fin_df[\"Result_FinancialStatement CashFlowsFromOperatingActivities\"] / fin_df[\n","    #     \"Result_FinancialStatement TotalAssets\"])\n","    # fin_df[\"f_cf_ratio\"] = (fin_df[\"Result_FinancialStatement CashFlowsFromFinancingActivities\"] / fin_df[\n","    #     \"Result_FinancialStatement TotalAssets\"])\n","    # fin_df[\"i_cf_ratio\"] = (fin_df[\"Result_FinancialStatement CashFlowsFromInvestingActivities\"] / fin_df[\n","    #     \"Result_FinancialStatement TotalAssets\"])\n","\n","    feat3 = [\"r_pm1\", \"r_roe1\", \"r_roa1\", \"f_pm1\", \"f_roe1\", \"f_roa1\",\n","             \"r_pm2\", \"r_roe2\", \"r_roa2\", \"f_pm2\", \"f_roe2\", \"f_roa2\",\n","             \"r_pm3\", \"r_roe3\", \"r_roa3\", \"f_pm3\", \"f_roe3\", \"f_roa3\",\n","             \"r_cost1\", \"r_cost2\", \"r_cost3\", \"f_cost1\", \"f_cost2\", \"f_cost3\",\n","             \"r_turn\", \"f_turn\", \"equity_ratio\", ] # \"o_cf_ratio\", \"f_cf_ratio\", \"i_cf_ratio\"]\n","\n","    # Inf値をNan値化\n","    fin_df = fin_df.replace([np.inf, -np.inf], np.nan)\n","\n","    # 差分系列\n","    d_feat2 = []\n","\n","    for f in feat2:\n","        fin_df[\"d_\" + f] = fin_df[f].diff(1)\n","        d_feat2.append(\"d_\" + f)\n","\n","    d_feat3 = []\n","    for f in feat3:\n","        fin_df[\"d_\" + f] = fin_df[f].diff(1)\n","        d_feat3.append(\"d_\" + f)\n","\n","    d_feat4 = [\"m_sales\", \"m_ope_income\", \"m_ord_income\", \"m_net_income\", \"m_expense1\", \"m_expense2\", \"m_expense3\",\n","               \"m_pm1\", \"m_pm2\", \"m_pm3\", \"m_roe1\", \"m_roe2\", \"m_roe3\", \"m_roa1\", \"m_roa2\", \"m_roa3\",\n","               \"m_cost1\", \"m_cost2\", \"m_cost3\"]\n","\n","    fin_df[\"m_sales\"] = fin_df[\"r_sales\"] - fin_df[\"f_sales\"].shift(1)\n","    fin_df[\"m_ope_income\"] = fin_df[\"r_ope_income\"] - fin_df[\"f_ope_income\"].shift(1)\n","    fin_df[\"m_ord_income\"] = fin_df[\"r_ord_income\"] - fin_df[\"f_ord_income\"].shift(1)\n","    fin_df[\"m_net_income\"] = fin_df[\"r_net_income\"] - fin_df[\"f_net_income\"].shift(1)\n","    fin_df[\"m_expense1\"] = fin_df[\"r_expense1\"] - fin_df[\"f_expense1\"].shift(1)\n","    fin_df[\"m_expense2\"] = fin_df[\"r_expense2\"] - fin_df[\"f_expense2\"].shift(1)\n","    fin_df[\"m_expense3\"] = fin_df[\"r_expense3\"] - fin_df[\"f_expense3\"].shift(1)\n","\n","    fin_df[\"m_pm1\"] = fin_df[\"r_pm1\"] - fin_df[\"f_pm1\"].shift(1)\n","    fin_df[\"m_pm2\"] = fin_df[\"r_pm2\"] - fin_df[\"f_pm2\"].shift(1)\n","    fin_df[\"m_pm3\"] = fin_df[\"r_pm3\"] - fin_df[\"f_pm3\"].shift(1)\n","    fin_df[\"m_roe1\"] = fin_df[\"r_roe1\"] - fin_df[\"f_roe1\"].shift(1)\n","    fin_df[\"m_roe2\"] = fin_df[\"r_roe2\"] - fin_df[\"f_roe2\"].shift(1)\n","    fin_df[\"m_roe3\"] = fin_df[\"r_roe3\"] - fin_df[\"f_roe3\"].shift(1)\n","    fin_df[\"m_roa1\"] = fin_df[\"r_roa1\"] - fin_df[\"f_roa1\"].shift(1)\n","    fin_df[\"m_roa2\"] = fin_df[\"r_roa2\"] - fin_df[\"f_roa2\"].shift(1)\n","    fin_df[\"m_roa3\"] = fin_df[\"r_roa3\"] - fin_df[\"f_roa3\"].shift(1)\n","    fin_df[\"m_cost1\"] = fin_df[\"r_cost1\"] - fin_df[\"f_cost1\"].shift(1)\n","    fin_df[\"m_cost2\"] = fin_df[\"r_cost2\"] - fin_df[\"f_cost2\"].shift(1)\n","    fin_df[\"m_cost3\"] = fin_df[\"r_cost3\"] - fin_df[\"f_cost3\"].shift(1)\n","\n","    feat = [\"base_date\", \"Local Code\"]\n","    feat.extend(feat1)\n","    feat.extend(feat2)\n","    feat.extend(feat3)\n","    feat.extend(d_feat2)\n","    feat.extend(d_feat3)\n","    feat.extend(d_feat4)\n","\n","    col_names = [\"datetime\", \"code\"]\n","    col_names.extend(feat1)\n","    col_names.extend(feat2)\n","    col_names.extend(feat3)\n","    col_names.extend(d_feat2)\n","    col_names.extend(d_feat3)\n","    col_names.extend(d_feat4)\n","\n","    fin_df = fin_df[feat]\n","    fin_df.columns = col_names\n","    fin_df[\"datetime\"] = pd.to_datetime(fin_df[\"datetime\"])\n","    fin_df = fin_df.set_index([\"datetime\", \"code\"])\n","    return fin_df\n","\n","# example\n","# get_financial(stock_fin, example_codes[1]).tail()"]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"9PQvRRyIPmH1","pycharm":{"name":"#%% md\n"}},"source":["生成した特徴量とラベルデータを用いて学習を行います。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"2-DEtA9iWTHL"},"outputs":[],"source":["# 予測対象はuniverse_comp2とします\n","stock_codes = sorted(stock_list.loc[stock_list[\"universe_comp2\"] == True, \"Local Code\"])"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"85H5XxlxPmH1","pycharm":{"name":"#%%\n"}},"outputs":[],"source":["def get_df_merge(stock_price:pd.DataFrame, stock_fin:pd.DataFrame, stock_labels:pd.DataFrame, stock_codes, train:bool=True):\n","    df_technical = []\n","    for code in stock_codes:\n","        df_technical.append(get_technical(stock_price, code))\n","    df_technical = pd.concat(df_technical)\n","\n","    df_financial = []\n","    for code in stock_codes:\n","        df_financial.append(get_financial(stock_fin, code))\n","    df_financial = pd.concat(df_financial)\n","\n","    # TypeOfDocument=NumerialCorrectionなどの場合において、　Disclosed Dateが同じのドキュメントが同一銘柄で存在したりする。\n","    # datetime-codeをuniqueなindexとしたいので一旦そういった重複日のドキュメントは排除する\n","    df_financial = df_financial[~df_financial.index.duplicated(keep='first')]\n","    \n","    if train:\n","        df_label = stock_labels.copy()\n","        feat = [\"base_date\", \"Local Code\", \"label_high_20\", \"label_low_20\"]\n","        df_label = df_label[feat]\n","        df_label.columns = [\"datetime\", \"code\", \"label_high_20\", \"label_low_20\"]\n","\n","        df_label[\"datetime\"] = pd.to_datetime(df_label[\"datetime\"])\n","        df_label = df_label.set_index([\"datetime\", \"code\"])\n","\n","        df_merge = pd.concat([df_financial,\n","                              df_technical[df_technical.index.isin(df_financial.index)],\n","                              df_label[df_label.index.isin(df_financial.index)]\n","                              ], axis=1)\n","    else:\n","        df_merge = pd.concat([df_financial,\n","                              df_technical[df_technical.index.isin(df_financial.index)],\n","                              ], axis=1)\n","\n","    df_merge = df_merge.reset_index()\n","    return df_merge\n","\n","def get_df_for_ml(stock_price:pd.DataFrame, stock_fin:pd.DataFrame, stock_labels:pd.DataFrame, stock_codes, train=True):\n","    df_merge = get_df_merge(stock_price, stock_fin, stock_labels, stock_codes, train=train)\n","    df_merge = df_merge.replace([np.inf, -np.inf], np.nan)\n","    df_merge = df_merge.fillna(0)\n","    return df_merge\n","\n","def save_model(model, label, model_path):\n","    os.makedirs(model_path, exist_ok=True)\n","    with open(os.path.join(model_path, f\"my_model_{label}.pkl\"), \"wb\") as f:\n","        pickle.dump(model, f)\n","\n","def train_and_save_model(stock_price:pd.DataFrame, stock_fin:pd.DataFrame, stock_labels:pd.DataFrame, train_end_dt:datetime, stock_codes, model_path):\n","    model_h_final_path = os.path.join(model_path, f\"my_model_model_h_final.pkl\")\n","    model_l_final_path = os.path.join(model_path, f\"my_model_model_l_final.pkl\")\n","    if os.path.isfile(model_h_final_path) and os.path.isfile(model_l_final_path):\n","        # 学習済みモデルが存在する場合はスキップします。\n","        return\n","\n","    # 特徴量を作成\n","    df_for_ml = get_df_for_ml(stock_price, stock_fin, stock_labels, stock_codes, train=True)\n","\n","    train_df = df_for_ml[df_for_ml[\"datetime\"] <= train_end_dt].copy()\n","\n","    model_h_final = XGBRegressor(max_depth=6, learning_rate=0.01, n_estimators=3000, n_jobs=-1,\n","                                 colsample_bytree=0.1, random_state=0)\n","    model_l_final = XGBRegressor(max_depth=6, learning_rate=0.01, n_estimators=3000, n_jobs=-1,\n","                                 colsample_bytree=0.1, random_state=0)\n","\n","    x_feats = [f for f in df_for_ml.columns if f not in [\"datetime\", \"code\", \"label_high_20\", \"label_low_20\"]]\n","    y_labels = [\"label_high_20\", \"label_low_20\"]\n","\n","    model_h_final.fit(train_df[x_feats], train_df[\"label_high_20\"])\n","    model_l_final.fit(train_df[x_feats], train_df[\"label_low_20\"])\n","\n","    save_model(model_h_final, \"model_h_final\", model_path=model_path)\n","    save_model(model_l_final, \"model_l_final\", model_path=model_path)\n","\n","train_and_save_model(stock_price, stock_fin, stock_labels, train_end_dt=train_end_dt, stock_codes=stock_codes, model_path=model_path)"]},{"cell_type":"markdown","metadata":{"collapsed":false,"id":"O3X_GpN5PmH1","pycharm":{"name":"#%% md\n"}},"source":["## 推論実行\n","\n","predictを行います。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"uOcatdpLPmH1","pycharm":{"name":"#%%\n"}},"outputs":[],"source":["def get_recent_statements(df, dt, latest_n=100):\n","    for cnt in range(100):\n","        tmp_df = df[(df[\"datetime\"] > dt - timedelta(cnt)) & (df[\"datetime\"] <= dt)].copy()\n","        if len(tmp_df) >= latest_n:\n","            break\n","    tmp_df[\"datetime\"] = dt\n","    return tmp_df\n","\n","def get_df_for_predict(stock_price:pd.DataFrame, stock_fin:pd.DataFrame, stock_labels:pd.DataFrame, stock_codes, train=False, latest_n=100):\n","    df_merge = get_df_merge(stock_price, stock_fin, stock_labels, stock_codes, train=train)\n","    df_merge = df_merge.replace([np.inf, -np.inf], np.nan)\n","    df_merge = df_merge.fillna(0)\n","    # dt_list = pd.to_datetime(sorted(df_merge[\"datetime\"].unique()))\n","    # dt_list = pd.Series(dt_list)\n","    # # FRIDAY = 4\n","    # dt_list = dt_list[dt_list.dt.dayofweek == FRIDAY]\n","    # tmp_dfs = []\n","    # for dts in dt_list:\n","    #     tmp = get_recent_statements(df_merge, dts, latest_n)\n","    #     tmp_dfs.append(tmp)\n","    # df_merge = pd.concat(tmp_dfs)\n","    # df_merge = df_merge[df_merge[\"datetime\"]>=\"2020-01-01\"]\n","    return df_merge\n","\n","def get_model(model_path):\n","    models = {}\n","    labels = [\"model_h_final\", \"model_l_final\"]\n","    for label in labels:\n","        m = os.path.join(model_path, f\"my_model_{label}.pkl\")\n","        with open(m, \"rb\") as f:\n","            models[label] = pickle.load(f)\n","    return models[\"model_h_final\"], models[\"model_l_final\"]\n","\n","def get_predict(df_for_ml, models_h, models_l):\n","    tmp_df = df_for_ml.copy()\n","\n","    x_feats = [f for f in tmp_df.columns if f not in [\"datetime\", \"code\", \"label_high_20\", \"label_low_20\"]]\n","\n","    tmp_df[\"pred_high_20\"] = models_h.predict(tmp_df[x_feats])\n","    tmp_df[\"pred_low_20\"] = models_l.predict(tmp_df[x_feats])\n","\n","    tmp_df = tmp_df.set_index([\"datetime\"])\n","    cols = [\"code\", \"pred_high_20\", \"pred_low_20\"]\n","    tmp_df = tmp_df[cols]\n","\n","    return tmp_df\n","\n","def predict(stock_price:pd.DataFrame, stock_fin:pd.DataFrame, stock_labels:pd.DataFrame, stock_codes, latest_n=100):\n","    # 特徴量を作成\n","    # df_for_ml = get_df_for_ml(stock_price, stock_fin, stock_labels, stock_codes, train=False)\n","    df_for_ml = get_df_for_predict(stock_price, stock_fin, stock_labels, stock_codes, train=False, latest_n=latest_n)\n","\n","    # 予測\n","    models_h, models_l = get_model(model_path)\n","    df = get_predict(df_for_ml, models_h, models_l)\n","    df = df.sort_values([\"datetime\", \"pred_high_20\", \"code\"], ascending=True)\n","\n","    return df\n","\n","result:pd.DataFrame = predict(stock_price, stock_fin, stock_labels, stock_codes)\n"]},{"cell_type":"markdown","metadata":{"id":"sWDmmOsKmrg5"},"source":["## 予測の確認\n","\n","出力された予測を確認します。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"lMUDrdfZmSjj"},"outputs":[],"source":["# 予測を出力します\n","result.tail(10)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"wmAZfvml3s_A"},"outputs":[],"source":["# 予測とラベルを結合します\n","df_m = pd.merge(result, stock_labels, left_on=[\"datetime\", \"code\"], right_on=[\"base_date\", \"Local Code\"]).sort_values([\"base_date\", \"pred_high_20\", \"Local Code\"])"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"LbjWrzD1BkAl"},"outputs":[],"source":["df_m.head(2)"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"_BMsu9vBR74F"},"outputs":[],"source":["# JPX (8697) の予測とラベルを描画してみる\n","df_m.loc[\n","    (df_m[\"code\"] == 8697) & (df_m[\"base_date\"] >= test_start_dt),\n","    [\"base_date\", \"pred_high_20\", \"pred_low_20\", \"label_high_20\", \"label_low_20\"],\n","].set_index(\"base_date\").plot(figsize=(20, 8), title=\"JPX (8697): pred/label\")"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"by5WaF4_R7vL"},"outputs":[],"source":[]},{"cell_type":"markdown","metadata":{"id":"zfsqMHIpmzLb"},"source":["## 予測の評価\n","\n","日次で直近n個の予測と正解ラベルとの順位相関を確認します。\n","簡易化するために pred_high_20 のみを評価しています。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"WSs1JguEBwnV"},"outputs":[],"source":["# ラベルの存在する営業日を取得します\n","dt_list = sorted(stock_labels.loc[stock_labels[\"label_high_20\"].notna(), \"base_date\"].unique())"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"eayfjy3hB_Bi"},"outputs":[],"source":["# 評価用に日次で直近n個の予測をグループ化します\n","df_m2 = df_m.sort_values([\"base_date\", \"pred_high_20\", \"code\"])\n","# label が存在しないレコードを除外\n","df_m2 = df_m2.loc[df_m2[\"label_high_20\"].notna()]\n","# グループ化\n","dfs_ret = {}\n","for latest_n in [10, 20, 50, 100, 200, 300, 400]:\n","    buff = []\n","    for dt in dt_list:\n","        tmp_df = df_m2.loc[df_m2[\"base_date\"] <= dt].sort_values([\"base_date\", \"pred_high_20\", \"code\"]).tail(latest_n).copy()\n","        tmp_df[\"eval_date\"] = dt\n","        buff.append(tmp_df)\n","    df_ret = pd.concat(buff)\n","    df_ret.index = df_ret[\"eval_date\"]\n","    df_ret.index.name = \"date\"\n","\n","    # テスト期間に絞り込み\n","    df_ret = df_ret.loc[df_ret.index >= test_start_dt]\n","    dfs_ret[latest_n] = df_ret"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"CAcJiXV_bjBl"},"outputs":[],"source":["from scipy.stats import spearmanr\n","\n","def _spearmanr_high_20(df):\n","    return spearmanr(df[\"pred_high_20\"].values, df[\"label_high_20\"].values)[0]\n","\n","\n","# def _spearmanr_low_20(df):\n","#     return spearmanr(df[\"pred_low_20\"].values, df[\"label_low_20\"].values)[0]"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"2eref0SaNsP7"},"outputs":[],"source":["for latest_n in [10, 20, 50, 100, 200, 300, 400]:\n","    display(f\"latest_n: {latest_n}\")\n","    # display(dfs_ret[latest_n].groupby(\"eval_date\").apply(_spearmanr).head(2))\n","    # display(dfs_ret[latest_n].groupby(\"eval_date\").apply(_spearmanr).tail(2))\n","    display(\"label_high_20\")\n","    display(dfs_ret[latest_n].groupby(\"eval_date\").apply(_spearmanr_high_20).describe())\n","    dfs_ret[latest_n].groupby(\"eval_date\").apply(_spearmanr_high_20).plot(figsize=(20, 8), title=f\"spearman: label_high_20, latest {latest_n}\")\n","    plt.show()\n","    dfs_ret[latest_n].groupby(\"eval_date\").apply(_spearmanr_high_20).cumsum().plot(figsize=(20, 8), title=f\"cumsum(spearman): label_high_20, latest {latest_n}\")\n","    plt.show()\n","    # display(\"label_low_20\")  # low側を評価するにはグループ化方法もlowに合わせたもので実施するのがよいです\n","    # display(dfs_ret[latest_n].groupby(\"eval_date\").apply(_spearmanr_low_20).describe())\n","    # dfs_ret[latest_n].groupby(\"eval_date\").apply(_spearmanr_low_20).plot(figsize=(20, 8), title=f\"spearman: label_low_20, latest {latest_n}\")\n","    # plt.show()\n","    # dfs_ret[latest_n].groupby(\"eval_date\").apply(_spearmanr_low_20).cumsum().plot(figsize=(20, 8), title=f\"cumsum(spearman): label_low_20, latest {latest_n}\")\n","    # plt.show()"]},{"cell_type":"markdown","metadata":{"id":"ddAbINJh5Ufl"},"source":["本ノートブックは以上となります。\n","\n","\n","用途に合わせて予測対象銘柄やラベルを変更したり、特徴量を変更したりすることで、さまざまな予測を実施することができます。\n","ぜひいろいろなモデルを作成して評価を実施してみてください。\n","\n","日本取引所グループで公開されている「[株式分析チュートリアル](https://japanexchangegroup.github.io/J-Quants-Tutorial/)」も大変参考となりますのでぜひ御一読ください。\n","\n","本ノートブックの作成においてご支援を賜りましたJPX総研のJ-Quants運営チームのみなさま、モデルの移植作業を実施いただいた @akkie30 および本ノートブックへのモデルの移植をご快諾いただいたUKIさん (@blog_uki) に心からの感謝を申し上げます。\n","\n","本ノートブックがみなさまのモデルを構築する際のご支援となれば幸いでございます。\n","\n","最後までご覧いただきありがとうございました。"]},{"cell_type":"code","execution_count":null,"metadata":{"id":"i7Nu-1ueljz6"},"outputs":[],"source":[]}],"metadata":{"colab":{"machine_shape":"hm","provenance":[]},"gpuClass":"standard","kernelspec":{"display_name":"Python 3","language":"python","name":"python3"},"language_info":{"codemirror_mode":{"name":"ipython","version":2},"file_extension":".py","mimetype":"text/x-python","name":"python","nbconvert_exporter":"python","pygments_lexer":"ipython2","version":"2.7.6"}},"nbformat":4,"nbformat_minor":0}