Feature: support new historical(download) providers (#478)

* feat: new command in download parameters

* feat: new historical provider TradeStation and Alpaca

* feat: add default market if try to download crypto (USA not support)

* refactor: prompt market name in download command

* revert: extra argument in data download command

* feat: help message of market if not provided

README.md

@@ -145,7 +145,7 @@ Options:
   -d, --detach                    Run the backtest in a detached Docker container and return immediately
   --debug [pycharm|ptvsd|debugpy|vsdbg|rider|local-platform]
                                   Enable a certain debugging method (see --help for more information)
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
                                   Update the Lean configuration file to retrieve data from the given historical provider
   --ib-user-name TEXT             Your Interactive Brokers username
   --ib-account TEXT               Your Interactive Brokers account id
@@ -209,6 +209,12 @@ Options:
   --bybit-api-secret TEXT         Your Bybit API secret
   --bybit-vip-level [VIP0|VIP1|VIP2|VIP3|VIP4|VIP5|SupremeVIP|Pro1|Pro2|Pro3|Pro4|Pro5]
                                   Your Bybit VIP Level
+  --trade-station-environment [live|paper]
+                                  Whether Live or Paper environment should be used
+  --trade-station-account-type [Cash|Margin|Futures|DVP]
+                                  Specifies the type of account on TradeStation
+  --alpaca-environment [live|paper]
+                                  Whether Live or Paper environment should be used
   --download-data                 Update the Lean configuration file to download data from the QuantConnect API, alias
                                   for --data-provider-historical QuantConnect
   --data-purchase-limit INTEGER   The maximum amount of QCC to spend on downloading data during the backtest when using
@@ -822,7 +828,7 @@ Usage: lean data download [OPTIONS]
   https://www.quantconnect.com/datasets
 
 Options:
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
                                   The name of the downloader data provider.
   --ib-user-name TEXT             Your Interactive Brokers username
   --ib-account TEXT               Your Interactive Brokers account id
@@ -886,6 +892,12 @@ Options:
   --bybit-api-secret TEXT         Your Bybit API secret
   --bybit-vip-level [VIP0|VIP1|VIP2|VIP3|VIP4|VIP5|SupremeVIP|Pro1|Pro2|Pro3|Pro4|Pro5]
                                   Your Bybit VIP Level
+  --trade-station-environment [live|paper]
+                                  Whether Live or Paper environment should be used
+  --trade-station-account-type [Cash|Margin|Futures|DVP]
+                                  Specifies the type of account on TradeStation
+  --alpaca-environment [live|paper]
+                                  Whether Live or Paper environment should be used
   --dataset TEXT                  The name of the dataset to download non-interactively
   --overwrite                     Overwrite existing local data
   -y, --yes                       Automatically confirm payment confirmation prompts
@@ -895,7 +907,8 @@ Options:
                                   Specify the resolution of the historical data
   --security-type [Equity|Index|Forex|Cfd|Future|Crypto|CryptoFuture|Option|IndexOption|Commodity|FutureOption]
                                   Specify the security type of the historical data
-  --market TEXT                   Specify the market name for tickers (e.g., 'USA', 'NYMEX', 'Binance')
+  --market TEXT                   Specify the market name for tickers (e.g., 'USA', 'NYMEX', 'Binance') (if not provided
+                                  or empty the default market for the requested security type will be used)
   --ticker TEXT                   Specify comma separated list of tickers to use for historical data request.
   --start TEXT                    Specify the start date for the historical data request in the format yyyyMMdd.
   --end TEXT                      Specify the end date for the historical data request in the format yyyyMMdd. (defaults
@@ -1237,7 +1250,7 @@ Options:
                                   The brokerage to use
   --data-provider-live [Interactive Brokers|Tradier|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Zerodha|Samco|Terminal Link|Trading Technologies|Kraken|TDAmeritrade|IQFeed|Polygon|IEX|CoinApi|ThetaData|Custom data only|Bybit|TradeStation|Alpaca]
                                   The live data provider to use
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Bybit]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Bybit|TradeStation|Alpaca]
                                   Update the Lean configuration file to retrieve data from the given historical provider
   --ib-user-name TEXT             Your Interactive Brokers username
   --ib-account TEXT               Your Interactive Brokers account id
@@ -1673,7 +1686,7 @@ Options:
   --parameter <TEXT FLOAT FLOAT FLOAT>...
                                   The 'parameter min max step' pairs configuring the parameters to optimize
   --constraint TEXT               The 'statistic operator value' pairs configuring the constraints of the optimization
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
                                   Update the Lean configuration file to retrieve data from the given historical provider
   --download-data                 Update the Lean configuration file to download data from the QuantConnect API, alias
                                   for --data-provider-historical QuantConnect
@@ -1750,6 +1763,12 @@ Options:
   --bybit-api-secret TEXT         Your Bybit API secret
   --bybit-vip-level [VIP0|VIP1|VIP2|VIP3|VIP4|VIP5|SupremeVIP|Pro1|Pro2|Pro3|Pro4|Pro5]
                                   Your Bybit VIP Level
+  --trade-station-environment [live|paper]
+                                  Whether Live or Paper environment should be used
+  --trade-station-account-type [Cash|Margin|Futures|DVP]
+                                  Specifies the type of account on TradeStation
+  --alpaca-environment [live|paper]
+                                  Whether Live or Paper environment should be used
   --lean-config FILE              The Lean configuration file that should be used (defaults to the nearest lean.json)
   --verbose                       Enable debug logging
   --help                          Show this message and exit.
@@ -1853,7 +1872,7 @@ Usage: lean research [OPTIONS] PROJECT
 
 Options:
   --port INTEGER                  The port to run Jupyter Lab on (defaults to 8888)
-  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit]
+  --data-provider-historical [Interactive Brokers|Oanda|Bitfinex|Coinbase Advanced Trade|Binance|Kraken|IQFeed|Polygon|FactSet|IEX|AlphaVantage|CoinApi|ThetaData|QuantConnect|Local|Terminal Link|Bybit|TradeStation|Alpaca]
                                   Update the Lean configuration file to retrieve data from the given historical provider
   --ib-user-name TEXT             Your Interactive Brokers username
   --ib-account TEXT               Your Interactive Brokers account id
@@ -1917,6 +1936,12 @@ Options:
   --bybit-api-secret TEXT         Your Bybit API secret
   --bybit-vip-level [VIP0|VIP1|VIP2|VIP3|VIP4|VIP5|SupremeVIP|Pro1|Pro2|Pro3|Pro4|Pro5]
                                   Your Bybit VIP Level
+  --trade-station-environment [live|paper]
+                                  Whether Live or Paper environment should be used
+  --trade-station-account-type [Cash|Margin|Futures|DVP]
+                                  Specifies the type of account on TradeStation
+  --alpaca-environment [live|paper]
+                                  Whether Live or Paper environment should be used
   --download-data                 Update the Lean configuration file to download data from the QuantConnect API, alias
                                   for --data-provider-historical QuantConnect
   --data-purchase-limit INTEGER   The maximum amount of QCC to spend on downloading data during the research session

lean/commands/data/download.py

@@ -15,7 +15,7 @@
 
 from docker.types import Mount
 from typing import Any, Dict, Iterable, List, Optional
-from click import command, option, confirm, pass_context, Context, Choice
+from click import command, option, confirm, pass_context, Context, Choice, prompt
 from lean.click import LeanCommand, ensure_options
 from lean.components.util.json_modules_handler import config_build_for_name
 from lean.constants import DEFAULT_ENGINE_IMAGE
@@ -437,14 +437,15 @@ def _get_download_specification_from_config(data_provider_config_json: Dict[str,
 
 
 def _get_user_input_or_prompt(user_input_data: str, available_input_data: List[str], data_provider_name: str,
-                              prompt_message_helper: str) -> str:
+                              prompt_message_helper: str, skip_validation: Optional[bool] = False) -> str:
     """
     Get user input or prompt for selection based on data types.
 
     Args:
     - user_input_data (str): User input data.
     - available_input_data (List[str]): List of available input data options.
     - data_provider_name (str): Name of the data provider.
+    - skip_validation (Optional[bool]): Whether to skip validation of user input data. Default is False.
 
     Returns:
     - str: Selected data type or prompted choice.
@@ -454,11 +455,15 @@ def _get_user_input_or_prompt(user_input_data: str, available_input_data: List[s
     """
 
     if not user_input_data:
+        if skip_validation:
+            return prompt(prompt_message_helper, "")
         # Prompt user to select a ticker's security type
         options = [Option(id=data_type, label=data_type) for data_type in available_input_data]
         return container.logger.prompt_list(prompt_message_helper, options)
 
     elif user_input_data.lower() not in [available_data.lower() for available_data in available_input_data]:
+        if skip_validation:
+            return user_input_data
         # Raise ValueError for unsupported data type
         raise ValueError(
             f"The {data_provider_name} data provider does not support {user_input_data}. "
@@ -531,7 +536,8 @@ def _replace_data_type(ctx, param, value):
 @option("--security-type", type=Choice(QCSecurityType.get_all_members(), case_sensitive=False),
     help="Specify the security type of the historical data")
 @option("--market", type=str,
-        help="Specify the market name for tickers (e.g., 'USA', 'NYMEX', 'Binance')")
+        help="Specify the market name for tickers (e.g., 'USA', 'NYMEX', 'Binance')"
+             " (if not provided or empty the default market for the requested security type will be used)")
 @option("--ticker",
         type=str,
         help="Specify comma separated list of tickers to use for historical data request.")
@@ -638,7 +644,7 @@ def download(ctx: Context,
                                                                                     QCResolution.get_all_members(),
                                                                                     "resolutions")
         data_provider_support_markets = _get_download_specification_from_config(data_provider_config_json,
-                                                                                ["USA"], "markets")
+                                                                                [""], "markets")
 
         security_type = _get_user_input_or_prompt(security_type, data_provider_support_security_types,
                                                   data_provider_historical, "Select a Ticker's security type")
@@ -647,7 +653,7 @@ def download(ctx: Context,
         resolution = _get_user_input_or_prompt(resolution, data_provider_support_resolutions,
                                                data_provider_historical, "Select a Resolution")
         market = _get_user_input_or_prompt(market, data_provider_support_markets,
-                                           data_provider_historical, "Select a Market")
+                                           data_provider_historical, "Select a Market", True)
 
         if not ticker:
             ticker = ','.join(DatasetTextOption(id="id",
@@ -704,9 +710,11 @@ def download(ctx: Context,
                          "--start-date", start.value.strftime("%Y%m%d"),
                          "--end-date", end.value.strftime("%Y%m%d"),
                          "--security-type", security_type,
-                         "--market", market,
                          "--resolution", resolution,
                          "--tickers", ticker]
+        # If no market is specified, Lean will use a default market value based on the SecurityType
+        if market != "":
+            dll_arguments.extend(["--market", market])
 
         run_options["commands"].append(' '.join(dll_arguments))