lean cli optimize Result was not reached

[alexschwantes]

When running lean optimize myproject the optimization proceeds as expected however it ends in Optimization has ended. Result was not reached

This is the folder structure under myproject/optimizations/

There are no errors produced during the execution. Lean CLI runs fine for other commands such as backtests/push/pull etc. environment is Windows.

I have tested with multiple optimization strategies and for a longer backtest duration with more trades in each backtest, always the same result.

Looking at LeanOptimizer.cs, it seems this happens when the result is null.

Verbose logs log.txt

optimize.py issues

Looking at the optimize.py file I found some things that could potentially be bugs or as a result of the above issue.

1. Line 352 incorrect regex. The log file it is reading looks like the following:

2024-10-03T03:57:27.9511359Z TRACE:: Config.Get(): Configuration key not found. Key: data-channel - Using default value: 
2024-10-03T03:57:27.9534607Z TRACE:: Config.Get(): Configuration key not found. Key: optimization-id - Using default value: c0f3f98c-92da-40eb-8445-09d8b3ce7658
2024-10-03T03:57:28.0682248Z TRACE:: Config.GetValue(): maximum-concurrent-backtests - Using default value: 16
2024-10-03T03:57:28.0685627Z TRACE:: Config.Get(): Configuration key not found. Key: out-of-sample-max-end-date - Using default value: 
2024-10-03T03:57:28.0688406Z TRACE:: Config.GetValue(): out-of-sample-days - Using default value: 0
2024-10-03T03:57:28.0694766Z TRACE:: Config.Get(): Configuration key not found. Key: optimization-launcher - Using default value: ConsoleLeanOptimizer
2024-10-03T03:57:28.0732074Z TRACE:: Config.GetValue(): optimization-update-interval - Using default value: 10
2024-10-03T03:57:28.0763313Z TRACE:: Config.Get(): Configuration key not found. Key: lean-binaries-location - Using default value: /Lean/Optimizer.Launcher/bin/Debug/../../../Launcher/bin/Debug/QuantConnect.Lean.Launcher
2024-10-03T03:57:28.0771597Z TRACE:: Config.Get(): Configuration key not found. Key: algorithm-type-name - Using default value: 
2024-10-03T03:57:28.0773812Z TRACE:: Config.Get(): Configuration key not found. Key: algorithm-language - Using default value: 
2024-10-03T03:57:28.0775694Z TRACE:: Config.Get(): Configuration key not found. Key: algorithm-location - Using default value: 
2024-10-03T03:57:28.0777518Z TRACE:: Config.GetValue(): estimate - Using default value: False
2024-10-03T03:57:28.1009795Z TRACE:: LeanOptimizer.LaunchLeanForParameterSet(OID c0f3f98c-92da-40eb-8445-09d8b3ce7658): launched backtest 'abdc04c3-c14b-419d-a2cb-20eb923de584' with parameters 'custom_alpha:1'
2024-10-03T03:57:28.1037947Z TRACE:: LeanOptimizer.LaunchLeanForParameterSet(OID c0f3f98c-92da-40eb-8445-09d8b3ce7658): launched backtest 'b6ad281a-0906-4c03-b7fb-4c3d20e53b31' with parameters 'custom_alpha:2'
2024-10-03T03:57:28.1042958Z TRACE:: LeanOptimizer.Start(OID c0f3f98c-92da-40eb-8445-09d8b3ce7658): start ended. Waiting on 2 backtests
2024-10-03T03:57:28.1065607Z TRACE:: ConsoleLeanOptimizer.SendUpdate(): 2 Completed:0, Failed:0, Running:2, In Queue:0, Average Length:00:00:00, Total Runtime:00:00:00
2024-10-03T03:57:38.4727132Z TRACE:: ConsoleLeanOptimizer.SendUpdate(): 2 Completed:1, Failed:0, Running:1, In Queue:0, Average Length:00:00:10, Total Runtime:00:00:10
2024-10-03T03:57:39.3784596Z TRACE:: LeanOptimizer.TriggerOnEndEvent(OID c0f3f98c-92da-40eb-8445-09d8b3ce7658): Optimization has ended. Result was not reached

The regex on that line is looking for r"ParameterSet: \(([^)]+)\) backtestId '([^']+)'" but this does not exist in the log. the only thing that exists is r"ParameterSet\(([^)]+)\): launched backtest '([^']+)'"

My guess is perhaps that due to no result being reached, that the line containing the original parameter set is not output to the log?

2. line 365 and 366: following on from point 1, when loading the runtime statistics for a given backtests, these keys don't exist in the dictionary:

runtimeStatistics=optimal_results["RuntimeStatistics"],
statistics=optimal_results["Statistics"])

but these do:

runtimeStatistics=optimal_results["runtimeStatistics"],
statistics=optimal_results["statistics"])

3. line 263: config["results-destination-folder"] = "/Results"

I'm really not sure of the purpose of this, as I don't see any /Results folder in the /optimizations folder. Is this a byproduct of no result being reached?

[Martin-Molinero]

Thank you for the report @alexschwantes, we will look into it

[Marinovsky]

Closed as solved by QuantConnect/Lean#8382