System.ArgumentException when executing multiple offscreen browsers in the same application

[peter2asp]

• What version of the product are you using?
  Nuget

  <package id="cef.redist.x64" version="3.3239.1723" targetFramework="net462" />
  <package id="cef.redist.x86" version="3.3239.1723" targetFramework="net462" />
  <package id="CefSharp.Common" version="63.0.1" targetFramework="net462" />
  <package id="CefSharp.OffScreen" version="63.0.1" targetFramework="net462" />
  <package id="CefSharp.WinForms" version="63.0.1" targetFramework="net462" />

• What architecture x86 or x64?
  x86

• On what operating system?

  • Win10

• Are you using WinForms, WPF or OffScreen?
  OffScreen

• What steps will reproduce the problem?
  We are using CefSharp browsers as clients when doing load testing of a web product.
  We run multiple offscreen browsers in different threads. We use 10 different urls that we run the tests on.
  When running more than about 70-80 browser instances some browser sub controls crashes.
  No error information is "noted" by the client browser control.
  When running the cpu usage is about 40% and there are about 16GB memory free.

• What is the expected output? What do you see instead?
  To be able to run any number of instances, as long as the machine has resources.

• Please provide any additional information below.
  From Windows Application Event log:

Application: CefSharp.BrowserSubprocess.exe
Framework Version: v4.0.30319
Description: The process was terminated due to an unhandled exception.
Exception Info: System.ArgumentException
   at System.ThrowHelper.ThrowArgumentException(System.ExceptionResource)
   at System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Insert(System.__Canon, System.__Canon, Boolean)
   at System.Collections.Generic.Dictionary`2[[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089],[System.__Canon, mscorlib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089]].Add(System.__Canon, System.__Canon)
   at <Module>.CefSharp.CefAppUnmanagedWrapper.OnProcessMessageReceived(CefSharp.CefAppUnmanagedWrapper*, scoped_refptr<CefBrowser>*, cef_process_id_t, scoped_refptr<CefProcessMessage>*)
   at <Module>.CefExecuteProcess(CefMainArgs*, scoped_refptr<CefApp>*, Void*)
   at CefSharp.BrowserSubprocess.SubProcess.Run()
   at CefSharp.BrowserSubprocess.Program.Main(System.String[])`

Cef log contains no information.
Code used to initialize the browser control:

this.browser = new ChromiumWebBrowser();
this.browser.ConsoleMessage += this.BrowserConsoleMessage;
this.browser.LoadError += this.OnLoadError;
CefSharpSettings.LegacyJavascriptBindingEnabled = true;
this.browser.RegisterJsObject("external", this);

var life = new LifespanHandler();
this.browser.LifeSpanHandler = life;


this.browser.LoadingStateChanged += this.WebBrowserLoadCompleted;
this.browser.StatusMessage += delegate(object sender, StatusMessageEventArgs args) { Logger.Debug($"{this.TestId}: StatusMessageChanged: {args.Value}"); };
this.browser.FrameLoadStart += (sender, args) => { Logger.Debug($"{this.TestId}: FrameLoadStart: {args.TransitionType}, {args.Url}"); };
this.browser.FrameLoadEnd += (sender, args) => { Logger.Debug($"{this.TestId}: FrameLoadEnd: {args.HttpStatusCode}, {args.Url}"); };
this.browser.TitleChanged += this.WebBrowserTitleChanged;
this.browser.BrowserInitialized += this.Browser_BrowserInitialized;

• Does this problem also occur in the CEF Sample Application from http://opensource.spotify.com/cefbuilds/index.html?
  No

[amaitland]

We run multiple offscreen browsers in different threads. We use 10 different urls that we run the tests on.

How many render processes are spawned? Are the urls you are testing all the same domain?

at System.ThrowHelper.ThrowArgumentException(System.ExceptionResource)

The stacktrace seems to be missing some details, like the actual exception message which is a little strange. The only ExceptionResource I can see in Dictionary.Insert is ExceptionResource.Argument_AddingDuplicate. Likely a problem with how the LegacyBinding handles storing of objects.

https://github.com/cefsharp/CefSharp/blob/cefsharp/63/CefSharp.BrowserSubprocess.Core/CefAppUnmanagedWrapper.cpp#L453

Older versions had one render process per ChromiumWebBrowser instance, so there was never any conflict.
https://github.com/cefsharp/CefSharp/blob/cefsharp/57/CefSharp.BrowserSubprocess.Core/CefAppUnmanagedWrapper.cpp#L404

You can use a RequestContext to isolate your browser instances in the short term, long term this will be addresses in #2306

To be able to run any number of instances, as long as the machine has resources.

If you are using x86 you'll be limited to 2gb of memory by default, ~3gb with LargeAddressAware set. Just because your machine has resources available, doesn't mean your application can use them. Actually seems unrelated in this case though.

When running more than about 70-80 browser instances some browser sub controls crashes.

Windows will limit you to running something like 100 sub processes, even if you had unlimited resources, there a OS limits in place.

[peter2asp]

How many render processes are spawned? Are the urls you are testing all the same domain?
We use 10 different urls/domains (basically we use same url with different ports using a LB).
We used one sub-process per browser.

Yes we changed so we use a context for each browser:
new RequestContext(new RequestContextSettings { CachePath = string.Empty }).
That seems to fix the problem.

[amaitland]

Closing as #2306 will be used to track this.