Skip to content
This repository has been archived by the owner on Oct 11, 2023. It is now read-only.

Cannot read property 'address' of undefined / Detach already received / transfer after detach #205

Closed
dan-v-elevate opened this issue Feb 14, 2019 · 4 comments
Closed

Cannot read property 'address' of undefined / Detach already received / transfer after detach #205

dan-v-elevate opened this issue Feb 14, 2019 · 4 comments

Comments

@dan-v-elevate
Copy link

dan-v-elevate commented Feb 14, 2019 •
edited
Loading

Describe the bug
We are using this library 'azure/event-hub' in a v2 Azure function with an event hub trigger. The function essentially reads in batches of data, transforms it, and outputs it to a number of other event hubs. Note that we are using the consumption plan, so we have somewhat limited insight into the underlying hosts where this is running. The issue we are seeing results in the azure function host crashing and not recovering (opened an issue for that first Azure/azure-functions-host#4079) and we get a very limited set of errors bubbled up that appears to possibly be related to this library. The errors: Error: Detach already received, Error: transfer after detach, and TypeError: Cannot read property 'address' of undefined is what we are seeing in the stack trace which looks pretty similar to another issue already posted here #171 . To be clear, I don't know for certain this is the issue based on the limited stack trace, but if you could provide any input for helping further debug if it is or isn't that would be really appreciated.

Stack trace from azure functions 
[  
   {  
      "severityLevel":"Error",
      "outerId":"0",
      "message":"Exception while executing function: Functions.iot_intake",
      "type":"Microsoft.Azure.WebJobs.Host.FunctionInvocationException",
      "parsedStack":[  
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":0,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__17.MoveNext",
            "level":1,
            "line":293,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":2,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":3,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":4,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":5,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<TryExecuteAsync>d__14.MoveNext",
            "level":6,
            "line":89,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         }
      ],
      "id":"5240712"
   },
   {  
      "severityLevel":"Error",
      "outerId":"5240712",
      "message":"Failed to start language worker for: node",
      "type":"Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerChannelException",
      "parsedStack":[  
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":0,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":1,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":2,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":3,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+<InvokeCore>d__6.MoveNext",
            "level":4,
            "line":74,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":5,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":6,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":7,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":8,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext",
            "level":9,
            "line":84,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":10,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":11,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":12,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":13,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext",
            "level":14,
            "line":20,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":15,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":16,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":17,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":18,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__9.MoveNext",
            "level":19,
            "line":63,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":20,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":21,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":22,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":23,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeAsync>d__25.MoveNext",
            "level":24,
            "line":556,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":25,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":26,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":27,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":28,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__24.MoveNext",
            "level":29,
            "line":503,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":30,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":31,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":32,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":33,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__23.MoveNext",
            "level":34,
            "line":439,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":35,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":36,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":37,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":38,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__17.MoveNext",
            "level":39,
            "line":249,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":40,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":41,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":42,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":43,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+<InvokeCore>d__6.MoveNext",
            "level":44,
            "line":74,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":45,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":46,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":47,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":48,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext",
            "level":49,
            "line":84,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":50,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":51,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":52,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":53,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext",
            "level":54,
            "line":20,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":55,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":56,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":57,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":58,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__9.MoveNext",
            "level":59,
            "line":63,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":60,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":61,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":62,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":63,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeAsync>d__25.MoveNext",
            "level":64,
            "line":556,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":65,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":66,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":67,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":68,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__24.MoveNext",
            "level":69,
            "line":503,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":70,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":71,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":72,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":73,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__23.MoveNext",
            "level":74,
            "line":439,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":75,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":76,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":77,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":78,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__17.MoveNext",
            "level":79,
            "line":249,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":80,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":81,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":82,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":83,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+<InvokeCore>d__6.MoveNext",
            "level":84,
            "line":74,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":85,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":86,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":87,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":88,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext",
            "level":89,
            "line":84,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":90,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59468,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__24.MoveNext",
            "level":59469,
            "line":503,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59470,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59471,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59472,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59473,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__23.MoveNext",
            "level":59474,
            "line":439,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59475,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59476,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59477,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59478,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__17.MoveNext",
            "level":59479,
            "line":249,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59480,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59481,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59482,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":59483,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+<InvokeCore>d__6.MoveNext",
            "level":59484,
            "line":74,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59485,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59486,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59487,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":59488,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext",
            "level":59489,
            "line":84,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59490,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59491,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59492,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59493,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext",
            "level":59494,
            "line":20,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59495,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59496,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59497,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":59498,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__9.MoveNext",
            "level":59499,
            "line":63,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59500,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59501,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59502,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":59503,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeAsync>d__25.MoveNext",
            "level":59504,
            "line":556,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59505,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59506,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59507,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59508,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__24.MoveNext",
            "level":59509,
            "line":503,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59510,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59511,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59512,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59513,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__23.MoveNext",
            "level":59514,
            "line":439,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59515,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59516,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59517,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59518,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__17.MoveNext",
            "level":59519,
            "line":249,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59520,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59521,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59522,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":59523,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.WorkerLanguageInvoker+<InvokeCore>d__6.MoveNext",
            "level":59524,
            "line":74,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\Rpc\\WorkerLanguageInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59525,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59526,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59527,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":59528,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Script, Version=2.0.0.0, Culture=neutral, PublicKeyToken=null",
            "method":"Microsoft.Azure.WebJobs.Script.Description.FunctionInvokerBase+<Invoke>d__24.MoveNext",
            "level":59529,
            "line":84,
            "fileName":"C:\\azure-webjobs-sdk-script\\src\\WebJobs.Script\\Description\\FunctionInvokerBase.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59530,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59531,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59532,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59533,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.VoidTaskMethodInvoker`2+<InvokeAsync>d__2.MoveNext",
            "level":59534,
            "line":20,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\VoidTaskMethodInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59535,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59536,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59537,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":59538,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionInvoker`2+<InvokeAsync>d__9.MoveNext",
            "level":59539,
            "line":63,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionInvoker.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59540,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59541,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59542,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter`1.GetResult",
            "level":59543,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<InvokeAsync>d__25.MoveNext",
            "level":59544,
            "line":556,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59545,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59546,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59547,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59548,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithWatchersAsync>d__24.MoveNext",
            "level":59549,
            "line":503,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59550,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59551,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59552,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59553,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__23.MoveNext",
            "level":59554,
            "line":439,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw",
            "level":59555,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess",
            "level":59556,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification",
            "level":59557,
            "line":0
         },
         {  
            "assembly":"System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e",
            "method":"System.Runtime.CompilerServices.TaskAwaiter.GetResult",
            "level":59558,
            "line":0
         },
         {  
            "assembly":"Microsoft.Azure.WebJobs.Host, Version=3.0.4.0, Culture=neutral, PublicKeyToken=31bf3856ad364e35",
            "method":"Microsoft.Azure.WebJobs.Host.Executors.FunctionExecutor+<ExecuteWithLoggingAsync>d__17.MoveNext",
            "level":59559,
            "line":249,
            "fileName":"C:\\projects\\azure-webjobs-sdk-rqm4t\\src\\Microsoft.Azure.WebJobs.Host\\Executors\\FunctionExecutor.cs"
         }
      ],
      "id":"57175771"
   },
   {  
      "severityLevel":"Error",
      "outerId":"57175771",
      "message":"One or more errors occurred. (node exited with code 1\n [error] Worker 2407da9e-1a4c-460f-a5c4-c34846a4f4de uncaught exception: TypeError: Cannot read property 'address' of undefined) (node exited with code 1\n [error] Worker 8d64531a-c854-41e6-b06a-a7179d8583f7 uncaught exception: { Error: Unhandled \"error\" event. ([object Object]), { _context: Error: transfer after detach) (node exited with code 1\n [error] Worker 2537693d-d57a-4d81-898e-952329111b3a uncaught exception: { Error: Unhandled \"error\" event. ([object Object]), { _context: Error: Detach already received)",
      "type":"System.AggregateException",
      "id":"45787762"
   },
   {  
      "severityLevel":"Error",
      "outerId":"45787762",
      "message":"node exited with code 1\n [error] Worker 2407da9e-1a4c-460f-a5c4-c34846a4f4de uncaught exception: TypeError: Cannot read property 'address' of undefined",
      "type":"Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException",
      "id":"55567947"
   },
   {  
      "severityLevel":"Error",
      "outerId":"45787762",
      "message":"node exited with code 1\n [error] Worker 8d64531a-c854-41e6-b06a-a7179d8583f7 uncaught exception: { Error: Unhandled \"error\" event. ([object Object]), { _context: Error: transfer after detach",
      "type":"Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException",
      "id":"58998898"
   },
   {  
      "severityLevel":"Error",
      "outerId":"45787762",
      "message":"node exited with code 1\n [error] Worker 2537693d-d57a-4d81-898e-952329111b3a uncaught exception: { Error: Unhandled \"error\" event. ([object Object]), { _context: Error: Detach already received",
      "type":"Microsoft.Azure.WebJobs.Script.Rpc.LanguageWorkerProcessExitException",
      "id":"57767042"
   }
]

To Reproduce
We don't have a simple reproduction of the issue at the moment, but it has happened very regularly for us over the past month. In the Azure function we establish a few connections to different event hubs (eventHubClient.createFromConnectionString) outside of the function handler and then during function executions we are sending data to these event hubs (await client.send({ body: output, partitionKey: partitionKey)). Sometimes after recycling hosts we will hit this issue immediately and other times we can go a few days without hitting it.

Expected behavior
Expect connections to event hub to keep being able to send data without throwing these exceptions.

Package-name: azure/event-hubs
Package-version: 1.0.8
node.js version: 8.11.1
OS name and version: Running in Azure functions consumption plan on Windows (32 bit)

Additional Information
Additionally I'm also seeing the issue #186 fairly frequently and not sure if this is related/contributing to this issue.

Stack trace for InvalidOperationError 
Timestamp: 2019-02-13T12:22:48.2390823Z
Inocation ID: dc2f9399-9f5e-4a61-bc1a-23d5f7d21ccd
{ InvalidOperationError: Error sending batch to rules event hub -- A link to connection '237673' $cbs node has already been opened.
    at Object.translate (D:\home\site\wwwroot\node_modules\@azure\amqp-common\dist\lib\errors.js:527:17)
    at CbsClient.<anonymous> (D:\home\site\wwwroot\node_modules\@azure\amqp-common\dist\lib\cbs.js:87:32)
    at Generator.throw (<anonymous>)
    at rejected (D:\home\site\wwwroot\node_modules\tslib\tslib.js:105:69)
    at <anonymous>
    at process._tickCallback (internal/process/next_tick.js:188:7)
  name: 'InvalidOperationError',
  translated: true,
  retryable: false,
  info: null,
  condition: 'amqp:not-allowed' }

@ramya-rao-a
Copy link
Contributor

@dan-v-elevate Apologies for not responding sooner.

We have just released a new version 2.0.0 for the Event Hubs library which includes the fix for #186
Please see #186 (comment)

Do you still see the other errors that you mentioned in the issue description?

The TypeError: Cannot read property 'address' of undefined should have an accompanying stack trace, which will be helpful for us to figure out what the problem is.

The other errors Error: Detach already received and Error: transfer after detach are related to the underlying message transfer details in AMQP which is hard for us to debug without more logs. If you still see this after updating to version 2.0.0 of this library, then I can work with you to enable logging to understand the issue better

@dan-v-elevate
Copy link
Author

@ramya-rao-a - thanks for the update on this! It seems like the majority of our issues are coming from 'A link to connection '.....' $cbs node has already been opened' issue. We will pull in 2.0 and see if that helps.

@dan-v-elevate
Copy link
Author

@ramya-rao-a - just wanted to let you know that 2.0 seems to have fixed our issues. Thanks!

@ramya-rao-a
Copy link
Contributor

That's great to hear, thanks @dan-v-elevate!

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@ramya-rao-a @dan-v-elevate