Skip to content

Commit

Permalink
Micellaneous cleanup
Browse files Browse the repository at this point in the history
Remove tutorial overview
Use dotnetcli or console as appropriate
Add applies to comment
Add preparing sections
  • Loading branch information
sdmaclea committed Oct 17, 2019
1 parent f9e971f commit 49fb20a
Show file tree
Hide file tree
Showing 7 changed files with 88 additions and 99 deletions.
35 changes: 24 additions & 11 deletions docs/core/diagnostics/app_is_leaking_memory_eventual_crash.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,39 +3,52 @@ title: Debugging a memory leak - .NET Core
description: A tutorial walk-through, debugging a memory leak in .NET Core.
author: sdmaclea
ms.author: stmaclea
ms.date: 08/27/2019
ms.date: 10/14/2019
---
# Debugging a memory leak

**This article applies to:** .NET Core 3.0 SDK and later versions

In this scenario, the endpoint will slowly start leaking memory and eventually will result in an out of memory exception. To diagnose this scenario, we need several key pieces of diagnostics data.

## Preparing

The tutorial uses:

- [Sample debug target](sample-debug-target.md) to trigger the scenario.
- [dotnet-trace](dotnet-trace.md) to list processes.
- [dotnet-counters](dotnet-counters.md) to check managed memory usage.
- [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file.

The tutorial assumes the sample and tools are ready to use and you are running on **Linux**.

## Memory counters

Before we dig into collecting diagnostics data to help us root cause this scenario, we need to convince ourselves that what we are actually seeing is a memory leak (memory growth). We can use the [dotnet-counters](dotnet-counters.md) tool to get at this information.

Lets run the [Sample debug target](sample-debug-target.md).

```bash
```dotnetcli
dotnet run
```

Then find the process ID using:

```bash
```dotnetcli
dotnet-trace list-processes
```

Before causing the leak, lets check our managed memory counters:

```bash
```dotnetcli
dotnet-counters monitor --refresh-interval 1 -p 4807
```

4807 is the process ID that was found using `dotnet-trace list-processes`. The refresh-interval is the number of seconds between refreshes.

The output should be similar to:

```
```console
Press p to pause, r to resume, q to quit.
System.Runtime:
CPU Usage (%) 4
Expand All @@ -49,11 +62,11 @@ The output should be similar to:

Here we can see that right after startup, the managed heap memory is 4 MB.

Now, let's hit the URL (http://localhost:5000/api/diagscenario/memleak/200000)
Now, let's hit the URL http://localhost:5000/api/diagscenario/memleak/200000

Rerun the dotnet-counters command. We should see an increase in memory usage as shown below:

```
```console
Press p to pause, r to resume, q to quit.
System.Runtime:
CPU Usage (%) 4
Expand All @@ -75,7 +88,7 @@ When analyzing possible memory leaks, we need access to the apps memory heap. We

Using the previous [Sample debug target](sample-debug-target.md) started above, run the following command to generate a core dump:

```bash
```dotnetcli
sudo ./dotnet-dump collect -p 4807
```

Expand All @@ -88,7 +101,7 @@ sudo ./dotnet-dump collect -p 4807

Now that we have a core dump generated, use the [dotnet-dump)](dotnet-dump.md) tool to analyze the dump:

```bash
```dotnetcli
dotnet-dump analyze core_20190430_185145
```

Expand All @@ -99,8 +112,8 @@ Where `core_20190430_185145` is the name of the core dump you want to analyze.
You'll be presented with a prompt where you can enter SOS commands. Commonly, the first thing we want to look at is the overall state of the managed heap:

```bash
dumpheap -stat
```console
> dumpheap -stat
```

The (partial) output can be seen below:
Expand Down
30 changes: 20 additions & 10 deletions docs/core/diagnostics/app_running_slow_highcpu.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,31 +3,41 @@ title: Debugging high CPU usage - .NET Core
description: A tutorial walk-through, debugging high CPU usage in .NET Core.
author: sdmaclea
ms.author: stmaclea
ms.date: 08/27/2019
ms.date: 10/14/2019
---
# Debugging high CPU usage

**This article applies to:** .NET Core 3.0 SDK and later versions

In this scenario, the [sample debug target](sample-debug-target.md) will consume excessive CPU. To diagnose this scenario, we need several key pieces of diagnostics data.

The tutorial uses:

- [Sample debug target](sample-debug-target.md) to trigger the scenario.
- [dotnet-trace](dotnet-trace.md) to list processes and generate a profile.
- [dotnet-counters](dotnet-counters.md) to monitor cpu usage.

The tutorial assumes the sample and tools are ready to use.

## CPU counters

Before we dig into collecting diagnostics data, we need to convince ourselves that what we are actually seeing is a high CPU condition.

Lets run the [sample debug target](sample-debug-target.md).

```bash
```dotnetcli
dotnet run
```

Then find the process ID using:

```bash
```dotnetcli
dotnet-trace list-processes
```

Before hitting the above URL that will cause the high CPU condition, lets check our CPU counters using the [dotnet-counters](dotnet-counters.md) tool:

```bash
```dotnetcli
dotnet-counters monitor --refresh-interval 1 -p 22884
```

Expand All @@ -47,7 +57,7 @@ Rerun the [dotnet-counters](dotnet-counters.md) command. We should see an increa

Throughout the execution of that request, CPU hovers at around 30%.

```bash
```dotnetcli
dotnet-counters monitor System.Runtime[cpu-usage] -p 22884 --refresh-interval 1
```

Expand All @@ -61,7 +71,7 @@ When analyzing a slow request, we need a diagnostics tool that can give us insig

We can use the [dotnet-trace](dotnet-trace.md) tool. Using the previous [sample debug target](sample-debug-target.md), hit the URL (http://localhost:5000/api/diagscenario/highcpu/60000) again and while its running within the 1-minute request, run:

```bash
```dotnetcli
dotnet-trace collect -p 2266 --providers Microsoft-DotNETCore-SampleProfiler
```

Expand All @@ -83,28 +93,28 @@ Set the `COMPlus_PerfMapEnabled` to cause the .NET Core app to create a `map` fi

Run the [sample debug target](sample-debug-target.md) in the same terminal session.

```bash
```dotnetcli
export COMPlus_PerfMapEnabled=1
dotnet run
```

Hit the URL (http://localhost:5000/api/diagscenario/highcpu/60000) again and while its running within the 1-minute request, run:

```bash
```console
sudo perf record -p 2266 -g
```

This command will start the perf collection process. Let it run for about 20-30 seconds and then hit CTRL-C to exit the collection process.

You can use the same perf command to see the output of the trace.

```bash
```console
sudo perf report -f
```

You can also generate a flame-graph by using the following commands:

```bash
```console
git clone --depth=1 https://github.com/BrendanGregg/FlameGraph
sudo perf script | FlameGraph/stackcollapse-perf.pl | FlameGraph/flamegraph.pl > flamegraph.svg
```
Expand Down
48 changes: 0 additions & 48 deletions docs/core/diagnostics/diagnostic-scenarios.md

This file was deleted.

52 changes: 28 additions & 24 deletions docs/core/diagnostics/hung_app.md
Original file line number Diff line number Diff line change
Expand Up @@ -3,57 +3,61 @@ title: Debugging deadlock - .NET Core
description: A tutorial walk-through, debugging locking issues in .NET Core.
author: sdmaclea
ms.author: stmaclea
ms.date: 08/27/2019
ms.date: 10/14/2019
---
# Debugging deadlock

In this scenario, the endpoint will experience a hang and thread accumulation. We'll show how you can use the existing tools to analyze the problem.
**This article applies to:** .NET Core 3.0 SDK and later versions

In this scenario, the endpoint will experience a hang and thread accumulation. We'll show how you can use the tools to analyze the problem.

## Preparing

The tutorial uses:

- [Sample debug target](sample-debug-target.md) to trigger the scenario.
- [dotnet-trace](dotnet-trace.md) to list processes.
- [dotnet-dump](dotnet-dump.md) to collect and analyze a dump file.

The tutorial assumes the sample and tools are ready to use and you are running on **Linux**.

## Core dump generation

To investigate hung application, a memory dump allows us to inspect the state of its threads and any possible locks that may have contention issues.

Lets run the [Sample debug target](sample-debug-target.md).

```bash
```dotnetcli
dotnet run
```

Then find the process ID using:

```bash
```dotnetcli
dotnet-trace list-processes
```

Before causing the leak, lets check our managed memory counters:

```bash
dotnet-counters monitor --refresh-interval 1 -p 4807
```

4807 is the process ID that was found using `dotnet-trace list-processes`.

Navigate to the following URL:

http://localhost:5000/api/diagscenario/deadlock

Let the request run for about 10-15 seconds then create the dump:

```bash
```dotnetcli
sudo ./dotnet-dump collect -p 4807
```

### Analyzing the core dump
## Analyzing the core dump

To start our investigation, let's open the core dump using dotnet-dump analyze:

```bash
```dotnetcli
./dotnet-dump analyze ~/.dotnet/tools/core_20190513_143916
```

Since we're looking at a potential hang, we want an overall feel for the thread activity in the process. We can use the threads command as shown below:

```
```console
threads
*0 0x1DBFF (121855)
1 0x1DC01 (121857)
Expand Down Expand Up @@ -102,13 +106,13 @@ The next step is to get a better understanding of what the threads are currently

Run:

```bash
clrstack -all
```console
> clrstack -all
```

A representative portion of the output looks like:

```
```console
...
...
...
Expand Down Expand Up @@ -190,7 +194,7 @@ OS Thread Id: 0x1dc88

Eye balling the callstacks for all 300+ threads shows a pattern where a majority of the threads share a common callstack:

```bash
```console
OS Thread Id: 0x1dc88
Child SP IP Call Site
00007F2ADFFAE680 00007f305abc6360 [GCFrame: 00007f2adffae680]
Expand All @@ -206,7 +210,7 @@ The callstack seems to show that the request arrived in our deadlock method that

The next step then is to find out which thread is actually holding the monitor lock. Since monitors typically store lock information in the sync block table, we can use the `syncblk` command to get more information:

```bash
```console
> syncblk
Index SyncBlock MonitorHeld Recursion Owning Thread Info SyncBlock Owner
41 000000000143D038 603 1 00007F2B542D28C0 1dc1d 20 00007f2e90080fb8 System.Object
Expand All @@ -217,11 +221,11 @@ Free 0

```

The two interesting columns are the `MonitorHeld` and the `Owning Thread Info` columns. The `MonitorHeld` shows whether a monitor lock is acquired by a thread and the number of waiting threads. The `Owning Thread Info` shows which thread currently owns the monitor lock. The thread info has three different subcolumns. The second subcolumn shows operating system thread ID.
The two interesting columns are the `MonitorHeld` and the `Owning Thread Info` columns. The `MonitorHeld` shows whether a monitor lock is acquired by a thread and the number of waiting threads. The `Owning Thread Info` shows which thread currently owns the monitor lock. The thread info has three different sub-columns. The second sub-column shows operating system thread ID.

At this point, we know two different threads (0x1dc1d and 0x1dc1e) hold a monitor lock. The next step is to take a look at what those threads are doing. We need to check if they're stuck indefinitely holding the lock. Let's use the `setthread` and `clrstack` commands to switch to each of the threads and display the callstacks:

```bash
```console
> setthread 0x1dc1d
> clrstack
OS Thread Id: 0x1dc1d (20)
Expand All @@ -240,7 +244,7 @@ OS Thread Id: 0x1dc1d (20)

Lets look at the first thread. The deadlock function is waiting to acquire a lock, but it already owns the lock. It's in deadlock waiting for the lock it already holds.

```
```console
> setthread 0x1dc1e
> clrstack
OS Thread Id: 0x1dc1e (21)
Expand Down
12 changes: 11 additions & 1 deletion docs/core/diagnostics/index.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,4 +41,14 @@ The [dotnet-dump](dotnet-dump.md) tool is a way to collect and analyze Windows a

## .NET Core diagnostics walk throughs

The [.NET Core diagnostics walk throughs](diagnostic-scenarios.md) are designed to highlight key use cases.
### Debugging a memory leak

[Debugging a memory leak](app_is_leaking_memory_eventual_crash.md) walks through finding a memory leak. The [dotnet-counters](dotnet-counters.md) tool is used to confirm the leak. Then the [dotnet-dump](dotnet-dump.md) tool is used to diagnose the leak.

### Debugging a slow running application

[Debugging high CPU usage](app_running_slow_highcpu.md) walks through investigating high CPU usage. It uses the [dotnet-counters](dotnet-counters.md) tool to confirm the high CPU usage. It then walks through using [Trace for performance analysis utility (`dotnet-trace`)](dotnet-trace.md) or Linux `perf` to collect and view CPU usage profile.

### Debugging deadlock

The [debugging deadlock](hung_app.md) tutorial explores using the [dotnet-dump](dotnet-dump.md) tool to investigate threads and locks.
Loading

0 comments on commit 49fb20a

Please sign in to comment.