Skip to content

[Diangostics][dotnet-trace] Add collect-linux verb #47894

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 5 commits into
base: main
Choose a base branch
from
Draft

[Diangostics][dotnet-trace] Add collect-linux verb #47894

Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
a666306
[Diagnostics] Cleanup
mdh1418 Aug 7, 2025
a2e4f74
[Diagnostisc][dotnet-trace] Add collect-linux verb
mdh1418 Aug 7, 2025
e1a3ec7
[Diagnostics][dotnet-trace] Add event filter option
mdh1418 Aug 7, 2025
6c9fdb6
Revert "[Diagnostics][dotnet-trace] Add event filter option"
mdh1418 Aug 14, 2025
728afed
Address Feedback
mdh1418 Aug 15, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
230 changes: 212 additions & 18 deletions docs/core/diagnostics/dotnet-trace.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,8 @@ The `dotnet-trace` tool:
* Is a cross-platform .NET Core tool.
* Enables the collection of .NET Core traces of a running process without a native profiler.
* Is built on [`EventPipe`](./eventpipe.md) of the .NET Core runtime.
* Delivers the same experience on Windows, Linux, or macOS.
* Delivers the same experience on Windows, Linux, and macOS.
* Offers an additional command on Linux to collect Linux perf events.

## Options

Expand All @@ -55,15 +56,12 @@ The `dotnet-trace` tool:

Displays the version of the dotnet-trace utility.

- **`--duration`**

How long to run the trace. `--duration 00:00:00:05` will run it for 5 seconds.

## Commands

| Command |
|-----------------------------------------------------------|
| [dotnet-trace collect](#dotnet-trace-collect) |
| [dotnet-trace collect-linux](#dotnet-trace-collect-linux) |
| [dotnet-trace convert](#dotnet-trace-convert) |
| [dotnet-trace ps](#dotnet-trace-ps) |
| [dotnet-trace list-profiles](#dotnet-trace-list-profiles) |
Expand All @@ -76,13 +74,23 @@ Collects a diagnostic trace from a running process or launches a child process a
### Synopsis

```dotnetcli
dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--clrevents <clrevents>]
dotnet-trace collect
[--buffersize <size>]
[--clreventlevel <clreventlevel>]
[--clrevents <clrevents>]
[--dsrouter <ios|ios-sim|android|android-emu>]
[--format <Chromium|NetTrace|Speedscope>] [-h|--help] [--duration dd:hh:mm:ss]
[-n, --name <name>] [--diagnostic-port] [-o|--output <trace-file-path>] [-p|--process-id <pid>]
[--profile <profile-name>] [--providers <list-of-comma-separated-providers>]
[--format <Chromium|NetTrace|Speedscope>]
[-h|--help]
[--duration dd:hh:mm:ss]
[-n, --name <name>]
[--diagnostic-port]
[-o|--output <trace-file-path>]
[-p|--process-id <pid>]
[--profile <profile-name>]
[--providers <list-of-comma-separated-providers>]
[-- <command>] (for target applications running .NET 5 or later)
[--show-child-io] [--resume-runtime]
[--show-child-io]
[--resume-runtime]
[--stopping-event-provider-name <stoppingEventProviderName>]
[--stopping-event-event-name <stoppingEventEventName>]
[--stopping-event-payload-filter <stoppingEventPayloadFilter>]
Expand Down Expand Up @@ -119,7 +127,7 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
| ------------ | ------------------- |
| `gc` | `0x1` |
| `gchandle` | `0x2` |
| `fusion` | `0x4` |
| `assemblyloader` | `0x4` |
| `loader` | `0x8` |
| `jit` | `0x10` |
| `ngen` | `0x20` |
Expand All @@ -138,7 +146,7 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
| `gcheapdump` | `0x100000` |
| `gcsampledobjectallocationhigh` | `0x200000` |
| `gcheapsurvivalandmovement` | `0x400000` |
| `gcheapcollect` | `0x800000` |
| `managedheapcollect` | `0x800000` |
| `gcheapandtypenames` | `0x1000000` |
| `gcsampledobjectallocationlow` | `0x2000000` |
| `perftrack` | `0x20000000` |
Expand All @@ -152,13 +160,16 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--
| `compilationdiagnostic` | `0x2000000000` |
| `methoddiagnostic` | `0x4000000000` |
| `typediagnostic` | `0x8000000000` |
| `jitinstrumentationdata` | `0x10000000000` |
| `profiler` | `0x20000000000` |
| `waithandle` | `0x40000000000` |
| `allocationsampling` | `0x80000000000` |

You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md).

- **`--dsrouter {ios|ios-sim|android|android-emu}**

Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information.
Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information.

- **`--format {Chromium|NetTrace|Speedscope}`**

Expand Down Expand Up @@ -204,11 +215,13 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--

A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available:

| Profile | Description |
|---------|-------------|
|`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.|
|`gc-verbose`|Tracks GC collections and samples object allocations.|
|`gc-collect`|Tracks GC collections only at very low overhead.|
| Profile | Description |
|---------|-------------|
|`dotnet-default`|Combination of `dotnet-thread-time` and `dotnet-clr-default`. This is the default profile used when no other provider/event/profile options are specified.|
|`dotnet-thread-time`|Useful for tracking .NET thread-time information. This is the default profile used when no other provider/event/profile options are specified.|
|`dotnet-clr-default`|Useful for tracking a multitude of .NET events.|
|`gc-verbose`|Tracks GC collections and samples object allocations.|
|`gc-collect`|Tracks GC collections only at very low overhead.|

- **`--providers <list-of-comma-separated-providers>`**

Expand Down Expand Up @@ -259,6 +272,187 @@ dotnet-trace collect [--buffersize <size>] [--clreventlevel <clreventlevel>] [--

> - When specifying a stopping event through the `--stopping-event-*` options, as the EventStream is being parsed asynchronously, there will be some events that pass through between the time a trace event matching the specified stopping event options is parsed and the EventPipeSession is stopped.

## dotnet-trace collect-linux

Collects diagnostic traces using perf_events, a Linux OS technology. `collect-linux` requires admin privileges to capture kernel- and user-mode events, and by default, captures events from all processes.

This Linux-only command includes the same .NET events as [`dotnet-trace collect`](#dotnet-trace-collect), and it uses the kernel’s user_events mechanism to emit .NET events as perf events, enabling unification of user-space .NET events with kernel-space system events.

### Prerequisites

- Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+)
- Root permissions
- .NET 10+

### Synopsis

```dotnetcli
dotnet-trace collect-linux
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can prune away some of these options

  • buffersize - If OneCollect can reasonably pick the size then we may not need an option to explicitly set it.
  • diagnostics-port - I think we'd only need this in some advanced scenarios. Given that we could be profiling multiple processes maybe we'd even need multiple ports? I'd suggest lets leave it out for now and wait to see what happens.
  • resume-runtime - if we don't have diagnostics port then we shouldn't need this.
  • event-filters - I'd suggest we leave this out for now to keep things simpler.
  • tracepoint-configs - can we leave this out? If the user cares about the tracepoint names that imples they are going to use some other tool besides dotnet-trace to record the events. But if that is true I'm not sure why they'd want dotnet-trace to be creating a nettrace file as well?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would there be a scenario where users are capturing events with another tool, but they want to start capturing runtime events from their .NET app as well, and just use dotnet-trace as the medium to start a user_events EventPipeSession?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can conceive of that scenario, but I'm suggesting we don't do anything to proactively support it right now to keep the scope smaller.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pruned and grouped the options into functionality (the previous order already deviates from dotnet-trace collect --help, and unless there's a reason to keep the options in some sort of alphabetical order, I think the functional order is easier to understand)

[-h|--help]

# Provider/Event Specification
[--providers <list-of-comma-separated-providers>]
[--clreventlevel <clreventlevel>]
[--clrevents <clrevents>]
[--perf-event-tracepoints <list-of-perf-event-tracepoints>]
[--profile <profile-name>]

# Trace Collection
[--format <Chromium|NetTrace|Speedscope>]
[-o|--output <trace-file-path>]
[--duration dd:hh:mm:ss]

# .NET Process Target (Optional)
[-n, --name <name>]
[-p|--process-id <pid>]
[-- <command>]
[--show-child-io]
```

### Options

#### Provider/Event Specification Options

- **`--providers <list-of-comma-separated-providers>`**

A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile <profile-name>`. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profile.

This list of providers is in the form:

- `Provider[,Provider]`
- `Provider` is in the form: `KnownProviderName[:Flags[:Level][:KeyValueArgs]]`.
- `KeyValueArgs` is in the form: `[key1=value1][;key2=value2]`.

To learn more about some of the well-known providers in .NET, refer to [Well-known Event Providers](./well-known-event-providers.md).

- **`--clreventlevel <clreventlevel>`**

Verbosity of CLR events to be emitted.
The following table shows the available event levels.

| String value | Numeric value |
| --------------- | :-----------: |
| `logalways` | `0` |
| `critical` | `1` |
| `error` | `2` |
| `warning` | `3` |
| `informational` | `4` |
| `verbose` | `5` |

- **`--clrevents <clrevents>`**

A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect --clrevents gc+gchandle --clreventlevel informational`. The table below shows the list of available keywords:

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If someone specified "sched:sched_wakeup,sched:sched_switch" what provider name, event name, and field lists would we expect to show up in the nettrace file? (This level of detail may not go into the docs, but we should understand it ourselves to decide what info should go in the docs)

| Keyword String Alias | Keyword Hex Value |
| ------------ | ------------------- |
| `gc` | `0x1` |
| `gchandle` | `0x2` |
| `assemblyloader` | `0x4` |
| `loader` | `0x8` |
| `jit` | `0x10` |
| `ngen` | `0x20` |
| `startenumeration` | `0x40` |
| `endenumeration` | `0x80` |
| `security` | `0x400` |
| `appdomainresourcemanagement` | `0x800` |
| `jittracing` | `0x1000` |
| `interop` | `0x2000` |
| `contention` | `0x4000` |
| `exception` | `0x8000` |
| `threading` | `0x10000` |
| `jittedmethodiltonativemap` | `0x20000` |
| `overrideandsuppressngenevents` | `0x40000` |
| `type` | `0x80000` |
| `gcheapdump` | `0x100000` |
| `gcsampledobjectallocationhigh` | `0x200000` |
| `gcheapsurvivalandmovement` | `0x400000` |
| `managedheapcollect` | `0x800000` |
| `gcheapandtypenames` | `0x1000000` |
| `gcsampledobjectallocationlow` | `0x2000000` |
| `perftrack` | `0x20000000` |
| `stack` | `0x40000000` |
| `threadtransfer` | `0x80000000` |
| `debugger` | `0x100000000` |
| `monitoring` | `0x200000000` |
| `codesymbols` | `0x400000000` |
| `eventsource` | `0x800000000` |
| `compilation` | `0x1000000000` |
| `compilationdiagnostic` | `0x2000000000` |
| `methoddiagnostic` | `0x4000000000` |
| `typediagnostic` | `0x8000000000` |
| `jitinstrumentationdata` | `0x10000000000` |
| `profiler` | `0x20000000000` |
| `waithandle` | `0x40000000000` |
| `allocationsampling` | `0x80000000000` |

You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md).

- **`--perf-event-tracepoints <list-of-perf-event-tracepoints>`**

A comma-separated list of kernel perf event tracepoints to include in the trace. Available kernel events can be found under tracefs, which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available tracepoints or through the `events/` subdirectory for categorized events.

Example: `--perf-event-tracepoints syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup`

- **`--profile <profile-name>`**

A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available:

| Profile | Description |
|---------|-------------|
|`dotnet-default`|Combination of `dotnet-thread-time` and `dotnet-clr-default`. This is the default profile used when no other provider/event/profile options are specified.|
|`dotnet-thread-time`|Useful for tracking .NET thread-time information. This is the default profile used when no other provider/event/profile options are specified.|
|`dotnet-clr-default`|Useful for tracking a multitude of .NET events.|
|`gc-verbose`|Tracks GC collections and samples object allocations.|
|`gc-collect`|Tracks GC collections only at very low overhead.|
|`cpu-sampling`|Useful for tracking CPU usage.|

#### Trace Collection Options

- **`--format {Chromium|NetTrace|Speedscope}`**

Sets the output format for the trace file conversion. The default is `NetTrace`.

- **`-o|--output <trace-file-path>`**

The output path for the collected trace data. If not specified it defaults to `<appname>_<yyyyMMdd>_<HHmmss>.nettrace`, for example, `myapp_20210315_111514.nettrace``.

- **`--duration <time-to-run>`**

The time for the trace to run. Use the `dd:hh:mm:ss` format. For example `00:00:00:05` will run it for 5 seconds.

#### .NET Process Target Options

> ![NOTE]
> The default behavior for collect-linux is to collect .NET events and perf event tracepoints from ALL discoverable processes.

- **`-n, --name <name>`**

The name of the process to collect the trace from.

> [!NOTE]
> By default, `/tmp` will be searched to discover .NET processes diagnostics ports. Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application.

- **`-p|--process-id <PID>`**

The process ID to collect the trace from.

> [!NOTE]
> By default, `/tmp` will be searched to discover .NET processes diagnostics ports. Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application.

- **`-- <command>`**

After the collection configuration parameters, the user can append `--` followed by a command to start a .NET application. This may be helpful when diagnosing issues that happen early in the process, such as startup performance issue or assembly loader and binder errors.

- **`--show-child-io`**

Only applicable when `-- <command>` is used. Shows the input and output streams of a launched child process in the current console.

> [!NOTE]

> - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace.

> - To collect a trace using `dotnet-trace collect-linux`, it needs to be run with root permissions (`CAP_PERFMON`/`CAP_SYS_ADMIN`). Otherwise, the tool will fail to establish collect events.

## dotnet-trace convert

Converts `nettrace` traces to alternate formats for use with alternate trace analysis tools.
Expand Down