From a666306e40724817986416fd2bf5205b89057184 Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Thu, 7 Aug 2025 16:51:43 -0400 Subject: [PATCH 1/5] [Diagnostics] Cleanup --- docs/core/diagnostics/dotnet-trace.md | 38 +++++++++++-------- .../eventsource-collect-and-view-traces.md | 2 +- 2 files changed, 23 insertions(+), 17 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index b2f6512906ca9..a6f6e17b860dd 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -43,7 +43,7 @@ The `dotnet-trace` tool: * Is a cross-platform .NET Core tool. * Enables the collection of .NET Core traces of a running process without a native profiler. * Is built on [`EventPipe`](./eventpipe.md) of the .NET Core runtime. -* Delivers the same experience on Windows, Linux, or macOS. +* On Linux, provides additional integration with kernel user_events for native tracing tool compatibility. ## Options @@ -55,10 +55,6 @@ The `dotnet-trace` tool: Displays the version of the dotnet-trace utility. -- **`--duration`** - - How long to run the trace. `--duration 00:00:00:05` will run it for 5 seconds. - ## Commands | Command | @@ -76,13 +72,23 @@ Collects a diagnostic trace from a running process or launches a child process a ### Synopsis ```dotnetcli -dotnet-trace collect [--buffersize ] [--clreventlevel ] [--clrevents ] +dotnet-trace collect + [--buffersize ] + [--clreventlevel ] + [--clrevents ] [--dsrouter ] - [--format ] [-h|--help] [--duration dd:hh:mm:ss] - [-n, --name ] [--diagnostic-port] [-o|--output ] [-p|--process-id ] - [--profile ] [--providers ] + [--format ] + [-h|--help] + [--duration dd:hh:mm:ss] + [-n, --name ] + [--diagnostic-port] + [-o|--output ] + [-p|--process-id ] + [--profile ] + [--providers ] [-- ] (for target applications running .NET 5 or later) - [--show-child-io] [--resume-runtime] + [--show-child-io] + [--resume-runtime] [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] @@ -158,7 +164,7 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- - **`--dsrouter {ios|ios-sim|android|android-emu}** - Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information. + Starts [dotnet-dsrouter](dotnet-dsrouter.md) and connects to it. Requires [dotnet-dsrouter](dotnet-dsrouter.md) to be installed. Run `dotnet-dsrouter -h` for more information. - **`--format {Chromium|NetTrace|Speedscope}`** @@ -204,11 +210,11 @@ dotnet-trace collect [--buffersize ] [--clreventlevel ] [-- A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available: - | Profile | Description | - |---------|-------------| - |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.| - |`gc-verbose`|Tracks GC collections and samples object allocations.| - |`gc-collect`|Tracks GC collections only at very low overhead.| + | Profile | Description | + |---------|-------------| + |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.| + |`gc-verbose`|Tracks GC collections and samples object allocations.| + |`gc-collect`|Tracks GC collections only at very low overhead.| - **`--providers `** diff --git a/docs/core/diagnostics/eventsource-collect-and-view-traces.md b/docs/core/diagnostics/eventsource-collect-and-view-traces.md index 14e7dcbc6975a..fd6b2a32d9dc6 100644 --- a/docs/core/diagnostics/eventsource-collect-and-view-traces.md +++ b/docs/core/diagnostics/eventsource-collect-and-view-traces.md @@ -214,7 +214,7 @@ into other formats, such as Chromium or [Speedscope](https://www.speedscope.app/ Trace completed. ``` - dotnet-trace uses the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in + dotnet-trace uses a comma-delimited variant of the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in the `--providers` argument. For more options on how to take traces using dotnet-trace, see the [dotnet-trace docs](./dotnet-trace.md#collect-a-trace-with-dotnet-trace). From a2e4f74e421bad25777e98de5608157becf1f2f4 Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Thu, 7 Aug 2025 16:52:00 -0400 Subject: [PATCH 2/5] [Diagnostisc][dotnet-trace] Add collect-linux verb --- docs/core/diagnostics/dotnet-trace.md | 129 ++++++++++++++++++++++++++ 1 file changed, 129 insertions(+) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index a6f6e17b860dd..bba276360ff3e 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -60,6 +60,7 @@ The `dotnet-trace` tool: | Command | |-----------------------------------------------------------| | [dotnet-trace collect](#dotnet-trace-collect) | +| [dotnet-trace collect-linux](#dotnet-trace-collect-linux) | | [dotnet-trace convert](#dotnet-trace-convert) | | [dotnet-trace ps](#dotnet-trace-ps) | | [dotnet-trace list-profiles](#dotnet-trace-list-profiles) | @@ -265,6 +266,134 @@ dotnet-trace collect > - When specifying a stopping event through the `--stopping-event-*` options, as the EventStream is being parsed asynchronously, there will be some events that pass through between the time a trace event matching the specified stopping event options is parsed and the EventPipeSession is stopped. +## dotnet-trace collect-linux + +Collects diagnostic traces from .NET applications using Linux user_events as a transport layer. This command provides the same functionality as [`dotnet-trace collect`](#dotnet-trace-collect) but routes .NET runtime events through the Linux kernel's user_events subsystem before writing them to `.nettrace` files. + +This transport approach enables automatic unification of user-space .NET events with kernel-space system events, since both are captured in the same kernel tracing infrastructure. Linux tools like `perf` and `ftrace` can monitor events in real-time while maintaining full compatibility with existing .NET profiling workflows. + +### Prerequisites + +- Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+) +- Appropriate permissions to access `/sys/kernel/tracing/user_events_data` +- .NET 10+ + +### Synopsis + +```dotnetcli +dotnet-trace collect-linux + [--buffersize ] + [--clreventlevel ] + [--clrevents ] + [--format ] + [-h|--help] + [--duration dd:hh:mm:ss] + [-n, --name ] + [--diagnostic-port] + [-o|--output ] + [-p|--process-id ] + [--profile ] + [--providers ] + [-- ] (for target applications running .NET 10 or later) + [--show-child-io] + [--resume-runtime] + [--stopping-event-provider-name ] + [--stopping-event-event-name ] + [--stopping-event-payload-filter ] + [--tracepoint-configs ] + [--kernel-events ] +``` + +### Options + +`dotnet-trace collect-linux` supports all the same options as [`dotnet-trace collect`](#dotnet-trace-collect), excluding `--dsrouter`, and additionally offers: + +- **`--tracepoint-configs ` (required)** + + Defines the explicit mapping between EventPipe providers and kernel tracepoints. Each provider in `--providers` must have a corresponding entry in `--tracepoint-configs` + + **Format:** `ProviderName::` + + Where: + - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) + - `DefaultTracepointName`: Default tracepoint name for this provider (can be empty to require explicit assignment) + - `TracepointSets`: Semi-colon delimited `TracepointName=` + - `EventIds`: Plus-delimited event IDs to route to that tracepoint + + > [!NOTE] + > All tracepoint names are automatically prefixed with the provider name to avoid collisions. For example, `gc_events` for the `Microsoft-Windows-DotNETRuntime` provider becomes `Microsoft_Windows_DotNETRuntime_gc_events`. + + **Examples:** + ``` + # Scenario: All events from provider go to a default tracepoint + --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" + # All enabled events from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_dotnet_runtime + + # Scenario: Split events by categories + --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3;jit_events=10+11+12" + # EventIDs 1, 2, and 3 will be written to Microsoft_Windows_DotNETRuntime_gc_events + # EventIDs 10, 11, and 12 will be written to Microsoft_Windows_DotNETRuntime_jit_events + + # Multiple providers (comma-separated) + --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3,MyCustomProvider:custom_events" + # EventIds 1, 2, and 3 from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_gc_events + # All enabled events from MyCustomProvider will be written to MyCustomProvider_custom_events + ``` + +- **`--kernel-events ` (optional)** + + A comma-separated list of kernel event categories to include in the trace. These events are automatically grouped into kernel-named tracepoints. Available categories include: + + | Category | Description | Linux Tracepoints | + |----------|-------------|-------------------| + | `syscalls` | System call entry/exit events | `syscalls:sys_enter_*`, `syscalls:sys_exit_*` | + | `sched` | Process scheduling events | `sched:sched_switch`, `sched:sched_wakeup` | + | `net` | Network-related events | `net:netif_rx`, `net:net_dev_xmit` | + | `fs` | Filesystem I/O events | `ext4:*`, `vfs:*` | + | `mm` | Memory management events | `kmem:*`, `vmscan:*` | + + These events correspond to Linux kernel tracepoints documented in the [Linux kernel tracing documentation](https://www.kernel.org/doc/html/latest/trace/index.html). For more details on available tracepoints, see [ftrace](https://www.kernel.org/doc/html/latest/trace/ftrace.html) and [tracepoints](https://www.kernel.org/doc/html/latest/trace/tracepoints.html). + + Example: `--kernel-events syscalls,sched,net` + +### Linux Integration + +**Tracepoint Configuration Requirements:** + +- **Mandatory Mapping**: Every provider must be explicitly mapped to at least a default tracepoint and/or exclusive tracepoint sets via `--tracepoint-configs` +- **Tracepoint Isolation**: Each tracepoint can only receive events from one provider +- **Event Routing**: Different event IDs within a provider can be routed to different tracepoints for granular control +- **Automatic Prefixing**: All tracepoint names are prefixed with the provider name to avoid collisions + +**Kernel Integration Points:** + +The kernel tracepoints can be accessed through standard Linux tracing interfaces: + +- **ftrace**: `/sys/kernel/tracing/events/user_events/` +- **perf**: Use `perf list user_events*` to see available events +- **System monitoring tools**: Any tool that can consume Linux tracepoints + +### Examples + +```dotnetcli +# All runtime events to one tracepoint +dotnet-trace collect-linux --process-id 1234 \ + --providers Microsoft-Windows-DotNETRuntime:0x8000:5 \ + --kernel-events syscalls,sched \ + --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" + +# Split runtime events by category +dotnet-trace collect-linux --process-id 1234 \ + --providers Microsoft-Windows-DotNETRuntime:0x8001:5 \ + --kernel-events syscalls,sched,net,fs \ + --tracepoint-configs "Microsoft-Windows-DotNETRuntime::exception_events=80;gc_events=1+2" + +# Multiple providers +dotnet-trace collect-linux --process-id 1234 \ + --providers "Microsoft-Windows-DotNETRuntime:0x8001:5,MyCustomProvider:0xFFFFFFFF:5" \ + --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime,MyCustomProvider:custom_events" +``` + ## dotnet-trace convert Converts `nettrace` traces to alternate formats for use with alternate trace analysis tools. From e1a3ec705989ce75b903018b57d9096373cc18f2 Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Thu, 7 Aug 2025 17:13:53 -0400 Subject: [PATCH 3/5] [Diagnostics][dotnet-trace] Add event filter option --- docs/core/diagnostics/dotnet-trace.md | 33 +++++++++++++++++++++++++++ 1 file changed, 33 insertions(+) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index bba276360ff3e..616f3d7e01754 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -93,6 +93,7 @@ dotnet-trace collect [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] + [--event-filters ] ``` ### Options @@ -256,6 +257,34 @@ dotnet-trace collect A string, parsed as [payload_field_name]:[payload_field_value] pairs separated by commas, that will stop the trace upon hitting an event containing all specified payload pairs. Requires `--stopping-event-provider-name` and `--stopping-event-event-name` to be set. for example, `--stopping-event-provider-name Microsoft-Windows-DotNETRuntime --stopping-event-event-name Method/JittingStarted --stopping-event-payload-filter MethodNameSpace:Program,MethodName:OnButtonClick` to stop the trace upon the first `Method/JittingStarted` event for the method `OnButtonClick` in the `Program` namespace emitted by the `Microsoft-Windows-DotNETRuntime` event provider. +- **`--event-filters `** + + Defines an additional optional filter for each provider's events. When no `--event-filters` is specified for a provider, all events allowed by the provider's keywords and level configuration are collected. Event filters provide additional granular control beyond the keyword/level filtering. + + **Format:** `ProviderName::` + + Where: + - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) + - `Enable` : Boolean value indicating whether EventIds will be enabled or disabled, defaults to false + - `EventIds`: Plus-delimited event IDs to enable or disable, defaults to empty. + + **Examples:** + ``` + # Scenario: Disable specific events from Microsoft-Windows-DotNETRuntime + --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3+4+5+6+7+8+9" + + # Scenario: Enable specific events from a provider + --event-filters "Microsoft-Windows-DotNETRuntime:true:80+129+130+250" + # Only events 80, 129, 130, and 250 will be collected from this provider (others are filtered out) + + # Scenario: Multiple providers with mixed filtering - some providers have no filters + --providers "Microsoft-Windows-DotNETRuntime:0xFFFFFFFF:5,System.Threading.Tasks.TplEventSource:0xFFFFFFFF:5,MyCustomProvider:0xFFFFFFFF:5" + --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3,System.Threading.Tasks.TplEventSource:true:7+8+9" + # Microsoft-Windows-DotNETRuntime: All events EXCEPT 1,2,3 are collected + # System.Threading.Tasks.TplEventSource: ONLY events 7,8,9 are collected + # MyCustomProvider: ALL events are collected (no filter specified - follows provider keywords/level) + ``` + > [!NOTE] > - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace. @@ -300,6 +329,7 @@ dotnet-trace collect-linux [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] + [--event-filters ] [--tracepoint-configs ] [--kernel-events ] ``` @@ -323,6 +353,9 @@ dotnet-trace collect-linux > [!NOTE] > All tracepoint names are automatically prefixed with the provider name to avoid collisions. For example, `gc_events` for the `Microsoft-Windows-DotNETRuntime` provider becomes `Microsoft_Windows_DotNETRuntime_gc_events`. + > [!TIP] + > Use `--event-filters` to disable specific events before they are routed to tracepoints. Event filtering happens before tracepoint routing - only events that pass the filter will be sent to their assigned tracepoints. + **Examples:** ``` # Scenario: All events from provider go to a default tracepoint From 6c9fdb6cc0dd668905fcaacf0bf11e8159a89cfc Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Thu, 14 Aug 2025 12:18:10 -0400 Subject: [PATCH 4/5] Revert "[Diagnostics][dotnet-trace] Add event filter option" This reverts commit e1a3ec705989ce75b903018b57d9096373cc18f2. --- docs/core/diagnostics/dotnet-trace.md | 33 --------------------------- 1 file changed, 33 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index 616f3d7e01754..bba276360ff3e 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -93,7 +93,6 @@ dotnet-trace collect [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] - [--event-filters ] ``` ### Options @@ -257,34 +256,6 @@ dotnet-trace collect A string, parsed as [payload_field_name]:[payload_field_value] pairs separated by commas, that will stop the trace upon hitting an event containing all specified payload pairs. Requires `--stopping-event-provider-name` and `--stopping-event-event-name` to be set. for example, `--stopping-event-provider-name Microsoft-Windows-DotNETRuntime --stopping-event-event-name Method/JittingStarted --stopping-event-payload-filter MethodNameSpace:Program,MethodName:OnButtonClick` to stop the trace upon the first `Method/JittingStarted` event for the method `OnButtonClick` in the `Program` namespace emitted by the `Microsoft-Windows-DotNETRuntime` event provider. -- **`--event-filters `** - - Defines an additional optional filter for each provider's events. When no `--event-filters` is specified for a provider, all events allowed by the provider's keywords and level configuration are collected. Event filters provide additional granular control beyond the keyword/level filtering. - - **Format:** `ProviderName::` - - Where: - - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) - - `Enable` : Boolean value indicating whether EventIds will be enabled or disabled, defaults to false - - `EventIds`: Plus-delimited event IDs to enable or disable, defaults to empty. - - **Examples:** - ``` - # Scenario: Disable specific events from Microsoft-Windows-DotNETRuntime - --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3+4+5+6+7+8+9" - - # Scenario: Enable specific events from a provider - --event-filters "Microsoft-Windows-DotNETRuntime:true:80+129+130+250" - # Only events 80, 129, 130, and 250 will be collected from this provider (others are filtered out) - - # Scenario: Multiple providers with mixed filtering - some providers have no filters - --providers "Microsoft-Windows-DotNETRuntime:0xFFFFFFFF:5,System.Threading.Tasks.TplEventSource:0xFFFFFFFF:5,MyCustomProvider:0xFFFFFFFF:5" - --event-filters "Microsoft-Windows-DotNETRuntime:false:1+2+3,System.Threading.Tasks.TplEventSource:true:7+8+9" - # Microsoft-Windows-DotNETRuntime: All events EXCEPT 1,2,3 are collected - # System.Threading.Tasks.TplEventSource: ONLY events 7,8,9 are collected - # MyCustomProvider: ALL events are collected (no filter specified - follows provider keywords/level) - ``` - > [!NOTE] > - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace. @@ -329,7 +300,6 @@ dotnet-trace collect-linux [--stopping-event-provider-name ] [--stopping-event-event-name ] [--stopping-event-payload-filter ] - [--event-filters ] [--tracepoint-configs ] [--kernel-events ] ``` @@ -353,9 +323,6 @@ dotnet-trace collect-linux > [!NOTE] > All tracepoint names are automatically prefixed with the provider name to avoid collisions. For example, `gc_events` for the `Microsoft-Windows-DotNETRuntime` provider becomes `Microsoft_Windows_DotNETRuntime_gc_events`. - > [!TIP] - > Use `--event-filters` to disable specific events before they are routed to tracepoints. Event filtering happens before tracepoint routing - only events that pass the filter will be sent to their assigned tracepoints. - **Examples:** ``` # Scenario: All events from provider go to a default tracepoint From 728afed1becae448451124c3d0f14a27a288d29e Mon Sep 17 00:00:00 2001 From: Mitchell Hwang Date: Fri, 15 Aug 2025 15:18:39 -0400 Subject: [PATCH 5/5] Address Feedback --- docs/core/diagnostics/dotnet-trace.md | 235 +++++++++++------- .../eventsource-collect-and-view-traces.md | 2 +- 2 files changed, 148 insertions(+), 89 deletions(-) diff --git a/docs/core/diagnostics/dotnet-trace.md b/docs/core/diagnostics/dotnet-trace.md index bba276360ff3e..9ce9802b2b7f9 100644 --- a/docs/core/diagnostics/dotnet-trace.md +++ b/docs/core/diagnostics/dotnet-trace.md @@ -43,7 +43,8 @@ The `dotnet-trace` tool: * Is a cross-platform .NET Core tool. * Enables the collection of .NET Core traces of a running process without a native profiler. * Is built on [`EventPipe`](./eventpipe.md) of the .NET Core runtime. -* On Linux, provides additional integration with kernel user_events for native tracing tool compatibility. +* Delivers the same experience on Windows, Linux, and macOS. +* Offers an additional command on Linux to collect Linux perf events. ## Options @@ -126,7 +127,7 @@ dotnet-trace collect | ------------ | ------------------- | | `gc` | `0x1` | | `gchandle` | `0x2` | - | `fusion` | `0x4` | + | `assemblyloader` | `0x4` | | `loader` | `0x8` | | `jit` | `0x10` | | `ngen` | `0x20` | @@ -145,7 +146,7 @@ dotnet-trace collect | `gcheapdump` | `0x100000` | | `gcsampledobjectallocationhigh` | `0x200000` | | `gcheapsurvivalandmovement` | `0x400000` | - | `gcheapcollect` | `0x800000` | + | `managedheapcollect` | `0x800000` | | `gcheapandtypenames` | `0x1000000` | | `gcsampledobjectallocationlow` | `0x2000000` | | `perftrack` | `0x20000000` | @@ -159,7 +160,10 @@ dotnet-trace collect | `compilationdiagnostic` | `0x2000000000` | | `methoddiagnostic` | `0x4000000000` | | `typediagnostic` | `0x8000000000` | + | `jitinstrumentationdata` | `0x10000000000` | + | `profiler` | `0x20000000000` | | `waithandle` | `0x40000000000` | + | `allocationsampling` | `0x80000000000` | You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md). @@ -213,7 +217,9 @@ dotnet-trace collect | Profile | Description | |---------|-------------| - |`cpu-sampling`|Useful for tracking CPU usage and general .NET runtime information. This is the default option if no profile or providers are specified.| + |`dotnet-default`|Combination of `dotnet-thread-time` and `dotnet-clr-default`. This is the default profile used when no other provider/event/profile options are specified.| + |`dotnet-thread-time`|Useful for tracking .NET thread-time information. This is the default profile used when no other provider/event/profile options are specified.| + |`dotnet-clr-default`|Useful for tracking a multitude of .NET events.| |`gc-verbose`|Tracks GC collections and samples object allocations.| |`gc-collect`|Tracks GC collections only at very low overhead.| @@ -268,131 +274,184 @@ dotnet-trace collect ## dotnet-trace collect-linux -Collects diagnostic traces from .NET applications using Linux user_events as a transport layer. This command provides the same functionality as [`dotnet-trace collect`](#dotnet-trace-collect) but routes .NET runtime events through the Linux kernel's user_events subsystem before writing them to `.nettrace` files. +Collects diagnostic traces using perf_events, a Linux OS technology. `collect-linux` requires admin privileges to capture kernel- and user-mode events, and by default, captures events from all processes. -This transport approach enables automatic unification of user-space .NET events with kernel-space system events, since both are captured in the same kernel tracing infrastructure. Linux tools like `perf` and `ftrace` can monitor events in real-time while maintaining full compatibility with existing .NET profiling workflows. +This Linux-only command includes the same .NET events as [`dotnet-trace collect`](#dotnet-trace-collect), and it uses the kernel’s user_events mechanism to emit .NET events as perf events, enabling unification of user-space .NET events with kernel-space system events. ### Prerequisites - Linux kernel with `CONFIG_USER_EVENTS=y` support (kernel 6.4+) -- Appropriate permissions to access `/sys/kernel/tracing/user_events_data` +- Root permissions - .NET 10+ ### Synopsis ```dotnetcli dotnet-trace collect-linux - [--buffersize ] + [-h|--help] + + # Provider/Event Specification + [--providers ] [--clreventlevel ] [--clrevents ] + [--perf-event-tracepoints ] + [--profile ] + + # Trace Collection [--format ] - [-h|--help] + [-o|--output ] [--duration dd:hh:mm:ss] + + # .NET Process Target (Optional) [-n, --name ] - [--diagnostic-port] - [-o|--output ] [-p|--process-id ] - [--profile ] - [--providers ] - [-- ] (for target applications running .NET 10 or later) + [-- ] [--show-child-io] - [--resume-runtime] - [--stopping-event-provider-name ] - [--stopping-event-event-name ] - [--stopping-event-payload-filter ] - [--tracepoint-configs ] - [--kernel-events ] ``` ### Options -`dotnet-trace collect-linux` supports all the same options as [`dotnet-trace collect`](#dotnet-trace-collect), excluding `--dsrouter`, and additionally offers: +#### Provider/Event Specification Options -- **`--tracepoint-configs ` (required)** +- **`--providers `** - Defines the explicit mapping between EventPipe providers and kernel tracepoints. Each provider in `--providers` must have a corresponding entry in `--tracepoint-configs` + A comma-separated list of `EventPipe` providers to be enabled. These providers supplement any providers implied by `--profile `. If there's any inconsistency for a particular provider, this configuration takes precedence over the implicit configuration from the profile. - **Format:** `ProviderName::` + This list of providers is in the form: - Where: - - `ProviderName`: The EventPipe provider name (e.g., `Microsoft-Windows-DotNETRuntime`) - - `DefaultTracepointName`: Default tracepoint name for this provider (can be empty to require explicit assignment) - - `TracepointSets`: Semi-colon delimited `TracepointName=` - - `EventIds`: Plus-delimited event IDs to route to that tracepoint + - `Provider[,Provider]` + - `Provider` is in the form: `KnownProviderName[:Flags[:Level][:KeyValueArgs]]`. + - `KeyValueArgs` is in the form: `[key1=value1][;key2=value2]`. - > [!NOTE] - > All tracepoint names are automatically prefixed with the provider name to avoid collisions. For example, `gc_events` for the `Microsoft-Windows-DotNETRuntime` provider becomes `Microsoft_Windows_DotNETRuntime_gc_events`. + To learn more about some of the well-known providers in .NET, refer to [Well-known Event Providers](./well-known-event-providers.md). - **Examples:** - ``` - # Scenario: All events from provider go to a default tracepoint - --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" - # All enabled events from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_dotnet_runtime - - # Scenario: Split events by categories - --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3;jit_events=10+11+12" - # EventIDs 1, 2, and 3 will be written to Microsoft_Windows_DotNETRuntime_gc_events - # EventIDs 10, 11, and 12 will be written to Microsoft_Windows_DotNETRuntime_jit_events - - # Multiple providers (comma-separated) - --tracepoint-configs "Microsoft-Windows-DotNETRuntime::gc_events=1+2+3,MyCustomProvider:custom_events" - # EventIds 1, 2, and 3 from Microsoft-Windows-DotNETRuntime will be written to Microsoft_Windows_DotNETRuntime_gc_events - # All enabled events from MyCustomProvider will be written to MyCustomProvider_custom_events - ``` +- **`--clreventlevel `** -- **`--kernel-events ` (optional)** + Verbosity of CLR events to be emitted. + The following table shows the available event levels. - A comma-separated list of kernel event categories to include in the trace. These events are automatically grouped into kernel-named tracepoints. Available categories include: + | String value | Numeric value | + | --------------- | :-----------: | + | `logalways` | `0` | + | `critical` | `1` | + | `error` | `2` | + | `warning` | `3` | + | `informational` | `4` | + | `verbose` | `5` | - | Category | Description | Linux Tracepoints | - |----------|-------------|-------------------| - | `syscalls` | System call entry/exit events | `syscalls:sys_enter_*`, `syscalls:sys_exit_*` | - | `sched` | Process scheduling events | `sched:sched_switch`, `sched:sched_wakeup` | - | `net` | Network-related events | `net:netif_rx`, `net:net_dev_xmit` | - | `fs` | Filesystem I/O events | `ext4:*`, `vfs:*` | - | `mm` | Memory management events | `kmem:*`, `vmscan:*` | +- **`--clrevents `** - These events correspond to Linux kernel tracepoints documented in the [Linux kernel tracing documentation](https://www.kernel.org/doc/html/latest/trace/index.html). For more details on available tracepoints, see [ftrace](https://www.kernel.org/doc/html/latest/trace/ftrace.html) and [tracepoints](https://www.kernel.org/doc/html/latest/trace/tracepoints.html). + A list of CLR runtime provider keywords to enable separated by `+` signs. This is a simple mapping that lets you specify event keywords via string aliases rather than their hex values. For example, `dotnet-trace collect --providers Microsoft-Windows-DotNETRuntime:3:4` requests the same set of events as `dotnet-trace collect --clrevents gc+gchandle --clreventlevel informational`. The table below shows the list of available keywords: - Example: `--kernel-events syscalls,sched,net` + | Keyword String Alias | Keyword Hex Value | + | ------------ | ------------------- | + | `gc` | `0x1` | + | `gchandle` | `0x2` | + | `assemblyloader` | `0x4` | + | `loader` | `0x8` | + | `jit` | `0x10` | + | `ngen` | `0x20` | + | `startenumeration` | `0x40` | + | `endenumeration` | `0x80` | + | `security` | `0x400` | + | `appdomainresourcemanagement` | `0x800` | + | `jittracing` | `0x1000` | + | `interop` | `0x2000` | + | `contention` | `0x4000` | + | `exception` | `0x8000` | + | `threading` | `0x10000` | + | `jittedmethodiltonativemap` | `0x20000` | + | `overrideandsuppressngenevents` | `0x40000` | + | `type` | `0x80000` | + | `gcheapdump` | `0x100000` | + | `gcsampledobjectallocationhigh` | `0x200000` | + | `gcheapsurvivalandmovement` | `0x400000` | + | `managedheapcollect` | `0x800000` | + | `gcheapandtypenames` | `0x1000000` | + | `gcsampledobjectallocationlow` | `0x2000000` | + | `perftrack` | `0x20000000` | + | `stack` | `0x40000000` | + | `threadtransfer` | `0x80000000` | + | `debugger` | `0x100000000` | + | `monitoring` | `0x200000000` | + | `codesymbols` | `0x400000000` | + | `eventsource` | `0x800000000` | + | `compilation` | `0x1000000000` | + | `compilationdiagnostic` | `0x2000000000` | + | `methoddiagnostic` | `0x4000000000` | + | `typediagnostic` | `0x8000000000` | + | `jitinstrumentationdata` | `0x10000000000` | + | `profiler` | `0x20000000000` | + | `waithandle` | `0x40000000000` | + | `allocationsampling` | `0x80000000000` | -### Linux Integration + You can read about the CLR provider more in detail on the [.NET runtime provider reference documentation](../../fundamentals/diagnostics/runtime-events.md). -**Tracepoint Configuration Requirements:** +- **`--perf-event-tracepoints `** -- **Mandatory Mapping**: Every provider must be explicitly mapped to at least a default tracepoint and/or exclusive tracepoint sets via `--tracepoint-configs` -- **Tracepoint Isolation**: Each tracepoint can only receive events from one provider -- **Event Routing**: Different event IDs within a provider can be routed to different tracepoints for granular control -- **Automatic Prefixing**: All tracepoint names are prefixed with the provider name to avoid collisions + A comma-separated list of kernel perf event tracepoints to include in the trace. Available kernel events can be found under tracefs, which is typically mounted at `/sys/kernel/tracing`, through `available_events` for all available tracepoints or through the `events/` subdirectory for categorized events. -**Kernel Integration Points:** + Example: `--perf-event-tracepoints syscalls:sys_enter_execve,sched:sched_switch,sched:sched_wakeup` -The kernel tracepoints can be accessed through standard Linux tracing interfaces: +- **`--profile `** -- **ftrace**: `/sys/kernel/tracing/events/user_events/` -- **perf**: Use `perf list user_events*` to see available events -- **System monitoring tools**: Any tool that can consume Linux tracepoints + A named pre-defined set of provider configurations that allows common tracing scenarios to be specified succinctly. The following profiles are available: -### Examples + | Profile | Description | + |---------|-------------| + |`dotnet-default`|Combination of `dotnet-thread-time` and `dotnet-clr-default`. This is the default profile used when no other provider/event/profile options are specified.| + |`dotnet-thread-time`|Useful for tracking .NET thread-time information. This is the default profile used when no other provider/event/profile options are specified.| + |`dotnet-clr-default`|Useful for tracking a multitude of .NET events.| + |`gc-verbose`|Tracks GC collections and samples object allocations.| + |`gc-collect`|Tracks GC collections only at very low overhead.| + |`cpu-sampling`|Useful for tracking CPU usage.| -```dotnetcli -# All runtime events to one tracepoint -dotnet-trace collect-linux --process-id 1234 \ - --providers Microsoft-Windows-DotNETRuntime:0x8000:5 \ - --kernel-events syscalls,sched \ - --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime" - -# Split runtime events by category -dotnet-trace collect-linux --process-id 1234 \ - --providers Microsoft-Windows-DotNETRuntime:0x8001:5 \ - --kernel-events syscalls,sched,net,fs \ - --tracepoint-configs "Microsoft-Windows-DotNETRuntime::exception_events=80;gc_events=1+2" - -# Multiple providers -dotnet-trace collect-linux --process-id 1234 \ - --providers "Microsoft-Windows-DotNETRuntime:0x8001:5,MyCustomProvider:0xFFFFFFFF:5" \ - --tracepoint-configs "Microsoft-Windows-DotNETRuntime:dotnet_runtime,MyCustomProvider:custom_events" -``` +#### Trace Collection Options + +- **`--format {Chromium|NetTrace|Speedscope}`** + + Sets the output format for the trace file conversion. The default is `NetTrace`. + +- **`-o|--output `** + + The output path for the collected trace data. If not specified it defaults to `__.nettrace`, for example, `myapp_20210315_111514.nettrace``. + +- **`--duration `** + + The time for the trace to run. Use the `dd:hh:mm:ss` format. For example `00:00:00:05` will run it for 5 seconds. + +#### .NET Process Target Options + +> ![NOTE] +> The default behavior for collect-linux is to collect .NET events and perf event tracepoints from ALL discoverable processes. + +- **`-n, --name `** + + The name of the process to collect the trace from. + + > [!NOTE] + > By default, `/tmp` will be searched to discover .NET processes diagnostics ports. Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. + +- **`-p|--process-id `** + + The process ID to collect the trace from. + + > [!NOTE] + > By default, `/tmp` will be searched to discover .NET processes diagnostics ports. Should the target application have a `TMPDIR` enviornment variable set, `dotnet-trace` will need to share the same `TMPDIR` value, otherwise events will not be collected from the target application. + +- **`-- `** + + After the collection configuration parameters, the user can append `--` followed by a command to start a .NET application. This may be helpful when diagnosing issues that happen early in the process, such as startup performance issue or assembly loader and binder errors. + +- **`--show-child-io`** + + Only applicable when `-- ` is used. Shows the input and output streams of a launched child process in the current console. + +> [!NOTE] + +> - Stopping the trace may take a long time (up to minutes) for large applications. The runtime needs to send over the type cache for all managed code that was captured in the trace. + +> - To collect a trace using `dotnet-trace collect-linux`, it needs to be run with root permissions (`CAP_PERFMON`/`CAP_SYS_ADMIN`). Otherwise, the tool will fail to establish collect events. ## dotnet-trace convert diff --git a/docs/core/diagnostics/eventsource-collect-and-view-traces.md b/docs/core/diagnostics/eventsource-collect-and-view-traces.md index fd6b2a32d9dc6..14e7dcbc6975a 100644 --- a/docs/core/diagnostics/eventsource-collect-and-view-traces.md +++ b/docs/core/diagnostics/eventsource-collect-and-view-traces.md @@ -214,7 +214,7 @@ into other formats, such as Chromium or [Speedscope](https://www.speedscope.app/ Trace completed. ``` - dotnet-trace uses a comma-delimited variant of the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in + dotnet-trace uses the [conventional text format](#conventions-for-describing-provider-configuration) for describing provider configuration in the `--providers` argument. For more options on how to take traces using dotnet-trace, see the [dotnet-trace docs](./dotnet-trace.md#collect-a-trace-with-dotnet-trace).