Replies: 2 comments
-
|
Here's the output of the reproducer (which wouldn't fit in the original issue report). |
Beta Was this translation helpful? Give feedback.
0 replies
-
|
I'm not fond of middleware either but removing it is more or less impossible. SDK v2 was designed around middleware, for better or worse. Every new middleware we add to an operation is going to be an additional stack frame. We can't just get rid of it, it's part of the public API and plently of people depend on custom middlewares in the wild today. We could try to "squash" some middlewares in the same phase together but that's going to break people that are I'm not seeing how we could get rid of those frames from |
Beta Was this translation helpful? Give feedback.
0 replies
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
-
Acknowledgements
go get -u github.com/aws/aws-sdk-go-v2/...)Describe the bug
aws-sdk-go-v2 is built with many layers of middleware. The middleware is structured as nested function calls. By the time the SDK has an HTTP request ready to send over the network (
s3.GetObjectfor example, without extra configuration), it's added about 120 functions to the active call stack. The net/http package adds about 8 more.Go's built-in tools for profiling and execution tracing have limits on how much of the call stack they'll look at. One (the CPU profiler) looks at 64 frames, and the others (without extra configuration, where possible) will look at 128 frames.
When Go apps that use aws-sdk-go-v2 do things that warrant an entry in those profiles (use CPU, allocate memory, wait for data from the network, etc), the SDK itself often fills the entirety of the call stack with its own functions. That doesn't leave any available for describing which part of the SDK user's application is responsible for the cost.
Regression Issue
Expected Behavior
The SDK should be more frugal with the frames it puts on the function call stack. It should put fewer than 100 nested function calls on the stack. Better would be to get it below 50 (for the CPU profiler to work). It should not waste frames with
github.com/aws/smithy-go/middleware.decoratedSerializeHandler.HandleSerialize(and similar) calls.The entry point to the SDK (such as
github.com/aws/aws-sdk-go-v2/service/s3.(*Client).GetObject) should be visible on the call stack of the heap profile, goroutine profiles, execution tracer, etc.Multiple frames of the caller of the SDK should be visible with those tools.
Current Behavior
The call stacks in Go apps that use aws-sdk-go-v2 are so deep that Go's built-in observability/diagnostic tools have no choice but to truncate them (to around 128 frames). Below is a call stack from an execution trace from a real production app. It includes 116 visible frames, and the root-most visible frame is something inside the AWS SDK, rather than the name of a function that was passed to the
gokeyword. The application appears to be usinggithub.com/aws/aws-sdk-go-v2/service/[email protected],github.com/aws/[email protected], andgithub.com/aws/[email protected]Several of the function names are from the
github.com/aws/aws-sdk-go-v2/service/s3package, so it's clear enough which AWS service the app is using. One of the function names isgithub.com/aws/aws-sdk-go-v2/service/s3.(*awsRestxml_serializeOpGetObject).HandleSerialize, so with a keen eye it's possible to get the name of the method (GetObject) that the app is invoking on that service.But there's no context on which part of the app is responsible for calling into the AWS SDK.
Reproduction Steps
Possible Solution
smithy-go isn't very frugal with its call frames. In the reproducer's output, I see 12 instances of
github.com/aws/smithy-go/middleware.decoratedFinalizeHandler.HandleFinalize, 11 of...decoratedSerializeHandler.HandleSerialize, 9 of...decoratedDeserializeHandler.HandleDeserialize, 8 each of...decoratedInitializeHandler.HandleInitializeand...decoratedBuildHandler.HandleBuild, and 6 of...decoratedHandler.Handle. Eliminating those would save 54 frames, which could instead be used on putting the SDK's resource usage into context within customers' applications. That might be enough for everything but the CPU profiler (which has a limit of 64 frames).Additional Information/Context
Below is the output of the reproducer on my machine (to be added as a comment, it's too long to post in this box). The traceback I take by hand (with
runtime.Callersand a huge buffer) shows 132 return PCs and expands into 132 function calls. The manual traceback includesgithub.com/aws/aws-sdk-go-v2/service/s3.(*Client).GetObject(the entry point to the SDK), andmain.main(representing the point within the user app that called the SDK). The allocs/heap profile hasgithub.com/aws/aws-sdk-go-v2/service/s3.(*Client).GetObjectas its root-most frame; the application doesn't appear at all. The goroutine profile has a few more functions at the leaf (within the runtime's implementation), so it showsgithub.com/aws/smithy-go/middleware.(*InitializeStep).HandleMiddlewareas the root-most frame.The limit for most profiles is documented as
profstackdepthin https://pkg.go.dev/runtime . The limits for the execution tracer and CPU profiler are set at https://github.com/golang/go/blob/go1.24.3/src/internal/trace/event/go122/event.go#L509 and https://github.com/golang/go/blob/go1.24.3/src/runtime/cpuprof.go#L22AWS Go SDK V2 Module Versions Used
module example.com/call-stack-depth
go 1.24
require github.com/aws/aws-sdk-go-v2/service/s3 v1.79.4
require (
github.com/aws/aws-sdk-go-v2 v1.36.3 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.10 // indirect
github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.34 // indirect
github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.34 // indirect
github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.34 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.12.3 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.7.2 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.12.15 // indirect
github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.18.15 // indirect
github.com/aws/smithy-go v1.22.3 // indirect
)
Compiler and Version used
go version go1.24.3 darwin/arm64
Operating System and version
macOS 14.5 (23F79)
Beta Was this translation helpful? Give feedback.
All reactions