Upgrade to AWS Java SDK v2

docs/conf.py

@@ -50,7 +50,8 @@
     'mail.md': 'notifications.md',
     'operator.md': 'reference/operator.md',
     'dsl1.md': 'migrations/dsl1.md',
-    'updating-syntax.md': 'strict-syntax.md'
+    'updating-syntax.md': 'strict-syntax.md',
+    'updating-spot-retries.md': 'guides/updating-spot-retries.md'
 }
 
 # Add any paths that contain templates here, relative to this directory.

docs/guides/aws-java-sdk-v2.md

@@ -0,0 +1,34 @@
+(aws-java-sdk-v2-page)=
+
+# AWS Java SDK v2
+
+AWS Java SDK v1 is reaching end of life at the end of 2025. Starting in version `25.06.0-edge`, Nextflow uses AWS Java SDK v2 in the `nf-amazon` plugin.
+
+This migration introduced several breaking changes to the `aws.client` config scope, including new options and removed options. This page describes these changes and how they affect your Nextflow configuraiton.
+
+## New HTTP client
+
+The HTTP client used by SDK v2 does not support overriding certain advanced HTTP options. As a result, the following config options are no longer supported:
+
+- `aws.client.protocol`
+- `aws.client.signerOverride`
+- `aws.client.socketRecvBufferSizeHint`
+- `aws.client.socketSendBufferSizeHint`
+- `aws.client.userAgent`
+
+## S3 transfer manager
+
+The *S3 transfer manager* is a subsystem of SDK v2 which handles S3 transfers, including S3 uploads and downloads.
+
+The concurrency and throughput of the S3 transfer manager can be configured manually using the `aws.client.maxConcurrency` and `aws.client.maxNativeMemory` config options. Alternatively, the `aws.client.targetThroughputInGbps` config option can be used to set the previous two options automatically based on a target throughput.
+
+## Multi-part uplaods
+
+Multi-part uploads are handled by the S3 transfer manager. The `aws.client.minimumPartSize` and `aws.client.multipartThreshold` config options can be used to control when and how multi-part uploads are performed.
+
+The following multi-part upload options are no longer supported:
+
+- `aws.client.uploadChunkSize`
+- `aws.client.uploadMaxAttempts`
+- `aws.client.uploadMaxThreads`
+- `aws.client.uploadRetrySleep`

docs/index.md

@@ -158,11 +158,19 @@ developer/plugins
 
 ```{toctree}
 :hidden:
-:caption: Guides
+:caption: Tutorials
 :maxdepth: 1
 
 data-lineage
-updating-spot-retries
 metrics
 flux
 ```
+
+```{toctree}
+:hidden:
+:caption: Guides
+:maxdepth: 1
+
+guides/aws-java-sdk-v2
+guides/updating-spot-retries
+```

docs/migrations/25-10.md

@@ -0,0 +1,13 @@
+(migrating-25-10-page)=
+
+# Migrating to 25.10 (preview)
+
+This page summarizes the upcoming changes in Nextflow 25.10, which will be released in October 2025.
+
+:::{note}
+This page is a work in progress and will be updated as features are finalized. It should not be considered complete until the 25.10 release.
+:::
+
+## Breaking changes
+
+- The AWS Java SDK used by Nextflow was upgraded from v1 to v2, which introduced some breaking changes to the `aws.client` config options. See {ref}`the guide <aws-java-sdk-v2-page>` for details.

docs/reference/config.md

@@ -219,13 +219,36 @@ The following settings are available:
   :::
 : The retrieval tier to use when restoring objects from Glacier, one of [`Expedited`, `Standard`, `Bulk`].
 
+`aws.client.maxConcurrency`
+: :::{versionadded} 25.06.0-edge
+  :::
+: The maximum number of concurrent S3 transfers used by the S3 transfer manager. By default, this setting is determined by `aws.client.targetThroughputInGbps`. Modifying this value can affect the amount of memory used for S3 transfers.
+
 `aws.client.maxConnections`
-: The maximum number of allowed open HTTP connections (default: `50`).
+: The maximum number of open HTTP connections used by the S3 transfer manager (default: `50`).
 
 `aws.client.maxErrorRetry`
 : The maximum number of retry attempts for failed retryable requests (default: `-1`).
 
+`aws.client.maxNativeMemory`
+: :::{versionadded} 25.06.0-edge
+  :::
+: The maximum native memory used by the S3 transfer manager. By default, this setting is determined by `aws.client.targetThroughputInGbps`.
+
+`aws.client.minimumPartSize`
+: :::{versionadded} 25.06.0-edge
+  :::
+: The minimum part size used by the S3 transfer manager for multi-part uploads (default: `8 MB`).
+
+`aws.client.multipartThreshold`
+: :::{versionadded} 25.06.0-edge
+  :::
+: The object size threshold used by the S3 transfer manager for performing multi-part uploads (default: same as `aws.cllient.minimumPartSize`).
+
 `aws.client.protocol`
+: :::{deprecated} 25.06.0-edge
+  This option is no longer supported.
+  :::
 : The protocol to use when connecting to AWS. Can be `http` or `https` (default: `'https'`).
 
 `aws.client.proxyHost`
@@ -234,6 +257,11 @@ The following settings are available:
 `aws.client.proxyPort`
 : The port to use when connecting through a proxy.
 
+`aws.client.proxyScheme`
+: :::{versionadded} 25.06.0-edge
+  :::
+: The protocol scheme to use when connecting through a proxy. Can be `http` or `https` (default: `'http'`).
+
 `aws.client.proxyUsername`
 : The user name to use when connecting through a proxy.
 
@@ -243,18 +271,27 @@ The following settings are available:
 `aws.client.requesterPays`
 : :::{versionadded} 24.05.0-edge
   :::
-: Use [Rrequester Pays](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html) for S3 buckets (default: `false`).
+: Use [Requester Pays](https://docs.aws.amazon.com/AmazonS3/latest/userguide/RequesterPaysBuckets.html) for S3 buckets (default: `false`).
 
 `aws.client.s3PathStyleAccess`
 : Use the path-based access model to access objects in S3-compatible storage systems (default: `false`).
 
 `aws.client.signerOverride`
+: :::{deprecated} 25.06.0-edge
+  This option is no longer supported.
+  :::
 : The name of the signature algorithm to use for signing requests made by the client.
 
 `aws.client.socketSendBufferSizeHint`
+: :::{deprecated} 25.06.0-edge
+  This option is no longer supported.
+  :::
 : The Size hint (in bytes) for the low level TCP send buffer (default: `0`).
 
 `aws.client.socketRecvBufferSizeHint`
+: :::{deprecated} 25.06.0-edge
+  This option is no longer supported.
+  :::
 : The Size hint (in bytes) for the low level TCP receive buffer (default: `0`).
 
 `aws.client.socketTimeout`
@@ -268,19 +305,44 @@ The following settings are available:
   :::
 : The AWS KMS key Id to be used to encrypt files stored in the target S3 bucket.
 
+`aws.client.targetThroughputInGbps`
+: :::{versionadded} 25.06.0-edge
+  :::
+: The target network throughput (in Gbps) used by the S3 transfer manager (default: `10`). This setting is not used when `aws.client.maxConcurrency` and `aws.client.maxNativeMemory` are specified.
+
+`aws.client.transferManagerThreads`
+: :::{versionadded} 25.06.0-edge
+  :::
+: Number of threads used by the S3 transfer manager (default `10`).
+
 `aws.client.userAgent`
+: :::{deprecated} 25.06.0-edge
+  This option is no longer supported.
+  :::
 : The HTTP user agent header passed with all HTTP requests.
 
 `aws.client.uploadChunkSize`
+: :::{deprecated} 25.06.0-edge
+  This option is no longer supported.
+  :::
 : The size of a single part in a multipart upload (default: `100 MB`).
 
 `aws.client.uploadMaxAttempts`
+: :::{deprecated} 25.06.0-edge
+  This option is no longer supported.
+  :::
 : The maximum number of upload attempts after which a multipart upload returns an error (default: `5`).
 
 `aws.client.uploadMaxThreads`
+: :::{deprecated} 25.06.0-edge
+  This option is no longer supported.
+  :::
 : The maximum number of threads used for multipart upload (default: `10`).
 
 `aws.client.uploadRetrySleep`
+: :::{deprecated} 25.06.0-edge
+  This option is no longer supported.
+  :::
 : The time to wait after a failed upload attempt to retry the part upload (default: `500ms`).
 
 `aws.client.uploadStorageClass`

modules/nextflow/src/main/groovy/nextflow/util/LoggerHelper.groovy

@@ -217,7 +217,7 @@ class LoggerHelper {
         final logger = createLogger(MAIN_PACKAGE, mainLevel)
 
         // -- set AWS lib level to WARN to reduce noise in the log file
-        final AWS = 'com.amazonaws'
+        final AWS = 'software.amazon'
         if( !debugConf.contains(AWS) && !traceConf.contains(AWS)) {
             createLogger(AWS, Level.WARN)
         }

modules/nf-commons/src/main/nextflow/file/FileHelper.groovy

@@ -880,7 +880,9 @@ class FileHelper {
 
             @Override
             FileVisitResult visitFile(Path fullPath, BasicFileAttributes attrs) throws IOException {
-                final path = folder.relativize(fullPath)
+                final path = fullPath.isAbsolute()
+                    ? folder.relativize(fullPath)
+                    : fullPath
                 log.trace "visitFiles > file=$path; includeFile=$includeFile; matches=${matcher.matches(path)}; isRegularFile=${attrs.isRegularFile()}"
 
                 if (includeFile && matcher.matches(path) && (attrs.isRegularFile() || (options.followLinks == false && attrs.isSymbolicLink())) && (includeHidden || !isHidden(fullPath))) {
@@ -912,7 +914,9 @@ class FileHelper {
     }
 
     static protected Path relativize0(Path folder, Path fullPath) {
-        def result = folder.relativize(fullPath)
+        final result = fullPath.isAbsolute()
+            ? folder.relativize(fullPath)
+            : fullPath
         String str
         if( folder.is(FileSystems.default) || !(str=result.toString()).endsWith('/') )
             return result

modules/nf-lang/src/main/java/nextflow/config/scopes/AwsClientConfig.java

@@ -49,6 +49,12 @@ The amount of time to wait (in milliseconds) when initially establishing a conne
     """)
     public String endpoint;
 
+    @ConfigOption
+    @Description("""
+        The maximum number of concurrency in S3 async clients.
+    """)
+    public int maxConcurrency;
+
     @ConfigOption
     @Description("""
         The maximum number of allowed open HTTP connections.
@@ -63,63 +69,63 @@ The amount of time to wait (in milliseconds) when initially establishing a conne
 
     @ConfigOption
     @Description("""
-        The protocol (i.e. HTTP or HTTPS) to use when connecting to AWS.
+        The maximum native memory used by the S3 asynchronous client for S3 transfers.
     """)
-    public String protocol;
+    public MemoryUnit maxNativeMemory;
 
     @ConfigOption
     @Description("""
-        The proxy host to connect through.
+        The minimum size of a single part in a multipart upload (default: `8 MB`).
     """)
-    public String proxyHost;
+    public MemoryUnit minimumPartSize;
 
     @ConfigOption
     @Description("""
-        The port on the proxy host to connect through.
+        The S3 Async client threshold to create multipart S3 transfers. Default is the same as `minimumPartSize`.
     """)
-    public int proxyPort;
+    public MemoryUnit multipartThreshold;
 
     @ConfigOption
     @Description("""
-        The user name to use when connecting through a proxy.
+        The proxy host to connect through.
     """)
-    public String proxyUsername;
+    public String proxyHost;
 
     @ConfigOption
     @Description("""
-        The password to use when connecting through a proxy.
+        The port on the proxy host to connect through.
     """)
-    public String proxyPassword;
+    public int proxyPort;
 
     @ConfigOption
     @Description("""
-        Enable the requester pays feature for S3 buckets.
+        The protocol scheme to use when connecting through a proxy (http/https).
     """)
-    public boolean requesterPays;
+    public String proxyScheme;
 
     @ConfigOption
     @Description("""
-        Enable the use of path-based access model that is used to specify the address of an object in S3-compatible storage systems.
+        The user name to use when connecting through a proxy.
     """)
-    public boolean s3PathStyleAccess;
+    public String proxyUsername;
 
     @ConfigOption
     @Description("""
-        The name of the signature algorithm to use for signing requests made by the client.
+        The password to use when connecting through a proxy.
     """)
-    public String signerOverride;
+    public String proxyPassword;
 
     @ConfigOption
     @Description("""
-        The size hint (in bytes) for the low level TCP send buffer.
+        Enable the requester pays feature for S3 buckets.
     """)
-    public int socketSendBufferSizeHint;
+    public boolean requesterPays;
 
     @ConfigOption
     @Description("""
-        The size hint (in bytes) for the low level TCP receive buffer.
+        Enable the use of path-based access model that is used to specify the address of an object in S3-compatible storage systems.
     """)
-    public int socketRecvBufferSizeHint;
+    public boolean s3PathStyleAccess;
 
     @ConfigOption
     @Description("""
@@ -141,33 +147,15 @@ The amount of time to wait (in milliseconds) for data to be transferred over an
 
     @ConfigOption
     @Description("""
-        The HTTP user agent header passed with all HTTP requests.
-    """)
-    public String userAgent;
-
-    @ConfigOption
-    @Description("""
-        The size of a single part in a multipart upload (default: `100 MB`).
-    """)
-    public MemoryUnit uploadChunkSize;
-
-    @ConfigOption
-    @Description("""
-        The maximum number of upload attempts after which a multipart upload returns an error (default: `5`).
-    """)
-    public int uploadMaxAttempts;
-
-    @ConfigOption
-    @Description("""
-        The maximum number of threads used for multipart upload.
+        The S3 Async client target network throughput in Gbps. This value is used to automatically set `maxConcurrency` and `maxNativeMemory` (default: `10`).
     """)
-    public int uploadMaxThreads;
+    public Double targetThroughputInGbps;
 
     @ConfigOption
     @Description("""
-        The time to wait after a failed upload attempt to retry the part upload (default: `500ms`).
+        The number of threads used by the S3 transfer manager (default: `10`).
     """)
-    public Duration uploadRetrySleep;
+    public int transferManagerThreads;
 
     @ConfigOption
     @Description("""