Upgrade to AWS Java SDK v2 #6165
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
✅ Deploy Preview for nextflow-docs-staging canceled.
|
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
Closed
Signed-off-by: jorgee <[email protected]>
bentsherman
changed the title
Signed-off-by: jorgee <[email protected]>
Signed-off-by: jorgee <[email protected]>
|
It is ready for review |
pditommaso
requested changes
Jun 18, 2025
plugins/nf-amazon/src/main/nextflow/cloud/aws/AwsClientFactory.groovy
Outdated
Show resolved
Hide resolved
plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3Client.java
Outdated
Show resolved
Hide resolved
plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3Client.java
Outdated
Show resolved
Hide resolved
plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3Client.java
Outdated
Show resolved
Hide resolved
plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3Client.java
Outdated
Show resolved
Hide resolved
plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3Client.java
Outdated
Show resolved
Hide resolved
plugins/nf-amazon/src/main/nextflow/cloud/aws/nio/S3Client.java
Outdated
Show resolved
Hide resolved
|
I have found an issue with multi-part uploads when uploading large files. I move to draft until I fix it. |
Signed-off-by: Ben Sherman <[email protected]>
bentsherman
reviewed
Jul 1, 2025
bentsherman
reviewed
Jul 1, 2025
Signed-off-by: Ben Sherman <[email protected]>
bentsherman
requested changes
Jul 1, 2025
There was a problem hiding this comment.
See my comments above.
I'm mainly confused about the use of the S3 transfer manager vs the old aws.client.upload* config options. The fact that the old options are still used by newOutputStream() , to a user this essentially means "sometimes we use the old options, sometimes we use the new options". It would be nice to deprecate these old options altogether.
It's also not clear to me how to select the async client vs sync client, as I don't see any explicit config option for this
Signed-off-by: jorgee <[email protected]>
pditommaso
reviewed
Jul 2, 2025
Signed-off-by: jorgee <[email protected]>
The sync client is used in all api call except the S3 transfer manager actions ( S3 copies, uploads and downloads). For transfer manager there is no option to use the sync client. I didn't changed all calls to async because it implied a lot of changes in the code |
|
I have added a subsection in the aws part of the documentation mentioning the most important changes in the v1 to v2 migration |
Signed-off-by: jorgee <[email protected]>
Signed-off-by: Ben Sherman <[email protected]>
bentsherman
approved these changes
Jul 3, 2025
Signed-off-by: Paolo Di Tommaso <[email protected]>
|
End to end tests fail badly. I've got this message unfortunately I'm not able to retried the full log |
The package is from v1. I am going to check if there is some place where I forgot someplace where the package is used |
|
I have realized the gradle tasks implementation inside nexflow that still uses v1. I am going to change it, but I think it shouldn't affect end-to-end tests. |
|
Yep, should be unrelated |
Signed-off-by: jorgee <[email protected]>
Signed-off-by: Paolo Di Tommaso <[email protected]>
pditommaso
force-pushed
the
aws-sdk-v2-fs-impl
branch
from
74f53d2 to
fd86ab4
Compare
|
I've realised some method like To address this, I've introduced a custom class to model the Job definition request . This allows to better bridge the new builder API model with the existing behaviour and minimise the changes. |
|
Merging this seems @jorgee let's discuss the plan for supporting the option |
|
I saw an integration test into another PR failing with this error Worth double checking |
|
It could be caused because in v2 content-length is now part of the request instead of metadata. The strange part is that it is not always happening. I am going to try to find when it is reproduced. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains the changes to port the Amazon plugin to AWS SDK version 2. Find below the most relevant changes:
S3Client.withForceGlobalBucketAcessEnabled(flag). In v2, it set the following flagsS3Client.Builder.crossRegionAccessEnabled(flag)andS3Configuration.multiRegionAccessEnabled(flag).-
AmazonS3Client.getS3AccountOwner()is not available in SDK v2. It was providing an ID used for checking the file access. In V2, the only way to retrieve the same ID is from a bucket owned by the user. To do it we need to list the buckets and get the owner field in the GetBucketACLResponse. If it is not possible to retrieve the ID because the user does not own any bucket, we perform the following fallback. In the case of READ access, it tries to retrieve the head of the object, It will fail if there isn't read access. In the case of writting, a warning is printed. It is the same as AWS NIO is doing to check the file access.The
setEndpointandsetRegionmethods in the S3Client wrapper are removed as it is not available in the v2 clients. They were only used in tests.CannedAccessControlListis split in two classes one for objects and another for buckets. In most of the code it has been substituted by ObjectCannedACL.ContentTypeandContentLengthare part of the request instead of theObjectMetadata, and they can be obtained invoking the S3client.headObject method in the SDK v2S3ClientConfigurationdoesn't exist in SDK v2. Two new classes have been created to emulate the same behaviour. They convert the properties to the SDK v2 sync and async client configurations.SsoCredentialsProviderV1class is not needed anymore as SDK v2 already manages the SSO credentials. The custom provider chain created in theS3FileSystemProvider.getCredetialsProvider0to include theSsoCredentialsProviderV1ihas been replace by theDefaultCredentialProviderin v2.Credentials and config are automatically merged by SDK v2. No option for NXF_DISABLE_AWS_CONFIG_MERGE.
In V2, clients and requests are immutable and must be generated with a builder class. Some helper methods have been modified to pass builder classes instead of requests, such as
makeJobDefRequest,configJobRefRequest,addVolumeMountsToContainer, etc.S3 Parallel Download was deprecated and S3CopyStream was not used. They have been removed.
In v1, the upload directory was performed by walking through the different directory files and uploading them one by one. In v2, it has been substituted by the uploadDirectory method in the SDK.