Use ECS properties instead of container properties

[jorgee]

Closes #5096

Ports the @tom-seqera implementation in #5391 to AWS SDK v2 implementation.

With the RegisterJobDefinitionModel and ContainerPropertiesModel introduced by @pditommaso, these changes shouldn't affect the xpack as the interface it extends remains untouched.

[netlify]

✅ Deploy Preview for nextflow-docs-staging canceled.

Name	Link
🔨 Latest commit	5c68064
🔍 Latest deploy log	https://app.netlify.com/projects/nextflow-docs-staging/deploys/689de93ec57d31000843819a

[pditommaso]

Nice, let's test using [e2e prod] in the commit message

[pditommaso]

@jorgee aws tests are failing

[jorgee]

It seems the tests are failing because of the job definition reuse. The old definition is using containerProperties and the new one is trying to override with ecsProperties. If I am not wrong, the job definition is reused if container name and nf-token is the same, isn't it? So, I think we should include something else to the token to ensure both are using the same container or ecs properties.

[jorgee]

Token was basically hashing the result of containerProperties.toString. I have added a fake property ecsMode=true in the new version. If there is a job definition with same values but with the previous version, the token will be different and it will generate a new revision. Now, the job is failing because of CI user role permissions as in the other CI pipelines.

[pditommaso]

Launching some e2e tests

[pditommaso]

@jorgee would not this impact also the xpack plugin for amazon?

[jorgee]

@jorgee would not this impact also the xpack plugin for amazon?

The xpack is overriding the configJobDefRequest method that is not changed. The changes are internal in the RegisterJobDefinitionModel.toBatchRequest method where it sets the ecs properties. But let me do a test to see if I miss something.

[jorgee]

When testing with xpack I have got this error. Not sure yet if it is related to this change or happening also with the master.

java.lang.ClassCastException: class software.amazon.awssdk.services.batch.model.MountPoint cannot be cast to class software.amazon.awssdk.services.batch.model.MountPoint (software.amazon.awssdk.services.batch.model.MountPoint is in unnamed module of loader org.pf4j.PluginClassLoader @4ea17147; software.amazon.awssdk.services.batch.model.MountPoint is in unnamed module of loader org.pf4j.PluginClassLoader @9a20cbd)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1597)
        at software.amazon.awssdk.services.batch.model.MountPointsCopier.copy(MountPointsCopier.java:32)
        at software.amazon.awssdk.services.batch.model.ContainerProperties$BuilderImpl.mountPoints(ContainerProperties.java:2578)
        at nextflow.cloud.aws.batch.model.ContainerPropertiesModel.toBatchContainerProperties(ContainerPropertiesModel.groovy:254)

[jorgee]

It was an xpack gradle problem. Working after applying the fix.

[jorgee]

It seems that something is broken with the e2e test triggering condition. It has not triggered the e2e tests.

[jorgee]

The problem was the git commit message was returning a string with escapes due to %q option in
echo "commit_message=$(printf "%q" "$COMMIT_MESSAGE")" | head -n 1 >> $GITHUB_OUTPUT
The logs of the get_commit_message show the escaped string:

GitHub event=pull_request
Commit message=\[e2e\ prod\]\ Merge\ branch\ \'master\'\ into\ migrate-ecs-properties

I have changed %q to %s to generate a plain string.