Skip to content

Run native tests without running tests on JVM first #765

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Run native tests without running tests on JVM first #765

wants to merge 2 commits into from

Conversation

dnestoro
Copy link
Collaborator

@dnestoro dnestoro commented Aug 22, 2025 •
edited
Loading

Problem: At the moment, every time we run native tests (with both Maven or Gradle), we must run tests on JVM first, in order to collect test-ids required for native run. With more complex tests, this additional step increases time required for tests execution.

Idea: We should use JUnit's dry-run mode to just collect test-ids without executing tests on JVM.

Solution: we can set junit.platform.execution.dryRun.enabled system property to true when executing tests on JVM (maven change, gradle change).

NOTE: Since this approach changes the default behavior, we should bump major version for the next release.

@dnestoro dnestoro self-assigned this Aug 22, 2025
@oracle-contributor-agreement oracle-contributor-agreement bot added the OCA Verified All contributors have signed the Oracle Contributor Agreement. label Aug 22, 2025
@dnestoro dnestoro linked an issue Aug 22, 2025 that may be closed by this pull request
The native JUnit tests cannot be executed without the prior run of JVM tests #764
Open
@dnestoro dnestoro changed the title Run native tests without running tests on JVM before Run native tests without running tests on JVM first Aug 22, 2025
@melix melix mentioned this pull request Aug 22, 2025
Open
melix
melix requested changes Aug 26, 2025
View reviewed changes
@dnestoro dnestoro force-pushed the dnestoro/run-native-tests-without-previous-jvm-run branch from 8dfa083 to 05b4dbd Compare August 28, 2025 13:24
@dnestoro dnestoro force-pushed the dnestoro/run-native-tests-without-previous-jvm-run branch from 05b4dbd to 23494be Compare August 28, 2025 13:33
dnestoro
dnestoro commented Aug 29, 2025
View reviewed changes
native-maven-plugin/src/main/java/org/graalvm/buildtools/utils/AgentUtils.java
public abstract class AgentUtils {

private static final String STANDARD = "standard";
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes in this file are not related to the original purpose of this PR. This is just a small clean-up triggered by some changes in Utils.java class.

@dnestoro dnestoro marked this pull request as ready for review August 29, 2025 11:11
@dnestoro dnestoro requested review from melix, vjovanov and olpaw August 29, 2025 11:11
@dnestoro
Copy link
Collaborator Author

Currently, the tests are failing because there are different number of discovered tests with and without junit.platform.execution.dryRun.enabled while we expect the same number of tests in both cases.

This only happens if the tests are annotated with annotations like @ParameterizedTest or @RepeatedTest.
Test ids discovered without dry-run:

[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testEvery(tests.VintageTests)]
[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testExpectedExceptionCause(tests.VintageTests)]
[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testExpectedException(tests.VintageTests)]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:accessMethodReflectively()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:resourceTest()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:accessFiledReflectively()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:callMethodFromOtherClass()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:repeatedTest(org.junit.jupiter.api.RepetitionInfo)]/[test-template-invocation:#1]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:repeatedTest(org.junit.jupiter.api.RepetitionInfo)]/[test-template-invocation:#2]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:repeatedTest(org.junit.jupiter.api.RepetitionInfo)]/[test-template-invocation:#3]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:testWithExternalFieldSource(java.lang.String)]/[test-template-invocation:#1]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:testWithExternalFieldSource(java.lang.String)]/[test-template-invocation:#2]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:testWithExternalFieldSource(java.lang.String)]/[test-template-invocation:#3]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:test(java.lang.String)]/[test-template-invocation:#1]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:test(java.lang.String)]/[test-template-invocation:#2]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:skipThisTest()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:singleFieldSource(java.lang.String)]/[test-template-invocation:#1]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[test-template:singleFieldSource(java.lang.String)]/[test-template-invocation:#2]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest1()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest2()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest3()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:firstTest()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:secondTest()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:thirdTest()]

Test ids discovered with dry-run:

[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testEvery(tests.VintageTests)]
[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testExpectedExceptionCause(tests.VintageTests)]
[engine:junit-vintage]/[runner:tests.VintageTests]/[test:testExpectedException(tests.VintageTests)]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:accessMethodReflectively()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:resourceTest()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:accessFiledReflectively()]
[engine:junit-jupiter]/[class:tests.ComplexTest]/[method:callMethodFromOtherClass()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:skipThisTest()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest1()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest2()]
[engine:junit-jupiter]/[class:tests.JUnitAnnotationsTests]/[method:beforeAndAfterEachTest3()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:firstTest()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:secondTest()]
[engine:junit-jupiter]/[class:tests.OrderTests]/[method:thirdTest()]

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@melix melix Awaiting requested review from melix

@vjovanov vjovanov Awaiting requested review from vjovanov

@olpaw olpaw Awaiting requested review from olpaw

Requested changes must be addressed to merge this pull request.

Assignees

@dnestoro dnestoro

Labels
OCA Verified All contributors have signed the Oracle Contributor Agreement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

The native JUnit tests cannot be executed without the prior run of JVM tests
2 participants
@dnestoro @melix