MaxMemoryPreload does not seem to be part of PR test baseline

[makortel]

While looking into the memory improvement of #48026, I noticed the max memory report being empty

https://cmssdt.cern.ch/SDT/jenkins-artifacts/baseLineComparisons/CMSSW_15_1_X_2025-05-06-2300+b04de8/68319/maxmem-comparison/

The PR test log (e.g. https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b04de8/45898/runTheMatrix-results/145.301_RunDisplacedJet2024E/step3_RunDisplacedJet2024E.log) but the baseline test log (https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-baseline-tests/CMSSW_15_1_X_2025-05-06-2300/el8_amd64_gcc12/AuthenticAMD/matrix-results/145.301_RunDisplacedJet2024E/step3_RunDisplacedJet2024E.log) does not.

Did the runTheMatrix option to run the MaxMemoryPreload got dropped from PR test baseline?

[makortel]

assign core

[makortel]

FYI @gartung

[cmsbuild]

New categories assigned: core

@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @makortel.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[gartung]

Could be related to changes made in this cms-bot PR 2458
or this change

[gartung]

cms-sw/cms-bot#2494

[gartung]

https://cmssdt.cern.ch/jenkins/job/ib-run-baseline/15890/consoleText

++ runTheMatrix.py -j 16 --job-reports -l 2500.201,250202.181,29834.999,29700.0,29696.0,29634.911,29634.75,29634.0,24834.911,17034.0,16834.0,14234.0,14034.0,13234.0,13034.0,12846.0,12834.7,12834.0,12434.0,11634.0,10224.0,1001.0,1000.0,25202.0,312.0,101.0,1330.0,1306.0,145.713,145.604,145.5,145.408,145.301,145.202,145.104,145.014,141.042,140.56,140.045,139.001,136.874,136.793,136.731,135.4,25.0,9.0,8.0,7.3,5.1,4.53,4.22 --command ' --maxmem_profile --customise Validation/Performance/TimeMemoryJobReport.customiseWithTimeMemoryJobReport '

Appears that last part of command wasn't propagated.

[gartung]

The command was customized without --maxmem_profile
https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-baseline-tests/CMSSW_15_1_X_2025-05-06-2300/el8_amd64_gcc12/AuthenticAMD/matrix-results/matrixTests.15879.log

+ runTheMatrix.py -j 16 --job-reports -l 2500.201,250202.181,29834.999,29700.0,29696.0,29634.911,29634.75,29634.0,24834.911,17034.0,16834.0,14234.0,14034.0,13234.0,13034.0,12846.0,12834.0,12434.0,11634.0,10224.0,1001.0,1000.0,25202.0,312.0,101.0,1330.0,1306.0,145.713,145.604,145.5,145.408,145.301,145.202,145.104,145.014,141.042,140.56,140.045,139.001,136.874,136.793,136.731,135.4,25.0,9.0,8.0,7.3,5.1,4.53,4.22 --command ' --customise Validation/Performance/TimeMemoryJobReport.customiseWithTimeMemoryJobReport '

[iarspider]

The command was customized without --maxmem_profile https://cmssdt.cern.ch/SDT/jenkins-artifacts/ib-baseline-tests/CMSSW_15_1_X_2025-05-06-2300/el8_amd64_gcc12/AuthenticAMD/matrix-results/matrixTests.15879.log

For that job, PRODUCTION_RELEASE (passed from ib-run-pr-tests) was false, which disabled checking for and enabling maxmem_profiling: https://github.com/cms-sw/cms-bot/blob/master/run-ib-pr-matrix.sh#L54 . But in ib-run-pr-tests PRODUCTION_RELEASE was correctly set to true.

[gartung]

What is setting PRODUCTION_RELEASE? For the same IB release it is set to true or false. Is it dependent on the workflows being used?

[iarspider]

What is setting PRODUCTION_RELEASE? For the same IB release it is set to true or false. Is it dependent on the workflows being used?

It defaults to false, and is set to true here: https://github.com/cms-sw/cms-bot/blob/641472a546ef87f50598bf129dba17cdffa34a95/pr_testing/test_multiple_prs.sh#L158-L160

[gartung]

It should default true unless ROCM is involved. The baseline memory profile is needed for comparison.