[ARM] WFs 145.X fail with Segmentation violation in XercesC

[iarspider]

We are observing occasional RelVal failures in WFs 145.x due to SIGSEGVs in XercesC code. Some examples: 1, 2, 3, 4.

The exact failure position in XercesC code differs from workflow to workflow, but in all cases XercesC code is called from XMLConfigReader::readPatterns(...)

[cmsbuild]

cms-bot internal usage

[cmsbuild]

A new Issue was created by @iarspider.

@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.

cms-bot commands are listed here

[iarspider]

assign L1Trigger/L1TMuonOverlap
@makortel FYI

[cmsbuild]

New categories assigned: l1

@BenjaminRS,@quinnanm you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

E.g. in DetectorDescription packages we call these functions

cmssw/Utilities/Xerces/src/Xerces.cc

Lines 8 to 28 in 1d7a156

	namespace cms {
	namespace concurrency {
	namespace {
	std::mutex g_xerces_mutex;
	}
	// We need to make sure these serialized are only called by one
	// thread at the time. Until the first Init succeeds the other threads
	// should not be allowed to proceed. We also do not want two different
	// threads calling Init and Finalize at the same time. We therefore simply
	// use a global mutex to serialize everything.
	void xercesInitialize() {
	std::unique_lock<std::mutex> l(g_xerces_mutex);
	XMLPlatformUtils::Initialize();
	}
	void xercesTerminate() {
	std::unique_lock<std::mutex> l(g_xerces_mutex);
	XMLPlatformUtils::Terminate();
	}
	} // namespace concurrency
	} // namespace cms

I don't see Utilities/Xerces being used in L1Trigger/L1TMuonOverlap package.

[dan131riley]

I don't see Utilities/Xerces being used in L1Trigger/L1TMuonOverlap package.

That's correct, it is calling XMLPlatformUtils::Initialize() directly, and that is almost certainly the cause of the crashes. Those calls need to be replaced with the cms::concurrency calls that wrap those with a mutex. These other places may have the same issue, depending on how they are used:

CondFormats/PPSObjects/src/CTPPSRPAlignmentCorrectionsMethods.cc
L1TriggerConfig/L1GtConfigProducers/src/L1GtTriggerMenuXmlParser.cc
L1Trigger/L1TCommon/src/XmlConfigParser.cc
L1Trigger/L1TMuonOverlapPhase1/src/Omtf/XMLConfigReader.cc
L1Trigger/L1TMuonOverlapPhase1/src/Omtf/XMLConfigWriter.cc
L1Trigger/L1TMuonOverlap/src/XMLConfigReader.cc
L1Trigger/L1TMuonOverlap/src/XMLConfigWriter.cc

I was working on opening an issue, but got distracted and @iarspider beat me to it.

It's also worth noting that these were easy to spot in the ROOT6 builds. They do happen with other release series, but they get hidden by all the expression parser segfaults unless you know where to look.

[makortel]

assign CondFormats/PPSObjects

[cmsbuild]

New categories assigned: alca

@atpathak,@perrotta you have been requested to review this Pull request/Issue and eventually sign? Thanks

[makortel]

One could also ask how much these codes really need to use xerces, or could they be migrated to use e.g. tinyxml2 (but that would, of course, need more effort)

[BenjaminRS]

One could also ask how much these codes really need to use xerces, or could they be migrated to use e.g. tinyxml2 (but that would, of course, need more effort)

We can follow up with the OMTF experts