Upgrade EvtGen to v03.00.00 by kawaho · Pull Request #10580 · cms-sw/cmsdist

kawaho · 2026-05-25T15:44:08Z

This commits upgrade EvtGen to v03.00.00, which

is now thread-safe, allowing multithreading in production
adds new final state photon radiation models in Sherpa/Photon++ and Vincia
includes new physics models and decay table implemented by Belle 2

cmsbuild · 2026-05-25T15:44:31Z

A new Pull Request was created by @kawaho for branch IB/CMSSW_17_0_X/master.

@akritkbehera, @cmsbuild, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

cmsbuild · 2026-05-25T15:44:31Z

cms-bot internal usage

akritkbehera · 2026-05-25T22:01:03Z

please test with cms-sw/cmssw#51029

cmsbuild · 2026-05-25T23:31:03Z

-1

Failed Tests: UnitTests RelVals AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53469/summary.html
COMMIT: 8089a4f
CMSSW: CMSSW_17_0_X_2026-05-25-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10580/53469/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 10 errors in the following unit tests:

---> test GeneratorInterfaceEvtGenInterfaceTest had ERRORS
---> test TestGeneratorInterfaceEvtGenInterface_bplus had ERRORS
---> test test_MC_22_crosscheck had ERRORS
and more ...

Failed RelVals

25202.0A fatal system signal has occurred: segmentation violation
25.0A fatal system signal has occurred: segmentation violation
1306.0A fatal system signal has occurred: segmentation violation

Expand to see more relval errors ...

Failed AddOn Tests

hlt_mc_Fake

A fatal system signal has occurred: segmentation violation

hlt_mc_Fake

----- Begin Fatal Exception 26-May-2026 00:52:06 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Attempted to open logical file file:RelVal_Raw_Fake_MC.root.
      [b] Failed to open the file with physical name file:RelVal_Raw_Fake_MC.root.
      [c] Fatal Root Error: @SUB=TStorageFactoryFile::ReadBuffer
read from Storage::xread returned 256. Asked to read n bytes: 300 from offset: 0 with file size: 256

----- End Fatal Exception -------------------------------------------------

hlt_mc_Fake

----- Begin Fatal Exception 26-May-2026 00:52:28 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Attempted to open logical file file:RelVal_Raw_Fake_MC.root.
      [b] Failed to open the file with physical name file:RelVal_Raw_Fake_MC.root.
      [c] Fatal Root Error: @SUB=TStorageFactoryFile::ReadBuffer
read from Storage::xread returned 256. Asked to read n bytes: 300 from offset: 0 with file size: 256

----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

kawaho · 2026-05-26T00:15:39Z

@akritkbehera how do I run these tests locally? They look strange because all the errors seem to be coming from GEANT4 which is not changed in this commit.

smuzaffar · 2026-05-26T07:27:52Z

please test with cms-sw/cmssw#51029 using full cmssw

lets build full cmssw

cmsbuild · 2026-05-26T11:54:10Z

-1

Failed Tests: UnitTests RelVals AddOn
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53479/summary.html
COMMIT: 8089a4f
CMSSW: CMSSW_17_0_X_2026-05-25-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10580/53479/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53479/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53479/git-merge-result

Failed Unit Tests

I found 10 errors in the following unit tests:

---> test test_MC_22_crosscheck had ERRORS
---> test test_MC_23_crosscheck had ERRORS
---> test GeneratorInterfaceEvtGenInterfaceTest had ERRORS
and more ...

Failed RelVals

25202.0A fatal system signal has occurred: segmentation violation
25.0A fatal system signal has occurred: segmentation violation
1306.0A fatal system signal has occurred: segmentation violation

Expand to see more relval errors ...

Failed AddOn Tests

hlt_mc_Fake

A fatal system signal has occurred: segmentation violation

hlt_mc_Fake

----- Begin Fatal Exception 26-May-2026 13:12:11 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Attempted to open logical file file:RelVal_Raw_Fake_MC.root.
      [b] Failed to open the file with physical name file:RelVal_Raw_Fake_MC.root.
      [c] Fatal Root Error: @SUB=TStorageFactoryFile::ReadBuffer
read from Storage::xread returned 256. Asked to read n bytes: 300 from offset: 0 with file size: 256

----- End Fatal Exception -------------------------------------------------

hlt_mc_Fake

----- Begin Fatal Exception 26-May-2026 13:12:33 CEST-----------------------
An exception of category 'FileOpenError' occurred while
   [0] Constructing the EventProcessor
   [1] Constructing input source of type PoolSource
   [2] Calling RootInputFileSequence::initTheFile()
   Additional Info:
      [a] Attempted to open logical file file:RelVal_Raw_Fake_MC.root.
      [b] Failed to open the file with physical name file:RelVal_Raw_Fake_MC.root.
      [c] Fatal Root Error: @SUB=TStorageFactoryFile::ReadBuffer
read from Storage::xread returned 256. Asked to read n bytes: 300 from offset: 0 with file size: 256

----- End Fatal Exception -------------------------------------------------

Expand to see more addon errors ...

kawaho · 2026-05-27T16:17:30Z

The GEANT4 crush is most likely due to the dependence on SimG4Core I introduced in GeneratorInterface/ExternalDecays. This should be resolved by #51041. Let's wait for it to be merged so I can modify my commit according to the new build

kawaho · 2026-05-28T17:01:18Z

@smuzaffar

Ready to test. ThreadHandoff is moved now in the new commit.

smuzaffar · 2026-05-28T18:45:09Z

please test with cms-sw/cmssw#51029 using full cmssw

cmsbuild · 2026-05-28T19:05:15Z

-1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53550/summary.html
COMMIT: 8089a4f
CMSSW: CMSSW_17_0_X_2026-05-28-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10580/53550/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed External Build

I found compilation warning when building: See details on the summary page.

kawaho · 2026-05-28T19:10:30Z

hepforge seems to be down right now....

cmsbuild · 2026-05-29T02:05:04Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53557/summary.html
COMMIT: 8089a4f
CMSSW: CMSSW_17_0_X_2026-05-28-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10580/53557/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed Unit Tests

I found 3 errors in the following unit tests:

---> test TestGeneratorInterfaceEvtGenInterface_bplus had ERRORS
---> test GeneratorInterfaceEvtGenInterfaceTest had ERRORS
---> test TestGeneratorInterfaceEvtGenInterface_external_bplus had ERRORS

Comparison Summary

Summary:

You potentially removed 173 lines from the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4198477
DQMHistoTests: Total failures: 9
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4198448
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 227 log files, 197 edm output root files, 53 DQM output files
TriggerResults: no differences found

kawaho · 2026-05-29T02:26:50Z

Unit tests failed because of MPI in Sherpa

*** The MPI_Comm_rank() function was called before MPI_INIT was invoked.
*** This is disallowed by the MPI standard.
*** Your MPI job will now abort.

It is fine when I build Sherpa without MPI....I need to find a way to disable it for the unit tests....

kawaho · 2026-05-29T02:36:15Z

@smuzaffar #10211 seems to be related...does OMPI_MCA_accelerator=null propagate when one does cmsDriver.py or cmsRun? Or are there tricks to turn it off?

smuzaffar · 2026-05-29T09:05:20Z

@kawaho , yes OMPI_MCA_accelerator=null is seen by cmsDriver/cmsRun. It is set when one runs cmsenv to set cmssw runtime env. As mentioned in #10211, we set this to make sure OpenMPI does not call setenv ( which is not threadsafe and we have seen runtime crashes due to that)

smuzaffar · 2026-05-29T14:56:20Z

please test with cms-sw/cmssw#51029, #10594

smuzaffar · 2026-05-29T14:57:43Z

please test workflows 534.0, 536.0 with cms-sw/cmssw#51029, #10594

cmsbuild · 2026-05-29T19:43:55Z

-1

Failed Tests: UnitTests
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53578/summary.html
COMMIT: 8089a4f
CMSSW: CMSSW_17_0_X_2026-05-29-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10580/53578/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53578/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53578/git-merge-result

Failed Unit Tests

I found 3 errors in the following unit tests:

---> test TestGeneratorInterfaceEvtGenInterface_bplus had ERRORS
---> test GeneratorInterfaceEvtGenInterfaceTest had ERRORS
---> test TestGeneratorInterfaceEvtGenInterface_external_bplus had ERRORS

Comparison Summary

Summary:

You potentially removed 175 lines from the logs
Reco comparison results: 10 differences found in the comparisons
DQMHistoTests: Total files compared: 55
DQMHistoTests: Total histograms compared: 4200255
DQMHistoTests: Total failures: 66
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4200169
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 54 files compared)
Checked 231 log files, 199 edm output root files, 55 DQM output files
TriggerResults: no differences found

kawaho · 2026-05-29T21:36:40Z

The unit tests now run fine with my latest commit. Tested locally with a build of sherpa with mpi

Singularity> scram b runtests_TestGeneratorInterfaceEvtGenInterface_bplus                                                                                                                                       
>> Local Products Rules ..... started
>> Local Products Rules ..... done
Creating test log file logs/el8_amd64_gcc13/testing.log
Pass  210s ... GeneratorInterface/EvtGenInterface/TestGeneratorInterfaceEvtGenInterface_bplus
Singularity> scram b runtests_GeneratorInterfaceEvtGenInterfaceTest      
>> Local Products Rules ..... started
>> Local Products Rules ..... done
Creating test log file logs/el8_amd64_gcc13/testing.log
Pass  281s ... GeneratorInterface/EvtGenInterface/GeneratorInterfaceEvtGenInterfaceTest
Singularity> scram b runtests_TestGeneratorInterfaceEvtGenInterface_external_bplus
>> Local Products Rules ..... started
>> Local Products Rules ..... done
Creating test log file logs/el8_amd64_gcc13/testing.log
Pass  220s ... GeneratorInterface/EvtGenInterface/TestGeneratorInterfaceEvtGenInterface_external_bplus

@smuzaffar can we test again

smuzaffar · 2026-05-29T22:13:59Z

please test with cms-sw/cmssw#51029

cmsbuild · 2026-05-30T03:19:49Z

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53590/summary.html
COMMIT: 8089a4f
CMSSW: CMSSW_17_0_X_2026-05-29-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmsdist/10580/53590/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53590/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-a29dd1/53590/git-merge-result

Comparison Summary

Summary:

You potentially removed 164 lines from the logs
Reco comparison results: 4 differences found in the comparisons
DQMHistoTests: Total files compared: 53
DQMHistoTests: Total histograms compared: 4198477
DQMHistoTests: Total failures: 69
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4198388
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
Checked 227 log files, 197 edm output root files, 53 DQM output files
TriggerResults: no differences found

Ka Wa Ho added 4 commits May 18, 2026 13:56

update spec file for EvtGen v3

fe13f1d

patch for pythia8 with the EvtGenv3

ae4090f

bump evtgen to 3.0.0; disable sherpa mpi

d845961

add back mpi to sherpa

8089a4f

cmsbuild added tests-pending externals-pending pending-signatures orp-pending labels May 25, 2026

kawaho mentioned this pull request May 25, 2026

upgrade EvtGen interface for v03.00.00 cms-sw/cmssw#51029

Open

cmsbuild added tests-started requires-external and removed tests-pending labels May 25, 2026

cmsbuild added tests-rejected and removed tests-started labels May 25, 2026

cmsbuild added tests-started and removed tests-rejected labels May 26, 2026

cmsbuild added tests-rejected and removed tests-started labels May 26, 2026

cmsbuild added tests-started and removed tests-rejected labels May 28, 2026

cmsbuild added tests-rejected and removed tests-started labels May 28, 2026

cmsbuild added tests-started and removed tests-rejected labels May 28, 2026

cmsbuild added tests-rejected and removed tests-started labels May 29, 2026

cmsbuild added tests-started and removed tests-rejected labels May 29, 2026

cmsbuild added tests-rejected and removed tests-started labels May 29, 2026

cmsbuild added tests-started and removed tests-rejected labels May 29, 2026

cmsbuild added tests-approved and removed tests-started labels May 30, 2026

Conversation

kawaho commented May 25, 2026

Uh oh!

cmsbuild commented May 25, 2026

Uh oh!

cmsbuild commented May 25, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

akritkbehera commented May 25, 2026

Uh oh!

cmsbuild commented May 25, 2026

Failed Unit Tests

Failed RelVals

Failed AddOn Tests

Uh oh!

kawaho commented May 26, 2026

Uh oh!

smuzaffar commented May 26, 2026

Uh oh!

cmsbuild commented May 26, 2026

Failed Unit Tests

Failed RelVals

Failed AddOn Tests

Uh oh!

kawaho commented May 27, 2026

Uh oh!

kawaho commented May 28, 2026

Uh oh!

smuzaffar commented May 28, 2026

Uh oh!

cmsbuild commented May 28, 2026

Failed External Build

Uh oh!

kawaho commented May 28, 2026

Uh oh!

cmsbuild commented May 29, 2026

Failed Unit Tests

Comparison Summary

Uh oh!

kawaho commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

kawaho commented May 29, 2026

Uh oh!

smuzaffar commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented May 29, 2026

Uh oh!

smuzaffar commented May 29, 2026

Uh oh!

cmsbuild commented May 29, 2026

Failed Unit Tests

Comparison Summary

Uh oh!

kawaho commented May 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

smuzaffar commented May 29, 2026

Uh oh!

cmsbuild commented May 30, 2026

Comparison Summary

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

cmsbuild commented May 25, 2026 •

edited

Loading

kawaho commented May 29, 2026 •

edited

Loading

smuzaffar commented May 29, 2026 •

edited

Loading

kawaho commented May 29, 2026 •

edited

Loading