ESMF_PIN_DE_TO_VAS not supported by ESMF_FieldEmptyComplete? (and ESMF_Array?)

[mathomp4]

Recently, MAPL had to implement a way to change the pinflag passed to ESMF_FieldEmptyComplete which was needed for some users to use MAPL (see ufs-community/ufs-weather-model#2619, GEOS-ESM/MAPL@v2.53.2...v2.53.3, GEOS-ESM/MAPL#3677)

In doing so, I decided to test MAPL with all of the flags:

• ESMF_PINFLAG: PET --> ESMF_PIN_DE_TO_PET
• ESMF_PINFLAG: VAS --> ESMF_PIN_DE_TO_VAS
• ESMF_PINFLAG: SSI --> ESMF_PIN_DE_TO_SSI
• ESMF_PINFLAG: SSI_CONTIG --> ESMF_PIN_DE_TO_SSI_CONTIG

In my testing with GEOSgcm, PET, SSI and SSI_CONTIG all "work" (in that things don't crash). However, if I try VAS:

 Integer*4 Resource Parameter: REFERENCE_DATE:20000414, (default value)
 Integer*4 Resource Parameter: REFERENCE_TIME:0, (default value)
pe=00040 FAIL at line=00379    Base_Base_implementation.F90             <status=518>
pe=00040 FAIL at line=06784    MAPL_Generic.F90                         <status=518>
pe=00040 FAIL at line=06412    MAPL_Generic.F90                         <status=518>
pe=00040 FAIL at line=01562    MAPL_Generic.F90                         <status=518>
pe=00040 FAIL at line=01126    MAPL_Generic.F90                         <status=518>
DYNInitialize                                 2562
SUPERDYNAMICSInitialize                        634
AGCMInitialize                                1229

where line 379 is in our call to ESMF_FieldEmptyComplete:

             call ESMF_FieldEmptyComplete(FIELD, typekind=ESMF_TYPEKIND_R4, &
                  gridToFieldMap=gridToFieldMap,              &
                  totalLWidth=haloWidth(1:griddedDims),     &
                  totalUWidth=haloWidth(1:griddedDims),     &
                  ungriddedLBound=(/lb3/), ungriddedUBound=(/ub3/),  &
                  pinflag=pinflag, _RC)
             call ESMF_FieldGet(FIELD, farrayPtr=VAR_3D, _RC)

From my reading of ESMF, it seems like ESMF_PIN_DE_TO_VAS is only valid for DELayout use of pinflag. For example, there are references in src/Infrastructure/DELayout/examples/ESMF_DELayoutEx.F90 and src/Infrastructure/DELayout/tests/ESMF_DELayoutWorkQueueUTest.F90.

But if I look at src/system_tests/ESMF_ArraySharedDeSSI/user_model1.F90, it has references to PET, SSI, and SSI_CONTIG but not VAS

So, I'm thinking maybe the docs should update things like:

[pinflag] Specify which type of resource DEs are pinned to. See section 51.2.1 for a list of valid pinning options. The default is to pin DEs to PETs, i.e. only the PET on which a DE was created considers the DE as local.

to specify the flags that are allowed?

Or, should VAS work in this case (even if nonsensically)? Not sure

[theurich]

After some thought, my conclusion is that ESMF_PIN_DE_TO_VAS is sensible and should be supported. However, it is somewhat of an edge case from a practical point of view because there aren't any actual components out there, that I know of, that utilize threaded VMs, i.e. VMs where groups of PETs run as threads under the same process, sharing their virtual address space (VAS). For the case where PETs are synonymous with MPI processes, ESMF_PIN_DE_TO_VAS and ``ESMF_PIN_DE_TO_PET` are identical.

My thinking is that we should implement support of ESMF_PIN_DE_TO_VAS on a high level, essentially making your test pass. There would be error checking further down, that would bail out if indeed a VM with threaded PETs is used in combination with ESMF_PIN_DE_TO_VAS in the DE sharing context (of Fields and Arrays). My plan is to get this into 8.9.0.