Add two-way ocean-wave coupling feature to the HAFS applications

[binli2337]

Commit Queue Requirements:

• Fill out all sections of this template.
• All sub component pull requests have been reviewed by their code managers.
• Run the full Intel+GNU RT suite (compared to current baselines) on either Hera/Derecho/Hercules
• Commit 'test_changes.list' from previous step

---

Description:

The ufs-weather-model will be updated to include two-way ocean-wave coupling capability for HAFS applications.

Commit Message:

  * CMEPS - CMEPS is updated to transfer ocean-current fields from MOM6 to WW3 and to transfer the Stokes drift components from WW3 to MOM6 for the HAFS applications.
  * MOM6 - MOM6 cap is updated to convert a missing value to zero for the imported Stokes drift components for the ufs.hafs applications.
  * WW3 - WW3 cap is updated to convert a missing value to zero for the imported ocean-current fields for the ufs.hafs applications.
 

Priority:

• Normal

Git Tracking

UFSWM:

• Closes Update HAFS applications to include 2-way ocean-wave coupling capability #2582

Sub component Pull Requests:

• CMEPS: Update CMEPS for 2-way ocean-wave coupling in HAFS NOAA-EMC/CMEPS#133
• MOM6: Update MOM6 cap for 2-way ocean-wave coupling for HAFS NOAA-EMC/MOM6#143
• WW3: Update WW3 cap for 2-way ocean-wave coupling NOAA-EMC/WW3#1348

UFSWM Blocking Dependencies:

• Blocked by #
• None

---

Changes

Regression Test Changes (Please commit test_changes.list):

• PR Updates/Changes Baselines.

Input data Changes:

• Updated input data files are located at
  /scratch1/NCEPDEV/hwrf/save/Bin.Li/UFS-WM_RT/NEMSfv3gfs/input-data-20250507/WW3_input_data_20250225/createmoddefs/ww3_grid.inp.natl_6m
  /scratch1/NCEPDEV/hwrf/save/Bin.Li/UFS-WM_RT/NEMSfv3gfs/input-data-20250507/WW3_input_data_20250225/mod_def.natl_6m

Library Changes/Upgrades:

• No Updates

---

Testing Log:

• RDHPCS
  • Hera
  • Orion
  • Hercules
  • Jet
  • GaeaC5
  • GaeaC6
  • Derecho
• WCOSS2
  • Dogwood/Cactus
  • Acorn
• CI
• opnReqTest (complete task if unnecessary)

[DeniseWorthen]

@binli2337 Don't you know the value which will be sent from non-overlapped regions (ie, the fillvalue). Why is it necessary to check for the difference between the field and the fillvalue ... for example in MOM6 : abs(stkx(i,j,ib)-fillValue) <= 0.01 ?

[binli2337]

@DeniseWorthen When comparing floating point numbers, it is generally recommended to check if the absolute difference between the two numbers is less than a tiny value. In this specific case, since stkx and fillvalue are constant, it is probably not necessary to check the absolute difference. Thanks for reviewing the code.

[DeniseWorthen]

@binli2337 If your argument about 'checking small differences' pertains, then shouldn't you make sure both components are tested?

if( (trim(casename) == "ufs.hafs") .and. (abs(stkx(i,j,ib)-fillValue) <= 0.01) ) then
              ice_ocean_boundary%ustkb(i,j,ib) = 0.0
              ice_ocean_boundary%vstkb(i,j,ib) = 0.0
            else

[binli2337]

@DeniseWorthen In the related CMEPS PR, the Stokes drift components for the x direction and the y direction are filled with the fill value in the same non-overlapped areas (from line 603 to line 610 in the esmFldsExchange_hafs_mod.F90 file). So in the MOM6 PR, when stkx has the fill value, the stky will always has the same fill value. Checking whether abs(stky(i,j,ib)-fillValue) is less than a small number is not needed.

[DeniseWorthen]

@binli2337 I'm confused how you are using waves in MOM6 w/o updating your MOM_input. When I look at the file
ufs.hafs.cpl.hi.2020-08-25-54000.nc in the hafs_regional_storm_following_1nest_atm_ocn_wav_mom6_intel test, there are no stokes fields exported to OCN. There are only stokes fields imported from WAV.

[DeniseWorthen]

See this code from the MOM cap

  call query_ocean_state(ocean_state, use_waves=use_waves, wave_method=wave_method)
  if (use_waves) then
    if (wave_method == "EFACTOR") then
      allocate( Ice_ocean_boundary%lamult(isc:iec,jsc:jec) )
      Ice_ocean_boundary%lamult          = 0.0
    else if (wave_method == "SURFACE_BANDS") then
      call query_ocean_state(ocean_state, NumWaveBands=Ice_ocean_boundary%num_stk_bands)
      allocate(Ice_ocean_boundary%ustkb(isc:iec,jsc:jec,Ice_ocean_boundary%num_stk_bands), source=0.0)
      allocate(Ice_ocean_boundary%vstkb(isc:iec,jsc:jec,Ice_ocean_boundary%num_stk_bands), source=0.0)
      allocate(Ice_ocean_boundary%stk_wavenumbers(Ice_ocean_boundary%num_stk_bands), source=0.0)
      call query_ocean_state(ocean_state, WaveNumbers=Ice_ocean_boundary%stk_wavenumbers, unscale=.true.)
    else
      call MOM_error(FATAL, "Unsupported WAVE_METHOD encountered in NUOPC cap.")
    endif
  endif

How is MOM6 importing and using waves when the use_waves parameter is not present in the MOM_input_hafs?

[jiandewang]

this parameter needs to be here (false is its default), other wise it will break mpi test (answer change with different PE #) on wcoss2. There is a FMA bug in the code but GFDL haven't figured it out yet.

[binli2337]

@jiandewang The file "MOM_input_hafs" has been updated. Thanks!

[binli2337]

@DeniseWorthen The file "MOM_input_hafs" has been updated for the 2-way ocean wave coupling test. The test name is hafs_regional_storm_following_1nest_atm_ocn_wav_mom6. Thanks!

[DeniseWorthen]

@binli2337 Are you using a new or updated WW3 mod_def for this test? If not, I believe you are exporting all 0s from WW3 for the stokes components. See this line

https://github.com/NOAA-EMC/WW3/blob/8e676278822292a97b2b69da336444ec7d04c06b/model/src/wav_import_export.F90#L808

You need to have the following line in your inp file for WW3 in order to calculate the stokes for export:

&OUTS USSP = 1, IUSSP = 3, STK_WN = 0.04, 0.110, 0.3305 /

How are you testing this? Have you verified non-zero stokes fields sent to OCN?

[binli2337]

@DeniseWorthen Yes, the model definition file has been updated with the stokes drift components. The updated file is from my input directory on Hera. You can verify the stokes drift fields from the mediator history file.

[DeniseWorthen]

@binli2337 Thanks, I found the mod_def (we need to be careful and commit the updated inp file too). But I still don't understand why the changes are necessary in WW3 or MOM6. I ran the mom6 RT w/ only your changes to CMEPS (to add the field exchanges). All the stokes drifts components are already 0.0 in the areas where there is no overlap.

[binli2337]

@DeniseWorthen Thank you for identifying some interpolation issues. The issues have been fixed.

[DeniseWorthen]

@binli2337 It's not clear what you mean. Do your latest changes mean that you now see non-zero stokes components where there is no overlap?

[binli2337]

@DeniseWorthen @JessicaMeixner-NOAA , The most recent HAFS-MOM6-WW3 test is located at /scratch1/NCEPDEV/stmp2/Bin.Li/FV3_RT/rt_3666111/hafs_regional_storm_following_1nest_atm_ocn_wav_mom6_intel. From the mediator history file "ufs.hafs.cpl.hi.2020-08-25-54000.nc" you can see that the Stokes drift fields (OCNEXP_SW_PSTOKES_X1) and ocean current fields (WAVEXP_SO_U) are all missing values in the non-overlapped regions. So the updates in MOM6 cap and WW3 cap are necessary to set the missing values to 0. Thanks!

[DeniseWorthen]

@binli2337 This is fundamentally an issue of non-matching land masks for the OCN and WAV models. We have the exact same issue in the global model, when we don't run the OCN and WAV on the same grids.

Using your run-directory, I added the config attribute to write the dststatus information (write_dststatus = true). In the figures below, pink=unmapped. They're unmapped because the destination is w/in a masked part of the source grid.

This is ocn->wav

And this is wav->ocn

If instead you switch your wav<-->ocn mapping to be bilinr_nstod, which is what we use in the global model, then the regions where your ocean and wave land masks do not match will be filled w/ the nearest neighbor instead of being filled w/ the fillvalue. This means that no changes to other components are required. That is a much cleaner solution. Did you test that?

See my run directory /scratch1/NCEPDEV/stmp2/Denise.Worthen/FV3_RT/hafs/test.nstod which is the current develop branch of ufs using only your change to the hafs fieldExchange (no changes to WAV or OCN), with this modifcation

diff --git a/mediator/esmFldsExchange_hafs_mod.F90 b/mediator/esmFldsExchange_hafs_mod.F90
index 0515c870..0dc6b9e7 100644
--- a/mediator/esmFldsExchange_hafs_mod.F90
+++ b/mediator/esmFldsExchange_hafs_mod.F90
@@ -379,7 +379,7 @@ contains
     use med_internalstate_mod , only : InternalState
     use med_internalstate_mod , only : mapbilnr, mapconsf, mapconsd, mappatch
     use med_internalstate_mod , only : mapfcopy, mapnstod, mapnstod_consd
-    use med_internalstate_mod , only : mapfillv_bilnr
+    use med_internalstate_mod , only : mapfillv_bilnr, mapbilnr_nstod
     use med_internalstate_mod , only : mapnstod_consf
     use esmFlds               , only : addmap_from => med_fldList_addmap_from
     use esmFlds               , only : addmrg_to   => med_fldList_addmrg_to
@@ -606,7 +606,7 @@ contains
        fldname = trim(S_flds(n))
        if ( fldchk(is_local%wrap%FBexp(compocn)        , fldname, rc=rc) .and. &
             fldchk(is_local%wrap%FBImp(compwav,compwav), fldname, rc=rc)) then
-           call addmap_from(compwav, fldname, compocn, mapfillv_bilnr, &
+           call addmap_from(compwav, fldname, compocn, mapbilnr_nstod, &
                 hafs_attr%mapnorm, 'unset')
            call addmrg_to(compocn, fldname, mrg_from=compwav, mrg_fld=fldname, mrg_type='copy')
        end if
@@ -648,7 +648,7 @@ contains
        fldname = trim(S_flds(n))
        if ( fldchk(is_local%wrap%FBexp(compwav)        , fldname, rc=rc) .and. &
             fldchk(is_local%wrap%FBImp(compocn,compocn), fldname, rc=rc)) then
-          call addmap_from(compocn, fldname, compwav, mapfillv_bilnr, &
+          call addmap_from(compocn, fldname, compwav, mapbilnr_nstod, &
                hafs_attr%mapnorm, 'unset')
           call addmrg_to(compwav, fldname, mrg_from=compocn, mrg_fld=fldname, mrg_type='copy')
        end if

[binli2337]

@DeniseWorthen We tried the mapbilnr_nstod method a few months ago. When the ocean domain and the wave domain are almost completely overlapped, the results from the interpolation look good when the mapbilnr_nstod method is used. But in one of our configurations used for operation, the wave domain (see fig. 1) is much smaller than the ocean domain (fig. 2). When the mapbilnr_nstod method is used to interpolate the Stokes drift field from the wave grid (fig. 1) to the ocean grid (fig. 2), some features appear in the non-overlapped regions. When the mapfillv_bilnr method is used and when the missing value is set to zero, the interpolated results look better (see fig. 3 and fig. 4).

fig.1 One of the Stokes drift components in WW3 domain

fig.2 Stokes drift field fter regridding using mapbilnr_nstod method from WW3 grid to MOM6 grid

fig.3 One of the Stokes drift components in WW3 domain

fig.4 Stokes drift field after regridding using mapfillv_bilnr method from WW3 grid to MOM6 grid

[DeniseWorthen]

@binli2337 I apologize for the long delay in having a chance to look at this PR again. Thanks for posting the figures above.

Looking at your test change list, why does any test other than the hafs_regional_storm_following_1nest_atm_ocn_wav_mom6 intel have changed baselines? The other tests either don't have a OCN model or use HYCOM.

Secondly, I was looking a different issue (#2676). I realized that no HAFS test is actually using the SSC to modify the surface stresses. Why is that? The feature was specifically added for HAFS but it is not being used in any of the RTs. This can be verified by running the HAFS RTs and checking that icplocn2atm=0 in all cases.

Finally, we need the new ww3_grid.inp.natl_6m file for the WW3 input directory, not just the new mod_def. The inp files are stored in the createmoddefs subdirectory. The one to produce the new natl_6m mod_def file includes the USSP OUTs, so we need that file also.

[DeniseWorthen]

@binli2337 I've created Issue #2742. Please take a look.

[binli2337]

@DeniseWorthen Thanks for the suggestions. I will revise the PR to use icplocn2atm=1 for the hafs_regional_storm_following_1nest_atm_ocn_wav_mom6 test so that the ocean currents are used in the air-sea flux calculation.
All the HAFS tests with the wave component will change results, because the revised model definition file (containing information about the Stokes drift components) will be used for all the HAFS tests with the wave component. I have prepared a revised ww3_grid.inp.natl_6m file and a new model definition file associated with the revised ww3_grid.inp.natl_6m file. Thanks!

[DeniseWorthen]

@binli2337 Thanks for the updates.

I checked which files failed comparison in your test_changes.list. I see that for everything but the MOM6 test, the change was only in the binary WW3 gridded output (due to the mod_def change), except for the hafs_regional_atm_ocn_wav_intel test. That one also changes the mediator restart file because the ocean/wav exchange fields are not w/in if (trim(coupling_mode) == 'hafs') then conditional. But all the other files (except the binary out.grd from ww3) are identical though, so it's not causing an issue.