Home >> Forums >> Coupling Copernicus Ocean data products (OPA-NEMO)

Coupling Copernicus Ocean data products (OPA-NEMO)

Hello, everyone,
I'm trying to couple a hydrodynamic model from marine.copernicus.eu to ICHTHYOP. Specifically OGCM (Ocean General Circulation Model) and MEDSEA_REANALYSIS_PHYS_006_004 product. The codes are NEMO-OPA version 3.2 and 3.4. The files are freely access and avalaible from marine.copernicus.eu. The commnon pattern filename is "sv03-med-ingv-cur-rean-d". At the end, I attach the basic information of one of these files.
I've tried coupling the models without success. I get some error messages in the NetBeans console, which I copy here. I also provide the necessary files to reproduce the error message.
So, What is the correct way to couple Copernicus' OPA NEMO with ICHTHYOP? Can someone provide a generic configuration file to manage the hydrodynamic dataset and that it is correctly associated with the hydrodynamic file and the grid/mask? 
Many thanks in advance,
Dr. Andres Ospina-Alvarez
 
NEMO_hydrodynamic_file (64,4 Mb)

The vertical grids are described in the file: MEDmeshmask_SYS4e_T.nc. This file is freely available via HTTP at this link http://gnoodap.bo.ingv.it/myocean/MEDmeshmask_SYS4e_T.nc.gz. The relevant variables described in MEDmeshmask_SYS4e_T.nc file are:

  • - tmask (3D land/sea mask);
  • - Depthlevt (3D matrix with the depth of each grid point taking into account the partial steps)
  • - e3t (3D matrix with the Δz of each grid point, taking into account the partial steps)
    netcdf MEDmeshmask_SYS4e_T { dimensions:
    x = 821 ;
    y = 253 ;
    z = 72 ;
    t = UNLIMITED ; // (1 currently)
    variables:
    floatnav_lon(y, x) ; floatnav_lat(y, x) ; floatnav_lev(z) ; doubletime_counter(t) ; bytetmask(t, z, y, x) ; floatglamt(t, y, x) ; floatgphit(t, y, x) ; double e1t(t, y, x) ; double e2t(t, y, x) ; doubleff(t, y, x) ; shortmbathy(t, y, x) ; doublehdept(t, y, x) ; double e3t(t, z, y, x) ; double gdept_0(t, z) ; double e3t_0(t, z) ; doubleDepthlevt(t, z, y, x) ;

// global attributes: :DOMAIN_number_total = 1 ;
:DOMAIN_number = 0 ; :DOMAIN_dimensions_ids = 1, 2 ; :DOMAIN_size_global = 821, 253 ; :DOMAIN_size_local = 821, 253 ; :DOMAIN_position_first = 1, 1 ; :DOMAIN_position_last = 821, 253 ; :DOMAIN_halo_size_start = 0, 0 ; :DOMAIN_halo_size_end = 0, 0 ; :DOMAIN_type = "BOX" ;
}

 

Ichthyop version: 
Ichthyop 3.3alpha
Hydrodynamic dataset: 
NEMO3D - MEDSEA_REANALYSIS_PHYS_006_004
Java version: 
Java SE 8
Operating system: 
Mac OS X

Hi

I can't download the MEDmeshmask_SYS4e_T.nc file from the URL you provided.

Nicolas

And I also need the zone files please (I found the mesh file somewhere else)

 

I updated the previous post with the link to download the grid/mask file and the release zones file.
Thanks in advance.

Hi Nicolas and Andres,
I step in here because my question is directly related to the subject,
I am also using file from Copernicus : http://marine.copernicus.eu/services-portfolio/access-to-products/?option=com_csw&view=details&product_id=GLOBAL_REANALYSIS_PHY_001_030
I have been running Ichthyop several times but with ROMS or MARS3D models and it is my first time using NEMO outuputs, would it be possible to get some more information about what this mask, zgr and hgr files are doing and where to find them? is using the one shared in this post would make sense?
Thanks
Boris

Hi Andres

Your problem is due to a wrong definition of the gridU file pattern parameter. It uses the grid file instead of the data file. You must modify its value to sv03-med*.nc.

However, I could not check since the mesh file I have contains 72 vertical levels, while the data file has only 47 vertical levels.

Nicolas

Hi Boris.

The mask, mesh_hgr and mesh_zgr contains some informations about the NEMO grid, which may be irregular.

The mask file contains the land-sea mask for T/U/V points, the mesh_hgr contains some informations about the 2D structure of the grid (zonal and meridional extents of the cells, longitude, latitude), the mesh_zgr contains some informations about the vertical structure of the grid (thickness of the cells, depth of the cells, etc.)

In some cases, all these files are merged together in one file, which is the case in the MEDmeshmask_SYS4e_T.nc file.

Nicolas

Hi Nicolas, thanks for fast reply,
do you usually get this file with the model outputs? is it something that I should ask from the people who run the model or can I create it myself?
thanks
Boris

Hi Boris.

You cannot generate this file yourself. Usually, this file is generated by the NEMO model and should be provided with model outputs.

Nicolas

ok thanks,
In case someone else is looking for these files,
they can can be dowloaded directly from Copernicus website (nameofthedataset-statics)
Boris

Hi Nicolas, could you help me out here?
I got a Number of ranges in section (2) must be = 1 error,
I understand from a previous post that Ichthyop is expecting a 2-dim variable and found a 1-dim variable instead,
however I cannot find where is the mistake,
I attached the description of the files mask and coordinates that contain variables for mask, hgr and zgr files, I moved one variable deptho_lev, which I assume to be Depth at T points from mask to coordinates,
let me know if you need further details,
thanks,
Boris
 
 

if I delete value for Depth at W points, then I get a new error message which is e3t cannot be read because of undexpected dimensions, e3t doesn't have time dimension in my file should I create one?

Hi Boris

The e3t variable should indeed be of dimensions (1, nz, nlat, nlon), i.e the first dimension is a virtual time dimension.

I have checked, and there indeed is a bug in the reading of NEMO depthw: Ichthyop expects a (nz+1) array, but NEMO returns a (nz) array.

Until the bug is fixed, it is advised to reconstruct depthw from deptht.

Nicolas

Hi Nicolas,
I am getting the same Number of ranges in section (2) must be = 1 error.
What do you mean by reconstruct depthw from deptht? How do I do that?
Thanks,
Jasmine
 

Hi Nicolas,
I came back working on that today, using the recommendations that you gave to Jasmine and myself,
but I still get an error message (a different one),
" Illegal Range for dimension 2: last requested 780 > max 35 | ucar.ma2.InvalidRangeException: Illegal Range for dimension 2: last requested 780 > max 35"
I gather all variables in one file, add 1dim to e3t, and check that the dimensions fit simulation file dimensions,
could you please have a look?
thanks
Boris

Hi Jasmine.

Could you please try this .jar?

https://filesender.renater.fr/?s=download&token=20020ff3-4609-0ac9-fccb-...

And let me know if it works?

Thanks

Nicolas

Hi Nicolas,
Thanks for such a speedy response. I tried the .jar file that you sent and I got this error instead:
ichthyop[info] Ichthyop 3.3 (2017/03/23) - Welcome on board ! ichthyop[info] [Configuration] Opened C:\Users\12708235\Desktop\Copernicus\2018_08_01_Jasmine_config_nemo3d.xml ichthyop[info] ===== Simulation started ===== ichthyop[info] [Simulation] Run 1 / 1 ichthyop[info] [Simulation] Initialising... ichthyop[warning] Ichthyop will recalculate W variable from U and V ichthyop[warning] Ichthyop assumes that by default the NEMO NetCDF files must be opened in enhanced mode (with scale, offset and missing attributes). ichthyop[info] read lon, lat & mask from C:/Users/12708235/Desktop/Copernicus/GLO-MFC_001_030_coordinates.nc ichthyop[info] Depth of W points is read from NetCDF file ichthyop[severe] null | java.lang.NullPointerException ichthyop[severe] [Simulation] Initialisation failed (performing: SETUP)
Thanks,
Jasmine

Hi Jasmine.

Could you please send me your NetCDF files (mask file and one data file)?

I will try to reproduce the error.

Thanks

Nicolas

Hi Nicolas,
Here is a link to my NetCDF files (mask file and one data file).
https://www.dropbox.com/sh/37ht4b4plis1xr4/AAC4tzLitkQAKcihfOM17mspa?dl=0
 
The data file is for 01/08/2016 - 02/08/2016.
Thanks,
Jasmine

Hi again Jasmine.

Sorry for the inconvenience, but I need all the grid files (mask, coordinates, etc.).

Thanks

Nicolas

Sorry about that. They are too large for me to send but they can be downloaded from here:
http://marine.copernicus.eu/services-portfolio/access-to-products/?optio...
as global-reanalysis-phy-001-030-statics.
I haven't altered them in anyway since I downloaded them from the link above.
Thanks,
Jasmine

Hi Jasmine.

I have investigated the source of the error. In fact, you misinterpreted the depth variables. "Depth at T point" must be set as equal to "depth", i.e. the 1D array providing depths of T grid points. The "Depth at W point" is the 1D array providing the depths of W points. You provided a 2D array containing the index of the last ocean point. In your case, you don't have depths of W points, so you can leave it blank, so that depthW can be estimated by using depthT.

However, there are several problems:

- Your files are way too big. So I advise you to extract the region you are interested in.

- The time settings were not good (time origin was not good).

- Since Ichtyop expects NEMO data to be 4D (time, depth, lat, lon), the code won't work with your dataset since there is a mismatch between the depth dimensions of the static files and of the data file:

netcdf global-reanalysis-phy-001-030-daily_1538995684106 {
dimensions:
    longitude = 4320 ;
    latitude = 2041 ;
    depth = 1 ;
    time = UNLIMITED ; // (2 currently)

 

netcdf GLO-MFC_001_030_mask_bathy {
dimensions:
    depth = 50 ;
    latitude = 2041 ;
    longitude = 4320 ;

netcdf GLO-MFC_001_030_coordinates {
dimensions:
    latitude = 2041 ;
    longitude = 4320 ;
    depth = 50 ;

Therefore, you need to find the data that contains 50 depth instead of 1. Else, a new class to read NEMO 2D data should be written.

Just in case, I provide you with my updated file.

Nicolas

 

 

Hi Boris.

I assume that you are working with either Matlab or R? If so, here is a very helpful hint. Matlab and R are column specific langages (see https://eli.thegreenplace.net/2015/memory-layout-of-multi-dimensional-ar...). 

As a consequence, for optimisation purposes, when you save a Matlab/R matrix of dimensions (time, depth, lat, lon) into a NetCDF with Matlab/R, it is iffectively saved as a (lon, lat, depth, time) matrix. Which is the case in the files you sent me:

nbarrier@nbarrier-HP-ZBook ~/Bureau/Boris $ ncdump -h mask-coor.nc
netcdf mask-coor {
dimensions:
    latitude = 481 ;
    longitude = 2161 ;
    time = 1 ;
    depth = 36 ;
variables:
    float depth(depth) ;
    float deptho_lev(longitude, latitude) ;
    float longitude(longitude) ;
    float latitude(latitude) ;
    byte mask(longitude, latitude, depth) ;
    float e1t(longitude, latitude) ;
    float e2t(longitude, latitude) ;
    float e3t(longitude, latitude, depth, time) ;
}

 

This new order is not what is expected by Ichthyop. So, when you generate this file, transpose all the variables/dimensions and it sjould be ok. If you need help, please send me your script that generates this file.

Nicolas

Hi Nicolas,
ok thanks, I did that but still have a problem with one of the variable, error message "Number of ranges in section (3) must be = 2"
is there any variable in the file that I sent you before which does not have the nbr of dim expected by ichthyop?
would it be possible to create a file somewhere summarizing information about how ichthyop wants the parameters?
in case someone else get the same issue than me working with matlab or R, to fix it you need to create  variables entering dimension in counter order, then to permute your data before writing them in the variable,
here a post about that for matlab
Boris

Hi Boris.

Another problem is your configuration file. You wrongly set the "depth" parameter (deptho_level instead of depth, cf. one of the post of this forum thread).

But since I have no hydrodynamic data, I cannot fully check.

Here is the corrected configuration file.

As to the format of the data, Ichthyop is build to fit the "classical" NEMO output layout. The problems occur when data are re-formatted into another kind of format, which is usually the case with data from Copernicus.

Nicolas

Hi Nicolas, thanks, it works now using the .jar that you shared upper,
I still get an error though later in the initialization process, here are the 4 last lines
ichthyop[info] Action manager setup [OK]
ichthyop[info] Output manager setup [OK]
ichthyop[severe] null | java.lang.NullPointerException
ichthyop[severe] [Simulation] Initialisation failed (performing: INIT)
if you can me orient me a bit I can investigate by myself, or I can share the files with you,
thanks
Boris

Hi Boris.

Can you please try to run the code with the java console enabled? So that I have more informations about where the code crashes (https://techhelpkb.com/enable-the-java-console-on-windows/)

Thanks

Nicolas

Hi Nicolas,
I am not sure how to enable the Java consol, the link you posted seems to work only for web app,
here are all the files to reproduce the error:
https://www.dropbox.com/sh/3w8sen4lq83o836/AAAu0QGeSm-dLjkPN_3nblYJa?dl=0
thanks
Boris

Hi Boris

I was able to solve your problem:

- The main problem was that you did not set the "Averaged time since initialisation" parameter. This parameter's name is misleading (and should be changed): it is the name of the time variable (in NetCDF, there is a distinction between variables and dimensions). You properly set the time dimension name, but not the time variable name.

- I have also changed the "Beginning time of simulation" by replacing 14:00 by 12:00.

I enclose the updated configuration file (you will need to set back the proper file paths).

It now works, but the results might be weird since CFL warnings occur on the vertical advection. Therefore, you will need to consider using a smaller time step (currently 7200 seconds).

I also take the benefit for asking to please acknowledge Ichthyop in your publication, either by co-authoring or through a mention in the acknowledgements.

Thanks

Nicolas

Hi Nicolas,
thanks for your support along the parametrization process, and for sharing this interesting article about row- vs column-major, the time step was way too big indeed, I set it to 240 which seems to be an OK compromise
I will start doing intensive simulations in a month or so and might come back to you at that moment,
about acknowledging Ichthyop, I use to refer to Lett et al, 2008 and post the website address,
I am totally fine to also mention Ichthyop is the acknowledgement if you think it will give Ichthyop more credit, we can think about co-authoring if never it turns out I need more specific support latter,
best
Boris