Home >> Forums >> Reading a netCDF array all at once.

Reading a netCDF array all at once.

Hello
I do need to read a whole array of variables contained my input file: Mainly vertical dispersion coefficients and depths. Currently I read one by one, but that's slow and innefficient!
Say... I want to get an array that contains rho depths, so currently I do:
 

double[] depthArray = null;
for(int k=0;k<dataset.get_nz()+1;k++) {
depthArray[k] = dataset.z2depth(pGrid[0],pGrid[1],k);
}
 

But I want to do something like:
 

depthArray = dataset.getArray("NameOfVariable",time);

 
How would that work? Any hint?
Thank you,
Ignacio Vidal

Ichthyop version: 
Ichthyop 3.1
Hydrodynamic dataset: 
ROMS3D
Java version: 
Java SE 8
Operating system: 
CentOS

Hi Ignacio

To read a whole variable at once, you can find inspiration in the different functions of the dataset module (for instance NemoDataset.java).

Here is one example:

private double[][][] read_e3_field(NetcdfFile nc, String varname) throws InvalidRangeException, IOException {
 
        Variable ncvar;
        Index index;
        Array array;
 
        ncvar = nc.findVariable(stre3t);   // stre3t is the name of the variable you want to read
        switch (ncvar.getShape().length) {
            case 4:
                array = ncvar.read(new int[]{0, 0, jpo, ipo}, new int[]{1, nz, ny, nx}).flip(1).reduce();
                // ncvar.read(start array, shape array)
                // here we read time from index 0 to 1, depth from index 0 to nz, and so on
                // flip allows you to flip dimensions, and reduce remove dimensions of length one
               break;
            case 3:
                array = ncvar.read(new int[]{0, jpo, ipo}, new int[]{nz, ny, nx}).flip(0).reduce();
            default:
                throw new UnsupportedOperationException("Field " + varname + " cannot be read because of undexpected dimensions.");
        }
 
        // here, you recover the data
        index = array.getIndex();
        double[][][] field = new double[nz][ny][nx];
        for (int k = 0; k < nz; k++) {
            for (int j = 0; j < ny; j++) {
                for (int i = 0; i < nx; i++) {
                    index.set(k, j, i);
                    field[k][j][i] = array.getDouble(index);
                }
            }
        }
        return field;
    }

More information here: http://www.unidata.ucar.edu/software/thredds/current/netcdf-java/javadoc... and https://www.unidata.ucar.edu/software/netcdf/examples/programs/

Nice!
That was of great help. Thank you!

Well, on a second thought, that's highgly inefficient... That's what I was meaning with my question... You're iterating over something that has alread been loaded in memory.
I found this: https://www.unidata.ucar.edu/software/thredds/v4.3/netcdf-java/v4.2/java...
The copyToNDJavaArray() method could be what I was looking for.
That way the three nested loops could be replaced with a single line of code. I will try that over this month.

Thanks for the tip. We will try to have a look at it and see if it improves performance. If so, we will try to implement this everytime needed.

Gentlemen,

You are hitting a sensitive point with NetCDF Java library. It is all about a compromise between genericity and efficiency.

Efficient solution : you know that a NetCDF variable will ALWAYS be of a given type and dimension ==> go for copyToNDJavaArray() method. Exemple :

float[][][] u_tp1 = (float[][][]) ncIn.findVariable(strU).read(origin, shape).reduce().copyToNDJavaArray();

Downside of this solution : if the type of the variable changes or the dimension are different, the code is broken. In early versions of Ichthyop I mostly used this approach because I worked with limited number of people with hydrodynamics dataset showing always the same structure and all was well. Problem came when including new hydrodynamics plugins and when working with people who have different versions of the same hydrodynamics model. I had to go for a more generic approach.

Generic approach : load the variable using the NetCDF generic Array object and loop over all the dimensions to fill up my Java primitive array. Example:

arr = ncIn.findVariable(strU).read(origin, new int[]{1, nz, ny, (nx - 1)}).reduce();
            u_tp1 = new float[nz][ny][nx - 1];
            index = arr.getIndex();
            for (int k = 0; k < nz; k++) {
                for (int j = 0; j < ny; j++) {
                    for (int i = 0; i < nx - 1; i++) {
                        u_tp1[k][j][i] = arr.getFloat(index.set(k, j, i));
                    }
                }
            }

Downside of this solution as Ignacio pointed out is (in)efficiency.

In latest development (not online yet) I tried a new approach with a new HYCOM plugin that has both genericity and efficiency : I read my NetCDF dataset by tiles, only where I really need my data, instead of loading everything at once. And the reading is actually multithreaded and anticipated from one time step to the next one. Resultats are very promising. Ignacio is you are interested in this approach, please email us. It should be released on the website by the end of the year otherwise.

Cheers, Philippe

 

In fact, I was wondering why do we need to copy the Array into a new array? Can't we just use the Array instance? However, In Philippe's answer, there is someting I do not understand.

In both cases, you assume that you will get a float in the end, isnt'it? Except that using the 2nd way, you assume that you read a float from the file, while in the first case no assumption is made on the file content. You may read a double, but it will be converted into floats in the end.

A way to make the first methode more generical might be to use the getRank and getDataype functions to kwnow how the final array should be like:

Variable lat = nc.findVariable(varname);
DataType data = lat.getDataType();
 int ndims = lat.getRank();
    
Array var = lat.read().reduce();
if ((ndims == 1) && (data.equals(data.FLOAT)))
{
    float[] output = (float[]) var.copyToNDJavaArray();
}
        
if ((ndims == 2) && (data.equals(data.INT)))
{
    int[][] output = (int[][]) var.copyToNDJavaArray();

}