CDO installation and use

Climate Modelling course
Chair of Meteorology

Dánnell Quesada — 13/10/21
dquesadacr

Logo TU Dresden

SMWA ESF LO Sachsen 2019 QUER RGB

1. From the CDO documentation

CDO stands for Climate Data Operators
CDO is meant for Posix compatible operating systems (like Linux)
Nevertheless, here you can find the official recommendations to run
CDO on Windows
- The Windows Subsystem for Linux version 2 (wsl2) installs a Linux
  kernel to create a proper compatibility layer for Windows
- The following steps correspond to option #2 on the documentation
  and works for Windows 10
- For Windows 8, wsl1 should be installed (not yet tested for our purpose)
- Windows S is incompatible
Should also work in MacOS

2. Install wsl

Option #1: Windows 10 version 2004 and
higher, or Windows 11 check here
Option #2: older versions, check here
Note that these steps are for wsl version 2
Some visual aid is added

2.1. Option #1:

Open PowerShell as Administrator:

Run (type or copy — paste [right-click on PowerShell]
and then press enter):
```
wsl --install
```

This should already install Ubuntu version 20.04 (?), for other versions/distros check here

Since these changes are quite new, these steps have not been tested

2.2. Option #2

Open PowerShell as Administrator (as shown before)

Run:

dism.exe /online /enable-feature /featurename:
    Microsoft-Windows-Subsystem-Linux /all /norestart

This should be typed all together (one line)

2.3. Update to WSL 2 (check Windows version)

Windows 10 version 1903+, build 18362+. For most computers (x64)
Check with Win key + R, type winver → OK
Update if needed:

If you are running Windows 10 version 1903 or 1909, open "Settings" from your Windows menu, navigate to "Update & Security" and select "Check for Updates". Your Build number must be 18362.1049+ or 18363.1049+, with the minor build # over .1049.

2.4. Enable Virtual Machine feature

From the PowerShell as Administrator run:

dism.exe /online /enable-feature /featurename:VirtualMachinePlatform /all /norestart

Restart to complete WSL install and updates

2.5. Download the Linux kernel update package

Download WSL2 Linux kernel update package for x64 machines
Install as administrator (elevated permissions)

2.6. Set WSL 2 as your default version

From the PowerShell as Administrator run:
```
wsl --set-default-version 2
```
wsl2 setup finished!

2.7. Install Linux distro

Open the Microsoft Store and select a Linux distribution (distro)
Ubuntu 18.04 and 20.04 were tested for CDO. After this, Ubuntu
should be a searchable program in Windows

3. Setting up Ubuntu

Open the Ubuntu app
On the first lunch, it will show this:
Set credentials, user and password (twice)
Output should look like this:

3.2. Update and upgrade

Run from the Linux terminal:

sudo apt-get update
sudo apt-get upgrade -y

To run sudo you will be asked for the password set in the
previous step
This might take a while, you should see something like this:

4. Install and check CDO

To install, simply run from the Ubuntu terminal:
```
sudo apt-get install cdo -y
```

Installation finished!

The last step works also for other Ubuntu based distros
It is also available for other distros, e.g. Arch based

4.1. CDO version

Check that CDO is working with:
```
cdo --version
```

5. CDO installation in MacOS

The following solutions have not been tested by us but they reportedly work for MacOS.
This information was taken from the CDO wiki
1. With MacPorts simply run from the terminal:
  port install cdo +grib_api +magicspp +szip
2. With homebrew:
  brew tap moffat/sciencebits brew install cdo

If needed, to install homebrew:

To install with ruby, run this:

ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"

If the last command doesn’t work, try:

curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install > \
brew_install_script
ruby brew_install_script
rehash

6. Basic shell commands

To work from the terminal, some basic commands are needed:
1. mkdir → creates directory. The following creates the folder data in the current directory
  mkdir data
2. pwd → print working directory. To check the current working directory
3. cd → change directory. To access the created folder:
  cd data

ls → list contents of current directory
cp → copy file. This copies file1 to file2
```
cp file1 file2
```
mv → move files or directories (also to rename). The following moves file1 into the
created folder while renaming it to file2
```
mv file1 data/file2
```
cat → prints the content of a file (text)
- More info here and here

7. Accessing Windows files from Linux

Download the test.nc file (here, under nc_files)
If the file was downloaded to C:\User\Downloads, to access the downloads folder do:
```
cd /mnt/c/User/Downloads
```

Try some CDO commands as:

cdo sinfo test.nc # short description of the file
cdo griddes test.nc # grid description, output on next slide
cdo graph,device="png" -selgridcell,1 test.nc plot # quick plot

The CDO operators are here (explained) or type:
```
cdo --operators
```
If you encounter issues opening or accessing the file do:
```
sudo chown $USER test.nc
```

8. Accessing Linux files from Windows

Type \\wsl$ on the file explorer path (on the top), enter
Then click Ubuntu → home → user
Should look something like this (note the plot created before):

9. Exercise

Exploration of NetCDF files
Visualization of the files
- A set of example files are in this repo, nc_files folder
Manipulation of files
Selection of desired coordinates
Export data as text for further manipulation in other software
Download CORDEX files

10. Exploration

To have an idea of the contents of the file we can run some commands from
the terminal that will print some information to the screen
Commands to try:
- cdo sinfo file → short description of the contents
- cdo griddes file → description of the grid
- ncdump file → metadata description, not CDO but useful
  - Try with -h (header) and -c (coordinates)
See more options with cdo --operators
Check help of a operator with e.g. cdo --help sellonlatbox or here

11. Visualization

It is possible to create high-quality plots from CDO by using the Magics software
- Already included in the installation process explained before
The documentation of the use of CDO+Magics is here
There are 3 types of plots with this combination
- 2D Lon-Lat plots: shaded, grfill and contour
- 2D vector plots: plots vectors (e.g. wind fields) on 2D maps
- Line graph plots: generates time-series plots
Due to the rotated pole rotation embedded in CORDEX projections, 2D plotting
options within CDO are not available
- They only accept rectilinear grids and CORDEX is curvilinear
- Reprojection needed to plot them
  - It can be done with CDO
  - Or check this repo for an R solution

11.1. 2D Lon-Lat

From the terminal, access the folder where the 2m_temperature.nc file was downloaded
- This file is a small example of a rectangular grid taken from ERA5

Run the following command

cdo -shaded,device=pdf,lon_max=-50,lon_min=-110,lat_max=30,lat_min=-10,\
interval=2,colour_triad=cw,colour_min=violet,colour_max=red \
-seltimestep,42 2m_temperature.nc shaded

Resulting file shaded_2t.pdf should be created, shown on next slide
Note the arguments passed to CDO
- Two functions, shaded and seltimestep, are passed with corresponding parameters
- The - allows concatenation of several functions
- Here we selected only the 42^nd step.
- The \ are only to break the line, this command could be written in one line
Try adding ,step_freq=10 after red (no space) and removing -seltimestep,42
- Check resulting file
Try the contour and grfill commands

11.3. Line graph plots

These type of plots are possible with CORDEX files (no need for rectangular grids)
Let’s plot a time series for the city of Dresden
Run this command from the terminal:

cdo -graph,device=pdf,linewidth=4 -remapnn,lon=13.73/lat=51.05 \
mrso_EUR-44_ICHEC-EC-EARTH_rcp85_r12i1p1_SMHI-RCA4_v1_day_20960101-21001231.nc graph

The file graph.pdf should have been created in the working directory
Note the remapnn function → remap to the nearest neighbour
- Returns the time series related to the pixel in which the given coordinate pair is
Plot in next slide

12. Manipulation of files

12.1. Merge

Merge all the provided historical run files into one and save into new file

cdo -z zip -mergetime \
mrso_EUR-44_ICHEC-EC-EARTH_historical_r12i1p1_SMHI-RCA4_v1_day_19* \
historical_1961-1990.nc

Notes
- -z zip to compress the file
- -mergetime is the main function
- 19* wildcard to give as input all the historical files instead of typing each
- historical_1961-1990.nc is the name of the output file
Explore resulting file

12.2. Cropping

To select only a pixel where our location of interest is, again, for Dresden:

cdo -remapnn,lon=13.73/lat=51.05 \
mrso_EUR-44_ICHEC-EC-EARTH_rcp85_r12i1p1_SMHI-RCA4_v1_day_20960101-21001231.nc \
mrso_dresden.nc

To select the pixels inside a boundary box, Saxony (approx.) for example:

cdo -sellonlatbox,11.5,15.5,49.7,52.2 \
mrso_EUR-44_ICHEC-EC-EARTH_rcp85_r12i1p1_SMHI-RCA4_v1_day_20960101-21001231.nc \
mrso_saxony.nc

Check the files with griddes and sinfo

12.3. Exporting to text

For further analysis and plots, one might want to export some information in text format

For example, for the previous file for Dresden:

cdo -outputkey,year,month,day,lon,lat,value mrso_dresden.nc > table.txt

The file table.txt will be created and can be parsed with e.g. R, Python or Excel to do:
- Further statistical analysis
- Boxplots comparing historical vs future conditions
- Day of the year plots, etc.

13. Downloading CORDEX files

Create account for ESGF node
Login with your OpenID after receiving confirmation email
Register to the CORDEX group
Explore and select desired data (checked boxes, note for later, where the pointer is)

13.1. File by file

To download file by file through your browser click on list files
Then click on HTTP Download, this will open a prompt
Save file and repeat
Note that CORDEX files are usually on a 5 years basis (daily)

13.2. The automated way

Click on Add all displayed results to Data Cart (where the
pointer was in the penultimate image)
Go to your data cart
Select datasets to download. When ready, click on WGET Script
Depending on your dataset, there might be several WGET Scripts.
Click and save them.

13.3. Run script

Open a Linux terminal
Go to your downloads folder (see point 7)

Run the following command (change the name appropriately):

bash wget-20201026140831.sh -H
# The -H is to interactively enter your credentials

Give credentials, the full OpenID looks like:
https://esgf-node.llnl.gov/esgf-idp/openid/user
This will start the download process of the NetCDF files
This process will take a while and should look like this:

14. Homework

Download CORDEX files for your home region
- If not available, choose Lisbon or Delft, check here for domains
- Historical period: 1961-1990
- Projected: RCP 8.5 2070-2100
- Variables: temperature, precipitation and either run off or soil moisture content
Merge and crop files for your country and city
Perform statistical analysis for the 3 variables in another software
- Box plots of historical vs projected for your city

Plot time series for your home city
- Both periods and all variables
Extra points for:
- Probability density function plot for both periods
- Average Day of the year (1-365) comparison plot
- Monthly comparison boxplots
- 2D map of average difference between the periods in your home country
  - Tips: sellatlonbox, this repo, or reprojection with CDO (remapbil)

15. Questions — ideas for the extra points?

You can contact me!