Downloading from the ICARE server

Synchronising folder structure

ICARE.jl is meant for data that is arranged by years and dates with the following structure:

<root>/<product folder>/yyyy/yyyy_mm_dd

This folder structure is synchronised with the local system and data files are downloaded to the date folders at the lowest level. To minimize server communication and speed up download processes, a local .inventory.yaml file (hidden on Linux and MacOs) is created in the product folder. The .inventory.yaml contains information about the folder structure and file stats and should not be edited or deleted. The inventory is created before the first download of a given product. This process takes several minutes up to hours in extreme cases. After the initial setup, only dates outside the known date range are updated, which is much faster, unless a complete resynchronisation is forced.

Downloading data files

Use the sftp_download function to synchronise the AERIS/ICARE server with the local system. Mandatory arguments are the ICARE user credentials, the product, and the date span. Further fine-tuning of the downloads is possible with keyword arguments as described in the sftp_download help. The function returns the current full inventory with all available online dates over the full date range for further exploration.

The inventory has its own version number that follows the rules of semantic versioning. It is, however, considered part of the public API of the ICARE package. So, any breaking changes in the inventory, for which no transitioning from older versions is provided, is considered a breaking change in ICARE itself. If a transition is provided, it would be a major update of the inventory, but a minor update of ICARE.

sftp_download(
    user::String,
    password::String,
    product::String,
    startdate::Int,
    enddate::Int=-1;
    version::Union{Nothing,Real} = 4.51,
    remoteroot::String = "/SPACEBORNE/CALIOP/",
    localroot::String = ".",
    convert::Bool = true,
    resync::Bool = false,
    update::Bool = false,
    logfile::String = "downloads.log",
    loglevel::Symbol = :Debug
) -> SortedDict

Specifying the download product

By default, CALIOP data (/SPACEBORNE/CALIOP) will be downloaded, but can be changed with the remoteroot keyword argument. Data will be downloaded to the current folder or the folder specified by localroot. Use the third positional argument to specify the product, you want to download. Products are assumed in the format <name>.v<X.XX>, where X.XX is the version number with a two-digit minor version. By default, version 4.51 is assumed, but can be changed with the version keyword argument (e.g., upgrade to version 5 with version=5).

For custom formats, version can be set to nothing and the entire name and version string passed to the product argument.

Specifying a date range

For convenience, dates are given as integers, so users don't have to import the Dates package. You can either give one date range as fourth positional argument or a start and end date as fourth and fifth positional argument.

The date format is yyyy[mm[dd]], where the day and/or month part can be omitted. Defining a span is possible for:

• a whole year (yyyy)
• a whole month (yyyymm)
• a day (yyyymmdd)

If the the day and/or month are omitted in the start or end date, they will be filled with the earliest possible day for the startdate and the latest possible date for the enddate.

Some examples are:

• 20220212: Download the whole day of 2022-02-12
• 202004: Download the whole April of 2020
• 2020: Download the whole year 2020
• 2002, 200206: Download the first half of 2002
• 200207, 2002: Download the second half of 2002
• 200003, 20000315: Download the first half of March 2000
• 20000316, 200003: Download the second half of March 2000

Only complete days can be downloaded.

Updating the inventory

As mentioned before, updating the entire inventory including the parts already synced may take a long time and should not be necessary under normal circumstances. It can be achieved by setting resync to true.

If you want to check for updated data files on the server, you can set the update flag to true. This will also resync the complete inventory and, hence, take a long time to finish. If update is set, any newer file on the server compared to the modified time on the local system will be downloaded.

Logging

Some basic information about the current download session is printed to the screen with further comprehensive information in a log file. By default, log files are saved to the product folder as download_<timestamp>.log. You can change the file name with the logfile keyword argument. The file name will automatically be appended with the time stamp, when the download session started. If the file name includes a path, the logfile will be saved to this path. The path can be absolute or relative to your current location (where you started your julia session or where you changed to during your julia session), i.e. logfile = "~/icare.log" will create a logfile icare_<timeestamp>.log in your root directory. Note that the extension can be change as well to, e.g., .txt or .dat.

For the log file the verbosity can be set to 4 levels given as Symbol (with verbosity listed from least to most):

• Error
• Warn
• Info
• Debug

By default, all messages are printed allowing you to track the download status on screen and in the log file. The Debug level is used to inform about completed downloads.

Converting data files

During download

During an sftp_download, you can set the convert keyword argument to true. By default, this will save downloads in the HDF5 (.h5) format instead of the expected HDF4 (.hdf) format. You can overload the ICARE.convert_file function, see section about Converting to other file formats. However, you are allowed to save only one other format.

Separate file conversions

Routines exist to do batch conversions of the existing files and are described in the section about Analysing and manipulating the inventory.