The database inventory

load_inventory(
    path::AbstractString,
    logger::logex.AbstractLogger=logex.global_logger();
    save_migrations::Bool=true
) -> SortedDict

Load the database inventory from the path to the product folder (and a hidden yaml file) to a SortedDict, which can be processed by other ICARE functions. By default, older versions of the inventory are upgraded and saved to the .inventory.yaml. Updating the inventory file can be prevented by setting the kwarg save_migrations to false.

If a logger is provided, events are logged to logger instead of the global logger (typically the console).

See also: list_inventory

list_inventory(
    inventory::SortedDict;
    list_dates::Bool=true,
    list_gaps::Bool=true,
    list_ignored::Bool=true,
    list_extras::Bool=true
)

List the content of the inventory in a simplified tree structure showing available and already downloaded folders and files and statistics about the inventory content. Additionally, missing dates (gaps), ignored files, and extra files are listed.

For all but the overall stats, printing can be switched off with keyword arguments (list_dates, list_gaps, list_ignored, list_extras).

convert_inventory(
    root::AbstractString=".";
    sizecheck::Bool=false,
    logfile::AbstractString = "logs/conversions.log",
    loglevel::Symbol = :Debug
) -> SortedDict

Convert all files in root that are part of the inventory to a new format as defined in the inventory. If files of the new format already exist, they are skipped unless sizecheck is set to true, which will reconvert any file whose size differs from that listed in the inventory. Logging is written to logfile with the specified loglevel. A timestamp is added to the log file name to avoid overwriting existing logs. The logfile may include a path (either absolute or relative to the product folder). The function returns the updated inventory.

See also: convert_inventory!, sftp_download, ignore!, unignore!

convert_inventory!(
    inventory::SortedDict,
    sizecheck::Bool,
    logfile::AbstractString = "logs/conversions.log",
    loglevel::Symbol = :Debug
) -> SortedDict

Convert all files in the local database that are part of the inventory to a new format as defined in the inventory. If files of the new format already exist, they are skipped unless sizecheck is set to true, which will reconvert any file whose size differs from that listed in the inventory. Logging is written to logfile with the specified loglevel. A timestamp is added to the log file name to avoid overwriting existing logs. The logfile may include a path (either absolute or relative to the product folder). The function returns the updated inventory.

See also: convert_inventory, sftp_download, ignore!, unignore!

clean!(
    root::AbstractString=".",
    erase::Extension=none;
    logs::Bool=false,
    logfile::AbstractString = "logs/clean.log",
    loglevel::Symbol = :Debug
) -> SortedDict

clean!(
    inventory::SortedDict,
    erase::Extension=none;
    logs::Bool=false,
    logfile::AbstractString = "logs/clean.log",
    loglevel::Symbol = :Debug
) -> SortedDict

Clean a product folder recursively from all content not listed in the inventory, i.e. not available on the ICARE server, or not flagged as extra files in the inventory extra section with the attach! function. The function has two methods – you can either provide an AbstractString with the path of the product folder or the inventory as SortedDict of the product. The latter is more performant as the inventory doesn't need to be loaded first.

Both methods allow an optional second argument that specifies whether either the original or converted files should be additionally cleaned from the local database. The optional parameter values are predefined constants from the Extension enum. Both methods return the updated inventory for reference.

See also: attach!, detach!, ignore!, unignore!, convert_inventory!, convert_inventory, sftp_download

Keyword Arguments

• logs::Bool: Whether to clean (true) or keep (false) log files created during previous operations (default: false).
• logfile::AbstractString: The name of the log file (default: "logs/clean.log"; the name will be appended by the start timestamp). The name may include a path (either absolute or relative to the product folder).
• loglevel::Symbol: The log level for the download process (default: :Debug).

Enum Extension

Store general choices for available file extensions in the database.

ignore!(
    inventory::SortedDict,
    dates::AbstractDict{Date, Any};
    logfile::AbstractString = "logs/ignore.log",
    loglevel::Symbol = :Debug
) -> SortedDict

Flag the dates as ignored in the inventory and ensure they will not get downloaded. Log events with the specified loglevel to the logfile. A timestamp is appended to the log file name automatically. The logfile may include a path (either absolute or relative to the product folder). The function returns the updated inventory.

See also: unignore!, attach!, detach!, sftp_download

unignore!(
    inventory::SortedDict,
    dates::AbstractDict{Date, Any}=Dict{Date,Any}();
    logfile::AbstractString = "logs/ignore.log",
    loglevel::Symbol = :Debug
) -> SortedDict

Unflag the dates from being ignored in the inventory and allow them to be downloaded again. Log events with the specified loglevel to the logfile. A timestamp is appended to the log file name automatically. The logfile may include a path (either absolute or relative to the product folder). The function returns the updated inventory.

See also: ignore!, attach!, detach!, sftp_download

attach!(
    inventory::SortedDict,
    extras::Union{AbstractString,Vector{<:AbstractString}};
    logfile::AbstractString = "logs/extras.log",
    loglevel::Symbol = :Debug
) -> SortedDict

Attach extra files and folders to the inventory that should be kept during clean! operations. Files nested in foreign folders are recognised as well keeping the parent folders during clean! operations. The extras can be provided as a single AbstractString or as a vector of AbstractStrings. The function returns the updated inventory and logs events with the specified loglevel to the logfile. A timestamp is appended to the log file name automatically. The logfile may include a path (either absolute or relative to the product folder).

See also: detach!, clean!, ignore!, unignore!

detach!(
    inventory::SortedDict,
    extras::Union{AbstractString,Vector{<:AbstractString}}=String[];
    logfile::AbstractString = "logs/extras.log",
    loglevel::Symbol = :Debug
) -> SortedDict

Detach files and folders from the inventory that were previously marked as extra data to be kept during clean! operations. If no extras are provided, all extra data will be detached. For nested files and folders, the parent will be detached as well, if it contains no other extra data.

The function returns the updated inventory and logs events with the specified loglevel to the logfile. A timestamp is appended to the log file name automatically. The logfile may include a path (either absolute or relative to the product folder).

See also: attach!, clean!, ignore!, unignore!