uwsift.util package

Subpackages

Submodules

uwsift.util.common module

uwsift.util.common.create_scenes(scenes: dict, file_groups: dict) → List[DataID][source]

Create Scene objects for the selected files.

uwsift.util.common.format_clims(clims, unit_conv)[source]
uwsift.util.common.format_resolution(resolution: float) → str[source]

Return string representing the given resolution (interpreted in meters) in the unit km for values greater than 1 km, otherwise in m, including the unit symbol.

uwsift.util.common.format_wavelength(wl)[source]

Return wavelength formatted as string with unit.

A wavelength below 4.1 micrometers is formatted with two, above with one decimal.

uwsift.util.common.get_initial_gamma(info)[source]

Determine initial gamma value based on ‘info’

uwsift.util.common.get_reader_kwargs_dict(reader_names)[source]

Extract the reader_kwargs from the config for multiple readers.

uwsift.util.common.is_datetime_format(format_str)[source]
uwsift.util.common.is_same_proj(proj_a: str, proj_b: str) → bool[source]

Compare the given proj strings and consider them the same if numeric parameters do not differ in the first 9 significant digits.

The order of proj parameters and possible differences in case are ignored.

uwsift.util.common.normalize_longitude(lon: float) → float[source]

Return longitude centered around prime meridian, i.e. in the range ]-180.0, 180.0]

uwsift.util.common.preferred_units(info) → str[source]

Return unit string (i.e.: Kelvin) for a Product currently being loaded. :param info: metadata information describing the product currently being added. :return: String describing the preferred unit for the product described in info.

uwsift.util.common.range_hull(range_1: tuple, range_2: tuple) → tuple[source]

Get a range which contains the given ranges.

If the given ranges don’t overlap the resulting range also contains the range between them. Thus, the result can be more than the union of the given ranges.

Here a ‘range’ is defined by a pair of numbers a, b. The range is considered empty, if a > b.

uwsift.util.common.range_hull_no_fail(range_1: tuple | None, range_2: tuple | None, fallback: tuple) → tuple[source]

Get a range which contains the given ranges, safely.

Same as ‘range_hull()’ but if the range can not be computed from ‘range_1’ and ‘range_2’ then the given ‘fallback’ is returned. This happens if any of ‘range_1’ and ‘range_2’ is None or (None, None).

uwsift.util.common.unit_symbol(unit)[source]
uwsift.util.common.units_conversion(info)[source]

return UTF8 unit string, lambda v,inverse=False: convert-raw-data-to-unit, format string for converted value with unit.

uwsift.util.default_paths module

Utility functions and classes

Directory Constants:

  • WORKSPACE_DB_DIR: Default location to place the workspace metadata database(s)

  • WORKSPACE_CACHE_DIR: Default workspace cache directory Where any raster data may be cached.

  • DOCUMENT_SETTINGS_DIR: Default document user settings/profiles directory

uwsift.util.disk_management module

class uwsift.util.disk_management.DiskManagement(pid: int | None = None, cmdline: str | None = None)[source]

Bases: object

DiskManagement can be used to locate temporary files, which are not cleaned up by the application itself. When the program terminates a detailed report will be printed to stdout containing all open files grouped by their access mode.

Parameters:

  • pid – Process Identifier of the process to be traced

  • cmdline – command line of the process to be traced

Raises:

ValueError – both pid and cmdline are None, cmdline of process does not match with specified cmdline, process with the specified command line could not be found

collect_open_files(interval: float = 0.2) → None[source]

Collect a list of open files from all registered processes. If a file is opened in read only mode, then the access mode will be “r”. If a file is opened in read/write mode, then the access mode will be “r+”. Also, a file can be reopened with write privileges. Additionally, a message with the number of new files will be printed to stdout.

Parameters:

interval – time between checks in seconds

open_files: Dict[str, str] = {}

uwsift.util.heap_analyzer module

class uwsift.util.heap_analyzer.HeapAnalyzer[source]

Bases: object

combine_snapshot(snapshot: List[Statistic])[source]
create_plot(top_sort: str = 'size', limit: int = 25)[source]
dump_combined_snapshots(path: str)[source]
static get_conda_packages_path(stdlib_directory: str) → str | None[source]
static get_python_stdlib_path() → str | None[source]
get_top_stats(limit: int = 25) → Generator[str, None, None][source]
static get_uwsift_project_path() → str | None[source]
load_combined_snapshots(path: str)[source]
static load_snapshots(snapshot_directory: str) → Generator[List[Statistic], None, None][source]

uwsift.util.heap_profiler module

class uwsift.util.heap_profiler.HeapProfiler(sec_interval: float)[source]

Bases: object

cancel()[source]
snapshot_idx = 0
start()[source]
class uwsift.util.heap_profiler.RepeatableTimer(sec_interval: float, callback, *, args=None, kwargs=None)[source]

Bases: Thread

cancel()[source]
finished = <threading.Event at 0x7f5c2c2ad110: unset>
run()[source]

Method representing the thread’s activity.

You may override this method in a subclass. The standard run() method invokes the callable object passed to the object’s constructor as the target argument, if any, with sequential and keyword arguments taken from the args and kwargs arguments, respectively.

uwsift.util.heap_profiler.format_byte_count(byte_count: int)[source]

uwsift.util.logger module

uwsift.util.logger.configure_loggers() → None[source]

Configure all loggers with a certain log level, format and handler.

While all loggers are configured to have the same format and handler, the log level can be configured explicitly for “all” loggers and/or for individual ones. If not configured, the log level is left as is, i.e., loggers of third party libraries keep their default log level. If a logging.filename is given, logging goes to that file (if it is writable), to console otherwise.

uwsift.util.ps_analyzer module

uwsift.util.ps_analyzer.create_plot(df: DataFrame)[source]
uwsift.util.ps_analyzer.format_byte_count(byte_count: int)[source]

uwsift.util.ps_profiler module

class uwsift.util.ps_profiler.PsProfiler(cmdline: str)[source]

Bases: object

file = None
run(out_file: str, interval: float = 0.2)[source]

uwsift.util.storage_agent module

class uwsift.util.storage_agent.FileMetadata(path: str, size: int, mtime: float)[source]

Bases: object

FileMetadata stores the path, size and last data modification time of an entry from the filesystem. This class is an implementation detail of StorageAgent and not exposed as part of the public API surface.

Parameters:

  • path – absolute path to the filesystem entry

  • size – size in bytes

  • mtime – last data modification time in seconds

class uwsift.util.storage_agent.StorageAgent(files_lifetime: int, notification_cmd: str | None)[source]

Bases: object

The StoregeAgent can be used to cleanup temporary files, which are not deleted up by the application itself. Only files and directories which were not used for some time are deleted by this agent.

Parameters:

  • files_lifetime – number of seconds after which the entry is deleted

  • notification_cmd – command to send log messages to a reporting system

Raises:

ValueError – files_lifetime is negative or zero

dir_paths: List[str] = []
register_directory(dir_path: str) → bool[source]

Checks if the directory exists and if this process has write access. An IOError will be raised when either condition is violated. This can be bypassed by appending the dir_path to self.dir_paths.

Parameters:

dir_path – absolute path to a directory

Returns:

False if the directory doesn’t exist or has read-only access

run(interval: int | None) → None[source]

Start the scanning of the processes specified by the method register_directory. This method will block indefinitely and can only be exited using a KeyboardInterrupt Exception.

Parameters:

interval – time between the checks in seconds

uwsift.util.watchdog module

class uwsift.util.watchdog.Watchdog(config_dirs: List[str], cache_dir: str)[source]

Bases: object

allowed_mem_usage: int | None = None
ask_again_interval: timedelta | None = None
notification_cmd: str | None = None
restart_interval: timedelta | None = None
run()[source]

Run the watchdog in blocking mode. The watchdog will read the heartbeat file periodically and check whether the timestamp of the dataset creation is too old. Additional it will ensure that the application is restarted from time to time and that the memory consumption isn’t too high.

uwsift.util.watchdog.get_config_value(config: Config, key: str) → str[source]

Wrapper for the get method from the donfig library, which provides a more friendly error message if the key doesn’t exist.

Parameters:

  • config – Config object from donfig

  • key – key to the config value

Returns:

str or dict with the config value

Module contents

uwsift.util.check_grib_definition_dir()[source]
uwsift.util.check_imageio_deps()[source]
uwsift.util.get_base_dir()[source]

Return the uwsift installation/package base directory.

uwsift.util.get_package_data_dir()[source]

Return location of the package ‘data’ directory.

When frozen the data directory is placed in ‘sift_data’ of the root package directory.

uwsift.util.prefix_share_dir()[source]