HPSSPy API¶
hpsspy¶
Python interface to the HPSS system.
hpsspy.os¶
Reproduces some features of the Python built-in os
.
- hpsspy.os.chmod(path, mode)[source]¶
Reproduces the behavior of
os.chmod()
for HPSS files.- Parameters:
- Raises:
HpssOSError – If the underlying hsi reports an error.
- hpsspy.os.listdir(path)[source]¶
List the contents of an HPSS directory, similar to
os.listdir()
.- Parameters:
path (
str
) – Directory to examine.- Returns:
A list of
HpssFile
objects.- Return type:
- Raises:
HpssOSError – If the underlying hsi reports an error.
- hpsspy.os.lstat(path)[source]¶
Perform the equivalent of
os.lstat()
on the HPSS file path.- Parameters:
path (
str
) – Path to file or directory.- Returns:
An object that contains information similar to the data returned by
os.stat()
.- Return type:
- Raises:
HpssOSError – If the underlying hsi reports an error.
- hpsspy.os.makedirs(path, mode=None)[source]¶
Reproduces the behavior of
os.makedirs()
.- Parameters:
- Raises:
HpssOSError – If the underlying hsi reports an error.
Notes
Unlike
os.makedirs()
, attempts to create existing directories raise no exception.
- hpsspy.os.mkdir(path, mode=None)[source]¶
Reproduces the behavior of
os.mkdir()
.- Parameters:
- Raises:
HpssOSError – If the underlying hsi reports an error.
Notes
Unlike
os.mkdir()
, attempts to create existing directories raise no exception.
- hpsspy.os.stat(path, follow_symlinks=True)[source]¶
Perform the equivalent of
os.stat()
on the HPSS file path.- Parameters:
path (
str
) – Path to file or directory.follow_symlinks (
bool
, optional) – IfFalse
, makesstat()
behave likeos.lstat()
.
- Returns:
An object that contains information similar to the data returned by
os.stat()
.- Return type:
- Raises:
HpssOSError – If the underlying hsi ls reports an error.
- hpsspy.os.walk(top, topdown=True, onerror=None, followlinks=False)[source]¶
Traverse a directory tree on HPSS, similar to
os.walk()
.- Parameters:
- Returns:
This function can be used in the same way as
os.walk()
.- Return type:
iterable
hpsspy.os.path¶
Reproduces some features of the Python built-in os.path
.
- hpsspy.os.path.isdir(path)[source]¶
Reproduces the behavior of
os.path.isdir()
for HPSS files.
- hpsspy.os.path.isfile(path)[source]¶
Reproduces the behavior of
os.path.isfile()
for HPSS files.
- hpsspy.os.path.islink(path)[source]¶
Reproduces the behavior of
os.path.islink()
for HPSS files.
hpsspy.scan¶
Functions for scanning directory trees to find files in need of backup.
- hpsspy.scan._options()[source]¶
Parse command-line options.
- Returns:
The parsed command-line arguments.
- Return type:
- hpsspy.scan.compile_map(old_map, section)[source]¶
Compile the regular expressions in a map.
- Parameters:
- Returns:
A new dictionary containing compiled regular expressions.
- Return type:
- hpsspy.scan.extract_directory_name(filename)[source]¶
Extract a directory name from a HTAR filename that may contain various prefixes.
- hpsspy.scan.files_to_hpss(hpss_map_cache, section)[source]¶
Create a map of files on disk to HPSS files.
- Parameters:
- Returns:
A tuple contiaining the compiled mapping and an additional configuration dictionary.
- Return type:
tuple()
- hpsspy.scan.find_missing(hpss_map, hpss_files, disk_files_cache, missing_files, report=10000, limit=1024.0)[source]¶
Compare HPSS files to disk files.
- Parameters:
hpss_map (
dict
) – A mapping of file names to HPSS files.hpss_files (
dict
) – The list of actual HPSS files.disk_files_cache (
str
) – Name of the disk cache file.missing_files (
str
) – Name of the file that will contain the list of missing files.report (
int
, optional) – Print an informational message when N files have been scanned.limit (
float
, optional) – HPSS archive files should be smaller than this size (in GB).
- Returns:
True
if no serious problems were found.- Return type:
- hpsspy.scan.main()[source]¶
Entry-point for command-line scripts.
- Returns:
An integer suitable for passing to
sys.exit()
.- Return type:
- hpsspy.scan.physical_disks(release_root, config)[source]¶
Convert a root path into a list of physical disks containing data.
- hpsspy.scan.process_missing(missing_cache, disk_root, hpss_root, dirmode='2770', test=False)[source]¶
Convert missing files into HPSS commands.
- Parameters:
missing_cache (
str
) – Name of a JSON file containing the missing file data.disk_root (
str
) – Missing files are relative to this root on disk.hpss_root (
str
) – Missing files are relative to this root on HPSS.dirmode (
str
, optional) – Create directories on HPSS with this mode (defaultdrwxrws---
).test (
bool
, optional) – Test mode. Try not to make any changes.
- hpsspy.scan.scan_disk(disk_roots, disk_files_cache, overwrite=False)[source]¶
Scan a directory tree on disk and cache the files found there.
- hpsspy.scan.scan_hpss(hpss_root, hpss_files_cache, overwrite=False)[source]¶
Scan a directory on HPSS and return the files found there.
- hpsspy.scan.validate_configuration(config)[source]¶
Check the configuration file for validity.
- Parameters:
config (
str
) – Name of the configuration file.- Returns:
An integer suitable for passing to
sys.exit()
.- Return type:
hpsspy.util¶
Low-level utilities.
- class hpsspy.util.HpssFile(*args)[source]¶
This class is used to store and access an HPSS file’s metadata.
- Parameters:
args (iterable) – This object this will normally be initialized by a tuple produced by
hpsspy.os.listdir()
.
- htar_contents()[source]¶
Return (and cache) the contents of an htar file.
- Returns:
List containing the contents.
- Return type:
- property isdir¶
True
if the file is a directory or a symbolic link that points to a directory.
- property islink¶
True
if the file is a symbolic link.
- property name¶
Name of the file.
- property path¶
Full path to the file.
- property readlink¶
Destination of symbolic link.
- property st_mode¶
File permission mode.
- property st_mtime¶
File modification time.
- hpsspy.util.get_tmpdir(**kwargs)[source]¶
Return the path to a suitable temporary directory.
Resolves the path to the temporary directory in the following order:
If
tmpdir
is present as a keyword argument, the value is returned.If
TMPDIR
is set, its value is returned.If neither are set,
/tmp
is returned.
- hpsspy.util.hsi(*args, **kwargs)[source]¶
Run hsi with arguments.
- Parameters:
args (
tuple()
) – Arguments to be passed to hsi.tmpdir (
str
, optional) – Write temporary files to this directory. Defaults to the value returned byhpsspy.util.get_tmpdir()
. This option must be passed as a keyword!
- Returns:
The standard output from hsi.
- Return type:
- Raises:
KeyError – If the
HPSS_DIR
environment variable has not been set.