lstmcpipe.io package

Subpackages

• lstmcpipe.io.tests package
  • Submodules
  • lstmcpipe.io.tests.test_data_management module
    • create_tmp_dir()
    • create_tmp_subdir()
    • test_check_data_path_raise_value_error()
    • test_get_input_filelist()
  • lstmcpipe.io.tests.test_lstmcpipe_tree_path module
    • test_backup_log()
    • test_create_log_dir()
    • test_create_log_files()
    • test_update_scancel_file()
  • Module contents

Submodules

lstmcpipe.io.data_management module

lstmcpipe.io.data_management.check_and_make_dir(directory)

Check if a directory exists or contains data before to makedir. If exists, query the user to remove its content.

Parameters:

directory (str) – path to a directory

lstmcpipe.io.data_management.check_and_make_dir_without_verification(directory)

lstmcpipe.io.data_management.check_data_path(data_path, glob=None)

Check if the path to some data exists. Raise an Error if the path does not exist, is not a directory or does not contain data.

Parameters:

• data_path (str)

• glob (str) – Glob pattern to be passed

lstmcpipe.io.data_management.check_files_in_dir_from_file(directory, file)

Check that a list of files from a file exist in a dir

Parameters:

• directory

• file

lstmcpipe.io.data_management.get_input_filelist(data_path, glob_pattern=None)

Return list of files in data_path

Parameters:

• data_path (str) – Directory path

• glob_pattern (str) – Glob the given pattern. To Glob recursively, add “**/” in front of the string

Return type:

list

lstmcpipe.io.data_management.move_dir_content(src, dest)

lstmcpipe.io.data_management.query_continue(question, default='no')

Ask a question and if the answer is no, exit the program. Calls query_yes_no.

Parameters:

• question (str)

• default (str)

Return type:

bool - answer from query_yes_no

lstmcpipe.io.data_management.query_yes_no(question, default='yes')

Ask a yes/no question via raw_input() and return their answer.

Parameters:

• question (str) – question to the user

• default (str - "yes", "no" or None) – resumed answer if the user just hits <Enter>. “yes” or “no” will set a default answer for the user None will require a clear answer from the user

Return type:

bool - True for “yes”, False for “no”

lstmcpipe.io.data_management.read_lines_file(file)

lstmcpipe.io.lstmcpipe_tree_path module

lstmcpipe.io.lstmcpipe_tree_path.backup_log(file)

Backups a log file. This scenario should only happens if a MC prod is relaunched because it failed.

Parameters:

file (Path) – Path object to the file to be ‘backup-ed’

lstmcpipe.io.lstmcpipe_tree_path.create_log_dir(prod_id)

Parameters:

prod_id (str) – production identifier of the MC production to be launched

Returns:

log_dir – Path to prod_id log directory

Return type:

Path

lstmcpipe.io.lstmcpipe_tree_path.create_log_files(production_id)

Manages filenames (and overwrites if needed) log files.

Parameters:

production_id (str) – production identifier of the MC production to be launched

Returns:

• logs_files (dict) –

  Dictionary containing

  log_file: Path - path and filename of full log file debug_file: Path - path and filename of reduced (debug) log file

• scancel_file (Path) – path and filename of bash file to cancel all the scheduled jobs

• lstmcpipe_log_dir (Path) – Path to prod_ID log directory

lstmcpipe.io.lstmcpipe_tree_path.update_scancel_file(scancel_file, jobids_to_update)

Bash file containing the slurm command to cancel multiple jobs. The file will be updated after every batched stage and will be erased in case the whole MC prod succeed without errors.

Parameters:

• scancel_file (pathlib.Path) – filename that cancels the whole MC production

• jobids_to_update (str) – job_ids to be included into the the file

Module contents