lstmcpipe.stages package

lstmcpipe.stages package#

Submodules#

lstmcpipe.stages.mc_dl1_to_dl2 module#

lstmcpipe.stages.mc_dl1_to_dl2.batch_dl1_to_dl2(dict_paths, config_file, jobid_from_training, batch_config, logs)#

Function to batch the dl1_to_dl2 stage once the lstchain train_pipe batched jobs have finished.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • jobid_from_training (str) – string containing the jobid from the jobs batched in the train_pipe stage, to be passed to the dl1_to_dl2 function (as a slurm dependency)

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to dl1_dl2 function

  • logs (dict) – Dictionary with logs files

Returns:

jobid_for_dl2_to_dl3 – string containing the jobids to be passed to the next stage of the workflow (as a slurm dependency)

Return type:

str

lstmcpipe.stages.mc_dl1_to_dl2.dl1_to_dl2(input_file, output_dir, path_models, config_file, wait_jobid_train_pipe=None, batch_configuration='', extra_slurm_options=None)#

Convert onsite files from dl1 to dl2

Parameters:

  • input_file (str) – FILE DL1 path

  • output_dir (str) – DIR Dl2 path

  • path_models (str) – DIR trained models path

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • wait_jobid_train_pipe (str) – Comma-separated string with the batched jobid from the train stage to indicate the dependencies of the current job to be batched

  • batch_configuration (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the sbatch commands ! NOTE : train_pipe AND dl1_to_dl2 MUST BE RUN WITH THE SAME ENVIRONMENT

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_dl1_to_dl2 (dict) – log dictionary containing {jobid: batch_cmd} information

  • jobid_dl1_to_dl2 (str) – batched job_id to be passed to later stages

lstmcpipe.stages.mc_dl2_to_irfs module#

lstmcpipe.stages.mc_dl2_to_irfs.batch_dl2_to_irfs(dict_paths, config_file, job_ids_from_dl1_dl2, batch_config, logs)#

Batches the dl2_to_irfs stage (lstchain lstchain_create_irf_files script) once the dl1_to_dl2 stage had finished.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • config_file (str) – Path to lstchain-like config file

  • job_ids_from_dl1_dl2 (str) – Comma-separated string with the job ids from the dl1_to_dl2 stage to be used as a slurm dependency to schedule the current stage

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to dl2_to_irfs function

  • logs (dict) – Dictionary with logs files

Returns:

jobs_from_dl2_irf – Comma-separated jobids batched in the current stage

Return type:

str

lstmcpipe.stages.mc_dl2_to_irfs.dl2_to_irfs(gamma_file, electron_file, proton_file, outfile, config_file, options, batch_configuration, wait_jobs_dl1dl2, extra_slurm_options=None)#

Batches interactively the lstchain lstchain_create_irf_files entry point.

Parameters:

  • gamma_file (str)

  • electron_file (str)

  • proton_file (str)

  • outfile (str)

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • options (str | None) – options to pass to lstchain_create_irf_files as a string Most common: –irf-point-like

  • batch_configuration (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the sbatch commands

  • wait_jobs_dl1dl2 (str) – Comma separated string with the job ids of previous stages (dl1_to_dl2 stage) to be passed as dependencies to the create_irfs_files job to be batched.

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_dl2_to_irfs (dict) – Dictionary-wise log containing {‘job_id’: ‘batched_cmd’} items

  • job_id_dl2_irfs (str) – Job-id of the batched job to be passed to the last (MC prod check) stage of the workflow.

lstmcpipe.stages.mc_dl2_to_sensitivity module#

lstmcpipe.stages.mc_dl2_to_sensitivity.batch_dl2_to_sensitivity(dict_paths, job_ids_from_dl1_dl2, batch_config, logs)#

Batches the dl2_to_sensitivity stage (stages.script_dl2_to_sensitivity based in the pyIRF iib) once the dl1_to_dl2 stage had finished.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • job_ids_from_dl1_dl2 (str) – Comma-separated string with the job ids from the dl1_to_dl2 stage to be used as a slurm dependency to schedule the current stage

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to dl2_to_sensitivity function

  • logs (dict) – Dictionary with logs files

Returns:

jobid_for_check – Comma-separated jobids batched in the current stage

Return type:

str

lstmcpipe.stages.mc_dl2_to_sensitivity.dl2_to_sensitivity(input_paths, output, batch_configuration, wait_jobs_dl1_dl2, extra_slurm_options=None)#

Function to run the script_dl2_to_sensitivity for the gamma (and the different gamma offsets) and gamma-diffuse particles. Creates the sensitivity *.fits.gz files and the corresponding sensitivity curve plot.

Parameters:

  • input_paths (dict)

  • output (str)

  • batch_configuration (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the sbatch commands

  • wait_jobs_dl1_dl2 (str) – Comma-separated string with the jobs (dependency) to wait for before launching the cmd

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_dl2_to_sensitivity (dict) – Dictionary with job_id-slurm command key-value pair used for logging

  • job_id (str) – String with job_ids batched by the dl2_to_sensitivity script

lstmcpipe.stages.mc_merge_dl1 module#

lstmcpipe.stages.mc_merge_dl1.batch_merge_dl1(dict_paths, batch_config, logs, jobid_from_splitting, workflow_kind='lstchain')#

Function to batch the onsite_mc_merge_and_copy function once the all the r0_to_dl1 jobs (batched by particle type) have finished.

Batch 8 merge_and_copy_dl1 jobs ([train, test] x particle) + the move_dl1 and move_dir jobs (2 per particle).

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to merge_dl1 and compose_batch_command_of_script functions.

  • workflow_kind (str) – Defines workflow kind (lstchain, ctapipe, hiperta)

  • logs (dict) – Dictionary with logs files

  • jobid_from_splitting (str)

Returns:

jobids_for_train – Comma-separated str with all the job-ids to be passed to the next stage of the workflow (as a slurm dependency)

Return type:

str

lstmcpipe.stages.mc_merge_dl1.merge_dl1(input_dir, output_file, batch_configuration, wait_jobs_split='', merging_options=None, workflow_kind='lstchain', extra_slurm_options=None)#
Parameters:

  • input_dir (str)

  • output_file (str)

  • batch_configuration (dict)

  • wait_jobs_split (str)

  • merging_options (dict)

  • workflow_kind (str)

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_merge (dict)

  • jobid_merge (str)

lstmcpipe.stages.mc_process_dl1 module#

lstmcpipe.stages.mc_process_dl1.batch_process_dl1(dict_paths, conf_file, batch_config, logs, workflow_kind='lstchain', new_production=True)#

Batch the dl1 processing jobs by particle type.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • conf_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • batch_config (dict) – Dict with source environment (to select the desired conda environment to run the r0/1_to_dl1 stage), and the slurm user account.

  • workflow_kind (str) – One of the supported pipelines. Defines the command to be run on r0 files

  • new_production (bool) – Whether to analysis simtel or reprocess existing dl1 files.

  • logs (dict) – Dictionary con logs files

Returns:

jobids_dl1_processing_stage – string, separated by commas, containing all the jobids of this stage

Return type:

str

lstmcpipe.stages.mc_process_dl1.r0_to_dl1(input_dir, output_dir, workflow_kind='lstchain', config_file=None, batch_config=None, debug_mode=False, keep_rta_file=False, extra_slurm_options=None)#

R0 to DL1 MC onsite conversion. Organizes files and launches slurm jobs in two slurm arrays.

Parameters:

  • input_dir (str or Path) – path to the files directory to analyse

  • output_dir (str or Path) – Output path to store files

  • config_file (str or Path) – Path to the {lstchain, ctapipe, HiPeRTA} configuration file

  • batch_config (dict) – Dictionary containing the full (source + env) source_environment and the slurm_account strings. ! NOTE : train_pipe AND dl1_to_dl2 MUST be run with the same environment.

  • workflow_kind (str) – One of the supported pipelines. Defines the command to be run on r0 files

  • extra_slurm_options (dict) – Extra slurm options to be passed

  • ARGUMENTS (# HIPERTA)

  • keep_rta_file (bool) – Flag to indicate whether to keep (keep_rta_file = True) or remove (keep_rta_file = Flase ) the dl1v06_reorganized_*.h5 output file (hiperta_r0_dl1 and re-organiser stages). Argument to be passed to the hiperta_r0_to_dl1lstchain script.

  • debug_mode (bool) – Flag to activate debug_mode. Only compatible with hiperta workflow kind, i.e., HiPeRTA functionality. DEFAULT=False.

Returns:

  • jobid2log (dict) – dictionary log containing {jobid: batch_cmd} information

  • jobids_r0_dl1 – A list of all the jobs sent for input dir

lstmcpipe.stages.mc_process_dl1.reprocess_dl1(input_dir, output_dir, workflow_kind='lstchain', config_file=None, batch_config=None, dl1_files_per_job=50, extra_slurm_options=None)#

Reprocessing of existing dl1 files. Organizes files and launches slurm jobs in two slurm arrays. The same train/test split performed with the earlier r0 to dl1 processing is used.

Parameters:

  • input_dir (str) – path to the files directory to analyse

  • output_dir (str or Path) – Output path to store files

  • config_file (str) – Path to a configuration file. If none is given, the standard configuration of the selected pipeline is applied

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings. ! NOTE : train_pipe AND dl1_to_dl2 MUST be run with the same environment.

  • workflow_kind (str) – One of the supported pipelines. Defines the command to be run on r0 files

  • dl1_files_per_job (int) – Number of dl1 files to be processed per job array that was batched.

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • jobid2log (dict) – dictionary log containing {jobid: batch_cmd} information

  • jobids_dl1_dl1 – A list of all the jobs sent for input dir

lstmcpipe.stages.mc_process_dl1.submit_dl1_jobs(input_dir, output_dir, base_cmd, file_list, job_type_id, dl1_files_per_batched_job, job_logs_dir, batch_config, n_jobs_parallel=100, dl1_processing_type='r0_to_dl1', extra_slurm_options=None)#

Compose sbatch command and batches it

Parameters:

  • input_dir (str) – path to the files directory to analyse

  • output_dir (str) – Output path to store files

  • base_cmd (str) – command choosing the lstmcpipe core script type (script_batch_filelist_*)

  • file_list (list) – list of r0_dl1 or dl1ab filelist to be passed to the core script

  • job_type_id (str) – String for job naming depending on the workflow

  • dl1_files_per_batched_job (int) – Number of dl1 files to be processed per job array that was batched.

  • job_logs_dir (Path) – Directory for the logs of the core script output. Should be Path(output_dir).joinpath(“job_logs_*”)

  • batch_config (dict) – Dictionary containing the full (source + env) source_environment and the slurm_account strings. ! NOTE : train_pipe AND dl1_to_dl2 MUST be run with the same environment.

  • n_jobs_parallel (int) – Number of array jobs to be processed in parallel. Default = 100

  • dl1_processing_type (str) – String for job and filelist naming

  • extra_slurm_options (dict) – Extra slurm options to be passed

Returns:

  • jobid2log (dict)

  • jobid (str)

lstmcpipe.stages.mc_train module#

lstmcpipe.stages.mc_train.batch_plot_rf_features(dict_paths, config_file, batch_configuration, train_jobid, logs)#

Batches the plot_model_importance.py script that creates a .png with the RF feature’s importance models after the RF are trained. The plot is saved in the same dir in where the modes are stored.

Parameters:

dict_paths (dict) – Core dictionary with {stage: PATHS} information

config_file: str

Path to lstchain config file

batch_configurationdict

Dictionary containing the (full) source_environment and the slurm_account strings.

train_jobid: str

Single jobid from training stage.

logs: dict

Dictionary with logs files

Returns:

jobid – jobid of batched stage

Return type:

str

lstmcpipe.stages.mc_train.batch_train_pipe(dict_paths, jobids_from_merge, config_file, batch_config, logs)#

Function to batch the lstchain train_pipe once the proton and gamma-diffuse merge_and_copy_dl1 batched jobs have finished.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • jobids_from_merge (str) –

    string containing the jobids (*ONLY from proton and gamma-diffuse*) from the jobs batched in the

    merge_and_copy_dl1 stage, to be passed to the train_pipe function (as a slurm dependency)

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the train_pipe function.

  • logs (dict) – Dictionary with logs files

Returns:

jobid_4_dl1_to_dl2 – string containing the jobid to be passed to the next stage of the workflow (as a slurm dependency). For the next stage, however, it will be needed TRAIN + MERGED jobs

Return type:

str

lstmcpipe.stages.mc_train.train_pipe(gamma_dl1_train_file, proton_dl1_train_file, models_dir, config_file=None, batch_configuration='', wait_jobs_dl1=None, extra_slurm_options=None)#

Train RF from MC DL1 data (onsite LaPalma cluster)

Parameters:

  • gamma_dl1_train_file (str) – FILE path to the gamma

  • proton_dl1_train_file (str) – FILE path to the proton

  • models_dir – DIR trained models output path

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • batch_configuration (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the sbatch commands ! NOTE : train_pipe AND dl1_to_dl2 MUST BE RUN WITH THE SAME ENVIRONMENT

  • wait_jobs_dl1 (str) – A string (of chained job_ids separated by ‘,’ and without spaces between each element), containing all the job_ids of the merging stage

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_train (dict) – dictionary containing the jobid of the batched job as key and the run command (the lstchain_mc_train command with all its corresponding arguments) as value.

  • jobid_train (str) – jobid of the batched job to be send (for dependencies purposes) to the next stage of the workflow (onsite_mc_dl1_to_dl2)

lstmcpipe.stages.mc_train_test_splitting module#

lstmcpipe.stages.mc_train_test_splitting.batch_train_test_splitting(dict_paths, jobids_from_r0dl1, batch_config, logs)#
Parameters:

  • dict_paths (dict)

  • jobids_from_r0dl1 (str)

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to merge_dl1 and compose_batch_command_of_script functions.

  • logs (dict) – Dictionary with logs files

lstmcpipe.stages.mc_train_test_splitting.check_empty_dir(directory)#

Check if a directory is empty. If not, erase all its content.

Parameters:

directory (Path) – Path to dir to check

lstmcpipe.stages.mc_train_test_splitting.train_test_split(input_dir, output_dirs, batch_configuration, wait_jobid_r0_dl1=None, extra_slurm_options=None)#
Parameters:

  • input_dir (str)

  • output_dirs (dict)

  • batch_configuration (dict)

  • wait_jobid_r0_dl1 (str)

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Module contents#

lstmcpipe.stages.batch_dl1_to_dl2(dict_paths, config_file, jobid_from_training, batch_config, logs)#

Function to batch the dl1_to_dl2 stage once the lstchain train_pipe batched jobs have finished.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • jobid_from_training (str) – string containing the jobid from the jobs batched in the train_pipe stage, to be passed to the dl1_to_dl2 function (as a slurm dependency)

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to dl1_dl2 function

  • logs (dict) – Dictionary with logs files

Returns:

jobid_for_dl2_to_dl3 – string containing the jobids to be passed to the next stage of the workflow (as a slurm dependency)

Return type:

str

lstmcpipe.stages.batch_dl2_to_irfs(dict_paths, config_file, job_ids_from_dl1_dl2, batch_config, logs)#

Batches the dl2_to_irfs stage (lstchain lstchain_create_irf_files script) once the dl1_to_dl2 stage had finished.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • config_file (str) – Path to lstchain-like config file

  • job_ids_from_dl1_dl2 (str) – Comma-separated string with the job ids from the dl1_to_dl2 stage to be used as a slurm dependency to schedule the current stage

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to dl2_to_irfs function

  • logs (dict) – Dictionary with logs files

Returns:

jobs_from_dl2_irf – Comma-separated jobids batched in the current stage

Return type:

str

lstmcpipe.stages.batch_dl2_to_sensitivity(dict_paths, job_ids_from_dl1_dl2, batch_config, logs)#

Batches the dl2_to_sensitivity stage (stages.script_dl2_to_sensitivity based in the pyIRF iib) once the dl1_to_dl2 stage had finished.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • job_ids_from_dl1_dl2 (str) – Comma-separated string with the job ids from the dl1_to_dl2 stage to be used as a slurm dependency to schedule the current stage

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to dl2_to_sensitivity function

  • logs (dict) – Dictionary with logs files

Returns:

jobid_for_check – Comma-separated jobids batched in the current stage

Return type:

str

lstmcpipe.stages.batch_merge_dl1(dict_paths, batch_config, logs, jobid_from_splitting, workflow_kind='lstchain')#

Function to batch the onsite_mc_merge_and_copy function once the all the r0_to_dl1 jobs (batched by particle type) have finished.

Batch 8 merge_and_copy_dl1 jobs ([train, test] x particle) + the move_dl1 and move_dir jobs (2 per particle).

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to merge_dl1 and compose_batch_command_of_script functions.

  • workflow_kind (str) – Defines workflow kind (lstchain, ctapipe, hiperta)

  • logs (dict) – Dictionary with logs files

  • jobid_from_splitting (str)

Returns:

jobids_for_train – Comma-separated str with all the job-ids to be passed to the next stage of the workflow (as a slurm dependency)

Return type:

str

lstmcpipe.stages.batch_plot_rf_features(dict_paths, config_file, batch_configuration, train_jobid, logs)#

Batches the plot_model_importance.py script that creates a .png with the RF feature’s importance models after the RF are trained. The plot is saved in the same dir in where the modes are stored.

Parameters:

dict_paths (dict) – Core dictionary with {stage: PATHS} information

config_file: str

Path to lstchain config file

batch_configurationdict

Dictionary containing the (full) source_environment and the slurm_account strings.

train_jobid: str

Single jobid from training stage.

logs: dict

Dictionary with logs files

Returns:

jobid – jobid of batched stage

Return type:

str

lstmcpipe.stages.batch_process_dl1(dict_paths, conf_file, batch_config, logs, workflow_kind='lstchain', new_production=True)#

Batch the dl1 processing jobs by particle type.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • conf_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • batch_config (dict) – Dict with source environment (to select the desired conda environment to run the r0/1_to_dl1 stage), and the slurm user account.

  • workflow_kind (str) – One of the supported pipelines. Defines the command to be run on r0 files

  • new_production (bool) – Whether to analysis simtel or reprocess existing dl1 files.

  • logs (dict) – Dictionary con logs files

Returns:

jobids_dl1_processing_stage – string, separated by commas, containing all the jobids of this stage

Return type:

str

lstmcpipe.stages.batch_train_pipe(dict_paths, jobids_from_merge, config_file, batch_config, logs)#

Function to batch the lstchain train_pipe once the proton and gamma-diffuse merge_and_copy_dl1 batched jobs have finished.

Parameters:

  • dict_paths (dict) – Core dictionary with {stage: PATHS} information

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • jobids_from_merge (str) –

    string containing the jobids (*ONLY from proton and gamma-diffuse*) from the jobs batched in the

    merge_and_copy_dl1 stage, to be passed to the train_pipe function (as a slurm dependency)

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the train_pipe function.

  • logs (dict) – Dictionary with logs files

Returns:

jobid_4_dl1_to_dl2 – string containing the jobid to be passed to the next stage of the workflow (as a slurm dependency). For the next stage, however, it will be needed TRAIN + MERGED jobs

Return type:

str

lstmcpipe.stages.batch_train_test_splitting(dict_paths, jobids_from_r0dl1, batch_config, logs)#
Parameters:

  • dict_paths (dict)

  • jobids_from_r0dl1 (str)

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to merge_dl1 and compose_batch_command_of_script functions.

  • logs (dict) – Dictionary with logs files

lstmcpipe.stages.dl1_to_dl2(input_file, output_dir, path_models, config_file, wait_jobid_train_pipe=None, batch_configuration='', extra_slurm_options=None)#

Convert onsite files from dl1 to dl2

Parameters:

  • input_file (str) – FILE DL1 path

  • output_dir (str) – DIR Dl2 path

  • path_models (str) – DIR trained models path

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • wait_jobid_train_pipe (str) – Comma-separated string with the batched jobid from the train stage to indicate the dependencies of the current job to be batched

  • batch_configuration (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the sbatch commands ! NOTE : train_pipe AND dl1_to_dl2 MUST BE RUN WITH THE SAME ENVIRONMENT

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_dl1_to_dl2 (dict) – log dictionary containing {jobid: batch_cmd} information

  • jobid_dl1_to_dl2 (str) – batched job_id to be passed to later stages

lstmcpipe.stages.dl2_to_irfs(gamma_file, electron_file, proton_file, outfile, config_file, options, batch_configuration, wait_jobs_dl1dl2, extra_slurm_options=None)#

Batches interactively the lstchain lstchain_create_irf_files entry point.

Parameters:

  • gamma_file (str)

  • electron_file (str)

  • proton_file (str)

  • outfile (str)

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • options (str | None) – options to pass to lstchain_create_irf_files as a string Most common: –irf-point-like

  • batch_configuration (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the sbatch commands

  • wait_jobs_dl1dl2 (str) – Comma separated string with the job ids of previous stages (dl1_to_dl2 stage) to be passed as dependencies to the create_irfs_files job to be batched.

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_dl2_to_irfs (dict) – Dictionary-wise log containing {‘job_id’: ‘batched_cmd’} items

  • job_id_dl2_irfs (str) – Job-id of the batched job to be passed to the last (MC prod check) stage of the workflow.

lstmcpipe.stages.dl2_to_sensitivity(input_paths, output, batch_configuration, wait_jobs_dl1_dl2, extra_slurm_options=None)#

Function to run the script_dl2_to_sensitivity for the gamma (and the different gamma offsets) and gamma-diffuse particles. Creates the sensitivity *.fits.gz files and the corresponding sensitivity curve plot.

Parameters:

  • input_paths (dict)

  • output (str)

  • batch_configuration (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the sbatch commands

  • wait_jobs_dl1_dl2 (str) – Comma-separated string with the jobs (dependency) to wait for before launching the cmd

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_dl2_to_sensitivity (dict) – Dictionary with job_id-slurm command key-value pair used for logging

  • job_id (str) – String with job_ids batched by the dl2_to_sensitivity script

lstmcpipe.stages.merge_dl1(input_dir, output_file, batch_configuration, wait_jobs_split='', merging_options=None, workflow_kind='lstchain', extra_slurm_options=None)#
Parameters:

  • input_dir (str)

  • output_file (str)

  • batch_configuration (dict)

  • wait_jobs_split (str)

  • merging_options (dict)

  • workflow_kind (str)

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_merge (dict)

  • jobid_merge (str)

lstmcpipe.stages.r0_to_dl1(input_dir, output_dir, workflow_kind='lstchain', config_file=None, batch_config=None, debug_mode=False, keep_rta_file=False, extra_slurm_options=None)#

R0 to DL1 MC onsite conversion. Organizes files and launches slurm jobs in two slurm arrays.

Parameters:

  • input_dir (str or Path) – path to the files directory to analyse

  • output_dir (str or Path) – Output path to store files

  • config_file (str or Path) – Path to the {lstchain, ctapipe, HiPeRTA} configuration file

  • batch_config (dict) – Dictionary containing the full (source + env) source_environment and the slurm_account strings. ! NOTE : train_pipe AND dl1_to_dl2 MUST be run with the same environment.

  • workflow_kind (str) – One of the supported pipelines. Defines the command to be run on r0 files

  • extra_slurm_options (dict) – Extra slurm options to be passed

  • ARGUMENTS (# HIPERTA)

  • keep_rta_file (bool) – Flag to indicate whether to keep (keep_rta_file = True) or remove (keep_rta_file = Flase ) the dl1v06_reorganized_*.h5 output file (hiperta_r0_dl1 and re-organiser stages). Argument to be passed to the hiperta_r0_to_dl1lstchain script.

  • debug_mode (bool) – Flag to activate debug_mode. Only compatible with hiperta workflow kind, i.e., HiPeRTA functionality. DEFAULT=False.

Returns:

  • jobid2log (dict) – dictionary log containing {jobid: batch_cmd} information

  • jobids_r0_dl1 – A list of all the jobs sent for input dir

lstmcpipe.stages.reprocess_dl1(input_dir, output_dir, workflow_kind='lstchain', config_file=None, batch_config=None, dl1_files_per_job=50, extra_slurm_options=None)#

Reprocessing of existing dl1 files. Organizes files and launches slurm jobs in two slurm arrays. The same train/test split performed with the earlier r0 to dl1 processing is used.

Parameters:

  • input_dir (str) – path to the files directory to analyse

  • output_dir (str or Path) – Output path to store files

  • config_file (str) – Path to a configuration file. If none is given, the standard configuration of the selected pipeline is applied

  • batch_config (dict) – Dictionary containing the (full) source_environment and the slurm_account strings. ! NOTE : train_pipe AND dl1_to_dl2 MUST be run with the same environment.

  • workflow_kind (str) – One of the supported pipelines. Defines the command to be run on r0 files

  • dl1_files_per_job (int) – Number of dl1 files to be processed per job array that was batched.

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • jobid2log (dict) – dictionary log containing {jobid: batch_cmd} information

  • jobids_dl1_dl1 – A list of all the jobs sent for input dir

lstmcpipe.stages.train_pipe(gamma_dl1_train_file, proton_dl1_train_file, models_dir, config_file=None, batch_configuration='', wait_jobs_dl1=None, extra_slurm_options=None)#

Train RF from MC DL1 data (onsite LaPalma cluster)

Parameters:

  • gamma_dl1_train_file (str) – FILE path to the gamma

  • proton_dl1_train_file (str) – FILE path to the proton

  • models_dir – DIR trained models output path

  • config_file (str) – Path to a configuration file. If none is given, a standard configuration is applied

  • batch_configuration (dict) – Dictionary containing the (full) source_environment and the slurm_account strings to be passed to the sbatch commands ! NOTE : train_pipe AND dl1_to_dl2 MUST BE RUN WITH THE SAME ENVIRONMENT

  • wait_jobs_dl1 (str) – A string (of chained job_ids separated by ‘,’ and without spaces between each element), containing all the job_ids of the merging stage

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command

Returns:

  • log_train (dict) – dictionary containing the jobid of the batched job as key and the run command (the lstchain_mc_train command with all its corresponding arguments) as value.

  • jobid_train (str) – jobid of the batched job to be send (for dependencies purposes) to the next stage of the workflow (onsite_mc_dl1_to_dl2)

lstmcpipe.stages.train_test_split(input_dir, output_dirs, batch_configuration, wait_jobid_r0_dl1=None, extra_slurm_options=None)#
Parameters:

  • input_dir (str)

  • output_dirs (dict)

  • batch_configuration (dict)

  • wait_jobid_r0_dl1 (str)

  • extra_slurm_options (dict) – Extra slurm options to be passed to the sbatch command