Command line Scripts (scripts)

The scripts to be executed from the command line are described below:

Note

Currently both scripts and Tools are meant to be run from the command line. Please see also Command line Tools (tools) section for more information.

lstchain_check_dl1
lstchain_create_run_summary
lstchain_data_create_time_calibration_file
lstchain_data_r0_to_dl1
lstchain_dl1_muon_analysis
lstchain_dl1ab
lstchain_dump_config
lstchain_find_pedestals
lstchain_longterm_dl1_check
lstchain_mc_r0_to_dl1
lstchain_mc_r0_to_dl2
lstchain_mc_rfperformance
lstchain_mc_sensitivity
lstchain_mc_trainpipe
lstchain_merge_hdf5_files
lstchain_merge_muon_files
lstchain_tune_nsb

lstchain_check_dl1

Script to check the contents of LST DL1 files and associated muon ring files To run it, type e.g.: python lstchain_check_dl1.py –input-file dl1_LST-1.1.Run01881.0000.h5

or, for a whole run: python lstchain_check_dl1.py –input-file “dl1_LST-1.1.Run01881.*.h5”

It produces as output a datacheck_dl1_*.h5 file and a datacheck_dl1_*.pdf file containing data check information. If the input file is a single dl1 file, then the output file names contain the run and subrun index (otherwise, only the run index)

The script can also be run over one file of type datacheck_dl1_*.h5, and then only the plotting part is executed.

The muons_*.fits files which are produced together with the DL1 event files must be available in the same directory as the input files (of whatever type).

Usage

usage: lstchain_check_dl1 [-h] --input-file INPUT_FILE
                          [--output-dir OUTPUT_DIR] [--muons-dir MUONS_DIR]
                          [--max-cores MAX_CORES] [--omit-pdf] [--batch]
                          [--log LOG_FILE]

required arguments

--input-file: Path to DL1 data file(s) (containing pixel-wise charge information and image parameters) OR to datacheck_dl1_*.h5 files (only plotting part is executed in that case)

optional arguments

--output-dir

Directory where the output files will be written

Default: '.'

--muons-dir

Directory where the muon .fits files are located

--max-cores

Maximum number of processes spawned

Default: 4

--omit-pdf

Do NOT create the data check pdf file

Default: False

--batch, -b

Run the script without plotting output

Default: False

--log

Log file name

lstchain_create_run_summary

Create a run summary for a given date containing the number of subruns, the start time of the run, type pf the run: DATA, DRS4, CALI, and the reference timestamp and counter of the run.

Usage

Create run summary file

usage: lstchain_create_run_summary [-h] -d DATE [--r0-path R0_PATH]
                                   [-o OUTPUT_DIR] [--overwrite]
                                   [-t TCU_SERVER]

Named Arguments

-d, --date

Date for the creation of the run summary in format YYYYMMDD

--r0-path

Path to the R0 files. Default is /fefs/aswg/data/real/R0

Default: /fefs/aswg/data/real/R0

-o, --output-dir

Directory in which Run Summary file is written

Default: /fefs/aswg/data/real/monitoring/RunSummary

--overwrite

Overwrite existing Run Summary file

Default: False

-t, --tcu-server

TCU database server

Default: 'lst101-int'

lstchain_data_create_time_calibration_file

Script to create drs4 time correction coefficients

Input: fits.fz file
Output: time_calibration.hdf5 file

Usage: $> python lstchain_data_create_time_calibration_file.py –input-file LST-1.1.Run01625.0000.fits.fz or input-file LST-1.1.Run01625.000*.fits.fz –output-file time_calibration.Run1625.0000.hdf5

Usage

usage: lstchain_data_create_time_calibration_file [-h] --input-file INPUT_FILE
                                                  --output-file OUTPUT_FILE
                                                  [--max-events MAX_EVENTS]
                                                  [--config CONFIG]
                                                  --pedestal-file
                                                  PEDESTAL_FILE
                                                  --run-summary-path
                                                  RUN_SUMMARY_PATH
                                                  [--no-progress]
                                                  [--flatfield-heuristic]
                                                  [--no-flatfield-heuristic]

Named Arguments

--input-file

Path to fits.fz file used to create the time calibration file. Allowed to use regular expression in given path to process subruns

--output-file

Path where script creates the time calibration file

--max-events

Maximum numbers of events to read. Default = 20000

Default: 20000

--config, -c

Path to a configuration file. If none is given, a standard configuration is applied

--pedestal-file, -p

Path to drs4 pedestal file

--run-summary-path

Path to run summary file

--no-progress

Do not display a progress bar during event processing

Default: False

--flatfield-heuristic

If given, try to identify flatfield events from the raw data. Should be used only for data from before 2022

--no-flatfield-heuristic

If given, do not to identify flatfield events from the raw data. Should be used only for data from before 2022

lstchain_data_r0_to_dl1

Pipeline to calibrate and compute image parameters at single telescope level for real data. - Inputs are a protozfits input file and a drs4 pedestal/calibration/time calibration files - Output is a dataframe with dl1 data

Usage:

$> python lstchain_data_r0_to_dl1.py –input-file LST-1.1.Run02030.0000.fits.fz –output-dir ./ –pedestal-file drs4_pedestal.Run2028.0000.h5 –calibration-file calibration.Run2029.0000.hdf5 –time-calibration-file time_calibration.Run2029.0000.hdf5

Usage

R0 to DL1

usage: lstchain_data_r0_to_dl1 [-h] -f INPUT_FILE [-o OUTPUT_DIR]
                               [-p PEDESTAL_FILE]
                               [--calibration-file CALIBRATION_FILE]
                               [--time-calibration-file TIME_CALIBRATION_FILE]
                               [--systematic-correction-file SYSTEMATIC_CORRECTION_FILE]
                               [--config CONFIG_FILE]
                               [--pointing-file POINTING_FILE]
                               [--dragon-reference-time DRAGON_REFERENCE_TIME]
                               [--dragon-reference-counter DRAGON_REFERENCE_COUNTER]
                               [--dragon-module-id DRAGON_MODULE_ID]
                               [-r RUN_SUMMARY_PATH] [--max-events MAX_EVENTS]
                               [--pedestal-ids-path PEDESTAL_IDS_PATH]
                               [--flatfield-heuristic]
                               [--no-flatfield-heuristic]
                               [--default-trigger-type DEFAULT_TRIGGER_TYPE]

Named Arguments

-f, --input-file

Path to the .fits.fz file with the raw events

-o, --output-dir

Path where to store the reco dl1 events

Default: ./dl1_data/

-p, --pedestal-file, -p

Path to a pedestal file

--calibration-file, --calib

Path to a calibration file

--time-calibration-file, -t

Path to a calibration file for pulse time correction

--systematic-correction-file, --systematics

Path to the file with the calibration systematics corrections

--config, -c

Path to a configuration file. If none is given, a standard configuration is applied

--pointing-file, --pointing

Path to the Drive log file with the pointing information.

--dragon-reference-time

UCTS timestamp in nsecs, unix format and TAI scale of the first event of the run with valid timestamp. If none is passed, the start-of-the-run timestamp is provided, hence Dragon timestamp is not reliable.

--dragon-reference-counter

Dragon counter (pps + 10MHz) in nsecs corresponding to the first reliable UCTS of the run. To be provided along with ucts_t0_dragon.

--dragon-module-id

Dragon module to use for the timestamp calculation (pps + 10MHz) to the first reliable UCTS of the run. To be provided along with ucts_t0_dragon.

-r, --run-summary-path

Path to the run summary of the correct night. Used to extract dragon reference values

--max-events

Maximum number of events to be processed.

--pedestal-ids-path

Path to the file containing the event ids of interleaved pedestals

--flatfield-heuristic

If given, try to identify flatfield events from the raw data. Should be used only for data from before 2022

--no-flatfield-heuristic

If given, do not to identify flatfield events from the raw data. Should be used only for data from before 2022

--default-trigger-type

Alias for LSTEventSource.default_trigger_type, can be set to “tib” for runs with UCTS problems.

lstchain_dl1_muon_analysis

Script to perform the analysis of muon events.

Inputs are a DL1a data file (pixel information is needed) and a calibration file
Output is a table with muon parameters (to be updated to a dataframe!)

Usage:

$> python lstchain_muon_analysis_dl1.py –input-file dl1_Run01566_0322.h5 –output-file Data_table.fits –calibration-file calibration.Run2029.0000.hdf5

Usage

usage: lstchain_dl1_muon_analysis [-h] --input-file INPUT_FILE --output-file
                                  OUTPUT_FILE [--calibration-file CALIB_FILE]
                                  [--plot-rings] [--plots-path PLOTS_PATH]
                                  [--max-muons MAX_MUONS]

Named Arguments

--input-file, -f

Path to DL1a data file (containing charge information).

--output-file, -o

Path to create the output fits table with muon parameters

--calibration-file, --calib

Path to corresponding calibration file (containing bad pixel information).

--plot-rings

Plot figures of the stored rings

Default: False

--plots-path

Path to the plots

--max-muons

Maximum number of processed muon ring candidates

lstchain_dl1ab

Read a HDF5 DL1 file, recompute parameters based on calibrated images and pulse times and a config file and write a new HDF5 file Updated parameters are : Hillas paramaters, wl, r, leakage, n_islands, intercept, time_gradient - Input: DL1 data file. - Output: DL1 data file. Usage: $> python lstchain_dl1ab.py –input-file dl1_gamma_20deg_0deg_run8___cta-prod3-lapalma-2147m-LaPalma-FlashCam.simtel.gz

Usage

Recompute DL1b parameters from a DL1a file

usage: lstchain_dl1ab [-h] -f INPUT_FILE -o OUTPUT_FILE
                      [--catB-calibration-file CATB_CALIBRATION_FILE]
                      [--max-unusable-pixels MAX_UNUSABLE_PIXELS]
                      [-c CONFIG_FILE] [--no-image] [--no-pedestal-cleaning]

Named Arguments

-f, --input-file

path to the DL1a file

-o, --output-file

key for the table of new parameters

--catB-calibration-file

path to the Cat-B calibration file

--max-unusable-pixels

Maximum accepted number of unusable pixels. Default: 70 (= 10 modules)

Default: 70

-c, --config

Path to a configuration file. If none is given, a standard configuration is applied

--no-image

Pass this argument to avoid writing the images in the new DL1 files.

Default: False

--no-pedestal-cleaning

Disable pedestal cleaning. This is also done automatically for simulations.

Default: True

lstchain_find_pedestals

This script reads all DL1 subrun files (i.e. not merged) in the input directory, and finds the event_id’s of interleaved pedestals. It writes them out into one hdf5 file for each sub-run, called pedestal_ids_RunXXXXX.YYYY.h5, which contains a single table with one column, ‘event_id’

Usage

Interleaved Pedestal Finder

usage: lstchain_find_pedestals [-h] [-d SRCDIR] output-dir

Positional Arguments

output-dir: Path where to store the output hdf5 files

Named Arguments

-d, --input-dir

path to the directory of the DL1 files

Default: ./

lstchain_longterm_dl1_check

This script reads in LST1 DL1 datacheck files, of those containing info for a whole run, with name pattern datacheck_dl1_LST-1.Run?????.h5. It takes all files in the input directory. It also reads, in the directory indicated ( muons-dir), the corresponding muons*fits files if available.

The output is the file longterm_dl1_check.h5 file (the name can be modified via commandline), which contains tables with some run-wise summary values for plotting long-term evolution of the DL1 data.

It also produces an interactive web page, longterm_dl1_check.html with plots showing the evolution of many such values. If not in batch mode, the page is opened by the default browser at the end of execution.

It also produces a longterm_dl1_check.log file with warnings about values which are beyond certain limits.

Usage

DL1 multi-run data checker

usage: lstchain_longterm_dl1_check [-h] [--input-dir INPUT_DIR]
                                   [--muons-dir MUONS_DIR]
                                   [--output-file OUTPUT_FILE] [--batch]

Named Arguments

--input-dir, -d

path to the datacheck_dl1_LST-1.RunXXXXX.h5 files

Default: ./

--muons-dir

path to the muons_LST-1.RunXXXXX.YYYY.fits files

Default: ./

--output-file, -o

.h5 output file name

Default: ./longterm_dl1_check.h5

--batch, -b

Run the script without opening html output

Default: False

lstchain_mc_r0_to_dl1

Pipeline to calibrate and compute image parameters at single telescope level for MC. - Inputs are simtelarray files. - Output is a dataframe with dl1 data.

Usage:

$> python lstchain_mc_r0_to_dl1.py –input-file gamma_20deg_0deg_run8___cta-prod3-lapalma-2147m-LaPalma-FlashCam.simtel.gz

Usage

R0 to DL1

usage: lstchain_mc_r0_to_dl1 [-h] [--input-file INPUT_FILE]
                             [--output-dir OUTPUT_DIR] [--config CONFIG_FILE]

Named Arguments

--input-file, -f

Path to the simtelarray file

--output-dir, -o

Path where to store the reco dl1 events

Default: ./dl1_data/

--config, -c

Path to a configuration file. If none is given, a standard configuration is applied

lstchain_mc_r0_to_dl2

Pipeline to calibrate and compute image parameters at single telescope level for MC.

Inputs are simtelarray files and trained Random Forests. Output is a dataframe with DL2 data.

Usage:

$> python lstchain_mc_r0_to_dl2.py –input-file gamma_20deg_0deg_run8___cta-prod3-lapalma-2147m-LaPalma-FlashCam.simtel.gz –path-models ./trained_models

Usage

MC Pipeline R0 to DL2.

usage: lstchain_mc_r0_to_dl2 [-h] [--input-file DATAFILE]
                             [--path-models PATH_MODELS] [--no-dl1]
                             [--output-dir OUTPUT_DIR] [--config CONFIG_FILE]

Named Arguments

--input-file, -f

path to the file with simtelarray events

--path-models, -p

Path where to find the trained RF

Default: './trained_models'

--no-dl1

If given, the dl1 file is removed after creating the dl2 output

Default: False

--output-dir, -o

Path where to store the reco dl2 events

Default: ./dl2_data

--config, -c

Path to a configuration file. If none is given, a standard configuration is applied

lstchain_mc_rfperformance

Pipeline to test train three Random Forests destinated to Energy, disp reconstruction and Gamma/Hadron separation and test the performance of Random Forests.

Inputs are DL1 files Outputs are the RF trained models

Usage:

$>python lstchain_mc_rfperformance.py

Usage

Train and Apply Random Forests.

usage: lstchain_mc_rfperformance [-h] [--input-file-gamma-train GAMMAFILE]
                                 [--input-file-proton-train PROTONFILE]
                                 [--input-file-gamma-test GAMMATEST]
                                 [--input-file-proton-test PROTONTEST]
                                 [--no-save-models] [--batch BATCH]
                                 [--output_dir PATH_MODELS]
                                 [--config CONFIG_FILE]

Named Arguments

--input-file-gamma-train, --g-train

path to the dl1 file of gamma events for training

--input-file-proton-train, --p-train

path to the dl1 file of proton events for training

--input-file-gamma-test, --g-test

path to the dl1 file of gamma events for test

--input-file-proton-test, --p-test

path to the dl1 file of proton events for test

--no-save-models

Disable storing trained models

Default: True

--batch, -b

Boolean. True for running it without plotting output

Default: True

--output_dir, -o

Path to store the resulting RF

Default: './saved_models/'

--config, -c

Path to a configuration file. If none is given, a standard configuration is applied

lstchain_mc_sensitivity

Script to compute the LST sensitivity using MC.

Inputs are DL1/DL2 gamma and proton files

Usage:

$> python lstchain_mc_sensitivity.py –gdl2 dl2_gammas.h5 –pdl2 dl2_protons.h5 –o /output/path

Usage

Compute MC sensitivity curve.

usage: lstchain_mc_sensitivity [-h] [--input-file-gamma-dl2 DL2_FILE_G]
                               [--input-file-proton-dl2 DL2_FILE_P]
                               [--input-file-on-dl2 DL2_FILE_ON]
                               [--output_path OUTPUT_PATH]

Named Arguments

--input-file-gamma-dl2, --gdl2: path to reconstructed gammas dl2 file
--input-file-proton-dl2, --pdl2: path to reconstructed protons dl2 file
--input-file-on-dl2, --ondl2: path to reconstructed ON dl2 file
--output_path, --o: path where to save plot images

lstchain_mc_trainpipe

Pipeline to train three Random Forests destinated to Energy, disp reconstruction and Gamma/Hadron separation. The resulting RF models can be stored in files for later use on data.

Inputs are DL1 gamma and proton files Outputs are RF trained models

Usage:

$> python lst-trainpipe –input_file_gamma dl1_gamma_20deg_180deg_cta-prod3-demo-2147m-LaPalma-baseline-mono_off0.4_merge_test.h5 –input-file-proton dl1_proton_20deg_180degcta-prod3-demo-2147m-LaPalma-baseline-mono_merge_test.h5

Usage

Train Random Forests.

usage: lstchain_mc_trainpipe [-h] --input-file-gamma GAMMAFILE
                             --input-file-proton PROTONFILE [--no-save-models]
                             [--output-dir PATH_MODELS] [--config CONFIG_FILE]

Named Arguments

--input-file-gamma, --fg

Path to the dl1 file of gamma events for training

--input-file-proton, --fp

Path to the dl1 file of proton events for training

--no-save-models

Disable storing trained models

Default: True

--output-dir, -o

Path to store the resulting RF

Default: './trained_models/'

--config, -c

Path to a configuration file. If none is given, a standard configuration is applied

lstchain_merge_hdf5_files

Merge all HDF5 files resulting from parallel reconstructions present in a directory. Every dataset in the files must be readable with pandas.

Input: several hdf5 files.
Output single hdf5 file.

Usage:

$> python lstchain_merge_hdf5_files.py –input-dir ./

Usage

Merge HDF5 files

usage: lstchain_merge_hdf5_files [-h] -d INPUT_DIR [-o OUTPUT_FILE]
                                 [--no-image] [-r RUN_NUMBER] [-p PATTERN]
                                 [--no-progress] [--skip-checks]
                                 [--keys-to-copy [KEYS_TO_COPY ...]]

Named Arguments

-d, --input-dir

path to the source directory of files

-o, --output-file

Path of the resulting merged file

Default: 'merge.h5'

--no-image

Do not include images in output file

Default: False

-r, --run-number

Merge files run-wise if a run number is passedotherwise merge all files in the directory

-p, --pattern

Glob pattern to match files

Default: '*.h5'

--no-progress

Do not display the progress bar during event processing

Default: False

--skip-checks

Skip checks when merging files

Default: False

--keys-to-copy

List of duplicated keys to be copied and not to be merged

Default: []

lstchain_merge_muon_files

Merge all fits files resulting from parallel muon reconstructions present in a directory.

Input: several fits files.
Output single fits file.

Usage:

$> python lstchain_merge_muon_files.py –input-dir ./ –run-number 1881

Usage

Merge fits files

usage: lstchain_merge_muon_files [-h] [--input-dir SRCDIR]
                                 [--output-file OUTFILE]
                                 [--run-number RUN_NUMBER] [--pattern PATTERN]

Named Arguments

--input-dir, -d

path to the source directory of files

--output-file, -o

Path of the resulting merged file

Default: 'merge.fits'

--run-number, -r

Merge files run-wise if a run number is passed, otherwise merge all files in the directory

--pattern, -p

Glob pattern to match files

Default: '*.fits'

lstchain_tune_nsb

Usage:

$> python lstchain_tune_nsb.py –config config_file.json (must be the one used in the DL1 production) –input-mc simtel_file.simtel.gz simulation simtel file –input-data dl1_data.h5 real data DL1 file

Calculates the parameters needed to tune the NSB in MC DL1 files (in the DL1ab stage) to the level of NSB in a given data file

Usage

Tune NSB

usage: lstchain_tune_nsb [-h] --config CONFIG --input-mc INPUT_MC --input-data
                         INPUT_DATA [--output-file OUTPUT_FILE] [--overwrite]

Named Arguments

--config

Path to the configuration file for the production (must be the one used for calibration and DL1 creation)

--input-mc

Path to a simtel file of the production (must include the true p.e. images)

--input-data

Path to a data DL1 file of the production (must include DL1a)

--output-file, -o

Path to a output file where to dump the update config

--overwrite

Use to overwrite output-file

Default: False

lstchain_dump_config

Usage

Dump lstchain config in a file.

usage: lstchain_dump_config [-h] [-o OUTPUT_FILE] [--update-with UPDATE_WITH]
                            [--mc] [--src-dep] [--overwrite]

Named Arguments

-o, --output-file

path to the output file containing the config

Default: lstchain_config.json

--update-with

Path to a partial config to update the full config with

--mc

Use to dump a modified standard configuration for Monte-Carlo analysis

Default: False

--src-dep

Use to dump a modified standard configuration for source dependent analysis

Default: False

--overwrite

Overwrite existing output file

Default: False