Datachecks (datachecks)

Introduction

Module containing functions for checking the quality of the LST data.

DL1 data checks

Currently the checks are done at the DL1 level. The DL1 datacheck files are produced by running the following scripts sequentially:

  • lstchain.scripts.lstchain_check_dl1

    This takes as input a DL1 file (including DL1a information, i.e. camera images & times) from a data subrun, e.g.:

    lstchain_check_dl1 --input-file dl1_LST-1.1.Run14619.0000.h5 --output-dir OUTPUT_DIR --omit-pdf

    The script produces a data check file for the subrun, datacheck_dl1_LST-1.Run14619.0000.h5 which contains many quantities that can be used to judge the quality of the data (see class DL1DataCheckContainer)


  • lstchain.scripts.lstchain_check_dl1

    The same script is run again, but now providing as input the subrun-wise datacheck files produced earlier (all subruns of a given run must be provided). It also needs to know where the subrun-wise muons_LST-1.*.fits files (produced in the R0 to DL1 analysis step) which contain the muon ring information are stored (“MUONS_DIR”):

    lstchain_check_dl1 --input-file "datacheck_dl1_LST-1.Run14619.*.h5" --output-dir OUTPUT_DIR --muons-dir MUONS_DIR

    The output is now a data check file for the whole run, datacheck_dl1_LST-1.Run14619.h5 which contains all the information from the subrun-wise files. The script also produces a .pdf file datacheck_dl1_LST-1.Run14619.pdf with various plots of the quantities stored in the DL1DataCheckContainer objects, plus others obtained from the muon ring analysis. Note that the muon ring information is not propagated to the run-wise datacheck files, it is just used for the plotting.


  • lstchain.scripts.lstchain_longterm_dl1_check

    This script merges the run-wise datacheck files of (typically) one night, stored in INPUT_DIR, and produces a single .h5 file for the whole night as output (e.g. DL1_datacheck_20230920.h5). The “longterm” in the script name is a bit of an overstatement - in principle it can be run over data from many nights, but the output html (see below) becomes too heavy and some of the interactive features work poorly.

    lstchain_longterm_dl1_check --input-dir INPUT_DIR --muons-dir MUONS_DIR --output-file DL1_datacheck_20230920.h5 --batch

    The output .h5 file contains a (run-wise) summarized version of the information in the input files, including the muon ring .fits files. It also creates an .html file (e.g. DL1_datacheck_20230920.html) which can be opened with any web browser and which contains various interactive plots which allow to make a quick check of the data of a night. See an example of the .html file here (password protected).


  • lstchain_cherenkov_transparency

    This script analyzes the image intensity histograms (one per subrun) stored in the run-wise datacheck files (which must exist in INPUT_DIR)

    lstchain_cherenkov_transparency --update-datacheck-file DL1_datacheck_20230920.h5 --input-dir INPUT_DIR

    The script updates the night-wise datacheck .h5 file DL1_datacheck_20230920.h5 with a new table (with one entry per subrun) containing parameters related to the image intensity spectra for cosmic ray events (i.e., a Cherenkov-transparency - like approach, see e.g. https://arxiv.org/abs/1310.1639).

Using the datacheck files for selecting good-quality data

The night-wise datacheck .h5 files, DL1_datacheck_YYYYMMDD.h5 can be used to select a subsample of good quality data from a large sample of observations. The files are relatively light, 6 MB per night in average. A large sample of them can be processed with the notebook cta_lstchain/notebooks/data_quality.ipynb (instructions can be found inside the notebook)

Reference/API

lstchain.datachecks.containers Module

Containers for data check

Functions

count_trig_types(array)

Counts the trigger of each type inside array

Classes

DL1DataCheckContainer([prefix])

Container to store the subrun-wise outcome of the DL1 data check

DL1DataCheckHistogramBins([prefix])

Histogram bins for the DL1 Datacheck

lstchain.datachecks.dl1_checker Module

Functions to check the contents of LST DL1 files and associated muon ring files

Functions

check_dl1(filenames, output_path[, ...])

Check DL1 files

merge_dl1datacheck_files(file_list)

Merge DL1 Datacheck files

plot_datacheck(datacheck_filename[, ...])

Plot datacheck

plot_mean_and_stddev(table, camgeom, ...[, ...])

The subrun-wise mean and std dev values are used to calculate the run-wise (i.e. for all processed subruns which appear in the table) counterparts of the same, which are then plotted.

plot_trigger_types(dchecktables, ...)

Plot trigger types

process_dl1_file(filename, bins[, tel_id])

Process DL1 file

write_error_page(tablename, pagesize)

Write error page