Changelog#
This page shows the release notes for every released version of the package.
The first entry corresponds always to a future release and gives a glimpse of what can be found in the development version.
Please, check the project Milestones for a detailed roadmap of the next planned releases.
v0.5.0 (April, 2022)#
This release will brings huge improvements to the performance of the pipeline and the software package.
Some of the changes are related to the improvements in the interface to the DIRAC grid. Please, consult that for more details about its latest release.
The performance of the pipeline has been validated successfully on a Prod5b simulation for the Alpha configuration at the North site under specific observational settings. Tests on a similar configuration for the South site are ongoing. From now on, performance results will be showcased at this website.
The benchmarking has become a package module (protopipe.benchmarking) and a new protopipe-BENCHMARK script has been added to the suite of available commands.
The pipeline makes now use of ctapipe v0.11.0 and pyirf v0.5.0. For more details, please consult the merged Pull Requests listed below.
Please, remember that protopipe and its interface are open-source projects and anyone can contribute to it. It is possible to perform full-scale analyses and contribute to the production of CTA performances for different science cases or perform studies that require big amounts of data.
Contributors#
Michele Peresano (@HealthyPear)
Karl Kosack (@kosack)
What is changed since v0.4.0.post1#
Pull-requests containing changes of multiple nature are repeated.
π General features#
Add benchmarking notebook template (#181) @HealthyPear
Add book template material (#174) @HealthyPear
Add new script for benchmarking (#157) @HealthyPear
Update the entire benchmark suite (#166) @HealthyPear
Improve data scripts output and update analysis default settings (#165) @HealthyPear
Improve protopipe.mva and protopipe-MODELS (#164) @HealthyPear
Add image extraction status for TwoPassWindowSum (#163) @HealthyPear
Add support for prod5N files (#167) @HealthyPear
Add I/O functions to simplify benchmarking (#147) @HealthyPear
Apply CALIB_SCALE directly from ctapipe.io.SimtelEventSource (#145) @HealthyPear
Add progress bar to both TRAINING and DL2 production scripts (#150) @HealthyPear
Add option to enable/disable charge integration integration correction (#146) @HealthyPear
Calculate impact parameter always in the tilted frame (#143) @HealthyPear
Add support for calibscale (#140) @HealthyPear
Add optional LST stereo trigger requirement (#131) @HealthyPear
Update to ctapipe 0.11.0 (#136) @HealthyPear
Add configuration option to choose which cleaning to use to get training data (#135) @HealthyPear
Add choice of estimation weigths and standard deviation for RandomForestRegressor models (#134) @HealthyPear
Add concentration (#133) @HealthyPear
Improve models generation (#96) @HealthyPear
Support for ctapipe 0.10.5 (#124) @HealthyPear
Small improvements to modeling script (#114) @HealthyPear
π Pipeline applications#
Add support for prod5N files (#167) @HealthyPear
π GRID support#
Update README and documentation (grid interface) (#152) @HealthyPear
Make progress bar optional and add it also to DL2 script (#154) @HealthyPear
π Bug Fixes#
Add book template material (#174) @HealthyPear
Set effective area HDU name as gammapy wants (#144) @HealthyPear
Update CameraFrame-to-TelescopeFrame transformation and HillasReconstructor (#151) @HealthyPear
Fix showerβs core transformation to TiltedFrame (#160) @HealthyPear
Fix CTAMARS-like energy estimation (#156) @HealthyPear
Make progress bar optional and add it also to DL2 script (#154) @HealthyPear
Output NaN concentrations in case of HillasParameterizationError or similar (#155) @HealthyPear
Fix CALIB_SCALE key in analysis.yaml (#153) @HealthyPear
Ensure that estimated energy is always recorded in linear scale (#141) @HealthyPear
Add pandas to pip installation (#138) @HealthyPear
Fix classifier integration test (#120) @HealthyPear
Bugfixes and improvements to protopipe-MODELS (#122) @HealthyPear
Fix pipeline integration test workflow (#118) @HealthyPear
Fix documentation development version in docs (#112) @HealthyPear
π§° Maintenance#
Update documentation for release 0.5.0 (#175) @HealthyPear
Add Pull Request template (#187) @HealthyPear
Add new script for benchmarking (#157) @HealthyPear
Update README and documentation (grid interface) (#152) @HealthyPear
Fix documentation build (#158) @HealthyPear
Add integration test for DL2-to-DL3 step (#137) @HealthyPear
Add pandas to pip installation (#138) @HealthyPear
Implement DL2 integration tests (#126) @HealthyPear
Add issue templates (#121) @HealthyPear
Fix classifier integration test (#120) @HealthyPear
Improve debugging of integration testing (#119) @HealthyPear
Fix pipeline integration test workflow (#118) @HealthyPear
Setup of pipeline integration testing up to modeling (#116) @HealthyPear
Update training integration tests (#113) @HealthyPear
Fix documentation development version in docs (#112) @HealthyPear
Versioning, packaging and continuous deployment on PyPI (#105) @HealthyPear
Update TRAINING integration tests (#111) @HealthyPear
Update DOIs after latest release (#109) @HealthyPear
0.4.0.post1 (Mar 5th, 2021)#
Summary#
This is a post-release that takes care of project maintenance, so it doesnβt change the performance of the code.
Contributors#
Michele Peresano @HealthyPear
Changes from previous release#
Pull-requests that contain changes belonging to multiple classes are repeated.
π Bug Fixes#
Fix zenodo configuration file and add LICENSE file (#106) @HealthyPear
π§° Maintenance#
Update CHANGELOG (#108) @HealthyPear
Fix zenodo configuration file and add LICENSE file (#106) @HealthyPear
Prepare first upload to PyPI (#107) @HealthyPear
0.4.0 (Feb 22th, 2021)#
Summary#
This release brings many improvements of which the most relevant are summarised here depending on their scope within the pipeline workflow.
Performance-wise, protopipe
caught up with the EventDisplay
and CTAMARS
historical pipelines starting from about 500 GeV onwards.
Below this threshold, even if compatible with requirements, the sensitivity
diverges. The cause seems to be a low-energy effect delimited to
the steps before model training.
All pipeline
upgrade to the API of
ctapipe 0.9.1
documentation also on
readthedocs
and link toZenodo
Continuous Integration is now performed on
GitHub
New benchmarks have been added
Reference analysis and benchmarks results have been updated
Data training
calibration benchmarks need only
ctapipe-stage1-process
write_dl1
has becomedata_training
DL1 parameters and (optionally) images are merged in a single file
DL1 parameters names as in
ctapipe
and they are in degrees (TelescopeFrame
)scale correction with the effective focal length
fixed bugs and wrong behaviors
Modeling and DL2 production
fixed bugs and wrong behaviors
Added missing features to get closer to
CTAMARS
DL3
Contributors#
Michele Peresano @HealthyPear
Gaia Verna (@gaia-verna)
Alice Donini (@adonini)
Changes from previous release#
Pull-requests that contain changes belonging to multiple classes are repeated.
π General features#
Performance using Pyirf (#83) @gaia-verna & @adonini
Towards using Pyirf (#79) @gaia-verna & @adonini
Upgrade of DL2 production (#77) @HealthyPear
Upgrade calibration benchmarks (#59) @HealthyPear
Upgrade of data training (#58) @HealthyPear
π Bug Fixes#
Fix calibration benchmarking settings (#100) @HealthyPear
Fix plot of simulated signal and noise of 2nd pass image extraction (#99) @HealthyPear
Upgrade of DL2 production (#77) @HealthyPear
Upgrade of data training (#58) @HealthyPear
π§° Maintenance#
Fix zenodo configuration file and add LICENSE file (#106) @HealthyPear
Update documentation + general maintenance (#62) @HealthyPear
Use mamba to create virtual enviroment for the CI (#101) @HealthyPear
Upgrade all other notebooks and their docs version (#76) @HealthyPear
Upgrade calibration benchmarks (#59) @HealthyPear
Upgrade of data training (#58) @HealthyPear
Enable CI from GitHub actions (#84) @HealthyPear
0.3.0 (Nov 9th, 2020)#
Summary#
early improvements related to the DL1 comparison against the CTAMARS pipeline
improvements to basic maintenance
a more consistent approach for full-scale analyses
bug fixes
Contributors#
Michele Peresano @HealthyPear
Thierry Stolarczyk (@tstolarczyk)
Gaia Verna (@gaia-verna)
Karl Kosack (@kosack)
Thomas Vuillaume (@vuillaut)
Changes from previous release#
π General features#
Add missing variables in write_dl2 (#66) @HealthyPear
Add missing dl1 parameters (#41) @HealthyPear
Updates on notebooks (#47) @HealthyPear
New plots for calibration benchmarking (#43) @HealthyPear
Double-pass image extractor (#48) @HealthyPear
Notebooks for low-level benchmarking (#42) @HealthyPear
Improved handling of sites, arrays and cameras for all Prod3b simtel productions (#33) @HealthyPear
Change gain selection (#35) @HealthyPear
Changes for adding Cameras beyond LSTCam and NectarCam (#29) @tstolarczyk
π GRID support#
Update configuration files (#74) @HealthyPear
Update documentation for GRID support (#54) @HealthyPear
Rollback for GRID support (#52) @HealthyPear
π Bug Fixes#
Bugfix in Release Drafter workflow file (#71) @HealthyPear
Convert pointing values to float64 at reading time (#68) @HealthyPear
Rollback for GRID support (#52) @HealthyPear
Fix recording of DL1 image and record reconstruction cleaning mask (#46) @gaia-verna
consistent definition of angular separation to the source with config (#39) @vuillaut
Update write_dl1.py (#30) @tstolarczyk
π§° Maintenance#
Update benchmarks and documentation (#75) @HealthyPear
Bugfix in Release Drafter workflow file (#71) @HealthyPear
Add release drafter (#67) @HealthyPear
Add benchmark notebooks for medium and late stages (#55) @HealthyPear
Update documentation for GRID support (#54) @HealthyPear
Updated documentation (#50) @HealthyPear
Implementation of a first unit test (DL1) (#34) @HealthyPear
Updated documentation (Closes #23) (#32) @HealthyPear
Added Travis CI configuration file (#18) @HealthyPear
Update README.md (#28) @tstolarczyk
Added versioning to init.py and setup.py using the manual approach. (#20) @HealthyPear
Update README.md (#21) @tstolarczyk
0.2.1 (Oct 28th, 2019)#
Summary#
Released Oct 28, 2019
1 contributor
1 pull requests
Description
The ctapipe-based cleaning algorithm for the biggest cluster was crashing in case of cleaned images with no surviving pixel clusters.
Contributors:
In alphabetical order by first name:
Michele Peresano
Pull Requests#
(#16) Bugfix: Closes #15 (Michele Peresano)
0.2.0 (Oct 24th, 2019)#
Summary#
Released Oct 24, 2019
3 contributor(s)
7 pull requests
Description
protopipe 0.2 now fully supports the stable release of ctapipe 0.7.0.
The main improvements involve the calibration process (high gain selected by default), the direction reconstruction and new camera-type labels.
Code based on pywi/pywi-cta libraries, relevant for wavelet-based image cleaning, has been removed in favor of ctapipe or made completely optional where needed. Wavelet cleaning is still optional but will need those two libraries to be additionally installed. Tailcut-based cleaning is now faster.
The README has been improved with installation, basic use, and developer instructions.
Dependencies are listed in protopipe_environment.yaml
and have been simplified.
The auxiliary scripts merge_tables.py
and merge.sh
have been added to allow merging of DL1 and DL2 HDF5 tables.
The mars_cleaning_1st_pass
method is now imported from _ctapipe_.
Novel code using the largest cluster of survived pixels
(number_of_islands
and largest_island
methods in the
event_preparer
module) has been hardcoded in _protopipe_ and will
disappear with the next release of _ctapipe_.
Model estimators now load the camera types directly from the analysis .yaml
configuration file.
Contributors:
In alphabetical order by first name:
Alice Donini
Michele Peresano
Thierry Stolarczyk
Pull Requests#
This list is incomplete. Small improvements and bug fixes are not listed here.
The complete list is found here.
0.1.1 (Oct 1st, 2019)#
Summary#
Released Oct 1, 2019
X contributor(s)
X pull request(s)
Description
The write_dl1
and write_dl2
tools can now save an additional file
through the flag --save-images
when applied to a single run.
This file will contain the original and calibrated (after gain selection)
photoelectron images per event.
A new method save_fig
has been introduced in the utils
module,
so that model_diagnostic
can save images also in PNG format.
Additional docstrings and PEP8 formatting have been added throughout the code.
Contributors:
In alphabetical order by first name:
β¦
Pull Requests#
The development of protopipe on GitHub started out directly in the master branch, so there are no pull request we can list here.
0.1.0 (Sep 23th, 2019)#
Summary#
Released Sep 23, 2019
6 contributor(s)
1 pull request(s)
Description
First version of protopipe to be publicly release on GitHub. This version is based on ctapipe 0.6.2 (conda package stable version). Its performance has been shown in a presentation at the CTAC meeting in Lugano 2019.
Contributors:
In alphabetical order by first name:
David Landriu
Julien Lefacheur
Karl Kosack
Michele Peresano
Thomas Vuillaume
Tino Michael
Pull Requests#
(#2) Custom arrays, example configs and aux scripts (M.Peresano)