The ESO Data Processing System (EDPS): a unified system for science data processing

Wolfram Freudling

doi:10.1117/12.2630198

25 August 2022 The ESO Data Processing System (EDPS): a unified system for science data processing

Wolfram Freudling

Proceedings Volume 12186, Observatory Operations: Strategies, Processes, and Systems IX; 121860E (2022) https://doi.org/10.1117/12.2630198
Event: SPIE Astronomical Telescopes + Instrumentation, 2022, Montréal, Québec, Canada

Abstract

The ESO Data Processing System (EDPS) is a new software infrastructure to run the European Southern Observatory’s (ESO) science data processing modules (“pipeline recipes”) that is currently in an advanced status of development. These recipes are used for quality control purposes, unsupervised production of science and calibration products for the archive, and are embedded in an interactive data reduction system that is also offered to external science users. EDPS aims to unify all these use cases and thereby replace individual dedicated systems that are currently used at the European Southern Observatory (ESO). All pipelines are based on self-contained recipes with well-defined inputs, outputs, and parameters. They are strung together into different workflows and fed with data selected by pre-defined rules. Even though the dependencies of the recipes imply an execution cascade that is identical for all use cases, the actual workflows and data selection rules for the different use cases are significantly different. EDPS will be a Python based system that generates workflows for ESO’s science data processing use cases from a general specification for the dependencies of the recipes. By starting from a single specification for different workflows, EDPS will improve the reliability and reproducible of data processing. At the same time, EDPS is expected to drastically lower implementation and maintenance costs for the workflows.

Conference Presentation

Citation Download Citation

Wolfram Freudling "The ESO Data Processing System (EDPS): a unified system for science data processing", Proc. SPIE 12186, Observatory Operations: Strategies, Processes, and Systems IX, 121860E (25 August 2022); https://doi.org/10.1117/12.2630198

ACCESS THE FULL ARTICLE

INSTITUTIONAL
Select your institution to access the SPIE Digital Library.

SELECT YOUR INSTITUTION

PERSONAL
Sign in with your SPIE account to access your personal subscriptions or to use specific features such as save to my library, sign up for alerts, save searches, etc.

PERSONAL SIGN IN

No SPIE Account? Create one

PURCHASE THIS CONTENT

SUBSCRIBE TO DIGITAL LIBRARY

50 downloads per 1-year subscription

Members: $195

Non-members: $335 ADD TO CART

25 downloads per 1 - year subscription

Members: $145

Non-members: $250 ADD TO CART

PURCHASE SINGLE ARTICLE

Includes PDF, HTML & Video, when available