Reusing distributed computing software and patterns for midscale collaborative science

Many of the challenges faced by the LHC experiments (aggregation of distributed computing resources, management of data across multiple storage facilities, integration of experiment-specific workflow management tools across multiple grid services) are similarly experienced by "midscale" high energy physics and astrophysics experiments, particularly as their data set volumes are increasing at comparable rates. Often these (international, multi-institution) collaborations have outgrown the computing resources offered by their home laboratories, or the capacities of any single member institution. Unlike the LHC experiments, however, these collaborations often lack the manpower required to build, integrate and operate the systems required to meet their scale. In the Open Science Grid, we have organized a team designed to support collaborative science organizations re-use proven software and patterns in distributed processing and data management, often but not restricted to software developed for the LHC. Examples are re-use of the Rucio and FTS3 software for reliable data transfer and management, XRootD for data access and caching, Ceph for large scale pre-processing storage, and Pegasus for workflow management across heterogeneous resources. We summarize experience with the VERITAS gamma ray observatory, the South Pole Telescope (CMB detector), and the XENON dark matter search experiment.

Tags
Data and Resources
To access the resources you must log in

This item has no data

Identity

Description: The Identity category includes attributes that support the identification of the resource.

Field Value
PID https://www.doi.org/10.5281/zenodo.3599635
PID https://www.doi.org/10.5281/zenodo.3599634
URL https://zenodo.org/record/3599635
URL http://dx.doi.org/10.5281/zenodo.3599634
URL http://dx.doi.org/10.5281/zenodo.3599635
URL https://figshare.com/articles/Reusing_distributed_computing_software_and_patterns_for_midscale_collaborative_science/11705535
Access Modality

Description: The Access Modality category includes attributes that report the modality of exploitation of the resource.

Field Value
Access Right Open Access
Attribution

Description: Authorships and contributors

Field Value
Author Paschos, Paschalis
Author Rynge, Mats
Author Riedel, Benedikt
Author Wuerthwein, Frank
Author Gardner, Robert William
Publishing

Description: Attributes about the publishing venue (e.g. journal) and deposit location (e.g. repository)

Field Value
Collected From Datacite; figshare; FigShare
Hosted By Zenodo; figshare; FigShare
Publication Date 2019-11-05
Publisher Zenodo
Additional Info
Field Value
Language UNKNOWN
Resource Type Other literature type; Conference object
system:type publication
Management Info
Field Value
Source https://science-innovation-policy.openaire.eu/search/publication?articleId=dedup_wf_001::3a3981ed2cfbc011f50af67d4e019503
Author jsonws_user
Last Updated 26 December 2020, 09:34 (CET)
Created 26 December 2020, 09:34 (CET)