New scientific instruments are starting to generate an unprecedented amount of data. The Low Frequency Array (LOFAR), one of the Square Kilometre Array (SKA) pathfinders, is already producing data on a petabyte scale. The calibration of these data presents a huge challenge for final users: (a) extensive storage and computing resources are required; (b) the installation and maintenance of the software required for the processing is not trivial; and (c) the requirements of calibration pipelines, which are experimental and under development, are quickly evolving. After encountering some limitations in classical infrastructures like dedicated clusters, we investigated the viability of cloud infrastructures as a solution. We found that the installation and operation of LOFAR data calibration pipelines is not only possible, but can also be efficient in cloud infrastructures. The main advantages were: (1) the ease of software installation and maintenance, and the availability of standard APIs and tools, widely used in the industry; this reduces the requirement for significant manual intervention, which can have a highly negative impact in some infrastructures; (2) the flexibility to adapt the infrastructure to the needs of the problem, especially as those demands change over time; (3) the on-demand consumption of (shared) resources. We found that a critical factor (also in other infrastructures) is the availability of scratch storage areas of an appropriate size. We found no significant impediments associated with the speed of data transfer, the use of virtualization, the use of external block storage, or the memory available (provided a minimum threshold is reached). Finally, we considered the cost-effectiveness of a commercial cloud like Amazon Web Services. While a cloud solution is more expensive than the operation of a large, fully-utilized cluster completely dedicated to LOFAR data reduction, we found that its costs are competitive if the number of datasets to be analysed is not high, or if the costs of maintaining a system capable of calibrating LOFAR data become high. Coupled with the advantages discussed above, this suggests that a cloud infrastructure may be favourable for many users.

We acknowledge the useful comments of the anonymous referee. We would like to acknowledge the work of all the developers and packagers of the LOFAR software that constitute the core of the processing pipelines (including factor, prefactor, LSMTool, LoSoTo, and the Kern Suite), as well as the useful discussions with the participants in the LOFAR blank fields and direction dependent calibration teleconferences over the years. JS and PNB are grateful for financial support from STFC via grant ST/M001229/1. This work has been also supported by the projects ‘AMIGA5: gas in and around galaxies. Scientific and technological preparation for the SKA’ (AYA2014-52013-C2-1-R) and ‘AMIGA6: gas in and around galaxies. Preparation for SKA science and contribution to the design of the SKA data flow’ (AYA2015-65973-C3-1-R) both of which were co-funded by MICINN and FEDER funds and the Junta de Andalucía (Spain) grants P08-FQM-4205 and TIC-114. We would like to explicitly acknowledge Dr Jose Ruedas – chief of the computer centre and responsible of the computing and communications infrastructures at IAA-CSIC – and Rafael Parra – system administrator of the IAA computing cluster – for their technical assistance. We acknowledge the joint SKA and AWS Astrocompute proposal call that was used to fund all the tests in the AWS infrastructure with the projects “Calibration of LOFAR ELAIS-N1 data in the Amazon cloud” and “Amazon Cloud Processing of LOFAR Tier-1 surveys: Opening up a new window on the Universe”. This work made use of the University of Hertfordshire’s high-performance computing facility and the LOFAR-UK computing facility, supported by STFC [grant number ST/P000096/1]. This work benefited from services and resources provided by the fedcloud.egi.eu Virtual Organization, supported by the national resource providers of the EGI Federation. We acknowledge the resources and support provided by the STFC RAL Cloud infrastructure. LOFAR, the Low Frequency Array designed and constructed by ASTRON, has facilities in several countries, that are owned by various parties (each with their own funding sources), and that are collectively operated by the International LOFAR Telescope (ILT) foundation under a joint scientific policy.

Peer Reviewed

Award-winning