Showing 1–50 of 54 results for author: Berriman, G B

Integration of Data Reduction and Near Real-Time Archiving into the Keck Observing Model

Authors: Max Brodheim, John O'Meara, Jeffrey A. Mader, G. Bruce Berriman, Matthew Brown, Lucas Furhman, Tyler Tucker, Christopher R. Gelino, Meca S. Lynn, Melanie A. Swain

Abstract: The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory's Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw… ▽ More The W. M. Keck Observatory is welcoming a new era where data reduction and archiving are tightly integrated into our observing model, under the auspices of the Observatory's Data Services Initiative (DSI) project. While previously the Keck Observatory Archive (KOA) archived minimally processed, raw science data the day after observing, Keck is transitioning to a model in which it archives both raw frames and reduced data in near real-time. These data will be made available to observers and collaborators immediately upon ingestion through a dedicated new interface that will support collaboration and sharing among teams, as well as stream data directly to personal computers without access to WMKO's internal networks. Both the raw and science-ready data products will be made publicly available upon the expiration of data protections. △ Less

Submitted 6 June, 2024; originally announced June 2024.

Comments: 10 pages, 5 figures, SPIE Astronomical Telescopes + Instrumentation 2022

Journal ref: Proc. SPIE 12186, Observatory Operations: Strategies, Processes, and Systems IX, 121860H (25 August 2022)

The Observers' Data Access Portal at the Keck Observatory Archive

Authors: T. Oluyide, M. S. Lynn, T. Coda, G. B. Berriman, M. Brown, L. Fuhrman, C. Gelino, J. Good, J. Hayashi, C. -H. Lee, J. Mader, M. A. Swain

Abstract: For all active instruments, the Keck Observatory Archive (KOA) now ingests raw data from the Keck Telescopes within 1 minute of acquisition, quick-look reduced data within 5 minutes of creation, and science ready reduced data for four instruments as they are created by their automated pipelines. On August 1, 2023, KOA released the Observers Data Access Portal (ODAP), which enables observers at the… ▽ More For all active instruments, the Keck Observatory Archive (KOA) now ingests raw data from the Keck Telescopes within 1 minute of acquisition, quick-look reduced data within 5 minutes of creation, and science ready reduced data for four instruments as they are created by their automated pipelines. On August 1, 2023, KOA released the Observers Data Access Portal (ODAP), which enables observers at the telescope and their collaborators anywhere in the world to securely monitor and download science, calibration, and quick-look data as they are ingested into the archive. The portal is built using Python Socket IO.WebSockets that ensure metadata appear in the portal as the data themselves are ingested. The portal itself is a dynamic web interface built with React. It enables users to view and customize metadata fields, filter metadata according to data type, and download data as they are ingested or in bulk through wget scripts. Observers have used the ODAP since its release and have provided feedback that will guide future releases. △ Less

Submitted 6 February, 2024; originally announced February 2024.

Comments: 4 pages, 2 figs, ADASS 2024 proceedings

Processing All-Sky Images At Scale On The Amazon Cloud: A HiPS Example

Authors: G. Bruce Berriman, John C. Good

Abstract: We report here on a project that has developed a practical approach to processing all-sky image collections on cloud platforms, using as an exemplar application the creation of three-color Hierarchical Progressive Survey (HiPS) maps of the 2MASS data set with the Montage Image Mosaic Engine on Amazon Web Services. We will emphasize issues that must be considered by scientists wishing to use cloud… ▽ More We report here on a project that has developed a practical approach to processing all-sky image collections on cloud platforms, using as an exemplar application the creation of three-color Hierarchical Progressive Survey (HiPS) maps of the 2MASS data set with the Montage Image Mosaic Engine on Amazon Web Services. We will emphasize issues that must be considered by scientists wishing to use cloud platforms to perform such parallel processing, so providing a guide for scientists wishing to exploit cloud platforms for similar large-scale processing. A HiPS map is based on the HEALPix sky-tiling scheme. Progressive zooming of a HiPS map reveals an image sampled at ever smaller or larger spatial scales that are defined by the HEALPix standard. Briefly, the approach used by Montage involves creating a base mosaic at the lowest required HEALPix level, usually chosen to match as closely as possible the spatial sampling of the input images, then cutting out the HiPS cells in PNG format from this mosaic. The process is repeated at successive HEALPix levels to create a nested collection of FITS files, from which PNG files are created that are shown in HiPS viewers. Stretching FITS files to produce PNGs is based on an image histogram. For composite regions (up and including the whole sky), the histograms for each tile can be combined to create a composite histogram for the region. Using this single histogram for each of the individual FITS files means all the PNGs are on the same brightness scale and displaying them side by side in a HiPS viewer produces a continuous uniform map across the entire sky. △ Less

Submitted 6 February, 2024; originally announced February 2024.

Comments: 4 pages, 1 figure, ADASS 2024 proceedings

Real-time Data Ingestion at the Keck Observatory Archive (KOA)

Authors: G. Bruce Berriman, M. Brodheim, M. Brown, L. Fuhrman, C. R. Gelino, M. Kong, C. -H. Lee, M. S. Lynn, J. Mader, T. Oluyide, M. A. Swain, T. Tucker, A. Laity, J. Riley

Abstract: Since February of this year, KOA began to prepare, transfer, and ingest data as they were acquired in near-real time; in most cases data are available to observers through KOA within one minute of acquisition. Real-time ingestion will be complete for all active instruments by the end of Summer 2022. The observatory is supporting the development of modern Python data reduction pipelines, which when… ▽ More Since February of this year, KOA began to prepare, transfer, and ingest data as they were acquired in near-real time; in most cases data are available to observers through KOA within one minute of acquisition. Real-time ingestion will be complete for all active instruments by the end of Summer 2022. The observatory is supporting the development of modern Python data reduction pipelines, which when delivered, will automatically create science-ready data sets at the end of each night for ingestion into the archive. This presentation will describe the infrastructure developed to support real-time data ingestion, itself part of a larger initiative at the Observatory to modernize end-to-end operations. During telescope operations, the software at WMKO is executed automatically when a newly acquired file is recognized through monitoring a keyword-based observatory control system; this system is used at Keck to execute virtually all observatory functions. The monitor uses callbacks built into the control system to begin data preparation of files for transmission to the archive on an individual basis: scheduling scripts or file system related triggers are unnecessary. An HTTP-based system called from the Flask micro-framework enables file transfers between WMKO and NExScI and triggers data ingestion at NExScI. The ingestion system at NEXScI is a compact (4 KLOC), highly fault-tolerant, Python-based system. It uses a shared file system to transfer data from WMKO to NExScI. The ingestion code is instrument agnostic, with instrument parameters read from configuration files. It replaces an unwieldy (50 KLOC) C-based system that had been in use since 2004. △ Less

Submitted 5 December, 2022; originally announced December 2022.

Comments: 4 pages, 3 figures

Astronomical Image Processing at Scale With Pegasus and Montage

Authors: G. Bruce Berriman, John C. Good, Ewa Deelman, Ryan Tanaka, Karan Vahi

Abstract: Image processing at scale is a powerful tool for creating new data sets and integrating them with existing data sets and performing analysis and quality assurance investigations. Workflow managers offer advantages in this type of processing, which involves multiple data access and processing steps. Generally, they enable automation of the workflow by locating data and resources, recovery from fail… ▽ More Image processing at scale is a powerful tool for creating new data sets and integrating them with existing data sets and performing analysis and quality assurance investigations. Workflow managers offer advantages in this type of processing, which involves multiple data access and processing steps. Generally, they enable automation of the workflow by locating data and resources, recovery from failures, and monitoring of performance. In this focus demo we demonstrate how the Pegasus Workflow Manager Python API manages image processing to create mosaics with the Montage Image Mosaic engine. Since 2001, Pegasus has been developed and maintained at USC/ISI. Montage was in fact one of the first applications used to design Pegasus and optimize its performance. Pegasus has since found application in many areas of science. LIGO exploited it in making discoveries of black holes. The Vera C. Rubin Observatory used it to compare the cost and performance of processing images on cloud platforms. While these are examples of projects at large scale, small team investigations on local clusters of machines can benefit from Pegasus as well. △ Less

Submitted 22 November, 2021; originally announced November 2021.

Comments: 4 pages, 1 figure, Proceedings of ADASS XXXI

Best Practices for Data Publication in the Astronomical Literature

Authors: Tracy X. Chen, Marion Schmitz, Joseph M. Mazzarella, Xiuqin Wu, Julian C. van Eyken, Alberto Accomazzi, Rachel L. Akeson, Mark Allen, Rachael Beaton, G. Bruce Berriman, Andrew W. Boyle, Marianne Brouty, Ben Chan, Jessie L. Christiansen, David R. Ciardi, David Cook, Raffaele D'Abrusco, Rick Ebert, Cren Frayer, Benjamin J. Fulton, Christopher Gelino, George Helou, Calen B. Henderson, Justin Howell, Joyce Kim , et al. (20 additional authors not shown)

Abstract: We present an overview of best practices for publishing data in astronomy and astrophysics journals. These recommendations are intended as a reference for authors to help prepare and publish data in a way that will better represent and support science results, enable better data sharing, improve reproducibility, and enhance the reusability of data. Observance of these guidelines will also help to… ▽ More We present an overview of best practices for publishing data in astronomy and astrophysics journals. These recommendations are intended as a reference for authors to help prepare and publish data in a way that will better represent and support science results, enable better data sharing, improve reproducibility, and enhance the reusability of data. Observance of these guidelines will also help to streamline the extraction, preservation, integration and cross-linking of valuable data from astrophysics literature into major astronomical databases, and consequently facilitate new modes of science discovery that will better exploit the vast quantities of panchromatic and multi-dimensional data associated with the literature. We encourage authors, journal editors, referees, and publishers to implement the best practices reviewed here, as well as related recommendations from international astronomical organizations such as the International Astronomical Union (IAU) for publication of nomenclature, data, and metadata. A convenient Checklist of Recommendations for Publishing Data in the Literature is included for authors to consult before the submission of the final version of their journal articles and associated data files. We recommend that publishers of journals in astronomy and astrophysics incorporate a link to this document in their Instructions to Authors. △ Less

Submitted 16 April, 2022; v1 submitted 2 June, 2021; originally announced June 2021.

Comments: 19 pages, 1 figure, 3 tables, accepted for publication in ApJS

Cost Management on Commercial Cloud Platforms

Authors: G. Bruce Berriman, William O'Mullane, Arik Mitschang, Ivelina Momcheva

Abstract: Commercial cloud platforms are a powerful technology for astronomical research. Despite the benefits of cloud computing -- such as on-demand scalability and reduction of systems management overhead -- confusion over how to manage costs remains, for many, one of the biggest barriers to entry. This confusion is exacerbated by the rapid growth in services offered by commercial providers, by the growt… ▽ More Commercial cloud platforms are a powerful technology for astronomical research. Despite the benefits of cloud computing -- such as on-demand scalability and reduction of systems management overhead -- confusion over how to manage costs remains, for many, one of the biggest barriers to entry. This confusion is exacerbated by the rapid growth in services offered by commercial providers, by the growth in the number of these providers, and by storage, compute, and I/O metered at separate rates -- all of which can change without notice. As a rule, processing is very cheap, storage is more expensive, and downloading is very expensive. Thus, an application that produces large image data sets for download will be far more expensive than an application that performs extensive processing on a small data set. This Birds of a Feather (BoF) session aimed to quantify the above statement by presenting case studies of the costing of astronomy applications on commercial clouds that covered a range of processing scenarios; these presentations were the basis for discussion by the attendees. △ Less

Submitted 10 December, 2020; originally announced December 2020.

Comments: 4 pages; conference proceedings

The International Virtual Observatory Alliance (IVOA) in 2020

Authors: G. Bruce Berriman, The IVOA Executive Committee, The IVOA Technical Coordination Group, The IVOA Community

Abstract: The International Virtual Observatory Alliance (IVOA) develops the technical standards needed for seamless discovery of and access to astronomy data worldwide, according to the Findable, Accessible, Interoperable and Reusable (FAIR) principles, with the goal of realizing the Virtual Observatory (VO). There are 21 member organizations. The Netherlands VO applied for membership in 2020. Astronomical… ▽ More The International Virtual Observatory Alliance (IVOA) develops the technical standards needed for seamless discovery of and access to astronomy data worldwide, according to the Findable, Accessible, Interoperable and Reusable (FAIR) principles, with the goal of realizing the Virtual Observatory (VO). There are 21 member organizations. The Netherlands VO applied for membership in 2020. Astronomical communities from other nations have shown interest in joining the IVOA. This paper describes the activities of the IVOA in 2020 and summarizes the May and November 2020 virtual "interoperability meetings." The May meeting was the first to be held online and the first to have over 200 registrants. △ Less

Submitted 10 December, 2020; originally announced December 2020.

Comments: 4 pages; conference proceedings

Creating High Quality All-Sky Visualizations of Astronomy Image Data Sets: HiPS and Montage

Authors: G. Bruce Berriman, John C. Good, Vandana Desai, Steven L. Groom

Abstract: We describe a case study to use the Montage image mosaic engine to create maps of the ALLWISE image data set in the Hierarchical Progressive Survey (HiPS) sky-tesselation scheme. Our approach demonstrates that Montage reveals the science content of infrared images in greater detail than has hitherto been possible in HiPS maps. The approach exploits two unique (to our knowledge) characteristics of… ▽ More We describe a case study to use the Montage image mosaic engine to create maps of the ALLWISE image data set in the Hierarchical Progressive Survey (HiPS) sky-tesselation scheme. Our approach demonstrates that Montage reveals the science content of infrared images in greater detail than has hitherto been possible in HiPS maps. The approach exploits two unique (to our knowledge) characteristics of the Montage image mosaic engine: background modeling to rectify the time variable image backgrounds to common levels; and an adaptive image stretch to present images for visualization. The creation of the maps is supported by the development of four new tools that when fully tested will become part of the Montage distribution. The compute intensive part of the processing lies in the reprojection of the images, and we show how we optimized the processing for efficient creation of mosaics that are used in turn to create maps in the HiPS tiling scheme. We plan to apply our methodology to infrared image data sets such a those delivered by Spitzer, 2MASS, IRAS and Planck. △ Less

Submitted 28 January, 2020; originally announced January 2020.

Comments: 4 pages, 2 figures. Submitted to the Proceedings of ADASS2019, Groningen, NL. (Oct 6-10. 2019)

The International Virtual Observatory Alliance in 2019

Authors: Mark A. Allen, Patrick Dowler, Janet D. Evans, Chenzhou Cui, Tim Jenness, Bruno Merin, G. Bruce Berriman, J. J. Kavelaars

Abstract: The International Virtual Observatory Alliance (IVOA) held its bi-annual Interoperability Meetings in May 2019, and in October 2019 following the ADASS XXIX conference. We provide a brief report on the status of the IVOA and the activities of the Interoperability Meetings. The International Virtual Observatory Alliance (IVOA) held its bi-annual Interoperability Meetings in May 2019, and in October 2019 following the ADASS XXIX conference. We provide a brief report on the status of the IVOA and the activities of the Interoperability Meetings. △ Less

Submitted 3 December, 2019; originally announced December 2019.

Comments: 4 pages, submitted to proceedings of ADASS XXIV to be published in ASP Conf. Series

Joint Survey Processing of LSST, Euclid and WFIRST: Enabling a broad array of astrophysics and cosmology through pixel level combinations of datasets

Authors: R. Chary, G. Brammer, P. Capak, W. Dawson, A. Faisst, S. Fajardo-Acosta, H. C. Ferguson, C. J. Grillmair, S. Hemmati, A. Koekemoer, B. Lee, R. Lupton, S. Malhotra, P. Melchior, I. Momcheva, J. Newman, J. Masiero, R. Paladini, A. Prakash, J. Rhodes, B. Rusholme, M. Schneider, N. Stickley, A. Smith, W. M. Wood-Vasey , et al. (1 additional authors not shown)

Abstract: Joint survey processing (JSP) is the pixel level combination of LSST, Euclid, and WFIRST datasets. By combining the high spatial resolution of the space-based datasets with deep, seeing-limited, ground-based images in the optical bands, systematics like source confusion and astrometric mismatch can be addressed to derive the highest precision optical/infrared photometric catalogs. This white paper… ▽ More Joint survey processing (JSP) is the pixel level combination of LSST, Euclid, and WFIRST datasets. By combining the high spatial resolution of the space-based datasets with deep, seeing-limited, ground-based images in the optical bands, systematics like source confusion and astrometric mismatch can be addressed to derive the highest precision optical/infrared photometric catalogs. This white paper highlights the scientific motivation, computational and algorithmic needs to build joint pixel level processing capabilities, which the individual projects by themselves will not be able to support. Through this white paper, we request that the Astro2020 decadal committee recognize the JSP effort as a multi-agency project with the natural outcome being a collaborative effort among groups which are normally supported by a single agency. JSP will allow the U.S. (and international) astronomical community to manipulate the flagship data sets and undertake innovative science investigations ranging from solar system object characterization, exoplanet detections, nearby galaxy rotation rates and dark matter properties, to epoch of reionization studies. It will also result in the ultimate constraints on cosmological parameters and the nature of dark energy, with far smaller uncertainties and a better handle on systematics than by any one survey alone. △ Less

Submitted 2 October, 2019; originally announced October 2019.

Comments: Astro2020 APC White Paper, 11 pages

Image Processing in Python With Montage

Authors: John Good, G. Bruce Berriman

Abstract: The Montage image mosaic engine has found wide applicability in astronomy research, integration into processing environments, and is an examplar application for the development of advanced cyber-infrastructure. It is written in C to provide performance and portability. Linking C/C++ libraries to the Python kernel at run time as binary extensions allows them to run under Python at compiled speeds a… ▽ More The Montage image mosaic engine has found wide applicability in astronomy research, integration into processing environments, and is an examplar application for the development of advanced cyber-infrastructure. It is written in C to provide performance and portability. Linking C/C++ libraries to the Python kernel at run time as binary extensions allows them to run under Python at compiled speeds and enables users to take advantage of all the functionality in Python. We have built Python binary extensions of the 59 ANSI-C modules that make up version 5 of the Montage toolkit. This has involved a turning the code into a C library, with driver code fully separated to reproduce the calling sequence of the command-line tools; and then adding Python and C linkage code with the Cython library, which acts as a bridge between general C libraries and the Python interface. We will demonstrate how to use these Python binary extensions to perform image processing, including reprojecting and resampling images, rectifying background emission to a common level, creation of image mosaics that preserve the calibration and astrometric fidelity of the input images, creating visualizations with an adaptive stretch algorithm, processing HEALPix images, and analyzing and managing image metadata. △ Less

Submitted 26 August, 2019; originally announced August 2019.

Comments: 4 pages, 1 figure. Submitted to Proceedings of ADASS XXVIII

Increasing the Discovery Space in Astrophysics - A Collation of Six Submitted White Papers

Authors: G. Fabbiano, M. Elvis, A. Accomazzi, G. B. Berriman, N. Brickhouse, S. Bose, D. Carrera, I. Chilingarian, F. Civano, B. Czerny, R. D'Abrusco, B. Diemer, J. Drake, R. Emami Meibody, J. R. Farah, G. G. Fazio, E. Feigelson, F. Fornasini, Jay Gallagher, J. Grindlay, L. Hernquist, D. J. James, M. Karovska, V. Kashyap, D. -W. Kim , et al. (24 additional authors not shown)

Abstract: We write in response to the call from the 2020 Decadal Survey to submit white papers illustrating the most pressing scientific questions in astrophysics for the coming decade. We propose exploration as the central question for the Decadal Committee's discussions.The history of astronomy shows that paradigm changing discoveries are not driven by well formulated scientific questions, based on the kn… ▽ More We write in response to the call from the 2020 Decadal Survey to submit white papers illustrating the most pressing scientific questions in astrophysics for the coming decade. We propose exploration as the central question for the Decadal Committee's discussions.The history of astronomy shows that paradigm changing discoveries are not driven by well formulated scientific questions, based on the knowledge of the time. They were instead the result of the increase in discovery space fostered by new telescopes and instruments. An additional tool for increasing the discovery space is provided by the analysis and mining of the increasingly larger amount of archival data available to astronomers. Revolutionary observing facilities, and the state of the art astronomy archives needed to support these facilities, will open up the universe to new discovery. Here we focus on exploration for compact objects and multi messenger science. This white paper includes science examples of the power of the discovery approach, encompassing all the areas of astrophysics covered by the 2020 Decadal Survey. △ Less

Submitted 18 March, 2019; v1 submitted 15 March, 2019; originally announced March 2019.

Deep Learning for Multi-Messenger Astrophysics: A Gateway for Discovery in the Big Data Era

Authors: Gabrielle Allen, Igor Andreoni, Etienne Bachelet, G. Bruce Berriman, Federica B. Bianco, Rahul Biswas, Matias Carrasco Kind, Kyle Chard, Minsik Cho, Philip S. Cowperthwaite, Zachariah B. Etienne, Daniel George, Tom Gibbs, Matthew Graham, William Gropp, Anushri Gupta, Roland Haas, E. A. Huerta, Elise Jennings, Daniel S. Katz, Asad Khan, Volodymyr Kindratenko, William T. C. Kramer, Xin Liu, Ashish Mahabal , et al. (23 additional authors not shown)

Abstract: This report provides an overview of recent work that harnesses the Big Data Revolution and Large Scale Computing to address grand computational challenges in Multi-Messenger Astrophysics, with a particular emphasis on real-time discovery campaigns. Acknowledging the transdisciplinary nature of Multi-Messenger Astrophysics, this document has been prepared by members of the physics, astronomy, compu… ▽ More This report provides an overview of recent work that harnesses the Big Data Revolution and Large Scale Computing to address grand computational challenges in Multi-Messenger Astrophysics, with a particular emphasis on real-time discovery campaigns. Acknowledging the transdisciplinary nature of Multi-Messenger Astrophysics, this document has been prepared by members of the physics, astronomy, computer science, data science, software and cyberinfrastructure communities who attended the NSF-, DOE- and NVIDIA-funded "Deep Learning for Multi-Messenger Astrophysics: Real-time Discovery at Scale" workshop, hosted at the National Center for Supercomputing Applications, October 17-19, 2018. Highlights of this report include unanimous agreement that it is critical to accelerate the development and deployment of novel, signal-processing algorithms that use the synergy between artificial intelligence (AI) and high performance computing to maximize the potential for scientific discovery with Multi-Messenger Astrophysics. We discuss key aspects to realize this endeavor, namely (i) the design and exploitation of scalable and computationally efficient AI algorithms for Multi-Messenger Astrophysics; (ii) cyberinfrastructure requirements to numerically simulate astrophysical sources, and to process and interpret Multi-Messenger Astrophysics data; (iii) management of gravitational wave detections and triggers to enable electromagnetic and astro-particle follow-ups; (iv) a vision to harness future developments of machine and deep learning and cyberinfrastructure resources to cope with the scale of discovery in the Big Data Era; (v) and the need to build a community that brings domain experts together with data scientists on equal footing to maximize and accelerate discovery in the nascent field of Multi-Messenger Astrophysics. △ Less

Submitted 1 February, 2019; originally announced February 2019.

Comments: 15 pages, no figures. White paper based on the "Deep Learning for Multi-Messenger Astrophysics: Real-time Discovery at Scale" workshop, hosted at NCSA, October 17-19, 2018 http://www.ncsa.illinois.edu/Conferences/DeepLearningLSST/

A Study of the Efficiency of Spatial Indexing Methods Applied to Large Astronomical Databases

Authors: G. B. Berriman, J. C. Good, B. Shiao, T. Donaldson

Abstract: We report the results of a study to compare the performance of two common database indexing methods, HTM and HEALPix, on Solaris and Windows database servers installed with PostgreSQL, and a Windows Server installed with MS SQL Server. The indexing was applied to the 2MASS All-Sky Catalog and to the Hubble Source Catalog, which approximate the diversity of catalogs common in astronomy. On each ser… ▽ More We report the results of a study to compare the performance of two common database indexing methods, HTM and HEALPix, on Solaris and Windows database servers installed with PostgreSQL, and a Windows Server installed with MS SQL Server. The indexing was applied to the 2MASS All-Sky Catalog and to the Hubble Source Catalog, which approximate the diversity of catalogs common in astronomy. On each server, the study compared indexing performance by submitting 1 million queries at each index level with random sky positions and random cone search radius, which was computed on a logarithmic scale between 1 arcsec and 1 degree, and measuring the time to complete the query and write the output. These simulated queries, intended to model realistic use patterns, were run in a uniform way on many combinations of indexing method and indexing depth. The query times in all simulations are strongly I/O-bound and are linear with number of records returned for large numbers of sources. There are, however, considerable differences between simulations, which reveal that hardware I/O throughput is a more important factor in managing the performance of a DBMS than the choice of indexing scheme. The choice of index itself is relatively unimportant: for comparable index levels, the performance is consistent within the scatter of the timings. At small index levels (large cells; e.g. level 4; cell size 3.7 deg), there is large scatter in the timings because of wide variations in the number of sources found in the cells. At larger index levels, performance improves and scatter decreases, but the improvement at level 8 (14 arcmin) and higher is masked to some extent in the timing scatter caused by the range of query sizes. At very high levels (20; 0.0004 arsec), the granularity of the cells becomes so high that a large number of extraneous empty cells begin to degrade performance. △ Less

Submitted 22 June, 2018; originally announced June 2018.

Comments: 4 pages, 4 figures. Submitted for publication in the Proceedings of ADASS XXVII, Santiago, Chile (2017)

Sustaining the Montage Image Mosaic Engine Since 2002

Authors: G. Bruce Berriman, John C. Good

Abstract: This paper describes how we have sustained the Montage image mosaic engine (http://montage.ipac.caltech.edu) first released in 2002, to support the ever-growing scale and complexity of modern data sets. The key to its longevity has been its design as a toolkit written in ANSI-C, with each tool performing one distinct task, for easy integration into scripts, pipelines and workflows. The same code b… ▽ More This paper describes how we have sustained the Montage image mosaic engine (http://montage.ipac.caltech.edu) first released in 2002, to support the ever-growing scale and complexity of modern data sets. The key to its longevity has been its design as a toolkit written in ANSI-C, with each tool performing one distinct task, for easy integration into scripts, pipelines and workflows. The same code base now supports Windows, JavaScript and Python by taking advantage of recent advances in compilers. The design has led to applicability of Montage far beyond what was anticipated when Montage was first built, such as supporting observation planning for the JWST. Moreover, Montage is highly scalable and is in wide use within the IT community to develop advanced, fault-tolerant cyber-infrastructure, such as job schedulers for grids, workflow orchestration, and restructuring techniques for processing complex workflows and pipelines. △ Less

Submitted 11 June, 2018; originally announced June 2018.

Comments: 12 pages, 8 figures. Software and Cyberinfrastructure for Astronomy V (Conference 10707), SPIE SPIE Astronomical Telescopes + Instrumentation, Austin TX. June 10-15, 2018

The KELT Follow-Up Network and Transit False Positive Catalog: Pre-vetted False Positives for TESS

Authors: Karen A. Collins, Kevin I. Collins, Joshua Pepper, Jonathan Labadie-Bartz, Keivan Stassun, B. Scott Gaudi, Daniel Bayliss, Joao Bento, Knicole D. Colón, Dax Feliz, David James, Marshall C. Johnson, Rudolf B. Kuhn, Michael B. Lund, Matthew T. Penny, Joseph E. Rodriguez, Robert J. Siverd, Daniel J. Stevens, Xinyu Yao, George Zhou, Mundra Akshay, Giulio F. Aldi, Cliff Ashcraft, Supachai Awiphan, Özgür Baştürk , et al. (86 additional authors not shown)

Abstract: The Kilodegree Extremely Little Telescope (KELT) project has been conducting a photometric survey for transiting planets orbiting bright stars for over ten years. The KELT images have a pixel scale of ~23"/pixel---very similar to that of NASA's Transiting Exoplanet Survey Satellite (TESS)---as well as a large point spread function, and the KELT reduction pipeline uses a weighted photometric apertu… ▽ More The Kilodegree Extremely Little Telescope (KELT) project has been conducting a photometric survey for transiting planets orbiting bright stars for over ten years. The KELT images have a pixel scale of ~23"/pixel---very similar to that of NASA's Transiting Exoplanet Survey Satellite (TESS)---as well as a large point spread function, and the KELT reduction pipeline uses a weighted photometric aperture with radius 3'. At this angular scale, multiple stars are typically blended in the photometric apertures. In order to identify false positives and confirm transiting exoplanets, we have assembled a follow-up network (KELT-FUN) to conduct imaging with higher spatial resolution, cadence, and photometric precision than the KELT telescopes, as well as spectroscopic observations of the candidate host stars. The KELT-FUN team has followed-up over 1,600 planet candidates since 2011, resulting in more than 20 planet discoveries. Excluding ~450 false alarms of non-astrophysical origin (i.e., instrumental noise or systematics), we present an all-sky catalog of the 1,128 bright stars (6<V<10) that show transit-like features in the KELT light curves, but which were subsequently determined to be astrophysical false positives (FPs) after photometric and/or spectroscopic follow-up observations. The KELT-FUN team continues to pursue KELT and other planet candidates and will eventually follow up certain classes of TESS candidates. The KELT FP catalog will help minimize the duplication of follow-up observations by current and future transit surveys such as TESS. △ Less

Submitted 19 September, 2018; v1 submitted 5 March, 2018; originally announced March 2018.

Comments: Accepted for publication in AJ, 21 pages, 12 figures, 7 tables

The Astrophysics Source Code Library: What's new, what's coming

Authors: Alice Allen, G. Bruce Berriman, Kimberly DuPrie, Jessica Mink, Robert Nemiroff, P. Wesley Ryan, Judy Schmidt, Lior Shamir, Keith Shortridge, Mark Taylor, Peter Teuben, John Wallin, Rein H. Warmels

Abstract: The Astrophysics Source Code Library (ASCL, ascl.net), established in 1999, is a citable online registry of source codes used in research that are available for download; the ASCL's main purpose is to improve the transparency, reproducibility, and falsifiability of research. In 2017, improvements to the resource included real-time data backup for submissions and newly-published entries, improved c… ▽ More The Astrophysics Source Code Library (ASCL, ascl.net), established in 1999, is a citable online registry of source codes used in research that are available for download; the ASCL's main purpose is to improve the transparency, reproducibility, and falsifiability of research. In 2017, improvements to the resource included real-time data backup for submissions and newly-published entries, improved cross-matching of research papers with software entries in ADS, and expansion of preferred citation information for the software in the ASCL. △ Less

Submitted 8 December, 2017; originally announced December 2017.

Comments: 4 pages; to be published in proceedings of the October 22-26 2017 ADASS XXVII meeting in Santiago, Chile

Software metadata: How much is enough?

Authors: Alice Allen, Peter Teuben, G. Bruce Berriman, Kimberly DuPrie, Keith Shortridge, Rein Warmels

Abstract: Broad efforts are underway to capture metadata about research software and retain it across services; notable in this regard is the CodeMeta project. What metadata are important to have about (research) software? What metadata are useful for searching for codes? What would you like to learn about astronomy software? This BoF sought to gather information on metadata most desired by researchers and… ▽ More Broad efforts are underway to capture metadata about research software and retain it across services; notable in this regard is the CodeMeta project. What metadata are important to have about (research) software? What metadata are useful for searching for codes? What would you like to learn about astronomy software? This BoF sought to gather information on metadata most desired by researchers and users of astro software and others interested in registering, indexing, capturing, and doing research on this software. Information from this BoF could conceivably result in changes to the Astrophysics Source Code Library (ASCL) or other resources for the benefit of the community or provide input into other projects concerned with software metadata. △ Less

Submitted 6 December, 2017; originally announced December 2017.

Comments: 4 pages; to be published in ADASS XXVII (held Oct 22-26, 2017 in Santiago, Chile) proceedings

NASA's Long-Term Astrophysics Data Archives

Authors: L. M. Rebull, V. Desai, H. Teplitz, S. Groom, R. Akeson, G. B. Berriman, G. Helou, D. Imel, J. M. Mazzarella, A. Accomazzi, T. McGlynn, A. Smale, R. White

Abstract: NASA regards data handling and archiving as an integral part of space missions, and has a strong track record of serving astrophysics data to the public, beginning with the the IRAS satellite in 1983. Archives enable a major science return on the significant investment required to develop a space mission. In fact, the presence and accessibility of an archive can more than double the number of pape… ▽ More NASA regards data handling and archiving as an integral part of space missions, and has a strong track record of serving astrophysics data to the public, beginning with the the IRAS satellite in 1983. Archives enable a major science return on the significant investment required to develop a space mission. In fact, the presence and accessibility of an archive can more than double the number of papers resulting from the data. In order for the community to be able to use the data, they have to be able to find the data (ease of access) and interpret the data (ease of use). Funding of archival research (e.g., the ADAP program) is also important not only for making scientific progress, but also for encouraging authors to deliver data products back to the archives to be used in future studies. NASA has also enabled a robust system that can be maintained over the long term, through technical innovation and careful attention to resource allocation. This article provides a brief overview of some of NASA's major astrophysics archive systems, including IRSA, MAST, HEASARC, KOA, NED, the Exoplanet Archive, and ADS. △ Less

Submitted 27 September, 2017; originally announced September 2017.

Comments: To appear in ADASS 2016 conference proceedings (invited talk)

The Application of the Montage Image Mosaic Engine To The Visualization Of Astronomical Images

Authors: G. Bruce Berriman, J. C. Good

Abstract: The Montage Image Mosaic Engine was designed as a scalable toolkit, written in C for performance and portability across *nix platforms, that assembles FITS images into mosaics. The code is freely available and has been widely used in the astronomy and IT communities for research, product generation and for developing next-generation cyber-infrastructure. Recently, it has begun to finding applicabi… ▽ More The Montage Image Mosaic Engine was designed as a scalable toolkit, written in C for performance and portability across *nix platforms, that assembles FITS images into mosaics. The code is freely available and has been widely used in the astronomy and IT communities for research, product generation and for developing next-generation cyber-infrastructure. Recently, it has begun to finding applicability in the field of visualization. This has come about because the toolkit design allows easy integration into scalable systems that process data for subsequent visualization in a browser or client. And it includes a visualization tool suitable for automation and for integration into Python: mViewer creates, with a single command, complex multi-color images overlaid with coordinate displays, labels, and observation footprints, and includes an adaptive image histogram equalization method that preserves the structure of a stretched image over its dynamic range. The Montage toolkit contains functionality originally developed to support the creation and management of mosaics but which also offers value to visualization: a background rectification algorithm that reveals the faint structure in an image; and tools for creating cutout and down-sampled versions of large images. Version 5 of Montage offers support for visualizing data written in HEALPix sky-tessellation scheme, and functionality for processing and organizing images to comply with the TOAST sky-tessellation scheme required for consumption by the World Wide Telescope (WWT). Four online tutorials enable readers to reproduce and extend all the visualizations presented in this paper. △ Less

Submitted 8 February, 2017; originally announced February 2017.

Comments: 16 pages, 9 figures; accepted for publication in the PASP Special Focus Issue: Techniques and Methods for Astrophysical Data Visualization

Implementing Ideas for Improving Software Citation and Credit

Authors: Peter Teuben, Alice Allen, G. Bruce Berriman, Kimberly DuPrie, Jessica Mink, Thomas Robitaille, Keith Shortridge, Mark Taylor, Rein Warmels

Abstract: Improving software citation and credit continues to be a topic of interest across and within many disciplines, with numerous efforts underway. In this Birds of a Feather (BoF) session, we started with a list of actionable ideas from last year's BoF and other similar efforts and worked alone or in small groups to begin implementing them. Work was captured in a common Google document; the session or… ▽ More Improving software citation and credit continues to be a topic of interest across and within many disciplines, with numerous efforts underway. In this Birds of a Feather (BoF) session, we started with a list of actionable ideas from last year's BoF and other similar efforts and worked alone or in small groups to begin implementing them. Work was captured in a common Google document; the session organizers will disseminate or otherwise put this information to use in or for the community in collaboration with those who contributed. △ Less

Submitted 18 November, 2016; originally announced November 2016.

Comments: 4 pages; to be published in ADASS XXVI (held Oct 16-20, 2016) proceedings

Astrophysics Source Code Library: Here we grow again!

Authors: Alice Allen, G. Bruce Berriman, Kimberly DuPrie, Jessica Mink, Robert Nemiroff, Thomas Robitaille, Judy Schmidt, Lior Shamir, Keith Shortridge, Mark Taylor, Peter Teuben, John Wallin

Abstract: The Astrophysics Source Code Library (ASCL) is a free online registry of research codes; it is indexed by ADS and Web of Science and has over 1300 code entries. Its entries are increasingly used to cite software; citations have been doubling each year since 2012 and every major astronomy journal accepts citations to the ASCL. Codes in the resource cover all aspects of astrophysics research and man… ▽ More The Astrophysics Source Code Library (ASCL) is a free online registry of research codes; it is indexed by ADS and Web of Science and has over 1300 code entries. Its entries are increasingly used to cite software; citations have been doubling each year since 2012 and every major astronomy journal accepts citations to the ASCL. Codes in the resource cover all aspects of astrophysics research and many programming languages are represented. In the past year, the ASCL added dashboards for users and administrators, started minting Digital Objective Identifiers (DOIs) for software it houses, and added metadata fields requested by users. This presentation covers the ASCL's growth in the past year and the opportunities afforded it as one of the few domain libraries for science research codes. △ Less

Submitted 18 November, 2016; originally announced November 2016.

Comments: 4 pages, 2 figures; to be published in ADASS XXVI (held October 16-20, 2016) proceedings

The Astropy Problem

Authors: Demitri Muna, Michael Alexander, Alice Allen, Richard Ashley, Daniel Asmus, Ruyman Azzollini, Michele Bannister, Rachael Beaton, Andrew Benson, G. Bruce Berriman, Maciej Bilicki, Peter Boyce, Joanna Bridge, Jan Cami, Eryn Cangi, Xian Chen, Nicholas Christiny, Christopher Clark, Michelle Collins, Johan Comparat, Neil Cook, Darren Croton, Isak Delberth Davids, Éric Depagne, John Donor , et al. (129 additional authors not shown)

Abstract: The Astropy Project (http://astropy.org) is, in its own words, "a community effort to develop a single core package for Astronomy in Python and foster interoperability between Python astronomy packages." For five years this project has been managed, written, and operated as a grassroots, self-organized, almost entirely volunteer effort while the software is used by the majority of the astronomical… ▽ More The Astropy Project (http://astropy.org) is, in its own words, "a community effort to develop a single core package for Astronomy in Python and foster interoperability between Python astronomy packages." For five years this project has been managed, written, and operated as a grassroots, self-organized, almost entirely volunteer effort while the software is used by the majority of the astronomical community. Despite this, the project has always been and remains to this day effectively unfunded. Further, contributors receive little or no formal recognition for creating and supporting what is now critical software. This paper explores the problem in detail, outlines possible solutions to correct this, and presents a few suggestions on how to address the sustainability of general purpose astronomical software. △ Less

Submitted 10 October, 2016; originally announced October 2016.

The Next Generation of the Montage Image Mosaic Toolkit

Authors: G. Bruce Berriman, J. C. Good, B. Rusholme, T. Robitaille

Abstract: The scientific computing landscape has evolved dramatically in the past few years, with new schemes for organizing and storing data that reflect the growth in size and complexity of astronomical data sets. In response to this changing landscape, we are, over the next two years, deploying the next generation of the Montage toolkit ([ascl:1010.036]). The first release (October 2015) supports multi-d… ▽ More The scientific computing landscape has evolved dramatically in the past few years, with new schemes for organizing and storing data that reflect the growth in size and complexity of astronomical data sets. In response to this changing landscape, we are, over the next two years, deploying the next generation of the Montage toolkit ([ascl:1010.036]). The first release (October 2015) supports multi-dimensional data sets ("data cubes"), and insertion of XMP/AVM tags that allows images to "drop-in" to the WWT. The same release offers a beta-version of web-based interactive visualization of images; this includes wrappers for visualization in Python. Subsequent releases will support HEALPix (now standard in cosmic background experiments); incorporation of Montage into package managers (which enable automated management of software builds), and support for a library that will enable Montage to be called directly from Python. This next generation toolkit will inherit the architectural benefits of the current engine - component based tools, ANSI-C portability across Unix platforms and scalability for distributed processing. With the expanded functionality under development, Montage can be viewed not simply as a mosaic engine, but as a scalable, portable toolkit for managing, organizing and processing images. △ Less

Submitted 8 August, 2016; originally announced August 2016.

Comments: 4 pages, ADASS XXV, Sydney, Australia

A case study in adaptable and reusable infrastructure at the Keck Observatory Archive: VO interfaces, moving targets, and more

Authors: G. Bruce Berriman, Richard W. Cohen, Andrew Colson, Christopher R. Gelino, John C. Good, Mihseh Kong, Anastasia C. Laity, Jeffrey A. Mader, Melanie A. Swain, Hien D. Tran, Shin-Ywan Wang

Abstract: This paper describes how the Keck Observatory Archive (KOA) is extending open source software components to develop new services. In August 2015, KOA deployed a program interface to discover public data from all instruments equipped with an imaging mode. The interface complies with version 2 of the Simple Imaging Access Protocol (SIAP), under development by the International Virtual Observatory Al… ▽ More This paper describes how the Keck Observatory Archive (KOA) is extending open source software components to develop new services. In August 2015, KOA deployed a program interface to discover public data from all instruments equipped with an imaging mode. The interface complies with version 2 of the Simple Imaging Access Protocol (SIAP), under development by the International Virtual Observatory Alliance (IVOA), which defines a standard mechanism for discovering images through spatial queries. The heart of the KOA service is an R-tree-based, database-indexing mechanism prototyped by the Virtual Astronomical Observatory (VAO) and further developed by the Montage Image Mosaic project, designed to provide fast access to large imaging data sets as a first step in creating wide-area image mosaics. The KOA service uses the results of the spatial R-tree search to create an SQLite data database for further relational filtering. The service uses a JSON configuration file to describe the association between instrument parameters and the service query parameters, and to make it applicable beyond the Keck instruments. The R-tree program was itself extended to support temporal (in addition to spatial) indexing, in response to requests from the planetary science community for a search engine to discover observations of Solar System objects. With this 3D-indexing scheme, the service performs very fast time and spatial matches between the target ephemerides, obtained from the JPL SPICE service. Our experiments indicate these matches can be more than 100 times faster than when separating temporal and spatial searches. Images of the tracks of the moving targets, overlaid with the image footprints, are computed with a new command-line visualization tool, mViewer, released with the Montage distribution. The service is currently in test and will be released in Fall 2016. △ Less

Submitted 8 August, 2016; originally announced August 2016.

Comments: 16 pages, 8 figures, 5 tables. SPIE Astronomical Telescopes and Instrumentation, Software and Cyberinfrastructure for Astronomy

Improving Software Citation and Credit

Authors: Alice Allen, G. Bruce Berriman, Kimberly DuPrie, Jessica Mink, Robert Nemiroff, Thomas Robitaille, Lior Shamir, Keith Shortridge, Mark Taylor, Peter Teuben, John Wallin

Abstract: The past year has seen movement on several fronts for improving software citation, including the Center for Open Science's Transparency and Openness Promotion (TOP) Guidelines, the Software Publishing Special Interest Group that was started at January's AAS meeting in Seattle at the request of that organization's Working Group on Astronomical Software, a Sloan-sponsored meeting at GitHub in San Fr… ▽ More The past year has seen movement on several fronts for improving software citation, including the Center for Open Science's Transparency and Openness Promotion (TOP) Guidelines, the Software Publishing Special Interest Group that was started at January's AAS meeting in Seattle at the request of that organization's Working Group on Astronomical Software, a Sloan-sponsored meeting at GitHub in San Francisco to begin work on a cohesive research software citation-enabling platform, the work of Force11 to "transform and improve" research communication, and WSSSPE's ongoing efforts that include software publication, citation, credit, and sustainability. Brief reports on these efforts were shared at the BoF, after which participants discussed ideas for improving software citation, generating a list of recommendations to the community of software authors, journal publishers, ADS, and research authors. The discussion, recommendations, and feedback will help form recommendations for software citation to those publishers represented in the Software Publishing Special Interest Group and the broader community. △ Less

Submitted 24 December, 2015; originally announced December 2015.

Comments: Birds of a Feather session organized by the Astrophysics Source Code Library (ASCL, http://ascl.net/ ); to be published in Proceedings of ADASS XXV (Sydney, Australia; October, 2015). 4 pages

The First Data Release of the KODIAQ Survey

Authors: J. M. O'Meara, N. Lehner, J. C. Howk, J. X. Prochaska, A. J. Fox, M. A. Swain, C. R. Gelino, G. B. Berriman, H. Tran

Abstract: We present and make publicly available the first data release (DR1) of the Keck Observatory Database of Ionized Absorption toward Quasars (KODIAQ) survey. The KODIAQ survey is aimed at studying galactic and circumgalactic gas in absorption at high-redshift, with a focus on highly-ionized gas traced by OVI, using the HIRES spectrograph on the Keck-I telescope. KODIAQ DR1 consists of a fully-reduced… ▽ More We present and make publicly available the first data release (DR1) of the Keck Observatory Database of Ionized Absorption toward Quasars (KODIAQ) survey. The KODIAQ survey is aimed at studying galactic and circumgalactic gas in absorption at high-redshift, with a focus on highly-ionized gas traced by OVI, using the HIRES spectrograph on the Keck-I telescope. KODIAQ DR1 consists of a fully-reduced sample of 170 quasars at 0.29 < z_em < 5.29 observed with HIRES at high resolution (36,000 <= R <= 103,000) between 2004 and 2012. DR1 contains 247 spectra available in continuum normalized form, representing a sum total exposure time of ~1.6 megaseconds. These co-added spectra arise from a total of 567 individual exposures of quasars taken from the Keck Observatory Archive (KOA) in raw form and uniformly processed using a HIRES data reduction package made available through the XIDL distribution. DR1 is publicly available to the community, housed as a higher level science product at the KOA. We will provide future data releases that make further QSOs, including those with pre-2004 observations taken with the previous-generation HIRES detectors. △ Less

Submitted 13 May, 2015; originally announced May 2015.

Comments: 14 pages, 9 figures, Submitted to AJ. All data products available at the Keck Observatory Archive beginning May 15, 2015. URL: https://koa.ipac.caltech.edu/applications/KODIAQ

The Virtual Astronomical Observatory: Re-engineering Access to Astronomical Data

Authors: R. J. Hanisch, G. B. Berriman, T. J. W. Lazio, S. Emery Bunn, J. Evans, T. A. McGlynn, R. Plante

Abstract: The U.S. Virtual Astronomical Observatory was a software infrastructure and development project designed both to begin the establishment of an operational Virtual Observatory (VO) and to provide the U.S. coordination with the international VO effort. The concept of the VO is to provide the means by which an astronomer is able to discover, access, and process data seamlessly, regardless of its phys… ▽ More The U.S. Virtual Astronomical Observatory was a software infrastructure and development project designed both to begin the establishment of an operational Virtual Observatory (VO) and to provide the U.S. coordination with the international VO effort. The concept of the VO is to provide the means by which an astronomer is able to discover, access, and process data seamlessly, regardless of its physical location. This paper describes the origins of the VAO, including the predecessor efforts within the U.S. National Virtual Observatory, and summarizes its main accomplishments. These accomplishments include the development of both scripting toolkits that allow scientists to incorporate VO data directly into their reduction and analysis environments and high-level science applications for data discovery, integration, analysis, and catalog cross-comparison. Working with the international community, and based on the experience from the software development, the VAO was a major contributor to international standards within the International Virtual Observatory Alliance. The VAO also demonstrated how an operational virtual observatory could be deployed, providing a robust operational environment in which VO services worldwide were routinely checked for aliveness and compliance with international standards. Finally, the VAO engaged in community outreach, developing a comprehensive web site with on-line tutorials, announcements, links to both U.S. and internationally developed tools and services, and exhibits and hands-on training .... All digital products of the VAO Project, including software, documentation, and tutorials, are stored in a repository for community access. The enduring legacy of the VAO is an increasing expectation that new telescopes and facilities incorporate VO capabilities during the design of their data management systems. △ Less

Submitted 8 April, 2015; originally announced April 2015.

Comments: Abstract abridged for arXiv limit; 23 pages, 15 figures; to appear in a special issue of Astronomy & Computing

Significant problems in FITS limit its use in modern astronomical research

Authors: Brian Thomas, Tim Jenness, Frossie Economou, Perry Greenfield, Paul Hirst, David S. Berry, Erik Bray, Norman Gray, Demitri Muna James Turner, Miguel de Val-Borro, Juande Santander Vela, David Shupe, John Good, G. Bruce Berriman

Abstract: The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard is, however, showing its age. Developed in the late 1970s the FITS authors made a number of implementation choices for the format that, while common at the time, are now seen to limit its utility wit… ▽ More The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard is, however, showing its age. Developed in the late 1970s the FITS authors made a number of implementation choices for the format that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not appreciate the challenges which we would be facing today in astronomical computing. Difficulties we now face include, but are not limited to, having to address the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets and the need to capture significantly more complex metadata and data relationships. There are members of the community today who find some (or all) of these limitations unworkable, and have decided to move ahead with storing data in other formats. This reaction should be taken as a wakeup call to the FITS community to make changes in the FITS standard, or to see its usage fall. In this paper we detail some selected important problems which exist within the FITS standard today. It is not our intention to prescribe specific remedies to these issues; rather, we hope to call attention of the FITS and greater astronomical computing communities to these issues in the hopes that it will spur action to address them. △ Less

Submitted 20 February, 2015; originally announced February 2015.

Comments: Astronomical Data Analysis Software and Systems XXIII. Proceedings of a meeting held 29 September - 3 October 2013 at Waikoloa Beach Marriott, Hawaii, USA. Edited by N. Manset and P. Forshay ASP conference series, vol. 485, 2014, p.351

Learning from FITS: Limitations in use in modern astronomical research

Authors: Brian Thomas, Tim Jenness, Frossie Economou, Perry Greenfield, Paul Hirst, David S. Berry, Erik Bray, Norman Gray, Demitri Muna, James Turner, Miguel de Val-Borro, Juande Santander-Vela, David Shupe, John Good, G. Bruce Berriman, Slava Kitaeff, Jonathan Fay, Omar Laurino, Anastasia Alexov, Walter Landry, Joe Masters, Adam Brazier, Reinhold Schaaf, Kevin Edwards, Russell O. Redman , et al. (13 additional authors not shown)

Abstract: The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data.… ▽ More The Flexible Image Transport System (FITS) standard has been a great boon to astronomy, allowing observatories, scientists and the public to exchange astronomical information easily. The FITS standard, however, is showing its age. Developed in the late 1970s, the FITS authors made a number of implementation choices that, while common at the time, are now seen to limit its utility with modern data. The authors of the FITS standard could not anticipate the challenges which we are facing today in astronomical computing. Difficulties we now face include, but are not limited to, addressing the need to handle an expanded range of specialized data product types (data models), being more conducive to the networked exchange and storage of data, handling very large datasets, and capturing significantly more complex metadata and data relationships. There are members of the community today who find some or all of these limitations unworkable, and have decided to move ahead with storing data in other formats. If this fragmentation continues, we risk abandoning the advantages of broad interoperability, and ready archivability, that the FITS format provides for astronomy. In this paper we detail some selected important problems which exist within the FITS standard today. These problems may provide insight into deeper underlying issues which reside in the format and we provide a discussion of some lessons learned. It is not our intention here to prescribe specific remedies to these issues; rather, it is to call attention of the FITS and greater astronomical computing communities to these problems in the hope that it will spur action to address them. △ Less

Submitted 10 February, 2015; v1 submitted 3 February, 2015; originally announced February 2015.

Astrophysics Source Code Library Enhancements

Authors: Robert J. Hanisch, Alice Allen, G. Bruce Berriman, Kimberly DuPrie, Jessica Mink, Robert J. Nemiroff, Judy Schmidt, Lior Shamir, Keith Shortridge, Mark Taylor, Peter J. Teuben, John Wallin

Abstract: The Astrophysics Source Code Library (ASCL; ascl.net) is a free online registry of codes used in astronomy research; it currently contains over 900 codes and is indexed by ADS. The ASCL has recently moved a new infrastructure into production. The new site provides a true database for the code entries and integrates the WordPress news and information pages and the discussion forum into one site. Pr… ▽ More The Astrophysics Source Code Library (ASCL; ascl.net) is a free online registry of codes used in astronomy research; it currently contains over 900 codes and is indexed by ADS. The ASCL has recently moved a new infrastructure into production. The new site provides a true database for the code entries and integrates the WordPress news and information pages and the discussion forum into one site. Previous capabilities are retained and permalinks to ascl.net continue to work. This improvement offers more functionality and flexibility than the previous site, is easier to maintain, and offers new possibilities for collaboration. This presentation covers these recent changes to the ASCL. △ Less

Submitted 7 November, 2014; originally announced November 2014.

Comments: 4 pages; to be published in ADASS XXIV Proceedings. ASCL can be accessed at http://ascl.net/

Metadata and Data Management for the Keck Observatory Archive

Authors: H. D. Tran, J. Holt, R. W. Goodrich, J. A. Mader, M. Swain, A. C. Laity, M. Kong, C. R. Gelino, G. B. Berriman

Abstract: A collaboration between the W. M. Keck Observatory (WMKO) in Hawaii and the NASA Exoplanet Science Institute (NExScI) in California, the Keck Observatory Archive (KOA) was commissioned in 2004 to archive observing data from WMKO, which operates two classically scheduled 10 m ground-based telescopes. The observing data from Keck is not suitable for direct ingestion into the archive since the metada… ▽ More A collaboration between the W. M. Keck Observatory (WMKO) in Hawaii and the NASA Exoplanet Science Institute (NExScI) in California, the Keck Observatory Archive (KOA) was commissioned in 2004 to archive observing data from WMKO, which operates two classically scheduled 10 m ground-based telescopes. The observing data from Keck is not suitable for direct ingestion into the archive since the metadata contained in the original FITS headers lack the information necessary for proper archiving. Coupled with different standards among instrument builders and the heterogeneous nature of the data inherent in classical observing, in which observers have complete control of the instruments and their observations, the data pose a number of technical challenges for KOA. We describe the methodologies and tools that we have developed to successfully address these difficulties, adding content to the FITS headers and "retrofitting" the metadata in order to support archiving Keck data, especially those obtained before the archive was designed. With the expertise gained from having successfully archived observations taken with all eight currently active instruments at WMKO, we have developed lessons learned from handling this complex array of heterogeneous metadata that help ensure a smooth ingestion of data not only for current but also future instruments, as well as a better experience for the archive user. △ Less

Submitted 21 August, 2014; originally announced August 2014.

Comments: 9 pages, SPIE Astronomical Telescopes and Instrumentation 2014: Software and Cyberinfrastructure for Astronomy III, Proc. SPIE 9152-92

The Design and Operation of The Keck Observatory Archive

Authors: G. Bruce Berriman, Christopher R. Gelino, Robert W. Goodrich, Jennifer Holt, Mihseh Kong, Anastasia C. Laity, Jeffrey A. Mader, Melanie Swain, Hien D. Tran

Abstract: The Infrared Processing and Analysis Center (IPAC) and the W. M. Keck Observatory (WMKO) operate an archive for the Keck Observatory. At the end of 2013, KOA completed the ingestion of data from all eight active observatory instruments. KOA will continue to ingest all newly obtained observations, at an anticipated volume of 4 TB per year. The data are transmitted electronically from WMKO to IPAC f… ▽ More The Infrared Processing and Analysis Center (IPAC) and the W. M. Keck Observatory (WMKO) operate an archive for the Keck Observatory. At the end of 2013, KOA completed the ingestion of data from all eight active observatory instruments. KOA will continue to ingest all newly obtained observations, at an anticipated volume of 4 TB per year. The data are transmitted electronically from WMKO to IPAC for storage and curation. Access to data is governed by a data use policy, and approximately two-thirds of the data in the archive are public. △ Less

Submitted 4 August, 2014; originally announced August 2014.

Comments: 12 pages, 4 figs, 4 tables. Presented at Software and Cyberinfrastructure for Astronomy III, SPIE Astronomical Telescopes + Instrumentation 2014. June 2014, Montreal, Canada

Creating A Galactic Plane Atlas With Amazon Web Services

Authors: G. Bruce Berriman, Ewa Deelman, John Good, Gideon Juve, Jamie Kinney, Ann Merrihew, Mats Rynge

Abstract: This paper describes by example how astronomers can use cloud-computing resources offered by Amazon Web Services (AWS) to create new datasets at scale. We have created from existing surveys an atlas of the Galactic Plane at 16 wavelengths from 1 μm to 24 μm with pixels co-registered at spatial sampling of 1 arcsec. We explain how open source tools support management and operation of a virtual clus… ▽ More This paper describes by example how astronomers can use cloud-computing resources offered by Amazon Web Services (AWS) to create new datasets at scale. We have created from existing surveys an atlas of the Galactic Plane at 16 wavelengths from 1 μm to 24 μm with pixels co-registered at spatial sampling of 1 arcsec. We explain how open source tools support management and operation of a virtual cluster on AWS platforms to process data at scale, and describe the technical issues that users will need to consider, such as optimization of resources, resource costs, and management of virtual machine instances. △ Less

Submitted 23 December, 2013; originally announced December 2013.

Comments: 7 pages, 1 table, 2 figures. Submitted to IEEE Special Edition on Computing in Astronomy

The NASA Exoplanet Archive: Data and Tools for Exoplanet Research

Authors: R. L. Akeson, X. Chen, D. Ciardi, M. Crane, J. Good, M. Harbut, E. Jackson, S. R. Kane, A. C. Laity, S. Leifer, M. Lynn, D. L. McElroy, M. Papin, P. Plavchan, S. V. Ramirez, R. Rey, K. von Braun, M. Wittman, M. Abajian, B. Ali, C. Beichman, A. Beekley, G. B. Berriman, S. Berukoff, G. Bryden , et al. (10 additional authors not shown)

Abstract: We describe the contents and functionality of the NASA Exoplanet Archive, a database and tool set funded by NASA to support astronomers in the exoplanet community. The current content of the database includes interactive tables containing properties of all published exoplanets, Kepler planet candidates, threshold-crossing events, data validation reports and target stellar parameters, light curves… ▽ More We describe the contents and functionality of the NASA Exoplanet Archive, a database and tool set funded by NASA to support astronomers in the exoplanet community. The current content of the database includes interactive tables containing properties of all published exoplanets, Kepler planet candidates, threshold-crossing events, data validation reports and target stellar parameters, light curves from the Kepler and CoRoT missions and from several ground-based surveys, and spectra and radial velocity measurements from the literature. Tools provided to work with these data include a transit ephemeris predictor, both for single planets and for observing locations, light curve viewing and normalization utilities, and a periodogram and phased light curve service. The archive can be accessed at http://exoplanetarchive.ipac.caltech.edu. △ Less

Submitted 10 July, 2013; originally announced July 2013.

Comments: Accepted for publication in the Publications of the Astronomical Society of the Pacific, 4 figures

A Tale Of 160 Scientists, Three Applications, A Workshop and A Cloud

Authors: G. Bruce Berriman, Carolyn Brinkworth, Dawn Gelino, Dennis K. Wittman, Ewa Deelman, Gideon Juve, Mats Rynge, Jamie Kinney

Abstract: The NASA Exoplanet Science Institute (NExScI) hosts the annual Sagan Workshops, thematic meetings aimed at introducing researchers to the latest tools and methodologies in exoplanet research. The theme of the Summer 2012 workshop, held from July 23 to July 27 at Caltech, was to explore the use of exoplanet light curves to study planetary system architectures and atmospheres. A major part of the wo… ▽ More The NASA Exoplanet Science Institute (NExScI) hosts the annual Sagan Workshops, thematic meetings aimed at introducing researchers to the latest tools and methodologies in exoplanet research. The theme of the Summer 2012 workshop, held from July 23 to July 27 at Caltech, was to explore the use of exoplanet light curves to study planetary system architectures and atmospheres. A major part of the workshop was to use hands-on sessions to instruct attendees in the use of three open source tools for the analysis of light curves, especially from the Kepler mission. Each hands-on session involved the 160 attendees using their laptops to follow step-by-step tutorials given by experts. We describe how we used the Amazon Elastic Cloud 2 to run these applications. △ Less

Submitted 16 November, 2012; originally announced November 2012.

Comments: 4 pages, 1 figure, 1 table. Submitted to Astronomical Data Analysis Software and Systems XXII

Running a distributed virtual observatory: US Virtual Astronomical Observatory operations

Authors: Thomas A. McGlynn, Robert J. Hanisch, G. Bruce Berriman, Aniruddha R. Thakar

Abstract: Operation of the US Virtual Astronomical Observatory shares some issues with modern physical observatories, e.g., intimidating data volumes and rapid technological change, and must also address unique concerns like the lack of direct control of the underlying and scattered data resources, and the distributed nature of the observatory itself. In this paper we discuss how the VAO has addressed these… ▽ More Operation of the US Virtual Astronomical Observatory shares some issues with modern physical observatories, e.g., intimidating data volumes and rapid technological change, and must also address unique concerns like the lack of direct control of the underlying and scattered data resources, and the distributed nature of the observatory itself. In this paper we discuss how the VAO has addressed these challenges to provide the astronomical community with a coherent set of science-enabling tools and services. The distributed nature of our virtual observatory-with data and personnel spanning geographic, institutional and regime boundaries-is simultaneously a major operational headache and the primary science motivation for the VAO. Most astronomy today uses data from many resources. Facilitation of matching heterogeneous datasets is a fundamental reason for the virtual observatory. Key aspects of our approach include continuous monitoring and validation of VAO and VO services and the datasets provided by the community, monitoring of user requests to optimize access, caching for large datasets, and providing distributed storage services that allow user to collect results near large data repositories. Some elements are now fully implemented, while others are planned for subsequent years. The distributed nature of the VAO requires careful attention to what can be a straightforward operation at a conventional observatory, e.g., the organization of the web site or the collection and combined analysis of logs. Many of these strategies use and extend protocols developed by the international virtual observatory community. △ Less

Submitted 20 June, 2012; originally announced June 2012.

Comments: 7 pages with 2 figures included within PDF

The organization and management of the Virtual Astronomical Observatory

Authors: G. Bruce Berriman, Robert J. Hanisch, T. Joseph W. Lazio, Alexander Szalay, Giussepina Fabbiano

Abstract: The U.S. Virtual Astronomical Observatory (VAO; http://www.us-vao.org/) has been in operation since May 2010. Its goal is to enable new science through efficient integration of distributed multi-wavelength data. This paper describes the management and organization of the VAO, and emphasizes the techniques used to ensure efficiency in a distributed organization. Management methods include using an… ▽ More The U.S. Virtual Astronomical Observatory (VAO; http://www.us-vao.org/) has been in operation since May 2010. Its goal is to enable new science through efficient integration of distributed multi-wavelength data. This paper describes the management and organization of the VAO, and emphasizes the techniques used to ensure efficiency in a distributed organization. Management methods include using an annual program plan as the basis for establishing contracts with member organizations, regular communication, and monitoring of processes. △ Less

Submitted 18 June, 2012; originally announced June 2012.

Comments: 9 pages, 3 figures. SPIE Conference 8449: Modeling, Systems Engineering, and Project Management for Astronomy V

The role in the Virtual Astronomical Observatory in the era of massive data sets

Authors: G. Bruce Berriman, Robert J. Hanisch, T. Joseph W. Lazio

Abstract: The Virtual Observatory (VO) is realizing global electronic integration of astronomy data. One of the long-term goals of the U.S. VO project, the Virtual Astronomical Observatory (VAO), is development of services and protocols that respond to the growing size and complexity of astronomy data sets. This paper describes how VAO staff are active in such development efforts, especially in innovative s… ▽ More The Virtual Observatory (VO) is realizing global electronic integration of astronomy data. One of the long-term goals of the U.S. VO project, the Virtual Astronomical Observatory (VAO), is development of services and protocols that respond to the growing size and complexity of astronomy data sets. This paper describes how VAO staff are active in such development efforts, especially in innovative strategies and techniques that recognize the limited operating budgets likely available to astronomers even as demand increases. The project has a program of professional outreach whereby new services and protocols are evaluated. △ Less

Submitted 18 June, 2012; originally announced June 2012.

Comments: 10 pages, 3 figures. SPIE Conference 8448: Observatory Operations: Strategies, Processes, and Systems IV

How Will Astronomy Archives Survive The Data Tsunami?

Authors: G. Bruce Berriman, Steven L. Groom

Abstract: The field of astronomy is starting to generate more data than can be managed, served and processed by current techniques. This paper has outlined practices for developing next-generation tools and techniques for surviving this data tsunami, including rigorous evaluation of new technologies, partnerships between astronomers and computer scientists, and training of scientists in high-end software en… ▽ More The field of astronomy is starting to generate more data than can be managed, served and processed by current techniques. This paper has outlined practices for developing next-generation tools and techniques for surviving this data tsunami, including rigorous evaluation of new technologies, partnerships between astronomers and computer scientists, and training of scientists in high-end software engineering engineering skills. △ Less

Submitted 31 October, 2011; originally announced November 2011.

Comments: 8 pages, 3 figures; ACM Queue. Vol 9, Number 10, October 2011 (http://queue.acm.org/detail.cfm?id=2047483)

The NStED Periodogram Service and Interface for Public CoRoT Data

Authors: K. von Braun, M. Abajian, A. Beekley, G. B. Berriman, G. Bryden, B. Chan, D. R. Ciardi, J. Good, M. Harbut, S. R. Kane, A. Laity, C. Lau, M. Lynn, D. McElroy, P. Plavchan, M. Regelson, R. Rey, S. V. Ramirez, J. Stauffer, A. Zhang

Abstract: As part of the NASA-CNES agreement, the NASA Star and Exoplanet Database (NStED) serves as the official US portal for the public CoRoT data products. NStED is a general purpose archive with the aim of providing support for NASA's planet finding and characterization goals. Consequently, the NASA Exoplanet Science Institute (NExScI) developed, and NStED adapted, a periodogram service for CoRoT data… ▽ More As part of the NASA-CNES agreement, the NASA Star and Exoplanet Database (NStED) serves as the official US portal for the public CoRoT data products. NStED is a general purpose archive with the aim of providing support for NASA's planet finding and characterization goals. Consequently, the NASA Exoplanet Science Institute (NExScI) developed, and NStED adapted, a periodogram service for CoRoT data to determine periods of variability phenomena and create phased photometric light curves. Through the NStED periodogram interface, the user may choose three different period detection algorithms to use on any photometric time series product, or even upload and analyze their own data. Additionally, the NStED periodogram is remotely accessed by the CoRoT archive as part of its interface. NStED is available at {\bf http://nsted.ipac.caltech.edu}. △ Less

Submitted 11 July, 2011; originally announced July 2011.

Comments: 2 pages, 1 figure. To appear in "Transiting Planets, Vibrating Stars, and Their Connection", Conference Proceedings of the 2nd CoRoT Symposium, Eds: A. Baglin, M. Deleuil, E. Michel, C. Moutou

A Cross-Match of 2MASS and SDSS. II. Peculiar L Dwarfs, Unresolved Binaries, and the Space Density of T Dwarf Secondaries

Authors: Kerstin Geißler, Stanimir Metchev, J. Davy Kirkpatrick, G. Bruce Berriman, Dagny Looper

Abstract: We present the completion of a program to cross-correlate the SDSS Data Release 1 and 2MASS Point Source Catalog in search for extremely red L and T dwarfs. The program was initiated by Metchev and collaborators, who presented the findings on all newly identified T dwarfs in SDSS DR1, and estimated the space density of isolated T0--T8 dwarfs in the solar neighbourhood. In the current work we prese… ▽ More We present the completion of a program to cross-correlate the SDSS Data Release 1 and 2MASS Point Source Catalog in search for extremely red L and T dwarfs. The program was initiated by Metchev and collaborators, who presented the findings on all newly identified T dwarfs in SDSS DR1, and estimated the space density of isolated T0--T8 dwarfs in the solar neighbourhood. In the current work we present most of the L dwarf discoveries. Our red-sensitive (z-J > 2.75 mag) cross-match proves to be efficient in detecting peculiarly red L dwarfs, adding two new ones, including one of the reddest known L dwarfs. Our search also nets a new peculiarly blue L7 dwarf and, surprisingly, two M8 dwarfs. We further broaden our analysis to detect unresolved binary L or T dwarfs through spectral template fitting to all L and T dwarfs presented here and in the earlier work by Metchev and collaborators. We identify nine probable binaries, six of which are new and eight harbour likely T dwarf secondaries. We combine this result with current knowledge of the mass ratio distribution and frequency of substellar companions to estimate an overall space density of 0.005--0.05 pc^{-3} for individual T0--T8 dwarfs. △ Less

Submitted 6 March, 2011; originally announced March 2011.

Comments: Accepted by ApJ

Collaborative Astronomical Image Mosaics

Authors: Daniel S. Katz, G. Bruce Berriman, Robert G. Mann

Abstract: This chapter describes how astronomical imaging survey data have become a vital part of modern astronomy, how these data are archived and then served to the astronomical community through on-line data access portals. The Virtual Observatory, now under development, aims to make all these data accessible through a uniform set of interfaces. This chapter also describes the scientific need for one com… ▽ More This chapter describes how astronomical imaging survey data have become a vital part of modern astronomy, how these data are archived and then served to the astronomical community through on-line data access portals. The Virtual Observatory, now under development, aims to make all these data accessible through a uniform set of interfaces. This chapter also describes the scientific need for one common image processing task, that of composing individual images into large scale mosaics and introduces Montage as a tool for this task. Montage, as distributed, can be used in four ways: as a single thread/process on a single CPU, in parallel using MPI to distribute similar tasks across a parallel computer, in parallel using grid tools (Pegasus/DAGMan) to distributed tasks across a grid, or in parallel using a script-driven approach (Swift). An on-request web based Montage service is available for users who do not need to build a local version. We also introduce some work on a new scripted version of Montage, which offers ease of customization for users. Then, we discuss various ideas where Web 2.0 technologies can help the Montage community. △ Less

Submitted 23 November, 2010; originally announced November 2010.

Comments: 16 pages, 3 figures. To be published in "Reshaping Research and Development using Web 2.0-based technologies." Mark Baker, ed. Nova Science Publishers, Inc.(2011)

The Application of Cloud Computing to Astronomy: A Study of Cost and Performance

Authors: G. Bruce Berriman, Ewa Deelman, Gideon Juve, Moira Regelson, Peter Plavchan

Abstract: Cloud computing is a powerful new technology that is widely used in the business world. Recently, we have been investigating the benefits it offers to scientific computing. We have used three workflow applications to compare the performance of processing data on the Amazon EC2 cloud with the performance on the Abe high-performance cluster at the National Center for Supercomputing Applications (NCS… ▽ More Cloud computing is a powerful new technology that is widely used in the business world. Recently, we have been investigating the benefits it offers to scientific computing. We have used three workflow applications to compare the performance of processing data on the Amazon EC2 cloud with the performance on the Abe high-performance cluster at the National Center for Supercomputing Applications (NCSA). We show that the Amazon EC2 cloud offers better performance and value for processor- and memory-limited applications than for I/O-bound applications. We provide an example of how the cloud is well suited to the generation of a science product: an atlas of periodograms for the 210,000 light curves released by the NASA Kepler Mission. This atlas will support the identification of periodic signals, including those due to transiting exoplanets, in the Kepler data sets. △ Less

Submitted 22 October, 2010; originally announced October 2010.

Comments: 7 pages, accepted for publication in the Proceedings of the e-Science in Astronomy Conference (Brisbane, Australia, December 2010)

The Application of Cloud Computing to the Creation of Image Mosaics and Management of Their Provenance

Authors: G. Bruce Berriman, Ewa Deelman, Paul Groth, Gideon Juve

Abstract: We have used the Montage image mosaic engine to investigate the cost and performance of processing images on the Amazon EC2 cloud, and to inform the requirements that higher-level products impose on provenance management technologies. We will present a detailed comparison of the performance of Montage on the cloud and on the Abe high performance cluster at the National Center for Supercomputing Ap… ▽ More We have used the Montage image mosaic engine to investigate the cost and performance of processing images on the Amazon EC2 cloud, and to inform the requirements that higher-level products impose on provenance management technologies. We will present a detailed comparison of the performance of Montage on the cloud and on the Abe high performance cluster at the National Center for Supercomputing Applications (NCSA). Because Montage generates many intermediate products, we have used it to understand the science requirements that higher-level products impose on provenance management technologies. We describe experiments with provenance management technologies such as the "Provenance Aware Service Oriented Architecture" (PASOA). △ Less

Submitted 24 June, 2010; originally announced June 2010.

Comments: 15 pages, 3 figure

Journal ref: SPIE Conference 7740: Software and Cyberinfrastructure for Astronomy (2010)

Montage: a grid portal and software toolkit for science-grade astronomical image mosaicking

Authors: Joseph C. Jacob, Daniel S. Katz, G. Bruce Berriman, John Good, Anastasia C. Laity, Ewa Deelman, Carl Kesselman, Gurmeet Singh, Mei-Hui Su, Thomas A. Prince, Roy Williams

Abstract: Montage is a portable software toolkit for constructing custom, science-grade mosaics by composing multiple astronomical images. The mosaics constructed by Montage preserve the astrometry (position) and photometry (intensity) of the sources in the input images. The mosaic to be constructed is specified by the user in terms of a set of parameters, including dataset and wavelength to be used, locati… ▽ More Montage is a portable software toolkit for constructing custom, science-grade mosaics by composing multiple astronomical images. The mosaics constructed by Montage preserve the astrometry (position) and photometry (intensity) of the sources in the input images. The mosaic to be constructed is specified by the user in terms of a set of parameters, including dataset and wavelength to be used, location and size on the sky, coordinate system and projection, and spatial sampling rate. Many astronomical datasets are massive, and are stored in distributed archives that are, in most cases, remote with respect to the available computational resources. Montage can be run on both single- and multi-processor computers, including clusters and grids. Standard grid tools are used to run Montage in the case where the data or computers used to construct a mosaic are located remotely on the Internet. This paper describes the architecture, algorithms, and usage of Montage as both a software toolkit and as a grid portal. Timing results are provided to show how Montage performance scales with number of processors on a cluster computer. In addition, we compare the performance of two methods of running Montage in parallel on a grid. △ Less

Submitted 24 May, 2010; originally announced May 2010.

Comments: 16 pages, 11 figures

Journal ref: Int. J. Computational Science and Engineering. 2009

The Role of Provenance Management in Accelerating the Rate of Astronomical Research

Authors: G. Bruce Berriman, Ewa Deelman

Abstract: The availability of vast quantities of data through electronic archives has transformed astronomical research. It has also enabled the creation of new products, models and simulations, often from distributed input data and models, that are themselves made electronically available. These products will only provide maximal long-term value to astronomers when accompanied by records of their provenanc… ▽ More The availability of vast quantities of data through electronic archives has transformed astronomical research. It has also enabled the creation of new products, models and simulations, often from distributed input data and models, that are themselves made electronically available. These products will only provide maximal long-term value to astronomers when accompanied by records of their provenance; that is, records of the data and processes used in the creation of such products. We use the creation of image mosaics with the Montage grid-enabled mosaic engine to emphasize the necessity of provenance management and to understand the science requirements that higher-level products impose on provenance management technologies. We describe experiments with one technology, the "Provenance Aware Service Oriented Architecture" (PASOA), that stores provenance information at each step in the computation of a mosaic. The results inform the technical specifications of provenance management systems, including the need for extensible systems built on common standards. Finally, we describe examples of provenance management technology emerging from the fields of geophysics and oceanography that have applicability to astronomy applications. △ Less

Submitted 19 May, 2010; originally announced May 2010.

Comments: 8 pages, 1 figure; Proceedings of Science, 2010

The LAEX and NASA portals for CoRoT public data

Authors: E. Solano, K. von Braun, A. Velasco, D. R. Ciardi, R. Gutierrez, D. L. McElroy, M. Lopez, M. Abajian, M. Garcia, B. Ali, L. M. Sarro, G. B. Berriman, G. Bryden, B. Chan, J. Good, S. R. Kane, A. C. Laity, C. Lau, A. N. Payne, P. Plavchan, S. Ramirez, M. Schmitz, J. R. Stauffer, P. L. Wyatt, A. Zhang

Abstract: * Aims. We describe here the main functionalities of the LAEX (Laboratorio de Astrofisica Estelar y Exoplanetas/Laboratory for Stellar Astrophysics and Exoplanets) and NASA portals for CoRoT Public Data. The CoRoT archive at LAEX was opened to the community in January 2009 and is managed in the framework of the Spanish Virtual Observatory. NStED (NASA Star and Exoplanet Database) serves as the C… ▽ More * Aims. We describe here the main functionalities of the LAEX (Laboratorio de Astrofisica Estelar y Exoplanetas/Laboratory for Stellar Astrophysics and Exoplanets) and NASA portals for CoRoT Public Data. The CoRoT archive at LAEX was opened to the community in January 2009 and is managed in the framework of the Spanish Virtual Observatory. NStED (NASA Star and Exoplanet Database) serves as the CoRoT portal for the US astronomical community. NStED is a general purpose stellar and exoplanet archive with the aim of providing support for NASA planet finding and characterisation goals, and the planning and support of NASA and other space missions. CoRoT data at LAEX and NStED can be accessed at http://sdc.laeff.inta.es/corotfa/ and http://nsted.ipac.caltech.edu,respectively. * Methods. Based on considerable experience with astronomical archives, the aforementioned archives are designed with the aim of delivering science-quality data in a simple and efficient way. * Results. LAEX and NStED not only provide access to CoRoT Public Data but furthermore serve a variety of observed and calculated astrophysical data. In particular, NStED provides scientifically validated information on stellar and planetary data related to the search for and characterization of extrasolar planets, and LAEX makes any information from Virtual Observatory services available to the astronomical community. △ Less

Submitted 20 July, 2009; originally announced July 2009.

Comments: Accepted for publication in Astronomy & Astrophysics

The NASA/IPAC/NExScI Star and Exoplanet Database

Authors: G. B. Berriman, B. Ali, R. Baker, K. von Braun, N-M. Chiu, D. R. Ciardi, J. Good, S. R. Kane, M. Kong, A. C. Laity, D. L. McElroy, S. Monkewitz, A. N. Payne, S. Ramirez, M. Schmitz, J. S. Stauffer, P. L. Wyatt

Abstract: The NASA/IPAC/NExScI Star and Exoplanet Database (NStED) is a general purpose stellar archive which supports NASA planet-finding and planet-characterization goals, stellar astrophysics, and the planning of NASA and other space missions. There are two principal components of NStED: a database of 140,000 nearby stars and exoplanet-hosting stars, and an archive dedicated to high precision photometr… ▽ More The NASA/IPAC/NExScI Star and Exoplanet Database (NStED) is a general purpose stellar archive which supports NASA planet-finding and planet-characterization goals, stellar astrophysics, and the planning of NASA and other space missions. There are two principal components of NStED: a database of 140,000 nearby stars and exoplanet-hosting stars, and an archive dedicated to high precision photometric surveys for transiting exoplanets (NStED-ETSS). We present summaries of these components. The NStED stellar database currently serves published parameters for 140,000 stars. These parameters include coordinates, multiplicity, proper motion, parallax, spectral type, multiband photometry, radial velocity, metallicity, chromospheric and coronal activity index, rotation velocity/period, infrared excess. NStED-ETSS currently serves data from the TrES survey of the Kepler field as well as dedicated photometric surveys of four stellar clusters. NStED-ETSS aims to serve both the surveys and the broader astronomical community by archiving these data and making them available in a homogeneous format. △ Less

Submitted 13 March, 2009; originally announced March 2009.

Comments: 4 pages, AIP style

Journal ref: COOL STARS, STELLAR SYSTEMS AND THE SUN: Proceedings of the 15th Cambridge Workshop on Cool Stars, Stellar Systems and the Sun. AIP Conference Proceedings, Volume 1094, pp. 421-424 (2009)