AstroPT: Scaling Large Observation Models for Astronomy
Authors:
Michael J. Smith,
Ryan J. Roberts,
Eirini Angeloudi,
Marc Huertas-Company
Abstract:
This work presents AstroPT, an autoregressive pretrained transformer developed with astronomical use-cases in mind. The AstroPT models presented here have been pretrained on 8.6 million $512 \times 512$ pixel $grz$-band galaxy postage stamp observations from the DESI Legacy Survey DR8. We train a selection of foundation models of increasing size from 1 million to 2.1 billion parameters, and find t…
▽ More
This work presents AstroPT, an autoregressive pretrained transformer developed with astronomical use-cases in mind. The AstroPT models presented here have been pretrained on 8.6 million $512 \times 512$ pixel $grz$-band galaxy postage stamp observations from the DESI Legacy Survey DR8. We train a selection of foundation models of increasing size from 1 million to 2.1 billion parameters, and find that AstroPT follows a similar saturating log-log scaling law to textual models. We also find that the models' performances on downstream tasks as measured by linear probing improves with model size up to the model parameter saturation point. We believe that collaborative community development paves the best route towards realising an open source `Large Observation Model' -- a model trained on data taken from the observational sciences at the scale seen in natural language processing. To this end, we release the source code, weights, and dataset for AstroPT under the MIT license, and invite potential collaborators to join us in collectively building and researching these models.
△ Less
Submitted 23 May, 2024;
originally announced May 2024.
Innovative Science
Authors:
Donald W Braben,
John F Allen,
William Amos,
Richard Ball,
Hagan Bayley,
Tim Birkhead,
Peter Cameron,
Eleanor Campbell,
Richard Cogdell,
David Colquhoun,
Steve Davies,
Rod Dowler,
Peter Edwards,
Irene Engle,
Felipe Fernandez-Armesto,
Desmond Fitzgerald,
Jon Frampton,
Dame Anne Glover,
John Hall,
Pat Heslop-Harrison,
Dudley Herschbach,
Sui Huang,
H Jeff Kimble,
Sir Harry Kroto,
James Ladyman
, et al. (23 additional authors not shown)
Abstract:
Sir, We write as senior scientists about a problem vital to the scientific enterprise and prosperity. Nowadays, funding is a lengthy and complex business. First, universities themselves must approve all proposals for submission. Funding agencies then subject those that survive to peer review, a process by which a few researchers, usually acting anonymously, assess a proposal's chances that it will…
▽ More
Sir, We write as senior scientists about a problem vital to the scientific enterprise and prosperity. Nowadays, funding is a lengthy and complex business. First, universities themselves must approve all proposals for submission. Funding agencies then subject those that survive to peer review, a process by which a few researchers, usually acting anonymously, assess a proposal's chances that it will achieve its goals, is the best value for money, is relevant to a national priority and will impact on a socio-economic problem. Only 25% of proposals received by the funding agencies are funded. These protracted processes force researchers to exploit existing knowledge, severely discourage open-ended studies and are hugely time-consuming. They are also new: before 1970, few researchers wrote proposals. Now they are virtually mandatory.
△ Less
Submitted 23 September, 2015;
originally announced October 2015.
