Copernicus long-term archive in the cloud
OVHcloud & Werum
Long-term archive
in the cloud
24/7 Service for historical
Earth observation data
Scalability and high availability
in the public cloud
The background
Providing large amounts of data for worldwide users in a secure and reliable way
Werum Software & Systems AG, headquartered in Lüneburg, is one of the longest established independent software companies in Germany. Werum creates sophisticated software and systems for a global clientele, including numerous well-known companies from the automotive and aerospace industries as well as scientific institutes and clients from the public sector. The focus of their activities is on supporting customer-specific processes in the areas of measurement data and information management, test data management for mechanical engineering, Earth observation, eGovernment and enterprise information management. Their software solutions are based on platforms developed specifically for this purpose.
On behalf of the European Commission and the European Space Agency (ESA), Werum has set up a long-term archiving service for Copernicus in the OVHcloud Public Cloud. It covers the so-called Level-0 and auxiliary data of the Earth observation satellites Sentinel-1, Sentinel-2 and Sentinel-3. 220 million data sets totalling over 9 petabytes have been uploaded to the public cloud archive since its launch in October 2020. About 110 terabytes are still being added every month. In addition, its current capacity allows for 10 terabytes to be downloaded daily by specific users.
The challenge
High availability, high performance, and the ability to scale
In the past, Earth observation ground segments were set up anew for each mission. Part of these ground segments are long-term archives for which a large and very maintenance-intensive infrastructure had to be kept available. The necessary software services were run in dedicated data centres whose performance had to be optimised for the data management of a specific satellite mission. Operations further required a specially trained team to resolve critical hardware- and software-related incidents.
The new "archive-as-a-service" needed to provide the existing functionalities in the cloud in a scalable and portable manner. The requirements for the archive service were a high infrastructure availability, high performance in terms of bandwidth and storage requests as well as the ability to scale the infrastructure configuration. Additionally, the large amount of data must be stored ready for retrieval.
The archive service is run by Werum with highly specialised software and places specific requirements on their cloud provider. This provider must offer reliable 24/7 support as well as a flexible range of technical infrastructure in order to ensure that the software solution can be freely configured and optimised, as it is not based on typical cloud platforms.
In order to set up the innovative archive as a competitive service, Werum is cooperating with OVHcloud on this project.
The solution
Process and store several petabytes of data in the shortest time possible
With their many years of experience in creating and managing the necessary software, Werum already has the required expertise for these new requirements.
The cloud infrastructure is provided by OVHcloud. As part of this infrastructure, Werum has selected a high-performance bare-metal server for the centralised service orchestration. Data processing is scalable and distributed across several different public cloud instances, while the public cloud archive and public cloud object storage are used for equally scalable data storage. Data processing involves several steps such as integrity checks, metadata extraction, tracing to specific external services, and storage itself. It also ensures that the data can be read again. The Werum service team has subscribed to the OVHcloud Enterprise Support in order to quickly analyse and find solutions together in the event of problems with the infrastructure. The various cloud services in the Copernicus long-term archive help to tackle the challenges on a number of levels. First of all, the amount of data to be stored in itself is very large. Being able to generate several petabytes of storage capacity within a very short space of time was therefore a crucial factor for the selection of the provider. Thanks to the different public cloud instance options for data processing, it was and is possible in the project to create and guarantee short-term scalability in terms of both data processing speed and the required network capacities.
The result
Since October 2020, Werum has been operating the Copernicus long-term archive on behalf of ESA. This specialised software solution offers Werum a 24/7 service to provide users with the historical Earth observation data from various Sentinel satellite missions while at the same time ensuring a very high level of availability. The decision in favour of OVHcloud has already proven itself in the operational phase of the service, precisely because the technical requirements of the necessary software components can be flexibly mapped with the various server instances. Additionally, the good communication with the Technical Account Manager ensured targeted support. It enabled the service team to pass on any technical requirements to OVHcloud at an early stage – e.g. in situations where the forecast for the expected storage volumes changed significantly.
Following the successful first project phase, the archive operation in the cloud has been extended and will continue to provide the ESA user community with large volumes of data from the satellite missions beyond 2022.
