OGC Earth Observation Applications Pilot: Call for Participation (CFP)

The availability of free and open data, such as that provided by the Copernicus Sentinel fleet, together with the availability of affordable computing resources, create an opportunity for the wide adoption and use of Earth Observation (EO) data in all fields of our society.

ESA’s “EO Exploitation Platforms” initiative aims at achieving a paradigm shift from “bring the data to the user” (i.e. user downloads data locally) to “bring the user to the data” (i.e. move user exploitation to hosted environments with collocated computing and storage). This leads to a platform-based ecosystem that provides infrastructure, data, computing and software as a service. The resulting Exploitation Platform is where scientific and value-adding activities are conducted, to generate targeted outputs for end-users.

In order to unite the existing and future resources into an ‘Network of EO Resources’ there is a motivation to define standard interfaces that facilitate the federation and interoperation of these scattered resources – allowing the user to efficiently access and consume the disparate services of different providers seamlessly.

The approach can be applied to different domains (other than Earth Observation) and ESA is thus working closely with OGC in the definition of these standard interfaces and related solutions.

OGC Testbed activities in Testbed-13, Testbed-14, and the ongoing Testbed-15 have developed an architecture that allows the ad-hoc deployment and execution of applications close to the physical location of the source data. The goal is to minimize data transfer between data repositories and application processes. The following Engineering Reports describe the work accomplished in the Testbed-13/14:

Testbed-13 reports are referenced as they allow to understand the history of this work and design decisions in context, but are mostly superseded by Testbed-14 reports.

At the same time, significant progress towards more Web-oriented interfaces has been made in OGC with the emerging OGC APIs -Core, -Features, -Coverages, and -Processes. All of these APIs are using OpenAPI. These changes have not fully explored in the current architecture yet, which provides additional ground for experimentation as part of the pilot.

The following acronyms and abbreviations have been used in this report.

The following is a list of Reference Documents with a direct bearing on the content of this document.

Within this document and activity, the following roles are defined.

This pilot is conducted under the OGC Innovation Program and implements the OGC Innovation Program "Pilot" initiative policies and procedures. In this specific context, additional policies and procedures apply: The pilot is open to all organizations. Cost sharing funds can only be requested by entities from ESA member states, Canada, and Slovenia.

Within the context of this procurement ESA are acting as a sponsor to the OGC Earth Observation Applications Pilot. Participants for ESA activities within the pilot will be selected jointly by ESA/TVUK and the OGC from responses to this Call for Participation, and will become Subcontractors to OGC. Subcontractors will be referred to as Participants.

Once initiated, the pilot activities will be managed by OGC. The Participants will interact directly with OGC for technical, management, and contractual matters. ESA and TVUK should be copied on correspondence and invited to meetings.

Cost sharing payments to Participants selected by this call are funded by ESA. During and on completion of the pilot activities, OGC evaluates achievement of each milestone in order to trigger payments due under this contract. Payments will be made directly from OGC to the Participant.

The contractual conditions, at project level, come from the prime contract between ESA and TVUK, incorporate OGC pilot terms & conditions and then down to the Participant.

The pilot initiative works on a model of Participant in-kind contributions of engineering resources, technology, and/or facilities and cost-share funding provided by the Sponsors such as ESA.

TVUK, ESA or OGC will not cover the costs of equipment or software required for the normal development of participant software products. Nor will TVUK, ESA or OGC fund the purchase of additional hardware or software. OGC on behalf of TVUK and ESA will support the unique costs associated with engineering labour and travel (limited to kick-off and demonstration events) - noting that travel is often provided as an in-kind contribution for OGC activities.

This would include, but not necessarily be limited to: labour to develop engineering requirements, engineering specification reports, prototypical software component development exercising OGC specifications, instantiating new services based on existing software that exercises relevant OGC specifications, documentation of specifications, demonstrations of prototypical or existing software components that exercise OGC specifications or draft specifications and travel to demonstration events.

Proposals submitted under this call shall include a full breakdown of in-kind contributions by participants and required cost sharing contributions from OGC on behalf of TVUK/ESA.

The OGC Earth Observation Applications Pilot defines a set of interface specifications and data models working on the HTTP layer. The architecture allows application developers and consumers to interact with service endpoints that abstract from the underlying complexity of data handling, scheduling, resource allocation, or infrastructure management.

It consists of the following logical components:

Partly, these logical components can be implemented jointly. For example, it is likely that a data and processing platform provides exploitation platform elements, in particular as the various APIs are based on the same underlying OGC specifications.

Together, the architecture components allow the execution of the following steps:

The following figure illustrates all major components and the corresponding interactions.

The Execution Management Service (EMS) represents the front-end to both application developers and consumers. It makes available an OGC Web Processing Service interface that implements the emerging OGC Web API principles, i.e. provides a Web API that follows REST principles. The API supports the registration of new applications. The applications themselves are made available by reference in the form of containerized Docker images that are uploaded to Docker Hubs. These hubs may be operated centrally by Docker itself, by the cloud providers, or as private instances that only serve a very limited set of applications. Details mostly depend on security considerations. Initially developed to deploy applications only, the EMS has been developed into a workflow environment that allows application developers to re-use existing applications and orchestrate them into sequential workflows that can be made available as new applications again. This process is transparent to the application consumer.

The Application Package (AP) serves as the application meta data container that describes all essential elements of an application, such as its functionality, required satellite data, other auxiliary data, and input parameters to be set at execution time. The application package describes the output data and defines mount points to allow the execution environment to serve data to an application that is actually executed in a secure memory space; and to allow for persistent storage of results before a container is terminated.

The execution platform, which offers EMS functionality to application developers and consumers, acts itself as a client to the Application Deployment and Execution Services (ADES) offered by the data storing cloud platforms. The cloud platforms support the ad-hoc deployment and execution of Docker images that are pulled from the Docker hubs using the references made available in the deployment request.

Once application consumers request the execution of an app, the exploitation platform forwards the execution request to the processing clouds and makes final results available at standardized interfaces again, e.g. at Web Feature Service (WFS) or Web Coverage Service (WCS) instances. In the case of workflows that execute a number of applications sequentially, the exploitation platform realizes the transport of data from one process to the other. Upon completion, the application consumer is provided a data access service endpoint to retrieve the final results. All communication is established in a web-friendly way implementing the emerging next generation of OGC services that provides the various OGC APIs -Features, -Coverages, or -Processes.

BILLING AND QUOTING

Currently, satellite image processing still happens to a good extent on the physical machine of the end-user. This approach allows the end-user to understand all processing costs upfront. The hardware is purchased, prices per satellite product are known in advance, and actual processing costs are defined by the user’s time required to supervise the process. The approach is even reflected in procurement rules and policies at most organizations that often require a number of quotes before an actual procurement is authorized.
The new approach featured in this pilot requires a complete change of thinking. No hardware other than any machine with a browser (could even be a cell phone) needs to be purchased. Satellite imagery is not purchased or downloaded anymore, but rented just for the time of processing, and the final processing costs are set by the computational resource requirements of the process. Thus, most of the cost factors are hidden from the end-user, who does not necessarily know if his request results in a single satellite image process that can run on a tiny virtual machine, or a massive amount of satellite images that are processed in parallel on a 1000+ machines cluster.

The currently ongoing efforts to store Earth Observation data in Datacubes adds to the complexity to estimate the actual data consumption. The reason is that the old unit ”satellite image” is blurred when data is stored in multidimensional structures not made transparent to the user. Often, it is even difficult for the cloud operator to calculate exact costs prior to the completed execution of a process. This leads to the challenging situation for both, cloud operators that have to calculate costs upfront, and end-users that do not want to be negatively surprised by the final invoice for their processing request.

To address this challenge, OGC has started the integration of quoting and billing features into the cloud processing architecture. Specific service endpoints support quote requests that are executed prior to the actual execution requests. These allow a user to understand what costs will occur for a given service call, and they allow execution platforms to identify the most cost-effective cloud platform for any given application execution request.

Quoting and Billing information has been added to the Execution Management Service (EMS) and the Application Deployment and Execution Service (ADES). Both services are implemented in a web-friendly way as a Resource Oriented Architecture (ROA) Web API that resembles the behavior of the current transactional OGC Web Processing Service v2.0 (this version, v.3.0, is not published yet by OGC). The API is described by an OpenAPI 3.0 specification (see [OPENAPI]) that allows deploying and executing new processes by sending HTTP POST request to the DeployProcess operation or Execute operation endpoints. Following the same pattern, it allows posting similar requests against the Quota endpoint, which causes a JSON response with all quote related data. The sequence diagram in Figure 5 illustrates the workflow.