OGC Testbed-21: Call for Participation (CFP)

Standards such as ISO 19115 (for Geospatial Metadata) and ISO 19157 (for Geospatial Data Quality) have been used to help record the lineage and quality of geospatial datasets. However, to date, the absence of a shared vocabulary for describing data quality measures has prevented many systems from achieving a greater level of interoperability when exchanging metadata about the quality of a dataset. Leveraging existing ontologies, such a shared vocabulary could also facilitate interoperability between systems that support ISO 19115/ISO 19157 models and sector-specific models such as the NASA Unified Metadata Model (UMM).

One of the areas that could significantly benefit from a shared vocabulary for describing data quality measures is that of Machine Learning (ML) for Artificial Intelligence (AI). The quality of training datasets plays a key role in the performance of an ML model and thus acts as a requirement for a successful AI application. This is more so in Earth Observation applications since 'bad data' can often lead to a deluge of false positives in feature extraction from satellite imagery. Therefore the recent approval and release of the OGC Training Data Markup Language for AI (TrainingDML-AI) standard is expected to lead to better ML models and improvements in AI applications. To achieve such improvements in AI applications, there is a need for a shared vocabulary for describing data quality measures and the integration of those measures into an IPT framework.

The aim of this task is to:

The objectives of the task are therefore to:

Key research questions may include:

In an era of growing data volumes and geospatial analysis demands, maintaining IPT is critical, especially in distributed systems where data availability can fluctuate. The OGC Testbed-20 IPT activity addressed this need by developing resilient data services to safeguard IPT throughout the data lifecycle [2]. Two IPT Server instances were demonstrated in OGC Testbed-20, including the Federated Agile Collaborative Trusted System (FACTS) to manage Smart Certificates, and a Data Verification system that leveraged Decentralized Identifiers (DIDs) and Verifiable Credentials (VCs).

A number of previous OGC Testbeds explored interoperability solutions for Imagery Quality [3] and Quality Assessment Services [4]. Some of the lessons from those initiatives have led to a joint effort by OGC and ISO/TC 211 to develop a machine-actionable data quality measures register. ISO 19157-3 Geographic information — Data quality — Part 3: Data quality measures register will be the standard defining the components and content structure of a register for data quality measures, and the registration and maintenance procedure (ISO/AWI 19157-3). OGC will host the ISO 19157-3 register on OGC RAINBOW, a registry of terms and definitions. The register is in development, however experimentation during the July 2024 OGC Code Sprint demonstrated the referencing of registered data quality measures in documents conforming to the TrainingDML-AI standard.

The following subsections document all work items and deliverables of this task.

The primary goal of the OGC Testbed-21 GIMI task is to enable GIMI application development by continuing permissively-licensed open source contributions to the GStreamer and libheif libraries, building on the progress made during OGC Testbed-20. OGC Testbed-21 will include a new effort focused on investigating, implementing, and evaluating the GEOINT Media Standards Board’s (GMSB) emerging Imagery Domain Ontology (IDO) by implementing and testing ontology structured metadata sidecar files. These files provide descriptive metadata information for imagery inside GIMI files. The Imagery Domain Ontology is based on Basic Formal Ontology (BFO) and Common Core Ontologies (CCO).

The technical requirements for this testbed thread are listed below:

The GIMI format is very versatile as it is based in HEIF, which is structured in boxes that can contain both data and metadata. In the OGC Testbed-20 initiative some best practices on how to combine these boxes and propositions for new boxes were made. During the OGC Testbed-20 initiative, several implementations were developed to support generating image files conforming to the Geographic High Efficiency Image Format (GeoHEIF) candidate standard that is proposed for use in GIMI. The Testbed also benchmarked the reading of GeoHEIF files with specific open source libraries. In addition, a motion imagery use case for GIMI video support was analyzed.

Prior to OGC Testbed-20, the OGC Code Sprint held in the United Kingdom in October 2023 demonstrated the viability of the draft GIMI Profile of ISOBMFF as an alternative container of GEOINT data. As documented in the resulting Engineering Report (OGC 23-059), the Code Sprint participants identified several recommended future actions to advance GIMI as an industry standard. Amongst the recommendations was that with minor changes to popular open-source base libraries, a wide range of software could quickly make use of GIMI capabilities. Some of those libraries were enabled to support some GIMI capabilities by the OGC Testbed-20 initiative.

In detail, the following work items will be addressed:

OGC uses TEAM Engine (Test, Evaluation, And Measurement Engine) for conformance testing. TEAM Engine is a Java-based application for testing web services and other information resources. It executes test suites developed using the TestNG framework and, alternatively OGC Compliance Test Language (CTL) scripts. TEAM Engine can be used to test almost any type of service or information resource. It is the official test harness used by the OGC’s Compliance Program.

The primary goal of the OGC Testbed-21 Conformance Testing Tool Development (CTTG) task is to provide a GIMI conformance testing tool that can be integrated into TEAM Engine.

The task shall address the following research questions:

The previous OGC Testbed-17 initiative provided a thread to advance testing of implementations of OGC Standards. An executable test suite for OGC API – Processes — Part 1: Core was developed using the TestNG framework (the current recommended OGC approach), and another executable test suite was developed using NeoTL as an alternate approach [6].

In detail, the following work items will be addressed:

Reports and Change Requests (CR) will be prepared in accordance with OGC published templates. Reports will be delivered by posting on the (members-only) OGC Pending directory when complete and the document has achieved a satisfactory level of consensus among interested participants, contributors and editors. Reports are the formal mechanism used to deliver results of OGC Initiatives to Sponsors and to the OGC Standards Program for consideration by way of Standards Working Groups and Domain Working Groups.

Document content should follow this OGC Document Editorial Guidance (scroll down to view PDF file content). File names for documents posted to Pending should follow this pattern (replacing the document name and deliverable ID): OGC Testbed-21: Ontologies for Coverage Description Report (D204).

Component Implementations include services, clients, datasets, and tools. A service component is typically delivered by deploying an endpoint via an accessible URL. A client component typically exercises a service interface to demonstrate interoperability. Implementations should be developed and deployed in all threads for integration testing in support of the technical architecture.

Component implementations are often used as part of outreach demonstrations near the end of the timeline. To support these demos, component implementations are required to include Demo Assets. For clients, the most common approach to meet this requirement is to create a video recording of a user interaction with the client. These video recordings may optionally be included in a new YouTube Playlist such as this one for Testbed-15.

Since server components often do not have end-user interfaces, participants may instead support outreach by delivering static UML diagrams, wiring diagrams, screenshots, etc. In many cases, the images created for a Report will be sufficient as long as they are suitable for showing in outreach activities such as Member Meetings and public presentations. A server implementer may still choose to create a video recording to feature their organization more prominently in the new YouTube playlist. Another reason to record a video might be to show interactions with a "developer user" (since these interactions might not appear in a client recording for an "end user").

Note that all Participants are required to provide at least some level of in-kind contribution (costs for which no cost-share compensation has been requested). As a rough guideline, a proposal should include at least one dollar of in-kind contribution for every dollar of cost-share compensation requested. All else being equal, higher levels of in-kind contributions will be considered more favorably during evaluation. Participation may also take place by purely in-kind contributions (no cost-share request at all).

Once the proposals have been evaluated and cost-share funding decisions have been made, OGC Staff will begin notifying Bidders of their selection to enter negotiations to become and initiative Participant. Each selected bidder will enter into a Participation Agreement (PA), which will include a Statement of Work (SOW) describing the assigned deliverables.

OGC Staff will provide monthly progress reports to Sponsors. Ad hoc notifications may also occasionally be provided for urgent matters. To support this reporting, each testbed participant must submit (1) a Monthly Technical Report and (2) a Monthly Business Report by the first working day on or after the 3rd of each month. Templates and instructions for both of these report types will be provided.

The purpose of the Monthly Business Report is to provide initiative management with a quick indicator of project health from each participant’s perspective. OGC Staff will review action item status on a weekly basis with assigned participants. Initiative participants must remain available for the duration of the timeline so these contacts can be made.

Each Participant should submit a Final Summary Report by the milestone indicated in the Main Schedule. These reports should include the following information:

This report may be in the form of email text or a more formal attachment (at the Participant’s discretion).

Clicking on the Propose link will navigate to the Bid Submission Form. The first time through, the user should provide organizational information on the Organizational Background Page and click Update and Continue.

The following screenshot shows the Organizational Background page:

Clicking Update and Continue will navigate to an "Add Deliverable" page that will resemble the following:

The user should complete this form for each proposed deliverable.

On the far right, the Review link navigates to a page summarizing all the deliverables the Bidder is proposing. This Review tab won’t appear until the user has actually submitted at least one deliverable under the Propose tab first.

Once the Submit button is clicked, the user will receive an immediate confirmation on the website that their proposal has been received. The system will also send an email to the bidder and to OGC staff.

The user is afforded an opportunity under Done Adding Deliverables at the bottom of this page to attach an optional Attached Document of Explanation.

If this attachment is provided, it is limited to one per proposal and must be less than 5Mb.

This document could conceivably contain any specialized information that wasn’t suitable for entry into a Proposed Contribution field under an individual deliverable. It should be noted, however, that this additional documentation will only be read on a best-effort basis. There is no guarantee it will be used during evaluation to make selection decisions; rather, it could optionally be examined if the evaluation team feels that it might help in understanding any specialized (and particularly promising) contributions.

The Propose link takes the user to the first page of the proposal entry form. This form contains fields to be completed once per proposal such as names and contact information.

It also contains an optional Organizational Background field where Bidders (particularly those with no experience participating in an OGC initiative) may provide a description of their organization. It also contains a click-through check box where each Bidder will be required (before entering any data for individual deliverables) to acknowledge its understanding and acceptance of the requirements described in this appendix.

Clicking the Update and Continue button then navigates to the form for submitting deliverable-by-deliverable bids. On this page, existing deliverable bids can be modified or deleted by clicking the appropriate icon next to the deliverable name. Any attempt to delete a proposed deliverable will require scrolling down to click a Confirm Deletion button.

To add a new deliverable, the user would scroll down to the Add Deliverable section and click the Deliverable drop-down list to select the particular item.

The user would then enter the required information for each of the following fields (for this deliverable only). Required fields are indicated by an asterisk ("*"):

Cost-sharing funds may only be used for the purpose of offsetting burdened labor costs of development, engineering, documentation, and demonstration related to the Participant’s assigned deliverables. By contrast, the costs used to formulate the Bidder’s in-kind contribution may be much broader, including supporting labor, travel, software licenses, data, IT infrastructure, and so on.

Theoretically there is no limit on the size of the Proposed Contribution for each deliverable (beyond the raw capacity of the underlying hardware and software). But bidders are encouraged to incorporate content by reference where possible (rather than inline copying and pasting) to avoid overloading the amount of material to be read in each proposal. There is also a textbox on a separate page of the submission form for inclusion of Organizational Background information, so there is no need to repeat this information for each deliverable.

This field Proposed Contribution (Please include any proposed datasets) should also be used to provide a succinct description of what the Bidder intends to deliver for this work item to meet the requirements expressed in the Technical Architecture. This language could potentially include a brief elaboration on how the proposed deliverable will contribute to advancing the OGC standards baseline, or how implementations enabled by the specification embodied in this deliverable could add specific value to end-user experiences.

A Bidder proposing to deliver a Service Component Implementation can also use this field to identify what suitable datasets would be contributed (or what data should be acquired from another identified source) to support the proposed service.

A single bid may propose deliverables arising from any number of threads or tasks. To ensure that the full set of sponsored deliverables are made, OGC might negotiate with individual Bidders to drop and/or add selected deliverables from their proposals.