Data Contracts

Understanding Data Contracts: An Overview

Data contracts are abstraction layer agreements between data producers and data consumers. 

They can help us prevent data quality issues by formalizing interactions between different systems and teams handling data.

Often application and data teams are separated by ELT/ETL infrastructure and software developers have little visibility into how the data is being consumed.  For the data producers, the data platform is a black box.  Converting the term data contracts is a way to raise awareness of this issue even though the implementation is almost similar to what developers have already been doing using APIs.    

Data Contracts regulate:  

- the structure/schema of the data (semantic nature of entities, events, attributes)  

- quality  

- the terms of use  

As the business requirements evolve, changes can be made to both the data structure and the triggering logic. 

Structure/schema of the data 

To make use of any data, you need to know what the structure of the data is. This is typically defined in a schema. Schemas are already widely used, for example by relational database. You can use different Interface Definition Languages (IDLs) to define schemas for serialized data, including protocol buffers, Apache Thrift, and Apache Avro. There’s also JSON Schema for defining a schema over JSON documents, upon which standards such as OpenAPI and AsyncAPI are built. Alternatively, you could define your schemas in some custom or abstract format, for example in YAML or JSON or even in code.