ECHO

Andi Lamprecht ·2026-05-11· 2 min read· Accepted

ECHO - Ecosystem Central (data) wareHouse (and) Observability

The acronym is a stretch, but the name captures the concept.

DRAFT

Context

Investigating incidents is painful. Data is dispersed in multiple locations. Every system has its own logs.
Events are duplicated by systems in different formats.

“Why did the aircraft crash into the tree?” - Finding the answer using the structure above is no simple task.

We need a central repository for data and events. Then we can organize and simplify.

Decision

Part 1 : Data Lake

A data lake is a centralized storage system that can store, 
process, and analyze large amounts of data. Data lakes can store 
data in its original format, regardless of structure or size.

All systems should export their data to the data lake on a regular basis. For most systems,
GCP’s Datastream
will automate the process.

Part 2 : Pub/Sub

Archiver

The Archiver is a custom component that subscribes to all the topics of the ECHO pub/sub. Its job is to preserve
every event in the data lake.

Publishers

Systems send significant events to the ECHO pub/sub, along with associated data. The goal is to observe
significant events in one place. Subscribers are the driving force for what events are published, however,
observability may be reason enough to publish certain events.

Subscribers

Systems that are interested in a particular event can subscribe to the appropriate ECHO pub/sub topic.

Topics and Messages

• Echo pub/sub topic is created, via Terraform, by the publishing system.
• The schema of the event messages shall be documented in a format TBD (Apache Acro, JSON Schema, ProtoBuf),
  and published in a location TBD (DroneUp Docs, ???).
• The schemas must include a version number that is incremented with each change.

Examples

• UTM Airspace Updates
• HubOps Order Updated
• Mobile Commerce Order Creation
• Uncrew Mission Status
• UTM Flight Plan Status

Questions

• Should all log files go to the data lake?

Consequences

Real-time Events

ECHO pub/sub will have a latency of up to several seconds, which is not appropriate for real-time events.
For example, the DAA (Detect and Avoid) system needs aircraft telemetry as soon as possible. Real-time scenarios like
this will require alternative (direct-to-consumer) solutions. They may still publish events to Echo, however.

Alternatives Considered

• Apache Kafka
• RabitMQ