DATA Ingress DAA Telemetry
Originally
ADR-0080-DATA-INGRESS-DAA-TELEMETRY (v3) · Source on Confluence ↗UTM Data Ingress Pattern (DAA & Telemetry)
Context
Invalid Image Path
UTM is a multi-service system. When data enters the system, other services need to know that arrived and what the
data is.
Examples
DAA Plots
uAvionix shares plots over UDP. For reasons describe below, DAA is a separate service. Subscribers of
plot data subscribe through the API or gRPC service. The plot data needs to make it’s way from DAA
to API and gRPC as quickly as possible.
Telemetry
Telemetry data will enter the system via gRPC and make it’s way to API as quickly as possible.
Decisions
Data Ingress Pattern
- Persist the Data : Save it in the database.
- Notify : Post the ids of the saved data to Redis
DAA runs as a separate service.
Decision Drivers
Data Ingress Pattern
In the case of plots and telemetry, the information is ephemeral. That fact might lead one to put the data in a Redis
Stream where the data expires after a certain time. However, storing the data in Postgres offer the following benefits:
- Faster query options
- Less processing while reading data
- Data is more permanent and available for later research
- Easily transferable to archive
- Easily purged
Redis still plays a part. It is the nervous system of UTM. A service the receives data may notify the rest of the
system by posting the id of the newly saved record to a Redis stream. Services interested in the data can subscribe
to the stream and use the id to load the data from Postgres.
DAA as a Separate Service
The DAA service will listen on a UDP port for plot data. The networking concerns alone are reason enough to warrant
separating it. We don’t want to complicate existing services with the extra ingress and firewall restrictions. The
DAA service also has different scaling needs.
Use Case / Logic Flow
- A residential birthday party near Dallas Love Field Airport needs Pizza now. They place an order with their local pizzeria and request DroneUp delivery.
- Uncrew submits a flight intent, Mission-Pizza-Now, to UTM.
- Danny the DroneUp drone pilot is elected to fly Mission-Pizza-Now.
- Danny opens the UTM Spaces app and begins tracking Mission-Pizza-Now before taking off.
- Uncrew begins sending drone telemetry to UTM. Danny can see his drone moving on the Spaces map.
- Meanwhile, George, the GA pilot, is returning home from business trip to his base airport, Dallas Love Field.
- uAvionix sees George and sends plot data to UTM. We dive into this process below.
- Danny can see George’s aircraft in the vicinity of his Drone.
- The Drone is nearing its destination on the approach end of Dallas Love Field.
- George is on final approach and flying lower and he should be.
- UTM receives a plot in the DAA service. The plot is saved to Postgres and the ID is streamed through Redis.
- The API service receives the ID of the plot and loads the data from Postgres.
- API queries Postgres for active drone flights (telemetry) that are within a 2-mile radius of George in the past 60 seconds.
- API observes that Mission-Pizza-Now is on a collision trajectory with George.
- ALERT! API sends a message (websocket) to Danny’s Spaces session.
- Spaces draws a red circle (OCAV - Own-ship Collision Alerting Volume) around the drone and plays an audible alarm sound.
- Danny observes the alert and presses the “Defensive Crouch” button on his pilot app.
- The drone ducks out of the way.
- George flies safely above the drone and lands. Home at last.
- Mission-Pizza-Now completes its pizza delivery to a dozen hungry and happy children.
Consequences
- Performance: Sharing data over Redis is faster that saving it in Postgres and then loading it from Postgres.
- Postgres Load: Postgres may need extra maintenance to handle the higher number of write-connections and data churn.