DATA Pipeline ATLAS 6 Notificationservice

Andi Lamprecht ·2026-05-11· 2 min read· Accepted

Context

Atlas must notify it’s end users about changes done to the data

Decision

Atlas create a last step in the Airspace pipeline which serves as notification service. This step is going to utilize Change Data Feed feature of delta lake to extract latest changes done to the pipeline.

Invalid Image Path

Notification lifecycle

1. Get last_change_version
2. Read changes from airspace table that are greater by last_change_version
3. Create a message for each change with a content:

{
    airspace_uuid: <airspace_id>,
    change_type: <create/update/delete>,
    geometry: <wkt>
}

1. Publish the messages to a topic equal to ruleset name
2. Update last_change_version

Consequences

1. Subscribers are notified when the change is applied
2. Subscribers need to check if the change is in their area
3. Subscribers must read the full change information for db

Alternatives considered

1. Hub as a topic.
   This design is promising but require a separate DroneUp-maintained table that contains areas for interest for each hub. Having that Atlas could map a changes with each hub area, and remove the need of downstream processing.
2. Full change data in a message.
   Another alternative to be considered - this can increase the message size, especially for large change sets. Although if one message represents one change this could be implemented considering 10MB PubSub message limit, with acceptance the bigger traffic on a message bus.