Skip to content
Documentation
ADRs
SHARED
Airdex Sensor Ingest Architecture

Airdex Sensor Ingest Architecture

Andi Lamprecht Andi Lamprecht ·· 7 min read· Proposed
ADR-0264 · Author: Ihor Prozhoha · Date: 2026-02-19 · Products: shared
Originally ADR--0137--AirDEX Sensor Ingest Architecture (v2) · Source on Confluence ↗

Context

ATOMx requires integration with AirDEX sensors to provide real-time surveillance data for both drone and manned aircraft positions. AirDEX provides advisory aircraft position information via their v2/messages API endpoint, delivering telemetry data that must be ingested, processed, and displayed in Uncrew for situational awareness.

Current State:

  • No existing AirDEX integration in the traffic service architecture
  • Uncrew requires advisory aircraft display capability (display-only, no alerting initially)
  • Need alignment with ADR-001 Flight Traffic Exchange Architecture
  • First feature implementation for the traffic service

Requirements:

  • Ingest drone and aircraft position data from AirDEX v2/messages API
  • Process and normalize AirDEX telemetry into standardized traffic observations
  • Display advisory aircraft positions in Uncrew UI (read-only view)
  • Start with simplified architecture for rapid delivery
  • No alerting or collision detection in initial phase (display-only)

Decision

We will implement AirDEX sensor ingest as the first iteration of the Flight Traffic Exchange using a simplified architecture without Kafka. This initial implementation focuses on rapid delivery of display functionality using a single unified codebase with CLI-based component selection.

This is a first iteration approach that will eventually be enhanced to fully satisfy ADR-001: Flight Traffic Exchange Architecture with Kafka-based event sourcing, multi-source fusion, and comprehensive observability.

Architecture Overview 

┌─────────────────────────────────────────────────────────────────────┐
│                       AIRDEX PROVIDER                               │
│                                                                     │
│  ┌──────────────────────────────────────────────────────────────┐   │
│  │  AirDEX Provider                                             │   │
│  │  - Polls v2/messages API (configurable interval)             │   │
│  │  - Authenticates with AirDEX credentials                     │   │
│  │  - Parses aircraft/drone position data                       │   │
│  │  - Converts to standardized Observation format               │   │
│  │  - Forwards to WebGRPC Processor (internal gRPC)             │   │
│  └────────────────────────────┬─────────────────────────────────┘   │
└─────────────────────────────────┼─────────────────────────────────┘
                                  │
                                  │ gRPC (internal)
                                  │
                                  ▼
      ┌─────────────────────────────────────────────────────────────┐
      │                     WEBGRPC PROCESSOR                       │
      │                                                             │
      │  ┌──────────────────────────────────────────────────────┐   │
      │  │  WebGRPC Traffic Processor                           │   │
      │  │  - Receives observations via internal gRPC           │   │
      │  │  - Maintains in-memory cache (last 60s per aircraft) │   │
      │  │  - Basic filtering by geographic bounds              │   │
      │  │  - gRPC-Web server for external clients              │   │
      │  │  - Streams position updates to connected clients     │   │
      │  │  - No fusion/correlation in Phase 1                  │   │
      │  └────────────────────────────┬─────────────────────────┘   │
      └─────────────────────────────────┼─────────────────────────┘
                                        │
                                        │ gRPC-Web
                                        │
                                        ▼
                                ┌──────────────┐
                                │   Uncrew UI  │
                                │   - Display  │
                                │     aircraft │
                                │     positions│
                                │     on map   │
                                └──────────────┘

Implementation Strategy

Single Project Structure:

flight-traffic-exchanger/
├── cmd/
│   ├── provider/         # CLI entry point for provider mode
│   └── processor/        # CLI entry point for processor mode
├── internal/
│   ├── provider/
│   │   └── airdex/       # AirDEX API client & provider logic
│   ├── processor/
│   │   └── webgrpc/      # WebGRPC processor logic
│   ├── grpc/             # Internal gRPC client/server utilities
│   ├── models/           # Shared data models (Observation, Track)
│   └── config/           # Configuration management
├── go.mod
└── Makefile

CLI-Based Component Selection:

# Run as provider
./flight-traffic-exchanger provider airdex

# Run as processor
./flight-traffic-exchanger processor webgrpc

Component Details

1. AirDEX Provider

Responsibilities:

  • Poll AirDEX v2/messages API at configurable intervals (default: 5 seconds)
  • Authenticate using AirDEX API credentials (stored in environment/secrets)
  • Parse JSON responses containing aircraft/drone positions
  • Convert to standardized Observation protobuf message
  • Forward to WebGRPC Processor via internal gRPC stream
  • Implement rate limiting

Error Handling:

  • Exponential backoff on API failures (max 30s)
  • Continue operation on parse errors (log and skip malformed messages)
  • Circuit breaker pattern for sustained failures (open after 5 consecutive failures)

2. WebGRPC Processor

Responsibilities:

  • Receive observations from Provider via internal gRPC stream
  • Maintain in-memory cache of recent positions (last 60 seconds per aircraft)
  • Serve gRPC-Web endpoint for real-time streaming to external clients
  • Filter observations by geographic bounds (client-provided bounding box)
  • Push updates to connected clients on new data

Client Connection Flow:

1. Uncrew UI opens gRPC-Web connection to processor
2. Sends TrafficRequest with geographic bounds (map viewport)
3. Processor sends initial snapshot of aircraft in bounds
4. Processor streams incremental updates as new data arrives
5. Client updates map markers

Scaling Considerations (Future):

  • Current: Single provider → single processor (sufficient for Phase 1)
  • Future: Multiple processors with load balancing via Kafka consumer groups
  • Future: Shared state via Redis for multi-processor deployments

Phase 1 Scope (Initial Implementation)

In Scope:

  • ✅ AirDEX provider implementation (polling v2/messages)
  • ✅ Internal gRPC communication (provider → processor)
  • ✅ WebGRPC processor (basic streaming)
  • ✅ gRPC-Web API for Uncrew UI
  • ✅ Display aircraft positions on Uncrew map
  • ✅ Basic geographic filtering
  • ✅ Single unified codebase with CLI
  • ✅ In-memory cache (60s TTL, configurable)

Out of Scope (Future Phases):

  • ❌ Kafka event sourcing and durability
  • ❌ Track fusion (correlation with other sources)
  • ❌ Redis cluster for track state
  • ❌ Alerting/collision detection
  • ❌ Historical replay
  • ❌ Other providers (FlightAware, FAA SWIM, etc.)
  • ❌ Other processors (MQTT, ASTERIX, persistence)
  • ❌ Kalman filtering or position smoothing
  • ❌ Horizontal scaling (multi-processor deployments)

User Stories & Estimation

Epic: AirDEX Advisory Aircraft Display in Uncrew

Story 1: AirDEX Provider Implementation

As a traffic service operator
I want to ingest aircraft position data from AirDEX
So that advisory aircraft are visible in the system

Acceptance Criteria:

  • Provider polls AirDEX v2/messages API
  • Successfully authenticates with AirDEX API
  • Parses JSON response and extracts position data
  • Converts to standardized Observation protobuf
  • Forwards to processor via internal gRPC stream
  • Implements exponential backoff on failures
  • Logs all errors with structured logging and OTEL
  • Includes OTEL metrics (messages received, forwarded, errors) (OPTIONAL)
  • Rate limiter

Technical Tasks:

  • Set up Go project structure with CLI (provider mode)
  • Implement AirDEX API client with OAuth2
  • Define Observation protobuf schema
  • Implement internal gRPC client for streaming observations

Estimation: 5–6 days

Story 2: WebGRPC Processor Implementation

As an Uncrew user
I want to see real-time aircraft positions on the map
So that I have situational awareness of nearby traffic

Acceptance Criteria:

  • Processor receives observations via internal gRPC stream
  • Maintains in-memory cache of recent positions (60s TTL, configurable)
  • Exposes gRPC-Web endpoint for external clients
  • Supports bidirectional streaming (TrafficRequest/TrafficUpdate)
  • Filters aircraft by geographic bounds (bounding box)
  • Pushes updates to clients within 1 second of receiving observation
  • Includes OTEL metrics (active connections, messages sent) (OPTIONAL)

Technical Tasks:

  • Define gRPC service protobuf
  • Implement internal gRPC server to receive observations from provider
  • Build in-memory cache with TTL (using sync.Map or similar)
  • Implement external gRPC-Web server for client streaming
  • Add geographic bounding box filtering logic

Estimation: 4–5 days

Story 3: Uncrew UI Integration

As an Uncrew operator
I want to see advisory aircraft markers on the map
So that I can visualize nearby traffic

Acceptance Criteria:

  • Uncrew UI connects to WebGRPC processor via gRPC-Web
  • Sends bounding box based on current map viewport
  • Displays aircraft markers on map (unique icon for AirDEX data)
  • Shows aircraft details on marker click (callsign, altitude, speed)
  • Updates markers in real-time as position changes
  • Handles reconnection on connection loss

Technical Tasks:

  • Generate TypeScript types from proto files
  • Implement TrafficService client wrapper
  • Create aircraft marker component (React/MapboxGL)
  • Add marker clustering for high-density areas (OPTIONAL)
  • Implement position interpolation for smooth animation (OPTIONAL)
  • Add aircraft info popup component

Estimation: 3–4 days

Risk Factors:

  • ⚠️ AirDEX API documentation may be incomplete or have an issue (add 2–20 days for discovery)

Future Phases

Phase 2: Kafka Integration & Multi-Source Fusion

  • Introduce Kafka for event sourcing, durability, and replay (per ADR-001)
  • Refactor provider to publish to Kafka raw-observations topic
  • Refactor processor to consume from Kafka (consumer group for scaling)
  • Add FlightAware and FAA SWIM providers
  • Implement Fusion Workers with Redis cluster
  • Track correlation by ICAO address and position
  • Publish to fused-observations Kafka topic
  • Update processor to consume fused data

Phase 3: Advanced Features

  • MQTT Processor for IoT device distribution
  • Persistence Processor for InfluxDB storage
  • Archive Processor for S3 long-term storage
  • RBAC filtering for sensitive feeds
  • Historical replay capability

Alternatives Considered

1. Direct HTTP Polling to Uncrew

Architecture: Uncrew UI polls AirDEX directly via backend proxy

Pros:

  • Simpler, fewer components

Cons:

  • Not scalable
  • No provider/processor separation

Reason for Rejection:
Doesn’t support future multi-source architecture.

2. Separate Codebases for Provider and Processor

Architecture: Independent repositories for each component

Pros:

  • Independent deployment cycles
  • Language flexibility

Cons:

  • Code duplication
  • Complex dependency management
  • Harder to maintain

Reason for Rejection:
Premature optimization.

3. Immediate Kafka Implementation (Full ADR-001)

Architecture: Start with complete Kafka-based architecture from Day 1

Pros:

  • Full event sourcing, durability, replay from start

Cons:

  • Higher complexity
  • Longer time to first delivery
  • Requires Kafka infrastructure

Reason for Rejection:
Over-engineering for Phase 1 display-only requirements.

4. Native WebSocket Instead of gRPC-Web

Architecture: Native WebSocket protocol for processor-to-UI communication

Pros:

  • Simpler browser support
  • Mature libraries

Cons:

  • Manual protocol design
  • No type safety
  • Harder to version

Reason for Rejection:
gRPC-Web provides better type safety and ecosystem support; worth the additional complexity.

Cited by queries

Last updated on
Labelling Structure For Advisory DataADR 0000 Federated Design Infrastructure & Storage Architecture