Apollo Reversed gRPC

Andi Lamprecht ·2026-05-11· 4 min read· Accepted

gRPC between UAV and Avatar

Context

We’ve chosen gRPC as the main way of expressing Uncrew APIs and later noted that: If the UAV is to initiate a connection to the Avatar (and it has to because UAVs can be behind NAT) then the gRPC implementation forces it to act as a gRPC client and not server.

Conservatively and intuitively a UAV should be an RPC server because it offers itself as a resource that can be commanded to do things (e.g.: Land). Now that we’re forced to see a client in it, we could argue that, as the times progresses and UAVs become more autonomous, they become “sentient” beings (clients) that seeks someone (server) to control them.

Meanwhile and back-on-Earth however, we need to model a pilot requesting that the UAV Stops and if we want to model it with an RPC (gRPC specifically) then we really want the UAV to expose:

rpc Stop() returns Result

Decision

We chose to follow a PoC we’ve trail-blazed for gRPC client/server reversal where:

• UAV is the gRPC client and Avatar gRPC server.
• Avatar exposes an rpc with which the UAV can subscribe for a stream of commands/requests.
• Avatar exposes an rpc with which the UAV can tell it the result of each previously received command (by its ID).

Consequences

We manage to establish an interface between the UAV and its Avatar. The interface is efficient, but it suggests rather strongly that we abuse gRPC:

• Amplified by the grpc linter we’re ending up with requests being sent as responses and responses sent as requests.
• We’re forcing the MAVLINKShim to hold additional state (command_id) - state that would otherwise been held by the gRPC server implementation.
• We’re forcing the Avatar to hold additional state - (command_id) - state that would otherwise been held by the gRPC client implementation.

Alternatives Considered

gRPC Tunnel

PoC written that examines the use of grpctunnel as means of reversing the client/server relation. grpctunnel offers a tunnel that a client can establish with the server and then use it as a Connection to expose and rpc and act as a gPRC server. The PoC worked well in golang, but it failed in C++.

WebSocket Tunnel

A PoC was written (and copied to here) which examines the use of WebSockets as an underlying transport for gRPC. To build, do:

cd ./PoC-WebSocket && mkdir -p build && cd build
cmake -DCMAKE_BUILD_TYPE=Debug -DASAN_BUILD=ON ../

However, running the PoC was done manually.

The PoC was successful insomuch that the primary goal was achieved (gRPC tunneled over websocket) but revealed significant trouble with gRPC and Protobuf in C++ code.

Some of the recommended ideas weren’t achieved within the timeline provided and it wasn’t identified how to achieve tight control of the data and thread resources that gRPC creates (a requirement for Mercury projects).

• not able to hand an already-open raw file descriptor to gRPC to provide unauthenticated tunnel. Instead, the PoC must:

  • instruct gRPC to open a listening socket (requiring an additional listenening socket introduces security considerations)
  • connect to that listening socket, and then handle proxying

• not able to identify an easy way to handle message allocations to limit/prevent heap fragmentation or exhaustion

• did not look for a way to limit the number of threads that gRPC creates (it appears to create one thread per core by default)

• it eventually revealed the existence of some significant crash-bugs [UNCREW-149, UNCREW-410] in protobuf and/or gRPC

WebSockets without gRPC

The trouble with gRPC in C++ caused some of the team to want to reconsider gRPC.

No PoC written, but multiple references are provided to support conclusions:

• WebSocket RFC describes WebSocket’s capabilities
• One C++ API’s view into using websockets
• Knowledge that golang also has libraries for websockets

Websockets are a transport protocol, generally assumed to be built on TCP but designed to be capable of supporting UDP, which doesn’t describe RPC capabilities. It is, effectively, a socket API designed for web apps to utilize (which often come with HTTP requirements). WebSockets therefore include a required HTTP-upgrade path. HTTP itself assumes TCP but isn’t strictly required (you can find HTTP sent over local domain sockets (aka Unix sockets)) or encapsulated in UDP via HTTP3/QUIC.

Much of the attractiveness of websockets comes from easier/better support (in C++ in particular) and better conformance to web standards than gRPC. The tradeoffs were not significantly discussed in the form of proofs of concepts.

The lack of a well-defined RPC protocol is contentious. Several of the team consider it a drawback and a few consider it a benefit. The drawbacks are that we’d have to define our own RPC protocol. The benefits are that will allow us better flexibility as well as allowing us to use off-the-shelf tools to debug websockets-based protocols.

WebSockets also do not prescribe a data serialization format. That is a larger concerned shared by all of the team. Several options are being considered in a separate ADR.