jev-pilot: Handing a Drone to a Model That Can't Talk

Most "LLM flies a drone" demos look the same: you type an order, a chat model writes back a plan or some coordinates, and something parses the text and hopes for the best. My own earlier project, the LLM FPV Flight Assistant, was designed around a strict rule for exactly that reason: the LLM never sits in the real-time control loop. It proposes a command, a deterministic safety gate decides, and the model stays far away from anything that has to happen fast. With a chat model, that's the right call. It can invent a coordinate, reply with a verb you never offered, or take three seconds to answer while the drone drifts.

With jev-pilot I wanted to find out what it would take to break that rule safely. The answer I envisioned was a model that cannot write a single word, sitting inside the flight loop and making one small decision every 400 milliseconds, while plain code keeps everything that has to be exact. The safety gate from the earlier project stays exactly where it was.

The model is Jev, TypeSafe's "System One" decision model. You type fly to the tower, Jev reads the order, and then 2.5 times a second it picks the next manoeuvre from six options, in words, while code handles the geometry, the vetoes and the vehicle. Everything runs in PX4 SITL with Gazebo, and every decision is visible in a cockpit page as it happens:

That's a real recorded flight, replayed at 4×: 135 Jev decisions, 57 seconds, start to tower, nothing hit, for less than one US cent.

First, what Jev actually is

The simplest way I can put it: a chat model is a student who writes essays. Jev is a student who only takes multiple-choice exams, and next to every answer writes down how sure it is.

You don't send Jev a prompt. You send it a state (a JSON description of the situation) and a list of named questions. Each question is one of three types:

• choice: pick one of these options
• score: rate this on a scale
• noul: yes or no, as a probability

Jev answers all of them at once, in parallel, and returns a probability for every option. It never produces free text, and it can't produce a number either. The input side is just as narrow: text only. No images, no video, no audio. If something isn't written into the state, Jev doesn't know it exists. TypeSafe trains it specifically so those probabilities are calibrated: when it says 90%, it's supposed to be right about 90% of the time.

Here's the idea in its smallest form. This is one real decision from the flight, cut down to the parts that matter:

// IN: the state, i.e. the situation, in words
"direct_line_to_target": "blocked",
"maneuvers": {
  "hold_course": "direct line to target blocked",
  "gap_left": "clear (>15 m) toward far left",
  "climb": "nothing ahead to climb over",
  "gap_right": "clear (>15 m) toward right, nearer the target"
}
 
// IN: the question, with the options to pick from
"maneuver": {
  "type": "choice",
  "instructions": "Which manoeuvre should the vehicle fly?",
  "criteria": { "hold_course": "...", "gap_left": "...", ... }
}
 
// OUT: Jev's answer, 319 ms later
"maneuver": {
  "choice": "gap_right",
  "probabilities": { "gap_right": 0.92, "gap_left": 0.08 }
}

In plain words: "The straight way is blocked. There's room on the left, and room on the right that's closer to the tower. Which do you take?" Jev answers "right, 92% sure." That's the whole interaction. There's no conversation and no explanation, just a pick with a confidence attached. The full version of this call is further down.

Here's how it read my order, before the flight started:

One call, six questions: which action (take off, fly to, orbit, hold, return home, land, stop, or ask me), which target from the known landmarks, which altitude, radius and speed band, and one yes/no: is this order ambiguous? A reading above 0.9 just goes. Below that, it comes back as a proposal that wants a yes. If it's genuinely unclear, it becomes a question. go to blue came back as the blue pillar at 0.99. fly to the tower then land on the pad, two orders in one, comes back as a question, which is exactly what I wanted. Across a scorecard of 21 test orders, Jev got all 21 right at around 410 ms each, and the whole scorecard cost about $0.0015.

Why Jev is a natural fit for the pilot's seat

When I sketched the idea, three properties made Jev feel less like a compromise and more like the right shape for this job.

1. It's fast enough to live inside the loop. A typical answer took about 350 ms. That means the vehicle gets a fresh decision every 400 ms instead of a fresh plan every few seconds. It's the difference between someone reading you directions once at the start of a drive and a co-pilot calling every turn as it comes.

2. It's narrow enough to be safe. Jev can only pick from options the code offered. There's no text to parse, no coordinate it could hallucinate, no verb that wasn't on the list. The worst a wrong answer can do is choose a bad option from a list of flyable options, and the code still gets to say no to it.

3. It's honest about doubt. Because every option comes back with a probability, the system can act on confidence itself: go when Jev is sure, hover when it isn't, ask the operator when an order is unclear. A chat model that says "Sure! Flying to the tower now." in the same tone whether it's 99% or 51% sure gives you nothing to work with there.

There was also a precedent that convinced me this wasn't crazy: TypeSafe had shown Jev playing Doom, polled about ten times a second with a text report of the situation and a fixed set of questions (move, turn, fire), while ordinary code did the fine aiming. A drone loop has the same shape: a few hertz of high-level judgement on top of fast, exact low-level control.

The deal: geometry in code, judgement in Jev

Jev has one big weakness, and the whole design follows from it: it's bad with numbers. TypeSafe's own notes say so directly. It isn't a calculator, so do arithmetic and comparisons in code.

So I set one rule and never broke it: no raw number ever reaches Jev. Distances, bearings and sensor ranges are computed in code and handed over as words. A wall 4.2 m away becomes tight (3-6 m). A target 38° to the left becomes ahead-left. Jev reads the words and makes the call. Code turns the call back into numbers.

Here's the full version of the call from earlier: tick 18 of the recorded flight, about seven seconds into the course, with the straight line to the tower blocked. This is the entire input Jev gets, in two parts: the state (the situation, in words) and the questions (what I want to know, with the options spelled out). The state is exactly as sent; only the long question texts are trimmed:

{
  "model": "~typesafe/jev-latest",
  "state": {
    "goal": {
      "kind": "fly_to",
      "target": "tower",
      "speed_band": "normal"
    },
    "vehicle": {
      "altitude": "very low (1-5 m)",
      "speed": "moderate (3-6 m/s)",
      "battery": "good (>50%)",
      "health": "gps ok, estimator ok",
      "mode": "OFFBOARD",
      "armed": true
    },
    "scene": {
      "sectors": {
        "far left": "clear (>15 m)",
        "left": "open (6-15 m)",
        "ahead": "clear (>15 m)",
        "right": "clear (>15 m)",
        "far right": "open (6-15 m)"
      },
      "obstacle_ahead": "none",
      "direct_line_to_target": "blocked",
      "target": "visible, ahead"
    },
    "target": {
      "name": "tower",
      "distance": "medium (20-50 m)"
    },
    "maneuvers": {
      "hold_course": "direct line to target blocked",
      "gap_left": "clear (>15 m) toward far left",
      "gap_right":
        "clear (>15 m) toward right, nearer the target",
      "climb": "nothing ahead to climb over",
      "brake": "stop and hover",
      "reacquire": "target in view, not needed"
    },
    "recent_decisions": [
      { "maneuver": "gap_left", "risk": "clear" },
      { "maneuver": "gap_left", "risk": "tight" },
      { "maneuver": "gap_left", "risk": "tight" }
    ],
    "elapsed": "under 30 s"
  },
  "questions": {
    "maneuver": {
      "type": "choice",
      "instructions": {
        "question":
          "Which manoeuvre should the vehicle fly ...",
        "rules": [
          "... take a gap whose entry reads clear ...",
          "Never choose an option whose entry says ...",
          // ...four more rules of thumb
        ]
      },
      "criteria": {
        "hold_course": "keep flying toward the target",
        "gap_left":
          "steer into the open space on the left, then ...",
        "gap_right":
          "steer into the open space on the right, then ...",
        "climb": "rise over the obstacle ahead and continue",
        "brake": "stop and hover this tick",
        "reacquire": "turn in place toward the target, ..."
      }
    },
    "risk": {
      "type": "choice",
      "instructions":
        "How close is the vehicle to hitting something ...?",
      "criteria": {
        "clear": "...", "tight": "...",
        "close": "...", "about to hit": "..."
      }
    },
    "goal_reached": {
      "type": "noul",
      "instructions":
        "Is `target.distance` equal to `arrived (<5 m)`?"
    }
    // ...plus three more yes/no questions: route_blocked,
    //    target_truly_lost, emergency_stop
  }
}

Notice what's not in there. There are no coordinates, no metres as numbers, no image, and no history beyond the last three picks. Every option in maneuvers comes with one sentence of evidence for it, written by code. That turned out to be the most important part of the design: the annotations are the state. Jev isn't solving geometry. It's reading six short, honest arguments and picking the best one, which is precisely the kind of question it's good at.

And this is what came back, 319 ms later:

{
  "maneuver": {
    "choice": "gap_right",
    "confidence": 0.90,
    "probabilities": {
      "gap_right": 0.92, "gap_left": 0.08, "hold_course": 0,
      "climb": 0, "brake": 0, "reacquire": 0
    }
  },
  "risk": {
    "choice": "tight",
    "probabilities": {
      "tight": 0.89, "clear": 0.09,
      "close": 0.01, "about to hit": 0.01
    }
  },
  "route_blocked": 0.34,
  "target_truly_lost": 0.02,
  "goal_reached": 0.01,
  "emergency_stop": 0.04
}

Jev followed the rule it was given: both sides read clear, so take the one marked nearer the target, which is right, at 92%. But the drone had been going left for three ticks, and switching sides mid-manoeuvre is how you end up going nowhere, so the code held left. You'll see this exact tick again in the cockpit screenshot further down.

One tick, in plain words

Every 400 ms the system does this:

1. Look. A forward "sensor" casts rays and splits the view into five slices across 120°, each reporting the nearest obstacle within 15 m.
2. Put it into words. Every distance becomes a band: blocked, tight, open or clear.
3. Argue each option. Code writes the one-line evidence for each of the six manoeuvres: hold course, gap left, gap right, climb, brake, reacquire.
4. Ask Jev. One call, around 350 ms, a probability for every manoeuvre plus a risk level and a few yes/no checks.
5. Sanity-check it. Code turns the choice into a heading, a speed and an altitude, and applies its rules (more on those below).
6. Send it. The command goes to the flight-safety gate carried over from the earlier project, which is the only part of the system allowed to talk to the drone. A 10 Hz sender keeps re-sending the last command between decisions, and a reflex can still veto it if the world changed in the meantime.

Jev proposes, code decides

I never wanted Jev to be the last word, only the best opinion in the room. The code side has a few rules that can overrule it, and the cockpit shows every time they do:

• Risk is clamped to what the sensor actually sees. Jev can make the drone more careful than the geometry says, by one level at most. It can never make it less careful. It can slow you down, but it can't wave you through a wall.
• Once you pick a side, you keep it for 3 seconds. More on why below. That's the override in the screenshot.
• Unsure means hover. Under 50% confidence, the drone stops and hovers. If hovering for 2 seconds doesn't change the picture, Jev's top pick is flown at a crawl.
• A reflex that doesn't ask anyone. Something within 3 m straight ahead means no forward motion, whatever was decided.

Here's how often that mattered in the recorded flight:

34 of 135 picks were overruled: 29 by the side commitment, 4 for low confidence, 1 by the reflex. Everything else Jev asked for, it got.

The course and the flight

To make the test honest, the course is defined once as data and generates both the Gazebo world and the obstacle map the sensor reads, so the two can never disagree:

54 m from start to tower, with a pillar, a wall too tall to climb (gap on the left), a low bar meant to be climbed over, and another wall at the edge of the sensor's view.

Where it falls short

This is the part I care about most, because nearly every interesting design choice in this project is a direct answer to one of Jev's weaknesses. Here they are, as plainly as I can put them.

1. It can't see

The drone in the simulator has a camera, and the cockpit streams it, but that feed is only for me as the human watching. Jev never sees a single pixel. It only accepts text. So it can't look at the scene and judge it the way a person would. Code has to look first and describe the scene to it in words: ahead: tight (3-6 m), obstacle ahead: low (climbable), target: visible, ahead-left.

Imagine flying a drone blindfolded while a friend describes the view to you over the phone. You can make good calls, but only about the things your friend thought to mention. If the description leaves out a tree, you'll fly into the tree. That's Jev's situation on every tick, and it's why the way the scene is put into words matters so much in everything below.

This is also why the sensor in this project is a simple range sensor rather than a camera. Distances can be turned into words exactly, with no guessing. To use the camera, a separate vision model would first have to turn the picture into words, which adds time to every tick and adds a second thing that can be wrong before Jev even starts.

2. It can't count, so don't make it

Asking Jev "is 4.2 m less than 6 m?" is like asking a brilliant reader to do your taxes in their head. Sometimes it works. You shouldn't bet the drone on it. Hence the rule that no number ever reaches it. The cost is that everything Jev knows is rounded into buckets. It can't tell 3.1 m from 5.9 m, because to Jev both are just tight. Any precision that matters has to live in code.

3. It reads exactly what you wrote, and nothing more

Jev is a very literal reader. If the rule says "go forward when the target is ahead" and the target is ahead-left, Jev doesn't stretch the rule. It just doesn't go forward. One early flight did exactly that: a rule that named ahead but not ahead-left, combined with a turn rate that kept overshooting the ahead slice, gave a drone that turned back and forth and never moved forward.

Another one was nastier. A gap on the side was first worded band-first, something like tight (3-6 m) toward right, and Jev read that as a way through. It flew the drone toward a wall. The fix was purely about wording: a squeeze past an obstacle is now called a creep along the obstacle in its first words. Most of the tuning in this project was wording, not thresholds. Change one annotation and you should expect the behaviour to change with it.

I found the same pattern when I went through the recorded flight myself. Each tick Jev also answers a yes/no: "is the straight line to the target blocked by something the vehicle can't fly over?" That question comes with fine print (its criteria): answer yes only when the line is blocked and the obstacle straight ahead is too tall to climb. Here's how Jev answered, split by what the state said:

The interesting row is the last one. When the line was blocked by something off to the side, the plain-English question and its fine print disagree. The question says "yes, it's blocked". The fine print says "no, because nothing tall is straight ahead". Jev sided with the fine print on 41 of 52 ticks and with the plain English on the other 11. That isn't really Jev's mistake. The question contradicted itself, which TypeSafe's own docs warn against, and a literal reader showed it. This is also why I designed it so no rule reads that answer. It's only shown in the cockpit.

4. It has no memory, so it flip-flops

Every call is a fresh exam. Jev doesn't remember what it picked 400 ms ago, apart from a short "recent decisions" list I hand it. Over the recorded flight it picked "gap right" 38 times and "gap left" 37 times, and around the first wall it kept switching between them. A drone that turns left, then right, then left again cancels its own turning and goes nowhere. That's where the 3-second side commitment comes from: pick a side, keep it, and only switch if the other side starts to look clearly better. Without it the drone hesitates. With it, the drone commits like a person would.

5. Fast is not instant

350 ms is quick for a language model and slow for a flying object. About 1 in 5 answers took longer than a full 400 ms tick, and the worst took nearly a second. The system handles it by repeating the last command at 10 Hz until a new answer arrives (the flight-safety gate hovers the drone if commands stop for half a second), and it ignores any answer older than 1.5 s.

The harder lesson was about stale pictures, not slow answers. In the first version, the telemetry from the gate was effectively 3 seconds old, and Jev was asked "turn left or right?" off that old picture. The drone pivoted in place, endlessly correcting a heading it had already passed. It's like steering a car while watching a video feed from three seconds ago. The fix was to take steering away from Jev entirely: heading is now handled in code, from an estimate of where the drone is now (the last known position, moved forward by the commands sent since). Jev decides what to do. Code decides exactly how.

6. Its doubt can freeze it

Hovering when unsure sounds safe, and mostly it is. But there's a hole I'm leaving out in the open: if Jev is unsure and its top pick is brake, the "fly the top pick slowly" rule just produces another hover. In that one case, indecision never resolves on its own.

7. The honest fine print

• The sensor is virtual. Since Jev can't take images anyway, the "eyes" are a range sensor that casts rays against a known map, so it never misses an obstacle and never sees one that isn't on the map. No noise, no glare. This project is about the decision layer, not about perception.
• Simulation only. Everything here is PX4 SITL and Gazebo on one machine. Nothing has flown for real.
• One course, and it can be cheated. The walls are free-standing, so a pilot that keeps taking the roomiest side can go around them instead of through the gaps, and never really meets the climb station.
• "Arrived" means within 9 m. The tower is itself an obstacle, and the reflex keeps 3 m away from everything. An early run told to get within 5 m of a 6 m tower hovered just outside its own safety bubble until the clock ran out.

What I'd try next

Every tick is already recorded with Jev's full probabilities, its answer time and what was actually flown, so three next steps are close at hand:

• A calibration study. Jev is trained to be honest about its confidence. The recordings can check whether, in this setting, 80% really means right 80% of the time.
• A real range sensor behind the same interface, so the decisions come from what a lidar or depth camera sees instead of from a perfect map.
• Flip the roles: Jev as a guard. Let a conventional planner choose the manoeuvre, and ask Jev one yes/no per tick: "is this about to hit something?" The same typed question, used as a veto instead of a chooser. At $0.0085 for a whole flight, it's cheap enough to run beside almost anything.

The bigger takeaway for me is that the useful question isn't "can a language model fly a drone?" It's which small piece of judgement can you hand to a model, and how do you shape the world into words so that piece becomes an easy question? Jev turned out to be a surprisingly good fit for that piece, as long as everything around it treats its answers as the best opinion in the room, never as the final word.

The project, the cockpit and the recorded flight are on GitHub. jev-cockpit --demo replays the flight above with no simulator and no API key.