GPT-5.6 Luna is my new favorite model for Ladder Logic

After I published my last model bake-off, Gareth S. asked why I stopped after one answer. His comment inspired this eval. I gave five models up to five tries at each of 60 Ladder Logic exercises. After every failed try, the model saw the real compiler error or failed tests and could correct its answer. I wanted to see which model could reach working Ladder Logic, not only which model got the first answer right.

My earlier eval was still useful for comparing first answers and choosing models for rungs.dev Relay Chat. But it did not measure whether a model could use compiler and test feedback to recover from a failure.

There was another problem. The earlier eval used a generation version of the large prompt Relay uses in production. It listed Rungs Studio's instructions, explained the common traps, and supplied examples. I had built that prompt while working with Gemini-3-Flash-Preview. Was I measuring each model's Ladder Logic ability, or how well it matched a prompt built around Gemini?

I built a different eval to find out. Five models attempted all 60 playable Ladder Logic exercises in Rungs Studio. They got a deliberately small prompt and up to five tries. After each failure, they saw the real compiler error or failed tests and could fix their answer. I ran the full test three separate times.

Across all five models, only 377 of 900 first answers passed: 42%. When each model got up to five feedback-guided tries, 806 of 900 passed: 90%.

That 48-point difference is the result I was missing. The first answer is only half the story.

The prompt showed the format, not the answer

This was the complete system prompt:

Write Ladder Logic for the Logic routine of an Add-On Instruction, passing every test below. The tags listed are the AOI's own parameters and locals. Rungs run top to bottom, once per scan.

## Ladder DSL
- An instruction is NAME(args): a contact `XIC(In_Button)`, a coil `OTE(Out_Lamp)`, a block `MOVE(2000, Tmr.PRE)`.
- Adjacent instructions on a rung are in SERIES (AND): `XIC(In_Start) XIC(In_Run) OTE(Out_Motor)`.
- `[a, b]` is a PARALLEL branch (OR): `[XIC(In_Start), XIC(Out_Motor)] XIO(In_Stop) OTE(Out_Motor)` (seal-in).
- Each rung ends with `;`, which also separates rungs.

Output ONLY a JSON object, no prose, no markdown fence. The "logic" field is the LD DSL (rungs separated by ';'):
{"locals":[{"name":"Tmr_X","dataType":"TIMER","usage":"local"}],"logic":"XIC(In_X) TON(Tmr_X);\nMOVE(2000, Tmr_X.PRE);\nXIC(Tmr_X.DN) OTE(Out_Y);"}

The user message for each exercise supplied the required behavior, tag names, data types, and exact tests. That still gave the model some help. But the prompt did not list the available instructions, timer rules, comparison syntax, common traps, or worked answers. The model had to use what it learned during training.

The first try shows what the model already knows, with only enough help to write Rungs Studio's text format. The later tries measure the model, compiler, and tests working together.

How the five tries worked

Every exercise followed the same loop:

1. The model returned local tags and Ladder Logic as JSON.
2. Rungs Studio built the Add-On Instruction (AOI) and compiled the logic.
3. If compilation succeeded, the same tests that grade the exercise ran in the real test runner.
4. On failure, the model received either the compiler errors or the failed test messages, followed by: Fix the logic and reply with ONLY the corrected JSON object.
5. The loop stopped on a full pass or after five attempts.

No model graded another model. Pass means the logic compiled and every test passed. The 900 model–exercise pairs needed 1,846 tries and cost $8.67 in total.

I did not set a temperature because all five are reasoning models. The two Gemini models ran at medium thinking. GPT-5.6-Luna used its medium default, DeepSeek used its high default, and Qwen had thinking enabled. The models did not use equal reasoning settings, so this is not a perfectly controlled comparison. I will return to that limit.

After seeing how well Luna did at medium effort, I decided to run the same eval again at low and high effort. I kept the prompt, exercises, harness, and three-run format the same. These two follow-up runs added 360 model–exercise pairs. Across the original bake-off and the Luna effort sweep, the harness ran 1,260 tasks, made 2,431 attempts, and cost $9.56.

Luna was strongest at every effort level

Model	Thinking	Passed first try	Passed within five	Exercises passed in all 3 runs	Median reasoning tokens per try	Cost for 3 runs
GPT-5.6-Luna	High	97/180 (54%)	179/180 (99%)	59/60	459	$0.62
GPT-5.6-Luna	Low	92/180 (51%)	177/180 (98%)	57/60	264	$0.26
GPT-5.6-Luna	Medium	92/180 (51%)	176/180 (98%)	58/60	366	$0.37
Gemini-3-Flash-Preview	Medium	82/180 (46%)	168/180 (93%)	54/60	3,326	$3.91
Gemini-3.5-Flash-Lite	Medium	71/180 (39%)	161/180 (89%)	51/60	1,677	$2.89
DeepSeek-V4-Flash-0731	High	65/180 (36%)	154/180 (86%)	48/60	3,180	$0.70
Qwen3.7-Flash	Default	67/180 (37%)	147/180 (82%)	40/60	8,659	$0.80

Luna at high effort had the best accuracy. It passed 179 of 180 tasks within five tries and passed 59 of 60 exercises in all three runs. On this exercise mix, low effort gave the better value: 177 of 180 passes for $0.26, less than half the cost of high effort.

That result needs context. Only 12 of the 60 exercises were time-based, and 37 exercises passed every time for every model. These easier exercises dominate the overall pass rate and make Luna's effort levels look very close. High effort showed its main advantage on the hard tail, especially exercises involving time, scan order, and state changes. I suspect the difference between effort levels will grow as the learn.rungs.dev catalog expands with harder exercises, but that needs another test.

The extra tries also changed the order. Qwen slightly beat DeepSeek on the first try, 67 to 65. After up to five tries, DeepSeek finished seven passes ahead, 154 to 147.

Consistency separated them further. Only one of Luna's 60 results changed at medium effort, and one changed at high effort. At low effort, three changed. Qwen changed on 16. A model that sometimes finds a solution is not the same as one that finds it reliably, even when their average scores look close.

Cheaper tokens did not mean a cheaper result

Token efficiency matters because every extra token adds latency and cost. A low token price does not help if the model needs far more tokens to solve the task.

In this chart, one task means one model attempting one exercise once, including any fixes. I divided each configuration's cost for three runs by its 180 tasks.

Qwen reported a median 8,659 reasoning tokens per try, almost 24 times Luna's 366 at medium effort.

Qwen's median exercise took 180.2 seconds, compared with 8.8 seconds for Luna at medium effort. Qwen's slowest exercise ran for more than 24 minutes and still failed. Its three runs cost $0.80, more than twice Luna's $0.37 at medium effort.

Both Qwen and DeepSeek were cheaper per token than Luna. At the prices recorded for this eval, Qwen cost $0.03 per million input tokens and $0.13 per million output tokens. DeepSeek cost $0.13 and $0.26. Luna cost $0.20 and $1.20. Even so, Qwen cost $0.80 and passed 147 of 180 tasks, while DeepSeek cost $0.70 and passed 154. Luna at medium effort cost $0.37 and passed 176. The lower token prices could not make up for the much larger number of tokens they generated.

The Luna sweep is more useful because it compares the same model. Luna reported a median 264 reasoning tokens per try at low effort, 366 at medium, and 459 at high. High effort cleared more of the hard tail, while low effort gave the best cost and speed on this exercise mix. Three runs are too few for a broader conclusion.

The actual transport timeouts did not come from Qwen. All four came from DeepSeek. That is another reason to measure latency and reliability alongside accuracy. A better reasoning test would sweep effort within each model and verify that the gateway applied the setting by checking whether the returned reasoning-token count changed.

The hard part is time

Thirty-seven of the 60 exercises passed every time for every model. Almost all the difficulty came from the 12 exercises that use timers or advance over time.

Model	Timer/time exercises	Other exercises
GPT-5.6-Luna (High)	35/36 (97%)	144/144 (100%)
GPT-5.6-Luna (Low)	34/36 (94%)	143/144 (99%)
GPT-5.6-Luna (Medium)	32/36 (89%)	144/144 (100%)
Gemini-3-Flash-Preview	24/36 (67%)	144/144 (100%)
Gemini-3.5-Flash-Lite	21/36 (58%)	140/144 (97%)
Qwen3.7-Flash	18/36 (50%)	129/144 (90%)
DeepSeek-V4-Flash-0731	17/36 (47%)	137/144 (95%)

In the original five-model field, flasher was the hardest exercise: one pass in 15 model runs, and no first-answer passes. Luna later passed it in two of three low-effort runs and all three high-effort runs. rate-of-change passed 4 of 15 in the original field; run-hours and traffic-light-simple passed 5 of 15 each.

This matches the earlier benchmark and what I see from people using learn.rungs.dev. Time-based exercises also cause the most trouble for learners.

The models and learners should not take all the blame. Some test runner errors show what value was wrong without making it clear which scan, timer state, or state change caused the failure. The model receives that same message on its next try. If the message is unclear, both the model and the learner can keep making the wrong fix. Improving feedback for time-based test failures is what I will focus on next.

Five tries can still take too long

A correct answer only helps if it arrives soon enough. The time columns below cover a whole exercise, including every fix, on the two clean timing runs.

Model	Thinking	Median reasoning tokens per try	Median time per exercise	Slowest exercise	Completion tokens/s
Gemini-3.5-Flash-Lite	Medium	1,677	6.8s	159s	339
GPT-5.6-Luna	Low	264	8.0s	79s	70
GPT-5.6-Luna	Medium	366	8.8s	137s	89
GPT-5.6-Luna	High	459	11.2s	294s	94
Gemini-3-Flash-Preview	Medium	3,326	20.0s	179s	242
DeepSeek-V4-Flash-0731	High	3,180	28.2s	947s	106
Qwen3.7-Flash	Default	8,659	180.2s	1,464s	138

Completion tokens per second only shows how quickly a model generates tokens. It does not show how many tokens the model needs to solve the task. Qwen generated 138 completion tokens per second, compared with 89 for Luna at medium effort. But Qwen used almost 24 times more reasoning tokens per try, so its median exercise still took about 20 times longer and cost more. For Relay Chat, token efficiency and total waiting time matter more than raw generation speed.

This is why the final pass rate is not enough. Three minutes of waiting per exercise reaches the answer, but no student will sit through it. A useful test needs the first-try pass rate, final pass rate, tries needed, consistency, speed, and cost. Any one number hides part of the result.

The models also tested Rungs Studio's error messages

Without an instruction list, the models showed which instructions they expected Rungs Studio to support. Across all 1,260 tasks, including the Luna effort sweep, there were 1,154 uses of unavailable instruction names:

• 1,029 (89%) were old names such as GRT, LES, EQU, GEQ, and MOV. Rungs Studio already reports the correct replacement, so models usually recovered.
• 73 (6%) were real Logix instructions Rungs Studio does not implement.
• 52 (5%) came from other vendors or were invented.

CPT was the clearest missing feature. Every model tried to use it across many exercises, usually on its first try. CMP showed the same pattern: models wanted to write a calculation or comparison as an expression inside a rung.

This is why the small prompt is useful for more than ranking models. If several models cannot recover from the same compiler message, that message is worth reviewing for beginners too. The eval helps test Rungs Studio's error messages and find important missing instructions.

What this experiment does and does not prove

It does show that the full loop changes the result: 42% passed on the first try, while 90% passed within five feedback-guided tries. The first-try result and the final result measure different things.

It does not show how much of that gain came from the feedback itself. Five fresh tries without feedback would also improve the chance of finding a passing answer. This eval did not include that control.

It does not show that the earlier production prompt changed the ranking. That is still possible, but this run cannot prove it. The models changed, the exercise list grew from 40 to 60, and the test changed from one try to five.

It also does not measure all PLC programming. It measures one Ladder text format, Rungs Studio's instruction set, and the Rungs exercise catalog. Four DeepSeek attempts timed out and count as failures. The reasoning settings were different, and the effort sweep covered only Luna. These limits matter.

What I will improve next

My first priority is clearer test runner feedback for time-based exercises. A useful failure should identify the wrong state, the scan where it changed, and the timing boundary that the answer missed.

After that, I will identify the largest gaps exposed by the eval. One clear gap is mnemonics used before version 36 of Studio 5000 Logix Designer®, such as GRT, LES, and EQU. I will add small, targeted instructions to the Relay prompt, rerun the eval, and measure whether first-try results improve. The goal is a better first answer, not a larger prompt full of worked solutions.

This eval made GPT-5.6-Luna the leading candidate to replace Gemini-3-Flash-Preview in rungs.dev Relay Chat. Luna was more accurate, used fewer tokens, responded faster, and cost less here. But this eval only tested Ladder Logic. I still need to test Structured Text generation and the other Relay evals before changing the model.

The main lesson still holds. If an AI system can compile and run its answer, testing only the first answer misses an important part of the system. The useful question is not only can the model write Ladder Logic? It is also can it reach correct Ladder Logic after clear feedback, how often, and how long does it take?

For this field, GPT-5.6-Luna gave the best answer. Low effort gave the best value, while high effort cleared more of the hard cases. For the eval itself, Gareth's comment gave the best answer: put the model in a harness and measure where it converges.