Gray Area Labs
Gray Area Labs/BenchmarksRecorded through 2026-03-03
QISKIT-HUMANEVAL-HARD

Show the work.
Measure the result.

A closer look at quantum code generation: 151 tasks, 7 model and pipeline entries, and every recorded run behind the comparison.

Explore results ↓Read the methodology
151Tasks per run
19Completed runs
7Model & pipeline entries
02-17 — 03-03Recorded in 2026

Historical, internally recorded results. Configurations and execution budgets differ. Understand the comparison →

01 / THE COMPARISON

Results, with context.

Mean pass rate across completed runs. Expand an entry to inspect its individual scores and configuration.

7 entries Mean Individual run · scale 0–100%
01GrayGateVerification pipeline · 3 runs91.6%90.7–92.7%Mean pass rate; observed range 90.7 to 92.7 percent. Expand for individual runs.
PROVIDER graygateRECORDED MODEL ID graygate
RUN 01

92.7%

140 passed · 11 failed · 151 total

Qiskit
2.3.0
Parallel workers
1
114f676e433d
RUN 02

91.4%

138 passed · 13 failed · 151 total

Qiskit
2.3.0
Parallel workers
1
39faaa7f97d6
RUN 03

90.7%

137 passed · 14 failed · 151 total

Qiskit
2.3.0
Parallel workers
1
afa6dc5da65f

GrayGate is a generation and repair pipeline. Its base model and full execution budget are not recorded in these run rows.

02Gemini 3 ProPreview · 2 runs59.9%59.6–60.3%Mean pass rate; observed range 59.6 to 60.3 percent. Expand for individual runs.
PROVIDER googleRECORDED MODEL ID gemini-3-pro-preview
RUN 01

59.6%

90 passed · 61 failed · 151 total

Qiskit
2.0.0
Parallel workers
10
Temperature
0.0
Max tokens
8192
ca7be56f7ba9
RUN 02

60.3%

91 passed · 60 failed · 151 total

Qiskit
2.3.0
Parallel workers
1
Temperature
0.0
Max tokens
64000
ef4d271bf84f
03Gemini 3 FlashPreview · 3 runs55.6%55.6–55.6%Mean pass rate; observed range 55.6 to 55.6 percent. Expand for individual runs.
PROVIDER googleRECORDED MODEL ID gemini-3-flash-preview
RUN 01

55.6%

84 passed · 67 failed · 151 total

Qiskit
2.3.0
Parallel workers
7
Temperature
0.0
Max tokens
64000
801a1dfb333f
RUN 02

55.6%

84 passed · 67 failed · 151 total

Qiskit
2.3.0
Parallel workers
7
Temperature
0.0
Max tokens
64000
0bede3656bdc
RUN 03

55.6%

84 passed · 67 failed · 151 total

Qiskit
2.3.0
Parallel workers
7
Temperature
0.0
Max tokens
64000
e0bf96ad33f9
04GPT-5.2Direct model · 3 runs46.1%42.4–49.0%Mean pass rate; observed range 42.4 to 49.0 percent. Expand for individual runs.
PROVIDER openaiRECORDED MODEL ID gpt-5.2
RUN 01

47.0%

71 passed · 80 failed · 151 total

Qiskit
2.3.0
Parallel workers
10
Temperature
0.0
Max tokens
64000
7a0900fd9ba1
RUN 02

42.4%

64 passed · 87 failed · 151 total

Qiskit
2.3.0
Parallel workers
10
Temperature
0.0
Max tokens
64000
7546ea11e38e
RUN 03

49.0%

74 passed · 77 failed · 151 total

Qiskit
2.3.0
Parallel workers
10
Temperature
0.0
Max tokens
64000
356d051f4c00
05Qwen 3.5397B A17B · 3 runs38.4%37.7–39.1%Mean pass rate; observed range 37.7 to 39.1 percent. Expand for individual runs.
PROVIDER openrouterRECORDED MODEL ID qwen/qwen3.5-397b-a17b
RUN 01

37.7%

57 passed · 94 failed · 151 total

Qiskit
2.3.0
Parallel workers
10
Temperature
0.0
Max tokens
64000
cf53d5ece474
RUN 02

38.4%

58 passed · 93 failed · 151 total

Qiskit
2.3.0
Parallel workers
10
Temperature
0.0
Max tokens
64000
c20d396d6066
RUN 03

39.1%

59 passed · 92 failed · 151 total

Qiskit
2.3.0
Parallel workers
10
Temperature
0.0
Max tokens
64000
acf22b4c7727
06DeepSeek ReasonerDirect model · 3 runs32.5%31.1–33.8%Mean pass rate; observed range 31.1 to 33.8 percent. Expand for individual runs.
PROVIDER deepseekRECORDED MODEL ID deepseek-reasoner
RUN 01

33.8%

51 passed · 100 failed · 151 total

Qiskit
2.3.0
Parallel workers
10
Temperature
0.0
Max tokens
64000
7e84164d6dd7
RUN 02

31.1%

47 passed · 104 failed · 151 total

Qiskit
2.3.0
Parallel workers
10
Temperature
0.0
Max tokens
64000
fa08a44920b4
RUN 03

32.5%

49 passed · 102 failed · 151 total

Qiskit
2.3.0
Parallel workers
10
Temperature
0.0
Max tokens
64000
67cefdc021fb
07Qiskit Qwen 2.514B · local model · 2 runs25.8%25.8–25.8%Mean pass rate; observed range 25.8 to 25.8 percent. Expand for individual runs.
PROVIDER ollamaRECORDED MODEL ID hf.co/Qiskit/qwen2.5-coder-14b-qiskit-GGUF:latest
RUN 01

25.8%

39 passed · 112 failed · 151 total

Qiskit
2.3.0
Parallel workers
1
Temperature
0.0
Max tokens
64000
bbb8b311b8ba
RUN 02

25.8%

39 passed · 112 failed · 151 total

Qiskit
2.3.0
Parallel workers
1
Temperature
0.0
Max tokens
64000
168f3f7ad40b

The thin markers show individual runs, not confidence intervals. Ranking is by mean pass rate; it does not establish equal-budget model quality.

02 / THE RECORD

Every run behind the number.

Exact counts, dates, and identifiers. Filter the record to follow one model through its completed runs.

19 recorded runsDownload source JSON ↓
Completed benchmark runs grouped by model
Model / pipelineCompletedPass ratePassed / totalFailedQiskitRun ID
GrayGate2026-02-2392.7%140 / 151112.3.0114f676e433d
GrayGate2026-02-2791.4%138 / 151132.3.039faaa7f97d6
GrayGate2026-03-0390.7%137 / 151142.3.0afa6dc5da65f
Gemini 3 Pro2026-02-1759.6%90 / 151612.0.0ca7be56f7ba9
Gemini 3 Pro2026-02-2760.3%91 / 151602.3.0ef4d271bf84f
Gemini 3 Flash2026-02-2455.6%84 / 151672.3.0801a1dfb333f
Gemini 3 Flash2026-02-2555.6%84 / 151672.3.00bede3656bdc
Gemini 3 Flash2026-02-2555.6%84 / 151672.3.0e0bf96ad33f9
GPT-5.22026-02-2547.0%71 / 151802.3.07a0900fd9ba1
GPT-5.22026-02-2542.4%64 / 151872.3.07546ea11e38e
GPT-5.22026-02-2549.0%74 / 151772.3.0356d051f4c00
Qwen 3.52026-02-2537.7%57 / 151942.3.0cf53d5ece474
Qwen 3.52026-02-2538.4%58 / 151932.3.0c20d396d6066
Qwen 3.52026-02-2539.1%59 / 151922.3.0acf22b4c7727
DeepSeek Reasoner2026-02-2533.8%51 / 1511002.3.07e84164d6dd7
DeepSeek Reasoner2026-02-2531.1%47 / 1511042.3.0fa08a44920b4
DeepSeek Reasoner2026-02-2532.5%49 / 1511022.3.067cefdc021fb
Qiskit Qwen 2.52026-02-2425.8%39 / 1511122.3.0bbb8b311b8ba
Qiskit Qwen 2.52026-02-2525.8%39 / 1511122.3.0168f3f7ad40b
03 / THE TASK OUTCOMES

Look beneath the average.

One square per task. Choose a recorded run, then select a square to inspect the outcome behind it.

Passed Failed

Scroll within the map to see all tasks.

Loading the recorded task outcomes…

Ordered by recorded task ID; the map is not a difficulty ranking.Download task outcomes ↓
04 / THE METHODOLOGY

What these results tell you.
And what they don’t.

The record is useful when its assumptions are visible. Here is how to read and reproduce the arithmetic.

THE TASK SET

151 tasks. One recorded suite.

These completed runs are labeled Qiskit-HumanEval-Hard in the published snapshot. Each run includes 151 task outcomes. The dataset revision was not recorded, so exact task-set reproduction requires additional provenance.

THE CALCULATION

Counts first. Percentages second.

Run score = passed ÷ total × 100

For each entry, we take the arithmetic mean of its run scores, then round to one decimal place. Range means the lowest and highest observed run scores.

GrayGate: 415 passed ÷ 453 outcomes × 100 = 91.6%

THE COMPARISON

Different systems. Different budgets.

GrayGate uses a verification and repair pipeline. Other entries represent model runs. Token limits, parallelism, and Qiskit versions differ, so this is not a controlled single-attempt or equal-cost comparison.

THE LIMITS

A measured snapshot, not a guarantee.

These results are internally recorded and have not been independently reproduced. Two or three runs per entry do not establish statistical confidence. Passing this suite does not guarantee correctness on your own program.

Source, export, and missing fields

The published export validates each run’s passed and total counts against its task records. Both this page and the landing page use the same versioned snapshot. New database runs appear after a reviewed export.

Source
data/graybench-results.db
Recorded window
2026-02-17 through 2026-03-03; source timestamps do not specify a timezone.
Unavailable
Dataset revision, GrayGate base model, and a normalized cost or token budget. We do not substitute unrelated session totals.
Database SHA-256
a3710d04b607a78371399b2aca2b25c8bde8aaac90d6a895dd8590dfa32f8de7
TAKE THE NEXT STEP

Put the workflow to work.

Explore how GrayGate generates code and brings the checks together.

Explore GrayGate →Request access →