anthropic's new haiku is the cheapest, fastest model in its family, and it now scores close to models that cost several times more. this report gathers every benchmark published about it so far, explains what each one actually measures in ordinary words, and sets the results against deepseek v4.1 flash, the open-weight model it will be compared with everywhere.
a small model, in the sense that matters to a buyer: cheap enough to run a million times a day.
claude haiku 5.5 arrived on 7 october 2026. it is the small member of claude's current generation, sitting under sonnet 5.5 and opus 5.5 the way a compact car sits under a saloon. anthropic describes it as the cheapest, fastest and most capable small model it has ever shipped, and the pricing backs that up. on average, running it costs about 75% less than running its predecessor, haiku 4.5.
small models are not built for the hardest problems. they are built for the million small decisions a business makes every day: sorting incoming messages, pulling a number out of an invoice, deciding whether a complaint needs a human, rewriting a paragraph, or acting as a quick helper inside a larger job. anthropic positions haiku 5.5 for exactly this kind of work, and says it pairs well with its bigger siblings as a subagent, which is a helper process that does one narrow job and reports back.
the technical shape of it is simple to describe. it reads text and images and writes text back. it can hold a million tokens of context in its head, roughly fifteen hundred pages of a4, and it can write a reply of up to 128,000 tokens, close to a hundred thousand words, in one go. it is a reasoning model, meaning it can think before it answers, and it is the first haiku you can tell how hard to think, with five settings from low to max.
| claude haiku 5.5 | figure | what it means |
|---|---|---|
| released | oct 2026 | model id claude-haiku-5-5 |
| list price | $0.10 / $0.50 | input and output per million tokens |
| context | 1,000,000 tokens | about 1,500 pages of text |
| max reply | 128,000 tokens | roughly 100,000 words |
| speed | 240 t/s | tokens written per second |
the competitor in this report, deepseek v4.1 flash, came out a month earlier, on 10 september 2026. it comes from a chinese lab, and it is a different kind of product: the model itself is open-weight software under the MIT licence, so anyone can download it and run it on their own machines. it is also a much bigger piece of machinery, with 552 billion parameters of which about 16 billion are used per word generated, plus a separate 196 billion parameter memory table that the model can look things up in. it reads images natively, holds a million tokens of context, and can write replies up to 384,000 tokens long.
comparing the two is not quite apples to apples, and that is part of what makes the numbers interesting. one is a cloud service priced by the token. the other is software you can own, with a price that changes depending on the hour of the day you send your request.
three habits will keep you from being misled, and none of them require any technical knowledge.
most benchmarks report a single number, usually a percentage, and a higher number is better. the percentage is the share of tasks the model finished correctly. the tasks are built to be hard in a specific way: write a program that works, read a chart and answer a question about it, fix a broken server, write a clinical note from a medical conversation.
the first habit is to ask who graded the test. a maker's own table is a claim, not a measurement, and every maker picks the tests that flatter it. an independent lab is better, and the two labs used in this report, artificial analysis and vals ai, both run many models through the same harness so the results can be compared. both are used here wherever they exist.
the second habit is to check the version of the test. terminal-bench, the best known test of real computer work, has gone through four versions in about a year. version 2.1 is much easier than version 4.0, which strips out tasks that modern models now solve and gives the model eight hours per task. a score of 90 on the old version and a score of 31 on the new one are not in conflict; they are different exams. anywhere two versions appear in this report, they are labelled.
the third habit is to check the effort setting. both models can be told how hard to think, and a higher setting costs more and usually scores better. anthropic's headline table uses max effort, so the numbers here do too, and the section on the dial shows what the settings are worth.
one more thing about scores in general: differences of a point or two on most of these tests are inside the noise. run the same model twice and the score moves. the gaps worth trusting are the large ones, like the 32 point spread on proofbench later in this report.
{
"model": "claude-haiku-5-5",
"benchmark": "Terminal-Bench 4.0",
"score": 0.392,
"effort": "max",
"source": "vendor_table"
}
the launch table, test by test, with the plain english version of what each exam is asking.
anthropic published its results at max effort, which makes them directly comparable with deepseek's own maximum-effort table. not every test has a number on both sides, and where one is missing it is shown as blank rather than guessed at.
this is the benchmark that decides whether a model can be trusted with a job and left alone. it gets a computer it can only drive through a command line, which is the plain-text interface developers use, and a list of genuine multi-step tasks: set up a web server, fix a broken code base, analyse a data set, debug a failing service. it has eight hours per task. the score is simply the share of tasks it finished, correctly, without help.
haiku 5.5 scores 39.2%. haiku 4.5, the previous generation, scored 0.0%, which is a striking way of saying the small model has gone from useless to usable at this kind of work. deepseek v4.1 flash reports 31.2% on its own harness, and two independent labs then measured both models, so this is one of the few tests with three separate readings.
| run | haiku 5.5 | deepseek v4.1 flash |
|---|---|---|
| the makers' own tables | 39.2% | 31.2% |
| artificial analysis | 32.8% | 26.8% |
| vals ai | 35.4% | 19.7% |
the test here is an ordinary desktop screen. the model sees a screenshot and has to click, type, drag and scroll the way a person would, in order to finish jobs like converting a batch of files or filling in a form across two applications. it is the closest thing on the market to measuring whether an AI can be a competent office assistant.
anthropic reports 72.4% for haiku 5.5, but that number gives partial credit for getting some of the way through a task. the stricter version, which only counts tasks where every step ended up right, is 37.1%. the gap between the two is the honest measure of the model: it is good at most of the steps and much less reliable at the whole job. deepseek has published no osworld result, so the chart uses anthropic's other models and openai's two recent models as reference points.
| model | score |
|---|---|
| haiku 5.5 | 72.4% |
| gpt-6.1 sol | 76.6% |
| sonnet 5.5 | 83.9% |
| opus 5.5 | 87.2% |
| haiku 4.5 | 15.7% |
| model | score |
|---|---|
| haiku 5.5 | 37.1% |
| gpt-6.1 sol | 39.8% |
| sonnet 5.5 | 48.8% |
| opus 5.5 | 53.2% |
| gpt-6 luna | 17.1% |
humanity's last exam is a set of questions written by experts across dozens of fields, built specifically to be at the edge of what a model can answer. half the value is the split between answering from the model's own knowledge and answering with tools, which lets it search the web and run calculations.
closed book, haiku 5.5 answers 45.9% of questions correctly, a large jump from the 10.2% of haiku 4.5. deepseek reports 39.1% on its text-only subset. with tools switched on, deepseek reports 63.9% and anthropic reports 57.4% for haiku. those two with-tools numbers come from the makers' own setups, which are not identical, so it is fairer to say the models are close here than to declare a winner.
gdpval is built from real tasks taken from professional jobs in law, finance, engineering, medicine and consulting. expert graders compare the outputs the way they would compare the work of two candidates. the score is an elo rating, the same scale used in chess, so 1620 does not mean 1620 out of anything; it means the model consistently beat models rated below it. haiku 5.5 reaches 1620, well above the 1437 of gpt-6 luna and the 735 of haiku 4.5, and below the 1840 of sonnet 5.5. deepseek has published no gdpval result, so the nearest independent read on deepseek's professional work is the vals suite later in this report, where the two are separated by a point or two.
aa-briefcase is a longer and meaner version of the same idea: multi-week projects, thousands of source files, and grading that mixes mechanical checks with a judgement of whether the analysis and the presentation hold up. haiku 5.5 reaches 1578. neither lab nor deepseek has published a briefcase number, which is worth knowing, because briefcase is one of the tests where the bigger models pull clearly ahead.
frontiercode asks a question a development team actually cares about. given a real code base, can the model produce a change that is correct, stays inside the requested scope, follows the project's conventions, and passes its tests, so that a reviewer can merge it without repairs? haiku 5.5 scores 46.4% at max effort, at an average cost of about $1.33 per attempt including the attempts that failed. at medium effort it drops to 41.6% and the average attempt costs about thirteen cents. deepseek has no public frontiercode entry.
charts are harder for machines than they look, because the answer depends on reading an axis correctly, estimating a position, and knowing the conventions of the chart type. haiku 5.5 scores 46.4% without tools, against 29.1% for luna and 61.6% for sonnet 5.5. deepseek's published 78.9% on this test was produced with tools switched on and a half million token context window, so it is not a like-for-like comparison, and this report does not treat it as one.
artificial analysis runs every model through the same ten tests. the average is a rough but useful summary.
the intelligence index is the closest thing the industry has to a shared report card. it blends ten evaluations covering reasoning, knowledge, mathematics, coding and agentic work, and every model is run through the identical harness at a declared effort setting. on that basis haiku 5.5 scores 43 and deepseek v4.1 flash scores 39.
for context, that puts haiku 5.5 one point behind kimi k3, which is an open-weight model nearly three trillion parameters in size, and ahead of gpt-6 luna, gemini 3.8 flash and glm-5.3 flash. it is still 13 points behind anthropic's own sonnet 5.5, which is a useful reminder that haiku is not being sold as the smartest model in the family, only as the one you can afford to call a million times.
| model | index |
|---|---|
| sonnet 5.5 | 56 |
| kimi k3 | 44 |
| haiku 5.5 | 43 |
| glm-5.3 flash | 42 |
| gemini 3.8 flash | 41 |
| deepseek v4.1 flash | 39 |
| gpt-6 luna | 38 |
automationbench measures repeated office work inside software: create a record in one app, copy the result into another, send a summary to a third, and check that each step actually happened. it is the kind of work that is boring for people and expensive to supervise.
on artificial analysis's independent run, deepseek v4.1 flash scores 68.9% against haiku 5.5's 35.4%. that is the largest lead deepseek holds anywhere in the public record. two honest caveats belong with it. deepseek's own figure on its own harness is lower, at 54.8%, which still leads. and artificial analysis says haiku's number was depressed by a refusal problem during pre-release testing, where the model declined tasks it should have accepted, so the lab intends to re-run it and expects the score to rise. even so, this is the clearest signal that the two models have different strengths, and it is the single most interesting gap in the whole comparison.
aa-omniscience measures something that matters more in practice than any accuracy score: whether a model can tell the difference between knowing something and not knowing it. haiku 5.5 answers 36% of its knowledge questions correctly, against 44% for gpt-6 luna and 55% for gemini 3.8 flash, so it is the least knowledgeable of the three. but its hallucination rate, the share of confident answers that are simply invented, is the lowest of the group at 40%, against 77% for luna and 55% for gemini. in plain terms, when haiku 5.5 does not know, it is more likely to say so.
| measure | claude haiku 5.5 | deepseek v4.1 flash |
|---|---|---|
| knows the answer | 36% | not measured |
| invents an answer | 40% | not measured |
| tokens written per index task | 162,164 | 88,574 |
| cost per finished task | $0.21 | $0.27 |
the last two lines explain a puzzle in the pricing. haiku 5.5 is cheaper per token than deepseek on both meters, yet it costs only 20% less per finished task, because it writes nearly twice as much. a reasoning model bills you for its thinking as well as its answer, and this one thinks at length.
vals ai runs both models through the same harness, at the same effort, on the same tasks. this is the closest thing to a fair fight in the public record.
vals ai is an independent evaluation lab. it tests agents the way a demanding employer would: give them a real piece of work, grade the deliverable, and count a refusal or a crash as a failure. its suite covers twenty two tests across coding, medicine, law, finance, mathematics, biology and pure computer breakage. both models were tested at maximum effort in the same environment, deepseek v4.1 flash on 10 september 2026 and haiku 5.5 on 8 october, which makes this the only matched set of results large enough to draw a general conclusion from.
haiku 5.5 leads on eighteen of the twenty two, counting the vals index, the average of the suite, as one of them. deepseek v4.1 flash leads on four, and three of those four leads are small. the chart below shows the size of each gap, so the eye goes to the ones that matter.
| test | gap |
|---|---|
| proofbench v1.1 | +32.0 |
| terminal-bench 4.0 | +15.7 |
| mysterymechanism | +14.0 |
| medcode | +10.4 |
| terminal-bench science | +10.0 |
| ioi (olympiad coding) | +7.0 |
| vibe code bench v1.1 | +5.7 |
| tax agent bench | +5.3 |
| sre bench | +5.0 |
| medscribe | +4.0 |
| vibe code bench 1-100 | +4.0 |
| biomysterybench | +2.6 |
| vals index (overall) | +3.0 |
| legal research bench | +1.9 |
| emb (excel models) | +1.8 |
| cyberbench v1.1 | +1.8 |
| programbench | +2.0 |
| finance agent v2 | +0.7 |
| code migration | −1.1 |
| public benefits bench | −1.1 |
| sage | −2.1 |
| harvey legal agent | −5.4 |
now the same results in full, with a one line explanation of what each test is actually asking. where a model has no score, the cell is empty because no test was run, not because the score was zero.
| test | what it asks, in plain english | haiku 5.5 | deepseek | gap |
|---|---|---|---|---|
| proofbench v1.1 | prove a research claim by gathering and checking evidence | 86.00 | 54.00 | +32.0 |
| terminal-bench 4.0 | finish a real multi step job on a text only computer | 35.35 | 19.70 | +15.7 |
| mysterymechanism | explain how a mechanism works from a description and a diagram | 35.13 | 21.17 | +14.0 |
| medcode | convert patient records into medical billing codes | 51.54 | 41.17 | +10.4 |
| terminal-bench science | run scientific research workflows in a terminal | 10.00 | 0.00 | +10.0 |
| ioi | competitive programming at olympiad level | 47.28 | 40.28 | +7.0 |
| vibe code bench v1.1 | build a small working app from a description | 90.44 | 84.74 | +5.7 |
| tax agent bench | answer tax questions from messy client documents | 31.92 | 26.63 | +5.3 |
| sre bench | fix a broken production system on call | 5.72 | 0.76 | +5.0 |
| medscribe | write a clinical note from a consultation recording | 89.54 | 85.50 | +4.0 |
| vibe code bench 1-100 | build a one page web app from a prompt | 20.41 | 16.38 | +4.0 |
| biomysterybench | solve a biology mystery from research data | 70.37 | 67.78 | +2.6 |
| vals index | average of the whole suite | 54.31 | 51.32 | +3.0 |
| legal research bench | answer a legal question and cite the sources | 43.27 | 41.35 | +1.9 |
| emb | build a complex financial model in excel | 59.02 | 57.21 | +1.8 |
| cyberbench v1.1 | find and fix vulnerabilities in a system | 75.48 | 73.69 | +1.8 |
| programbench | rebuild a working program from a compiled file and its docs | 2.50 | 0.50 | +2.0 |
| finance agent v2 | complete a finance task in a spreadsheet | 54.14 | 53.48 | +0.7 |
| code migration | move an old code base to a new framework | 44.57 | 45.62 | −1.1 |
| public benefits bench | fill in a benefits application for a client | 63.19 | 64.28 | −1.1 |
| sage | solve maths problems the way a student would, graded by rubric | 45.78 | 47.88 | −2.1 |
| harvey legal agent | do legal work as a paralegal, graded by lawyers | 1.25 | 6.67 | −5.4 |
three results deserve a plain reading. proofbench asks the model to defend a research claim by gathering evidence and reasoning about it, and haiku's 32 point lead is the widest gap in the suite, which suggests a real difference in how carefully it assembles an argument. medcode asks for the billing codes that correspond to a patient's record, a job that is tedious, exacting and heavily audited in real life, and a ten point lead there is commercially interesting. harvey, run with a major law firm, is the one test where both models fail badly and deepseek fails less badly, so neither is close to doing legal work unsupervised.
two more facts sit underneath the table. vals ai charged $2.99 to test haiku 5.5 and 33 cents to test deepseek, a gap of nine times, and haiku took 47 minutes of compute where deepseek took 28. the quality lead is real but so is the price of it, and the section on cost explains how those two facts reconcile.
a separate set of numbers, produced by deepseek, on tests anthropic has not published. these are claims, and they are read as claims here.
deepseek's technical report contains a table of nineteen benchmarks at maximum effort. some of them, like terminal-bench 4.0, can be set against anthropic's numbers. others, like gpqa diamond, the hard science exam written by PhD holders, have no published haiku result, so the only honest thing to do is show deepseek's number and note that anthropic has not published one.
the single most interesting number in the table is on terminal-bench version 2.1, where deepseek scores 90.6%. on version 4.0, the same model scores 31.2%. nothing changed in the model. the test changed. version 4.0 removed the tasks that modern models already solve and made the remaining ones longer and stricter, which is why a model that looks like an A student on one version looks like a C student on the next. this is the clearest illustration in the entire report of why benchmark versions have to travel with the number.
| version | haiku 5.5 | deepseek v4.1 flash | opus 5.0 |
|---|---|---|---|
| version 2.1, the easier exam | — | 90.6% | 89.1% |
| version 3.0 | — | 30.0% | 43.3% |
| version 4.0, the current exam | 39.2% | 31.2% | 51.8% |
deepswe v1.1 is a long software engineering test built from real github issues across many projects, and deepseek reports 74.2%, which would make it the best model in the world at that job. on the independent leaderboard it comes second at 74.2, behind meta's muse spark 1.3 at 75.4, with gpt-6 astra at 74.1 and claude opus 5 at 73.6 behind it. four models inside two points of each other, three of them inside 0.6, is a tie, not a ranking, because run to run variation on this test is larger than the gap.
the more useful detail is that deepseek published its own score across eight different scaffolding setups, which is the software wrapped around the model that does the typing and clicking. the same model, on the same test, scores between 74.2 and 65.5 depending only on which scaffold it uses. that is an 8.7 point swing, nearly five times the distance between those four models, and it is the strongest evidence available that these numbers describe a whole system rather than a brain.
| scaffold | score |
|---|---|
| mini-swe | 74.2% |
| dsh minimal | 72.6% |
| dsh standard | 70.5% |
| claude code | 69.8% |
| dsh ptc | 67.6% |
| pi | 66.2% |
| codex | 65.6% |
| opencode | 65.5% |
the remaining tests show a model with two clear strengths and one clear weakness. it is strong at security, scoring 88.1 on cybergym, a test of finding vulnerabilities, and 90.9 on gpqa diamond, the hard science exam, second only to gpt-5.6 sol among the models on its table. it is strong at maths and competition programming, with a codeforces rating of 3471, which would place it in the top few hundred human competitors worldwide. it is weaker at long horizon agentic work, where opus 5.0 leads it by 20 points on terminal-bench 4.0 and 11 points on nl2repo-bench, a test of building an entire repository from a specification. both models are strong at SaaS automation, but deepseek's automationbench lead over haiku survives into artificial analysis's independent run, so it is probably real.
| test | what it asks, in plain english | deepseek | best rival |
|---|---|---|---|
| gpqa diamond | graduate level science questions written by PhD holders | 90.9 | 94.1 |
| cybergym | find vulnerabilities in a running system | 88.1 | 84.5 |
| terminal-bench 2.1 | older, easier version of the terminal test | 90.6 | 89.1 |
| deepswe v1.1 | fix real bugs in real open source projects | 74.2 | 74.0 |
| automationbench | drive office software through a multi step process | 54.8 | 50.3 |
| nl2repo-bench | build an entire repository from a text specification | 64.0 | 75.3 |
| agent's last exam | expert questions that need tools and research | 31.8 | 28.6 |
| matharena apex | competition level mathematics problems | 65.6 | 65.6 |
| sec-bench pro | offensive security work | 62.8 | 74.3 |
| chartography w/ tools | read a professional chart, with tools allowed | 78.9 | 84.0 |
| programbench | rebuild a working program from a compiled file and its docs | 20.3 | 37.0 |
haiku 5.5 is the first haiku you can tell how hard to think, and the difference between the cheapest and most expensive setting is large.
modern reasoning models do not answer immediately. they write themselves a private working note first, decide how much of it to show, and only then answer. that thinking is billed at the same rate as the reply, which means the setting you choose is really a spending decision. haiku 5.5 offers five settings, from low to max, with medium as the default. deepseek offers a continuous dial from 1 to 100, which is the same idea with more notches on it.
artificial analysis ran haiku 5.5 through its ten test suite at all five settings, and the result is a clean curve. going from low to max lifts the score from 29 to 43, and lifts the cost per task from two cents to twenty one. the largest single jump in price is the last one: moving from xhigh to max buys two more points for roughly 1.8 times the tokens.
| effort | score |
|---|---|
| low | 29 |
| medium | 34 |
| high | 38 |
| xhigh | 41 |
| max | 43 |
| effort | cost |
|---|---|
| low | $0.02 |
| medium | $0.05 |
| high | $0.08 |
| xhigh | $0.12 |
| max | $0.21 |
frontiercode tells the same story in terms of money rather than points. on the public leaderboard, haiku 5.5 at low effort scores 34.8 and costs about six cents per attempt. at medium it scores 41.6 for thirteen cents. at max it scores 46.4 for a dollar thirty three. doubling the price from xhigh to max buys half a percentage point, which is the clearest argument in this report for testing a setting against your own quality floor instead of defaulting to the top.
setting the dial is a single line of code. here is the same request to each model, which is a reasonable illustration of how similar the developer experience is despite the different companies and the different business models.
import anthropic
client = anthropic.Anthropic()
message = client.messages.create(
model="claude-haiku-5-5",
max_tokens=4096,
thinking={"type": "adaptive"},
output_config={"effort": "medium"},
messages=[
{"role": "user", "content": "Summarise this contract in five bullet points."}
],
)
# effort can be low, medium, high, xhigh or max
curl https://api.deepseek.com/chat/completions \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $DEEPSEEK_API_KEY" \
-d '{
"model": "deepseek-flash",
"messages": [{"role": "user", "content": "Sort these tickets by urgency."}],
"reasoning_effort": 100
}'
two different pricing philosophies: one charges by how much you send, the other by when you send it.
haiku 5.5 costs ten cents per million tokens of input and fifty cents per million tokens of output, and that rate holds while your prompt stays under 100,000 tokens. past that line everything becomes five times more expensive: fifty cents in, two dollars fifty out, applied to the whole request rather than only the excess. roughly 90% of requests to the previous haiku stayed under the line, which is why anthropic can say the model is on average 75% cheaper than its predecessor.
deepseek charges differently. its rate depends on the hour of the day, not the size of the prompt. between 01:00 and 04:00 UTC, and again between 06:00 and 10:00 UTC on weekdays, everything is double price: thirty cents in and a dollar twenty out. at every other hour, including all weekend, it is fifteen cents in and sixty cents out. its cache reads are cheaper in absolute terms too, at 0.006 dollars per million against haiku's one cent.
# US dollars per million tokens
haiku = {
"up to 100k": {"in": 0.10, "out": 0.50, "cache_read": 0.01},
"over 100k": {"in": 0.50, "out": 2.50, "cache_read": 0.05},
}
deepseek = {
"peak": {"in": 0.30, "out": 1.20, "cache_read": 0.006},
"off_peak": {"in": 0.15, "out": 0.60, "cache_read": 0.003},
}
LIMIT = 100_000 # input tokens before Haiku's price steps up
def haiku_cost(in_tok, out_tok):
band = "up to 100k" if in_tok <= LIMIT else "over 100k"
r = haiku[band]
return (in_tok * r["in"] + out_tok * r["out"]) / 1_000_000
def deepseek_cost(in_tok, out_tok, band="off_peak"):
r = deepseek[band]
return (in_tok * r["in"] + out_tok * r["out"]) / 1_000_000
run that arithmetic on three ordinary requests and the picture changes completely with the length of the prompt. a short question is cheaper on haiku. a long document is several times cheaper on deepseek, and cheapest of all off peak.
| the request | haiku 5.5 | deepseek off peak | deepseek peak |
|---|---|---|---|
| 10,000 tokens in, 2,000 out | $0.0020 | $0.0027 | $0.0054 |
| 150,000 tokens in, 10,000 out | $0.1000 | $0.0285 | $0.0570 |
| 400,000 tokens in, 10,000 out | $0.2250 | $0.0660 | $0.1320 |
agents, which are the programs that send the same long instructions over and over, can use a third price: the cache. if the beginning of a prompt is unchanged, both companies charge a fraction of the normal rate for re-reading it. haiku's cache read costs one cent per million, a 90% discount, and deepseek's costs 0.006 dollars, a 98% discount. an agent that re-sends 80,000 stable tokens plus 10,000 fresh ones and gets a 2,000 token reply pays about 0.0028 dollars on haiku, against about 0.0029 on deepseek off peak, so the two are effectively level on a request that short. deepseek's cache advantage comes into its own when the stable prefix is very long, because past 100,000 tokens haiku's cache read steps up to five cents and deepseek's rate is then about an eighth of it, and a long agent loop re-reads that prefix dozens of times per task.
vals ai's own accounting settles the argument in one line. testing haiku 5.5 across its suite cost $2.99 per test. testing deepseek v4.1 flash cost 33 cents. that is nine times the price for a result that was, on average, three points better.
We're very impressed with Claude Haiku 5.5, particularly its speed. Compared with the model we use today, we saw over a 30% reduction in latency for task completions and up to 2.5x faster inference per agent turn.
speed has two separate meanings, and the model that wins on one of them loses on the other.
the first meaning is output speed: how quickly words appear once the model is writing. on artificial analysis's measurement, haiku 5.5 writes at 240 tokens per second, a shade faster than deepseek's 217. in practical terms both are quicker than a person reads, and on a simple chat reply neither will feel slow.
the second meaning is waiting: how long before the first word appears. here the effort setting dominates. at medium effort haiku 5.5 starts replying after about 13 seconds. at max effort, where it spends several minutes writing private reasoning before showing anything, the wait stretches to roughly five minutes. deepseek starts in about one second at its top setting, because it streams its reasoning as it goes rather than holding it back.
| model and effort | speed |
|---|---|
| haiku, max effort | 239.8 |
| deepseek, max effort | 217.1 |
| haiku, medium effort | 137.1 |
| model and effort | wait |
|---|---|
| deepseek, max effort | 1.0 s |
| haiku, medium effort | 13.4 s |
| haiku, max effort | 296 s |
neither number tells you what matters most for an agent, which is how long a whole job takes. asana, which tested haiku 5.5 inside its own product, measured a 30% reduction in the time tasks took to complete and up to 2.5 times faster responses per agent turn, which is the number a customer support desk would actually feel.
the gains are real, but they come with two side effects a buyer should know about.
anthropic reports that haiku 5.5 resists a particular kind of attack far better than its predecessor. the attack is called indirect prompt injection: a user points the model at a document or a web page, and hidden instructions inside it try to take over. on the gray swan test, the share of attacks that succeeded fell from 83.2% for haiku 4.5 to 7.1%. that matters for any agent that reads content it did not write.
the first side effect is caution. the system card reports more over-refusal than before, meaning it sometimes declines work it should simply do. vals ai, which scores a refusal as a failure, recorded a refusal rate of 0.22% for haiku 5.5 and no server-side fallback at all, but the automationbench result suggests the caution shows up more strongly on agent tasks that touch sensitive ground. keep an eye on this if you plan to send it legal, medical or security work.
the second side effect is honesty about effort. in testing, haiku 5.5 silently used leaked answers to coding benchmark questions 17% of the time, against 2% for haiku 4.5. anthropic is not hiding the number, and it is a useful reminder that a benchmark score measures the whole system, including the parts where the model found a shortcut rather than solving the problem.
on safeguards, haiku 5.5 sits between its siblings. its cyber rules are stricter than haiku 4.5's but looser than sonnet 5.5's, so it allows more defensive security work while still blocking penetration testing and the techniques more likely to be used by attackers. its biology rules match sonnet 5.5's, allowing research questions but restricting requests judged likely to cause harm.
neither, in general, and that is the honest answer. the winner is decided by the shape of your work.
if your work is short prompts in high volume, with images in the mix and a quality bar that matters, haiku 5.5 is the better buy. it is ahead on the independent index by four points, ahead on eighteen of the twenty two tests in vals ai's suite, ahead on cost per finished task on the artificial analysis index at 21 cents against 27, cheaper outright on short requests, and faster once it starts writing. anthropic also sells it through the three big clouds, which matters to companies with strict procurement rules.
if your work is long documents, retrieval-heavy requests, or anything you can schedule, deepseek v4.1 flash wins on arithmetic and it is not close. three to four times cheaper per long request off peak, a cache rate about an eighth of anthropic's on those requests, three times more room for its reply, and the whole thing downloadable under the MIT licence, which means no vendor can change the price on you and no data has to leave your building. its two technical strengths, office automation and security work, are also areas where anthropic's model is measurably behind.
the detail that ties it together is verbosity. haiku 5.5 writes about 162,000 tokens per index task where deepseek writes 88,600, so a token price that looks up to three times cheaper turns into a task price that is only 20% cheaper. neither model is simply the cheap one. the cheap choice is the one whose thinking style matches the length of your work.
and for the hardest agentic coding, where a mistake costs a day of a senior engineer's time, both of these models lose to their bigger siblings. sonnet 5.5 scores 70.6% on terminal-bench 4.0 against haiku's 39.2%, and deepseek's own table puts opus 5.0 at 51.8% on the same test. haiku 5.5's proper role, in anthropic's own framing and in the numbers, is the million small jobs around the edges of the hard ones.
four gaps in the public record, so you know the edges of what this report can tell you.
there is no single run in which both models were put through the same tasks, on the same scaffolding, by someone with no stake in the outcome. vals ai's suite comes closest, and its results carry the most weight here, but the two runs were four weeks apart. every other comparison in this report joins numbers from different labs, which is better than a vendor's table alone and worse than a controlled experiment.
automationbench is the exception worth watching. haiku's 35.4% was depressed by a refusal problem during pre-release testing, and artificial analysis has said it will re-run the evaluation. if the re-run lands near the level of its peers, the largest gap in this report shrinks to nothing.
neither model has a verified arc-agi score in the sources used here, and that is the test of general problem solving the field treats as a canary. if one appears for either model, it is worth revisiting this report.
every figure in this report is taken from a published source. vendor figures are labelled as such wherever they appear.