The Devsmatcher Method

How we evaluate AI talent.

This is the actual system we use — published in full, so you can judge our judgment before you rely on it. If you read this and disagree, we've saved us both a call.

Why the method is public

Every recruiter claims to “screen thoroughly.” Claims are cheap. A method you can read, question, and hold us to is not.

Publishing it also keeps us honest: if a candidate passes our process, we can show you exactly what they passed. If they fail, we can show you why.

What we evaluate, in order

The order matters. Most hiring failures happen at step one — before a single candidate is contacted.

1.1

Role definition — before any candidate

A search against a wrong role produces a confident wrong hire. We start by pressure-testing the role itself.

What exactly is being built, and at what stage is it?
What does “production” mean in this company — traffic, users, revenue at stake?
Who does this hire report to, and who unblocks them?
What must ship in the first 90 days for this hire to be a success?
Is a full-time hire the right format at all — or is it advisory, fractional, or later?

1.2

Production evidence

We verify that claimed experience survived contact with real users. Keywords don't count; consequences do.

Systems that served real traffic — and the candidate's actual role in them
What broke in production, and what they personally did about it
Evaluation and monitoring: how they knew the system worked — and kept working
Awareness of inference cost and latency as engineering constraints
What happened to the system after launch — degradation, iteration, retirement

1.3

Technical depth

Depth beyond API calls. We probe until the answers stop being rehearsed.

Understanding of the stack below the framework they name-drop
Data reality: quality, pipelines, and the unglamorous majority of the work
Failure modes: hallucination, drift, silent degradation — and defenses against them
Follow-up resistance: does the third “why?” still get a real answer?

1.4

Judgment under trade-offs

Strong engineers know what not to build. We test decision-making where there is no clean answer.

When not to use an LLM — and what to use instead
Build vs. buy, fine-tune vs. prompt, quality vs. cost vs. latency
What they would cut under a hard deadline — and what they would refuse to cut

1.5

Business communication

AI teams fail on translation more often than on code. The hire has to work with people who don't read papers.

Can they explain a technical decision to a founder in two minutes?
Do they push back when the plan is wrong — with reasons, not vibes?
Do they write clearly? Most production coordination is written.

Production AI vs. demo AI

The single most expensive confusion in AI hiring. These are the signals we actually use to tell them apart:

Signals of production experience

Talks about evaluation pipelines and regressions, not just prompts
Mentions cost and latency numbers without being asked
Has failure stories — and owns their part in them
Knows what happened to the system months after launch
Describes data work as most of the job, because it was

Signals of demo experience

A portfolio of impressive demos with no traffic behind them
Framework name-dropping with no depth on any single one
“Prompt engineering” presented as the core skill
No real answer to “what broke?”
Model metrics, never business metrics

What fails a candidate

We reject for specific reasons, and we put them in writing:

F1Production claims with no verifiable production behind them
F2Cannot walk through a single technical decision they made and why
F3Depth that exists in the portfolio but collapses under follow-up questions
F4Treats evaluation, monitoring, or data quality as someone else's job
F5A team achievement presented as a personal one — checked through references

How we think about AI team architecture

“Who should we hire first?” has no universal answer — but it has a structured one. The first hire depends on what you're building and what already exists, and it sets the ceiling for everything after it.

Typical first-hire logic

Shipping an AI feature inside an existing product → an AI product engineer, not a researcher
Building an ML core where data is the moat → an ML engineer plus data foundations first
Still exploring whether AI fits at all → advisory or a fractional lead — not a full-time hire yet

What you receive

The output of the method is a decision you can defend, not a folder of résumés:

1A shortlist of 3–5 people we would hire ourselves
2Written judgment on each: strengths, risks, what we'd verify further
3Market context: compensation, availability, realistic timelines
4Our honest recommendation — including “don't hire yet” when that's the truth

Disagree with something here?

Good — that's exactly the conversation worth having. Bring your hardest hiring question to a strategy session, or send it in writing.

Book a strategy session Send your challenge