Trustdesk
Case Study · AI Deployment
Fills in security questionnaires from a company's own documentation. Every answer links to its source, and questions the documentation doesn't cover are marked as such.
Four questions, four outcomes. Each answer links to the section it was taken from.
The problem
Software vendors selling to large companies get the same spreadsheet over and over: a few hundred questions on encryption, hosting, certifications, backups. Someone spends days copying answers out of a security whitepaper and a data processing agreement.
Writing the text is the easy part to automate. The catch is that these answers end up in contracts, so a confident wrong "yes" is worse than an empty cell. Trustdesk is built around that: four possible outcomes, and never an answer without a source.
The documentation answers it, including a documented "no".
Part of it is documented. The answer says what is missing.
No source found, so no answer. It lists the sections it checked.
Two documents disagree. Both are quoted so a person can decide.
The documentation belongs to Vantrel, a software company I invented: twelve documents, written with the gaps, near-misses and one contradiction that real documentation always has.
How an answer is produced
1 · Search
Find the 5 closest sections
Search by meaning, so a French question still finds an English page.
2 · Draft
The model reads only those
It returns a status, the answer, and which sections it used.
3 · Check
Code verifies the sources
Any source the model wasn't given is removed. Without a valid source, the answer is blocked.
4 · Review
A person sends it
The tool only drafts. Uncertain answers are flagged for review.
How I know it works
Every change runs against the same 50 questions, each with a reference answer. Some are easy. Others are traps: rephrased questions, questions in French, answers spread across three documents, and twelve questions the documentation doesn't answer. One model drafts, the other grades, and plain code counts the answers that claim something the documentation doesn't say.
Invented answers on the 12 undocumented questions, by prompt version
Version 3 got the best grade of the three. Version 2 is the one in use, because it's the only one where neither model made anything up. Its cost: out of the 38 questions the documentation does answer, 2 were marked "not documented" with Mistral and 1 with Gemini. Those go to a person, with the sections that were checked. I picked the version whose mistakes are cheap, not the one with the highest score.
What the numbers changed
Each of these went against what I expected before measuring.
The textbook search lost
Share of the right sections found in the top 5 results. Mixing keyword and meaning search is the usual advice, but here meaning alone did better: keywords pushed the wrong sections up on rephrased questions.
I checked the graders against a person
I graded 13 answers by hand. Gemini agreed with me on 12, Ministral 14B on 2: it rejected correct answers. Its scores mostly reflected how strict it was, so Gemini now does the grading.
Free tiers decide the design
Two Mistral models I planned on had a quota of zero, and Gemini allows 500 calls a day. Every run now logs which model answered, since it isn't always the one requested.
Why I build these
I'm a functional consultant moving into Solutions Engineering, Pre-Sales and AI Deployment. Deploying a model is mostly deciding what "good" means for the people who will rely on it, then checking with numbers that you got there.