Choosing Broker Software: What to Actually Ask a Vendor

Every demo looks good. That is what a demo is for.

A demo is run on a clean file the vendor chose, in a workflow the vendor rehearsed, with a document the software already handles well. None of that is dishonest. It is just not your Tuesday.

The questions below are the ones that tend to separate tools that survive contact with a real book from tools that get quietly abandoned after a quarter. They are deliberately awkward. A vendor who answers them straight is worth more of your time than one who redirects to the roadmap.

Ask what happens to a document it cannot read

Every broker's file has scanned PDFs, faxed loss runs, and carrier forms with unusual layouts. The question is not whether the software handles all of them — nothing does. The question is what it does when it cannot.

There are two possible behaviours. It can tell you it could not read the document and stop. Or it can produce a confident-looking output with a plausible-looking number in it. The second is far more dangerous than an outright failure, because it looks finished. Ask to see the failure case, not the success case.

Ask where the number came from

If a tool tells you a sublimit is twenty-five thousand dollars, you should be able to ask it which page of which document that came from and get a straight answer.

This is the single most useful question in the whole evaluation. Anything that produces figures for client-facing documents without being able to point at the source is producing drafts, and you will end up checking every line by hand — which is the work you were trying to remove.

Ask what it does with your data

Three specifics, because the general answer is always reassuring:

• Is your client data used to train models, and is that answer in the contract or just on the website?
• Is your data separated from other brokerages' data, and how?
• Where does it physically sit, and does that satisfy your regulator and your larger clients' vendor reviews?

Commercial clients increasingly run vendor security reviews on their brokers. The answers you get here are the answers you will be repeating.

Ask how long before a producer is actually productive

Setup time is where a lot of broker software quietly fails. Not because the software is bad, but because it needs configuration nobody has time to do, and it sits unused until the subscription lapses.

Ask specifically: what does a producer have to do before they get value on their first file? If the honest answer involves weeks of configuration or a data migration, that is worth knowing before you sign rather than after.

Ask what happens when it is wrong

It will be wrong sometimes. Everything is. What matters is whether being wrong is visible.

Can you see what it did and why? Is there a record of what was produced, from what inputs, and when? If a client questions a document a year from now, can you reconstruct it? A system with no audit trail transfers all of the risk to you while removing your ability to explain what happened.

Ask who owns the output

Read the terms on this one rather than asking in the demo. Who owns the documents produced, what happens to your data if you leave, and can you export it in a usable form? "You can export a CSV" and "you can take your client files with you" are different promises.

Then run your own file through it

The most informative twenty minutes of any evaluation is putting one of your own messier policies through the tool and reading the output as though a client sent it to you. Not the clean one. The one with the odd endorsement schedule and the scanned pages.

Judge it the way you would judge a junior's work. Would you send this to a client with your name on it, or would you rewrite it? That answer is the whole evaluation, and no demo will give it to you.

The underlying question

All of the above is one question asked several ways: does this produce finished work, or does it produce something you finish? Both can be worth paying for. They are just very different purchases, and vendors are not always clear about which one they are selling.

If you want to test the questions above against something concrete, automated policy review and the integrations list are the two places where the answers are easiest to check.