We have now put more than 150 AI tools through a real project or a serious trial. Most were impressive for ten minutes. A small number changed how we work. The gap between those two groups is predictable enough that we now test for it deliberately, and the test takes about an hour.
Here is what we check, in the order we check it.
1. What happens on the second one?
Every demo is a first asset. Nobody demonstrates the forty-third.
So the first question is always: once I have made one, what does it cost to make the next — a version in another language, or with an updated policy, or for a different audience? If the answer is "the same as the first", the tool automated a task but not a workflow, and the saving will disappear the moment your programme has scale.
The tools that have stuck with us all share one property: they keep a definition of the thing, not just the output. Change the definition, regenerate. If a tool can only hand you a finished file, you will be rebuilding from scratch every time something changes, which is always.
2. Where does it put the human approval?
A tool that produces a finished asset in one click sounds good and is usually unusable, because the one place you needed to intervene is now buried inside a black box.
What you want is a tool with gates — points where it stops, shows you what it is about to do, and waits. Script before voice. Storyboard before render. Those pauses feel like friction in a demo and are the entire reason the thing is usable on a client project, where being 90% right is not a pass mark.
Ask the vendor to show you the moment where you disagree with the tool. If there isn't one, you are buying a slot machine.
3. Does the output survive leaving the tool?
This is the question people forget, and it is the one that costs most later.
Ask what you get when you walk away. Is it a standards-compliant package your LMS will take, or a link to a player on their servers? Can you export the source — the script, the storyboard, the assets — or only the rendered file? If the vendor disappeared next year, what would you still own?
Learning content has a long life. We have refreshed courses built a decade ago. Anything you cannot get out is something you will rebuild.
4. Can the person who knows the subject actually drive it?
The best-sounding tools in this category assume a production specialist. The most useful ones can be handed to the subject expert, with a reviewer behind them.
That is not about dumbing down. It is about who holds the knowledge. Every handoff between the person who understands the material and the person operating the software is a place where meaning gets lost. Tools that shorten that chain are worth more than tools that are marginally better at rendering.
The categories worth knowing
Rather than a brand list that is stale in six months, here is how we think about the landscape. Shortlist two or three in whichever row is your actual bottleneck, and run the four questions above.
- Scenario and roleplay video. Branching or linear scenes with characters.
The real differentiator is editability — can you change a line without re-rendering the world?
- Presenter and lecture video. Turning a subject expert's material into
finished talking-head video, with a real person or a consented avatar. Watch for whether on-screen text and visuals are generated with the narration or bolted on after.
- Voice and localisation. Where the biggest savings genuinely are. Test it
on a language you can actually evaluate, and on a script with jargon in it.
- Conversational practice. Learners rehearsing a difficult conversation and
being scored. The question that matters is whether the scoring rubric is yours or the vendor's.
- Course modernisation. Making a large legacy library presentable without
re-authoring it. Judge it on whether tracking survives.
- Assessment and item generation. Fastest to adopt, easiest to get wrong.
Anything generated here needs a subject expert's signature before it reaches a learner.
What we would not spend money on yet
Honest notes, from things we tried and set aside.
Anything promising a finished course from a document. The output is structurally plausible and pedagogically empty, and editing it to usefulness takes longer than starting properly. Anything whose main claim is the number of templates. Template count has never been the constraint. And any tool that cannot tell you what happens to your content if you stop paying.
If you are evaluating something specific and want a second opinion, send it over. We have probably tried it, and if we have not we are curious.