What is a Wizard of Oz Test?
A Wizard of Oz test gives customers what looks like a finished product: a form, an app screen or an email address that “does the work”. Behind it, a person (you, the wizard) does everything by hand. The name comes from The Wizard of Oz, where the mighty wizard turns out to be a man behind a curtain. Usability researchers have used the technique since the early 1980s, and John F. Kelley is credited with giving it the name while studying natural-language computer systems.
The test answers one question: when the product works the way you imagine, do people actually use it, and keep using it? You skip months of building the hard part (the matching engine, the AI model, the automation) and measure behaviour first. The well-known business version is early Zappos. Founder Nick Swinmurn has described photographing shoes in local stores, putting the photos online and buying the pair in the store once someone ordered, so the website looked like a shoe shop with stock it never held.
When should you use it?
Use it when the expensive part of your product is the “brain” in the middle, and the customer only sees the input and the output. Good fits:
- AI and automation ideas: anything that promises “upload this, get that back”.
- Matching and recommendation services: the customer fills in a profile, gets a match.
- Marketplaces where you can fulfil orders by hand at first.
Skip it when the customer needs to know a person is involved to trust the result (then run a concierge test), and when one request takes you hours, since you will not be able to keep the promise of a product.
Concierge or Wizard of Oz? In a concierge test the customer talks to you and knows you do the work. That teaches you the workflow. In a Wizard of Oz test the customer deals with an interface. That teaches you whether the product, used on its own, fits into their life.
How do you do it, step by step?
-
Draw the box. On one page, draw input on the left, output on the right and a box in the middle labelled “magic”. Everything inside the box is what you will do by hand, and everything around it must look and feel like the real product.
-
Build the front with no-code tools. A Tally form for the input, an email or calendar invite for the output, a one-page site with your product name. Give it a product voice: “Your plan is ready”, never “Hi, it’s Amara, here’s what I found”. A Figma clickable prototype can serve as the front for app ideas.
-
Promise a response time a human can keep. “Within 2 hours, 8:00 to 20:00” is honest and doable. “Instant” will break your test on day two. Our rule of thumb: promise double the time your fastest manual run takes.
-
Write the wizard’s rulebook. Software behaves the same way every time, so you must too. Write decision rules (“If the date is unclear, pick the earliest one and add a note”) and reply templates. This rulebook becomes the logic of the real product.
-
Be honest where it counts. Your privacy notice must say people review the requests, especially if users upload personal documents. Our recommendation: tell testers at the end of the test what happened behind the curtain, and thank them. You test behaviour, so hiding the human during the test is fine; hiding it forever is a trust problem.
-
Run it with 10 to 20 users for 2 to 3 weeks. Log every request in a Google Sheet: user, time in, time out, minutes of work, anything odd. Odd requests are gold: they show the edge cases your future software has to handle.
-
Measure use, then repeat use. Count requests per user per week and how many users come back in week 2 and week 3. One-time use means curiosity. Weekly use means habit.
What does a worked example look like?
Say Amara wants to build an app that turns photos of school letters into calendar events for busy parents. Building image reading that handles wrinkled paper and five languages would take months.
Her front: a page called “LetterSnap”, a Tally form where parents upload a photo, and a calendar invite that lands in their inbox “within 2 hours”. Behind it, Amara reads each letter, creates the event in Google Calendar and sends the invite from a product email address. Her rulebook covers unclear dates, multi-day trips and “bring €5” notes.
Over 3 weeks, 18 parents send 140 letters. 13 parents are still sending in week 3. The log shows what matters for the real build: 30% of uploads are handwritten teacher notes, 12% are in Polish or Arabic, and the most common problem is a missing year or a missing time. Amara now knows her first engineering task is reading dates in messy handwriting, and she has 140 real examples to test it on.
Which free tools help?
| Tool | What you use it for | Free plan |
|---|---|---|
| Tally | The input form, with file upload | Yes, file uploads up to 10 MB per file |
| Gmail | Sending outputs from a product address, with saved templates | Free |
| Google Sheets | Request log and the wizard’s rulebook | Free with a Google account |
| Google Calendar | Creating events or deadlines as outputs | Free |
| Figma | A clickable front end for app ideas | Yes, Starter plan |
What mistakes should you avoid?
- Answering in your own voice. One friendly personal note breaks the illusion and turns the test into a concierge test.
- No rulebook. If the wizard improvises, users get inconsistent results and you learn nothing about the logic to build.
- Promising speed you cannot keep. A missed response time feels like a broken product and kills repeat use.
- Fooling people about their data. If users share private documents, say in plain words that a person reviews them.
- Counting sign-ups instead of repeat use. Watch week-3 use; day-1 excitement tells you little.
How do you know it worked?
Our rule of thumb: go if at least half of your testers use the product again in week 2 and a third are still using it in week 3, and if your rulebook covers at least 80% of requests without improvising. That second number tells you the magic box can become software.
No-go: people try it once and stop. Rethink the problem. If they keep using it but your rulebook keeps growing, the product needs human judgment, so consider a service business instead.
What should you do next?
Once repeat use is clear, measure it properly with cohort retention and set up analytics before you build the real engine. If the wizard work becomes routine, a form-based service with light automation can carry you to the first paying customers. In the Fe/male Switch startup game you can plan a test like this with an AI co-founder that questions your rulebook.
FAQ
Is a Wizard of Oz test deceptive?
It becomes deceptive when users share personal data without knowing a person sees it, or when you never tell them. Mention human review in your privacy notice and tell testers afterwards what happened behind the scenes. Keep the reveal short and thank them for their help.
How is a Wizard of Oz test different from a concierge test?
In a concierge test the customer knows you personally deliver the service. A Wizard of Oz test gives them a product front end and hides the human work. Concierge tests teach you the workflow, Wizard of Oz tests show whether the product itself gets used.
How long should a Wizard of Oz test run?
Two to three weeks is our rule of thumb, long enough to see whether people come back. One week only shows first-time curiosity. Beyond a month you are running a manual business, and the work eats the time you need to build.
Updated 26 October 2026 · Written by the Fe/male Switch team