AI steps are the newest part of many automations. They read a customer's email and sort it, write a first reply, summarize a review, or pick an order number out of a message. Zapier, Make and n8n all offer them, and they can save hours a week.

They also work differently from every other step. An ordinary step does the same thing every time: copy this, send that. An AI step writes something new each time. Most of the time it's right. This guide is about the times it isn't, and how to notice before a customer does.

It's a check you can do yourself in about fifteen minutes. You don't change anything. You only read.

What it looks like when an AI answer is wrong

  • A reply promises something you don't offer, like a refund, a discount or a delivery date. AI fills gaps confidently, so a question with no clear answer can still get one.
  • An enquiry is sorted into the wrong pile, for example a complaint filed as a general question, so nobody answers it for days.
  • A customer's name, order number or price is slightly off, in a message that otherwise reads well.
  • The tone doesn't sound like you: too formal, too chatty, or in the wrong language.
  • A summary leaves out the one thing that mattered.

Why your automation won't tell you

The run history marks a run as successful when every step ran. An AI step that gives a wrong answer has still run, so it gets a green tick like any other. The tool can't tell a good answer from a wrong one. Only a person reading it can.

Three more things make AI steps easy to miss:

  • Answers can change while nobody touches anything. AI companies update and retire their models from time to time, so the same instructions can start giving different answers.
  • Unusual messages get unusual answers. An empty email, a message in Welsh or Polish, a long forwarded thread, or three questions in one message can all produce something odd.
  • Each answer has a cost. AI steps often use more of your plan's monthly allowance than ordinary steps, and some also have a charge from the AI company. This guide on monthly allowances shows where to see it.

The 15-minute check

1. List what each AI step does

For each automation with an AI step, write one line on what the AI decides or writes. Then add where its answer goes next: straight to a customer, into your inbox, or into a spreadsheet.

Anything that goes straight to a customer comes first in the rest of this check.

2. Read ten recent answers next to the originals

Open the run history (Zap history in Zapier, the scenario's History tab in Make, Executions in n8n). Open a recent run and look at what went into the AI step and what came out. Do this for ten runs, including a few from busy days.

For each one, ask:

  • Is everything it says true? Look hardest at prices, dates, policies and promises.
  • Did it use the right name and the right order number?
  • Would you have been happy to send it yourself?

Keep a tally. Even one answer you wouldn't have sent is worth knowing about.

3. Read the instructions it was given

Every AI step has written instructions, often called a prompt. Ask whoever set it up to show you. You're looking for two things:

  • It describes your business as it is today, with current prices, opening hours and policies.
  • It says what to do when the AI isn't sure. For example: "If the message isn't about an order, pass it to a person."

4. Note which customer details go to the AI company

An AI step sends what it reads to an AI company to be processed. Write down which details go: names, email addresses, order history, anything about health or money.

Then check that your privacy notice mentions it. Under UK GDPR, customers should be told who handles their details.

Habits that keep AI answers accurate and kind

  • Draft, then send. For anything a customer will read, have the AI write a draft that waits for a person to approve it. It still saves most of the time, and nothing goes out unread.
  • A clear way out. Ask whoever looks after it to give the AI step a "not sure" answer that hands the message to a person, rather than letting it guess.
  • Five answers a week. Put a short read of five recent answers in your diary, next to the weekly look at run history from the guide on automations that stop quietly.
  • A fresh read after any change. When someone edits the instructions or switches to a different AI model, read a few answers again that week.

What to do with what you found

If every answer you read was one you'd have sent, that's a good sign. Keep the weekly read going, because answers can drift over time.

If you found answers you wouldn't have sent, note the run, what came in and what the AI said. That's exactly what whoever looks after your automation needs to improve its instructions.

And if there's nobody to ask, I can read through it for you, read-only. Making AI steps dependable is a big part of my work. Here's an example of the findings note you'd get.