#4 What not to give away

Contracts, passwords, other people’s data and a recording of your voice. What never to send a chatbot, and how to recognise a call that only sounds like your boss.

The whole contract

  1. Wednesday, 9:40
    Martin: Jana, this chatbot is a miracle. I gave it our contract with the Westbrooks, and it summarised it in a minute.
    Jana: The whole contract, Martin?
  2. Martin: The whole thing. With the attachments. Why would I cut it?
    Jana: Because it has their names, ID numbers and prices in it, which nobody else is supposed to see.
  3. Jana: And where is that contract now?
    Martin: In the chatbot. Well… somewhere.
  4. A little later, at Jana’s desk
    Jana: So where is it? The contract Martin gave you.
    AI: Not with me. But quite possibly with the company that runs me.
  5. AI: What you write to me leaves your phone or computer and goes to someone else’s server. It may be stored there, a person may read it, and it may be used to train the next model.
    Jana: A person? I thought I was talking to you.
  6. Jana: And is that always the case?
    AI: It depends on the service and the account. A personal account, even a paid one, has different terms from a company contract.

  7. How it works

    Where your words go

    A chatbot does not run on your phone or computer. Everything you type or paste goes to the servers of the company that runs the service, and is processed there. What happens to the text after that is set by the service’s terms and the type of your account.

    With personal accounts, even paid ones, you should usually expect that conversations are stored, that in some cases they may be read by people who work on improving the service, and that they may be used to train new models. In most services you can switch off training on your conversations in the settings. That does not switch off storage, or checks for misuse. Company accounts often have different terms. What applies to you is in the service’s terms and your company’s rules. Do not guess.

    One thing is always true. Once you send a text, you lose control over where it ends up. So decide before you send it, not after.

    • In most services you can switch off training on your conversations in the settings. That does not switch off storage.
    • Send only the part you need, without names and numbers.

Fine or no way

  1. Back at Martin’s desk
    Jana: Let’s try this. I name something, and you tell me whether it belongs in a chatbot.
    Martin: I can do that. Probably.
  2. Your turn!

    Item 1 of 8

    Sort what could go into a chatbot and what could not. The characters will tell you what they think.

    A contract with a client, attachments included

  3. Jana: And what if I really do need that contract summarised?
    AI: Delete the names, numbers and prices. Or give me only the paragraph you don’t understand.
  4. Jana: And a company account? People say that one is safe.
    AI: It often has different terms from a personal account. Your company can tell you what applies to you. I can’t.
  5. Jana: Right. Other people’s data, passwords, secrets. I’m not giving any of that away again.
    AI: And it is not only about chatbots. Anything you let out can be used by someone. Even a recording of a voice.

  6. How it works

    What to do instead

    A chatbot is useful without sensitive data too. It can handle most tasks with a general request, or with a text from which you have deleted the names, numbers and prices. Instead of the whole contract, the paragraph you do not understand is enough, with the names left out.

    At work, your company’s rules decide. A company may have a contract with the service under which data is handled differently from a personal account, and a list of what may go into a chatbot. If you have no rules, ask. Under the EU’s AI Act, companies that use AI are expected to take measures to support the AI literacy of their staff, and clear rules for chatbots are a good way to do that.

    • Delete names, numbers, prices and anything that shows who it is about.
    • Send only the part you need, not the whole document.
    • Passwords belong in a password manager, never in a conversation.
    • Other people’s health, personal and business data is not yours to send anywhere.
    • At work, stick to the company account and the company rules. If there are none, ask.

    Sources: Regulation (EU) 2024/1689 on artificial intelligence (AI Act), Article 4 (AI literacy); Regulation (EU) 2026/1744 amending Article 4 (from 27 July 2026)

A call from the boss

  1. Thursday, 16:50, on the way home from work. The phone rings.
    Jana: The boss? But he’s on holiday all week.
    Martin: Then it must be important. Answer it.
  2. From the phone, in the boss’s voice: “Jana, I’m in a meeting and this is urgent. Send the supplier the deposit, I’m texting you the account details.”
    Jana: Right now? But you’re on holiday…
    Martin: I can hear him from here.
  3. “And don’t tell anyone. It isn’t public yet.”
    Jana: It sounds like him. Exactly like him.
    Martin: It is him. Who else would it be?
  4. Your turn!

    The boss’s voice, an urgent tone, a request for a quick payment and for silence. What do you do?

  5. Jana: How can someone speak in my boss’s voice?
    AI: In a similar way to how I write. A model learned how human speech sounds, and a short recording, sometimes a few seconds, is enough for it to copy a specific voice.
  6. Jana: A short recording? Where would they get one?
    AI: From a video on the company website. From a voicemail greeting. From a talk that someone filmed.
  7. Martin: I… posted the boss’s speech from the Christmas party on social media.
    Jana: Of course you did. The contract to a chatbot, the boss to social media. You give everything away.
  8. Jana: So I can’t trust a voice at all any more?
    AI: Treat a voice like an email from a stranger: it may be fake. When it asks for money, passwords or codes, check another way.

  9. How it works

    A voice to order

    Today, a widely available tool can copy a voice from a short recording, sometimes just a few seconds long. A scammer then calls in the voice of a boss, a child or a grandchild, usually in a hurry, asking for money, a password or a confirmation code, and insisting that it stays between the two of you. The boss’s real number may even show on the screen. That can be faked too.

    The defence does not depend on whether you recognise the voice. You cannot do that reliably. It depends on what you do next. You hang up and call back on a number you saved earlier, not the one that just called. At work and in the family, it helps to agree on a code word, in case someone really does need money quickly.

    It has really happened. In 2019, the head of a British energy company sent 220,000 euros to scammers who had copied the voice of his superior at the parent company over the phone. In 2024, the Hong Kong office of the company Arup lost tens of millions of dollars after a video call in which the finance director and the other colleagues were fakes.

    • The signs: hurry, silence, and money, passwords or codes.
    • Neither the voice nor the number on the screen is proof.
    • Hang up and call back on a number you knew before.
    • Agree on a code word for emergencies, at home and at work.
    • At work: no payment and no change to a supplier’s bank account based only on a phone call. Two people approve payments.

    Sources: FTC (US Federal Trade Commission): Scammers use AI to enhance their family emergency schemes (March 2023); McAfee: Beware the Artificial Impostor (May 2023); The Wall Street Journal, 30 August 2019: Fraudsters Used AI to Mimic CEO's Voice in Unusual Cybercrime Case; The Guardian, 17 May 2024: UK engineering firm Arup falls victim to £20m deepfake scam

Quiz

  1. AI: Five questions to finish. What you recall now, you will remember later.
    Jana: As long as nobody asks for my ID number.
  2. Quiz

    Question 1 of 5

    Five questions, no marks.

    Martin pasted a whole client contract into a chatbot on his personal account. Which of these is true?

    Choose one answer.

What to take away

  1. What you type into a chatbot leaves your phone or computer and goes to someone else’s server. It may be stored there, a person may read it, and it may be used to train the next model.
  2. Contracts, customer and colleague data, passwords, other people’s health information and anything confidential do not belong in a chatbot. General questions and your own text do.
  3. It depends on the service and the type of account. A personal account, even a paid one, is not a company contract with the service. What you may do at work is set by your company’s rules, not by the chatbot.
  4. It is not only text you give away. A voice can be copied from a short recording, such as a video on social media. For an urgent call “from the boss” about money, passwords or codes, one rule applies: hang up and call back on a number you know.

Next episode

#5 Why the chatbot forgets

The whole conversation fits in one notebook. What happens when it runs out of pages?

New episodes straight to your inbox

Every two weeks we send the new episodes, a short learning note and news from the world of AI. Free, and you can unsubscribe any time.

By submitting you agree to us processing your email address for the newsletter. More in the privacy policy.

Go to the home pageAll episodes