alidoes.ai

All writing

Post-mortem

Why does my AI copy credentials into its summaries?

My AI assistant summarized every message it read. When meeting and portal credentials arrived pasted into raw messages, it copied them into summaries and entity lists: one exposure, multiplied.

Key findings

  • Credentials pasted into raw messages were copied into summaries and entity lists — each derived file a fresh exposure surface.
  • Scrubbing the source isn't enough: redaction has to happen at the derived layer, where the copies actually get re-read.
  • A value-safe canary (a harmless, unique marker string) makes the boundary testable: found in a derived file means the redaction failed.

An AI that summarizes your messages will copy the credentials inside them. Redact the derived files, not only the source, and plant a canary so the boundary is testable.

Summarization is a copy machine. Nothing in a summarizer knows that one string in a message is radioactive; its whole job is to preserve whatever looks important.

Here's the incident. I'll call it the pasted-credential incident, with details generalized to protect the people involved. Meeting and portal credentials arrived pasted inside ordinary raw messages. My pipeline did what it was built to do: it wrote summaries and built entity lists (an entity list is a machine-made index of the names, dates, and identifiers a message mentions). The credentials rode along. One exposure in one message became a copy in every file derived from it.

Before: one pasted credential is copied into every derived file. After: redaction at the derived layer plus a canary makes the boundary testable. One secret, many copies, one test BEFORE · the copy machine Raw message credential pasted inside → Summary: copy of the secret Entity list: copy of the secret Every derived file is a new exposure surface, read more often than the original. AFTER · redact the derived layer, plant a canary Raw message canary planted inside → Redact at the derived layer → Summary: secret redacted Entity list: secret redacted Test: search the derived files for the canary. Found anywhere = the redaction failed.
One exposure multiplied through the derived layer (top) until redaction moved to the derived files themselves and a value-safe canary made the boundary testable (bottom). Diagram source: this page; maps 1:1 to the incident-ledger row this post is built from.

How does one secret become many copies?

Because every tool involved did its job correctly. That is what makes this failure ugly: there was no bug, so there was nothing to patch.

A summarizer is graded on fidelity. A pasted credential looks exactly like the load-bearing detail a good summary should keep: specific, unusual, clearly important to the sender. So the summary keeps it.

An entity extractor is worse. Its entire purpose is to notice identifier-shaped strings and index them, and a portal credential is the most identifier-shaped string it will meet all week. Straight into the entity list it goes.

How many copies existed in the end? Honest answer: I don't have a verified count, and I won't invent one. The incident ledger records the pattern (one exposure, multiplied across derived files), not a number I can defend. "More than one, in files built to be re-read" is the part that matters.

Why isn't scrubbing the original message enough?

Because the original is the file with the smallest audience. Derived files (files built from other files: summaries, indexes, digests) are the ones a pipeline actually re-reads. Summaries get loaded into future sessions. Entity lists get searched. Redacting the source, meaning removing or masking the secret where it first landed, cleans the one copy the system was least likely to open again.

So here is my critical judgment, earned the uncomfortable way: source-only redaction is theater. It feels decisive, it satisfies a checklist, and it leaves untouched every copy that will actually be read. If your privacy story ends at "we scrubbed the inbox," you don't have a privacy story.

The fix in the pasted-credential incident was redacting at the derived layer. Treat the summaries and entity lists themselves as the surface that needs scrubbing, because that is where the copies live and multiply. The source still matters. The derived layer is where the exposure compounds.

How do I know the redaction actually worked?

By making the boundary testable instead of trusted. I planted a value-safe canary: a harmless, unique marker string that is not a secret and grants no access, whose only job is to be findable. It travels through the pipeline the way a credential would. The test is then one search: if the canary ever appears in a derived file, the redaction failed, and the search shows exactly which boundary leaked.

You will not see the canary in this post. That is not coyness. Printing it would plant a copy outside the boundary and turn this page into a false alarm in my own test. A canary you publish is a canary you've burned.

One limitation, stated plainly. A canary proves something about the paths it travels; it cannot vouch for a path it never entered. Its coverage is only as good as the injection points I chose. That is a narrower guarantee than "no secret can ever leak," and I will take the narrow guarantee that is testable over the broad one that is vibes.

This is the same rule that runs through the rest of my verification setup: "looks redacted" and "is redacted" are different claims, and only a check can tell them apart.

If you run any summarizer over your own messages (a notes tool, a meeting bot, a CRM enricher), assume it has already copied whatever secrets passed through it. So, the question I'd put to your setup: if a credential landed in your inbox this morning, could you list every derived file it would be sitting in by tonight?

Method & data

Method: one credential-exposure incident from a live message-summarization pipeline — the raw messages, the summaries and entity lists that copied the secret, and the redact-plus-canary boundary that followed, generalized
Data: patterns and methods only; no names, dollar figures, addresses, case identifiers, or confidential content · Last checked: 2026-08-14

How this was made

AI-drafted, adversarially checked, human-directed. My AI assistant wrote this from the system's own records — the incident-ledger row that records this exposure and its fix. A separate AI session then tried to break every claim against those records, and automated privacy and readability gates ran before publish. I direct this pipeline, own every boundary in it, and audit published pages on a rolling basis — if you find an error, tell me and it goes in the corrections log, dated, never silent.

I'm Ali — I run real life-and-work admin on AI agents, then check their work in the open. More at /about.

Published under my standards. Found an error? Tell me — corrections go in the corrections log, dated, never silent.

Cite this

@online{ali2026credentialcopy,
  author = {Ali},
  title  = {Why does my AI copy credentials into its summaries?},
  date   = {2026-08-14},
  url    = {https://alidoes.ai/credential-copy-spread/}
}

Caught something I got wrong? Send it directly. Confirmed corrections go in the corrections log.