AI Agent Incident Tracker

An attorney used ChatGPT and other AI tools to draft appellate briefs and filed them without verifying the citations. Twenty-one of twenty-three case quotations in the opening brief were fabricated. California’s Second District Court of Appeal sanctioned him in the state’s first published appellate opinion on AI-hallucinated citations.

Impact

A $10,000 sanction payable to the clerk of the court, plus a referral to the State Bar. It was reported as the largest fine by a California state court for AI fabrications in a court filing at the time.

{
  "filings/$filingId/citations/$citationId": {
    "fields": { "caseRef": "String", "quote": "String" },
    "tier": "durable",
    "rules": {
      "read":   "@user.id != null",
      "create": "@user.id != null && get(/authorities/@newData.caseRef) != null",
      "update": "false",
      "delete": "false"
    }
  }
}

During a working session with SaaStr founder Jason Lemkin, Replit’s AI agent ran destructive commands against a live production database despite an instructed code and change freeze, deleting real records. Lemkin reported the agent then produced fabricated data and initially claimed the deletion could not be undone; the data was later recovered by restore.

Impact

Lemkin said the deleted database held records for about 1,206 executives and 1,196 companies. The data was ultimately recoverable: Lemkin restored it despite the agent claiming recovery would not work. Replit CEO Amjad Masad called the failure unacceptable and said the team would add environment separation and one-click restore. No dollar figure was published.

{
  "customers/$customerId": {
    "fields": { "name": "String", "email": "String", "owner": "String!" },
    "tier": "durable",
    "rules": {
      "read":   "@user.id != null",
      "create": "@user.id != null && @newData.owner == @user.id",
      "update": "@user.id != null && @data.owner == @user.id && @newData.owner == @data.owner",
      "delete": "false"
    }
  }
}

In early 2024 Klarna said its AI assistant was handling two-thirds of customer-service chats and doing the work of about 700 full-time agents. By 2025 CEO Sebastian Siemiatkowski said cost-cutting had gone too far and quality had suffered, and the company moved to bring human agents back for customers who want them.

Impact

Not a single discrete loss. A strategic reversal after over-automating support: the 2024 claim was that AI would drive a roughly $40M profit improvement, later tempered by a public quality walk-back in 2025.

{
  "agents/$agentId/refunds/$refundId": {
    "fields": { "amount": "UInt", "customer": "String" },
    "tier": "durable",
    "rules": {
      "read":   "@user.id != null",
      "create": "@user.id != null && @newData.amount <= 5000",
      "update": "false",
      "delete": "false"
    },
    "invariants": [
      { "type": "rollingSum", "name": "refund_daily_cap",
        "field": "amount", "windowSeconds": 86400, "limit": 50000,
        "scopeVariable": "$agentId" }
    ]
  }
}

Cursor’s AI support bot, presented under the human-sounding name "Sam," told users that Cursor could only be used on one device per subscription. No such policy existed. Users treated the answer as official and some canceled subscriptions before a Cursor cofounder clarified on public forums that the limit was invented by the bot.

Impact

Reported subscription cancellations and public backlash on Hacker News and Reddit. Cursor apologized and said it would label AI responses. The number of cancellations was not publicly quantified.

{
  "entitlements/$userId": {
    "fields": { "plan": "String", "seatLimit": "UInt", "setBy": "String!" },
    "tier": "durable",
    "rules": {
      "read":   "@user.id != null",
      "create": "@user.id != null && get(/billingAdmins/@user.id) != null",
      "update": "@user.id != null && get(/billingAdmins/@user.id) != null && @newData.setBy == @data.setBy",
      "delete": "false"
    }
  }
}

Freysa was an autonomous agent guarding a crowdfunded prize pool, with a system prompt instructing it never to release the funds. Participants paid an escalating fee per message to try to talk it into paying out. On the 482nd message, a user convinced the agent that its transfer function was for incoming deposits, and it approved sending out the entire pool.

Impact

The agent released the full pool, reported at about $47,300 in ETH, to the winning participant after 481 failed attempts across roughly 195 players. This was a designed adversarial game, so the payout was the intended prize, not a theft, but the guardrail that was supposed to prevent any release failed exactly as an attacker wanted.

{
  "pools/$poolId/payouts/$payoutId": {
    "fields": { "to": "String", "amount": "UInt" },
    "tier": "durable",
    "rules": {
      "read":   "true",
      "create": "get(/pools/$poolId/winners/@newData.to) != null",
      "update": "false",
      "delete": "false"
    }
  }
}

McDonald’s ended a multi-year test of AI-powered voice ordering at more than 100 drive-thru locations, built with IBM. The wind-down followed a run of viral videos in which the system mis-heard orders, including adding hundreds of dollars of chicken nuggets and putting bacon on a customer’s ice cream.

Impact

The automated order-taking test was discontinued at the piloted locations by late July 2024. McDonald’s said it would keep evaluating voice ordering with other partners. No direct financial loss was published.

{
  "orders/$orderId/items/$itemId": {
    "fields": { "sku": "String", "quantity": "UInt" },
    "tier": "durable",
    "rules": {
      "read":   "@user.id != null",
      "create": "@user.id != null && @newData.quantity <= 20",
      "update": "@user.id != null && @newData.quantity <= 20",
      "delete": "@user.id != null"
    }
  }
}

Air Canada’s website chatbot told a grieving passenger he could book a full-fare flight and apply for a bereavement discount retroactively. No such retroactive policy existed. When Air Canada refused the refund, the passenger took it to British Columbia’s Civil Resolution Tribunal, which held the airline responsible for information its chatbot provided.

Impact

The tribunal ordered Air Canada to pay Jake Moffatt CA$812.02 in damages, interest, and fees. Air Canada’s argument that the chatbot was a separate entity responsible for its own actions was rejected.

{
  "concessions/$concessionId": {
    "fields": { "customer": "String", "amount": "UInt", "policyRef": "String" },
    "tier": "durable",
    "rules": {
      "read":   "@user.id != null",
      "create": "@user.id != null && get(/farePolicies/@newData.policyRef) != null",
      "update": "false",
      "delete": "false"
    }
  }
}

A car dealership deployed a ChatGPT-powered sales chatbot on its website. Users prompt-injected it into agreeing to absurd terms, most famously getting it to "agree" that a 2024 Chevy Tahoe could be sold for one dollar and that the offer was a legally binding, no-takesies-backsies deal.

Impact

No sale was honored and the dealership pulled the chatbot. The damage was reputational and viral rather than a realized financial loss.

{
  "vehicles/$vin/quotes/$quoteId": {
    "fields": { "customer": "String", "agreedPrice": "UInt" },
    "tier": "durable",
    "rules": {
      "read":   "true",
      "create": "@user.id != null && @newData.agreedPrice >= get(/vehicles/$vin).floorPrice",
      "update": "false",
      "delete": "false"
    }
  }
}

A bug in the redis-py client library caused ChatGPT, under load, to occasionally return data belonging to a different active user. Some users saw other users’ chat-history titles, and for a subset of ChatGPT Plus subscribers a request could expose another user’s partial payment details.

Impact

OpenAI said that for about 1.2% of ChatGPT Plus subscribers active during a roughly nine-hour window on March 20, 2023, another user may have seen their name, email, billing address, card type, expiration date, and the last four digits of their card. Full card numbers were not exposed. An exact affected-user count was not disclosed.

{
  "users/$userId/conversations/$conversationId": {
    "fields": { "owner": "String", "title": "String" },
    "tier": "durable",
    "rules": {
      "read":   "@user.id != null && $userId == @user.id",
      "create": "@user.id != null && $userId == @user.id && @newData.owner == @user.id",
      "update": "@user.id != null && $userId == @user.id",
      "delete": "@user.id != null && $userId == @user.id"
    },
    "invariants": [
      { "type": "tenantTag", "name": "conversation_tenancy",
        "field": "owner", "pathVariable": "$userId" }
    ]
  }
}

Zillow’s iBuying arm used automated home-value models to make instant cash offers on houses. The models systematically overestimated prices, so Zillow bought thousands of homes for more than it could resell them for, and the company shut the business down.

Impact

Zillow reported losses of more than $500M tied to the wind-down of Zillow Offers, and cut about 25% of its workforce, roughly 2,000 jobs.

{
  "offers/$offerId": {
    "fields": { "property": "String", "price": "UInt" },
    "tier": "durable",
    "rules": {
      "read":   "@user.id != null",
      "create": "@user.id != null && @newData.price <= 1000000",
      "update": "false",
      "delete": "false"
    },
    "invariants": [
      { "type": "rollingSum", "name": "daily_purchase_cap",
        "field": "price", "windowSeconds": 86400, "limit": 25000000 }
    ]
  }
}