Trust Is a Budget

We had been digging into grant abuse again, and we wanted something the verification logs couldn't give us. Those logs tell you a health rate per function, which is the right number for deciding whether enforcement is safe and the wrong one for the question we actually had: which specific accounts were showing up on real hardware, and which were not? A free starter grant is worth farming, so that's an answer worth having. One Tuesday afternoon we shipped an alert. Whenever an authenticated call arrived carrying anApp Check provider we didn't recognize, it pinged a Slack channel. The logic felt obviously right: a real device produces a real attestation provider, so anyone who isn't is worth a look. The first messages came in within minutes.

The very first one flagged a customer who'd been paying us since 2023. We looked the account up: a real iOS device, an active subscription, nothing unusual about it at all. The provider we'd just called "suspicious" was device_check_app_attest, which is the value real iOS Release builds actually report, about the most trustworthy thing the system can produce. We'd seeded our allowlist from the documented provider names and missed the one combined variant the SDK really uses. Our brand-new fraud alert was paging us about our best customers.

A two-line patch fixed that particular embarrassment, and the alerts kept coming. For a couple of days we triaged them like incidents, looking up each account by hand, until the pattern got too obvious to ignore. Every alert was one of two things. Sometimes it was an abuser we'd already stopped from the signup side, the Apple Hide My Email account-cycling, blocked and handled. More often it was a real person whose call had landed in the half-second before App Attest finished provisioning, so their device reported no provider yet and a perfectly good one a moment later. We'd watch the same callable, ten seconds apart, return a different answer on the very same phone.

A single App Check observation, it turned out, folds at least four populations together: genuine abuse, a real device mid-bootstrap, a transient network or clock failure, and old clients that predate the rollout. Three of the four are completely benign, and they badly outnumber the fourth. The worst false positives came from our own earliest callables, the ones that fire the instant the app wakes up, before the phone has finished proving itself. We'd built a fraud siren and pointed it straight at noise.

Here's the honest part, the one we keep coming back to. As a gate, App Check has never caught a real faker for us. Not once. Every persistent bad signal we've chased resolved into something benign, or into an abuser we'd already stopped some other way. The machinery we built to keep people out has, so far, kept exactly nobody out.

That could read as a failure. It turned out to be the opposite, because of what we did next. We stopped asking each call whether it looked suspicious and started asking each account a better question: across all of your calls, what fraction proved a real hardware device? One observation is ambiguous; a thousand of them, summed per user, are not. Once we had that number, the whole thing turned over. The gate had caught nobody, but the measurement it had been throwing off all along, quietly, on every call, was the asset we actually wanted. A per-user trust score tells you who has earned the benefit of the doubt.

And that changes what you do with your effort. Instead of spending all of it trying to keep people out, you can spend some of it being generous to the accounts that have plainly earned it: the ones we've watched show up on real hardware a hundred times without fail can have their new-account friction eased. It's a safe kind of generous, too. Being wrong about easing friction costs us a little and reverses in seconds; holding a real customer at arm's length because they tripped a rule meant for someone else costs us something we don't get back. Our companion post, Trust Is Additive, builds that friction; the score here is one of the things that decides who to lift it for.

Trust is a budget. Spend it where being wrong is cheap to undo.

The per-user signal

The first post in this series read App Check's readiness signal from Cloud Logging, which gives you a health rate per function: the right metric for deciding whether enforcement is safe, but blind to any individual account. The verification log has a trace ID and a result, no UID.

The per-user signal lives somewhere else. The Firebase callable runtime exposes the verified token claims on request.app.token whenever a valid token comes in, and that object carries a provider field naming theattestation primitive that minted it: App Attest, DeviceCheck, Play Integrity, and so on. It's technically undocumented (a permissive index signature on DecodedAppCheckToken) but stable in production. A small server-side helper reads it, persists it to the user's profile, fails safe, needs no client changes, and works against the entire installed base on day one, including the four-week tail of older clients.

Aggregate per user, with the Wilson lower bound

A single observation is ambiguous; a stack of them tells a clear story. Twenty calls that all produced device_check_app_attest is a real iOS device. Twenty that all produced none is someone to look at. The borderline cases sort themselves out as they accumulate calls, because the legitimate majority of their traffic produces real providers.

So the helper keeps two lifetime counters on the profile: appCheckObservationCount for the total and appCheckSecureCount for the secure subset. The naive score is the raw ratio with a hard floor (under five observations, show nothing), but a floor is a guess at "enough" and is wrong at the edges. What we shipped does that job continuously: the Wilson score lower bound of the secure ratio, at a 95% confidence interval.

// Wilson score lower bound (95% CI). A confidence-aware ratio:
// 1/1 ranks at ~0.21, 60/60 at ~0.94, 100/100 at ~0.96.
function wilsonLowerBound(secure: number, total: number, z = 1.96): number {
  if (total === 0) return 0;
  const p = secure / total;
  const denom = 1 + (z * z) / total;
  const center = p + (z * z) / (2 * total);
  const margin = z * Math.sqrt((p * (1 - p)) / total + (z * z) / (4 * total * total));
  return (center - margin) / denom;
}

Wilson answers exactly what a sortable admin column needs: how trustworthy is this account, discounted for how little we've seen of it? A perfect 1/1 lands near 0.21, because one good call is barely any evidence; 60/60 lands near 0.94, and 100/100 near 0.96. The same 100% ratio spreads across the column by how much we've actually watched, so no floor is needed to keep a 1/1 account from ranking like a 100/100 one.

The helper does a transactional read-modify-write rather than a blind FieldValue.increment, because it needs the post-increment counts in hand to compute Wilson and persist the result in the same atomic write. The score lands on users/{uid}._sort.appCheckTrust, so admin tooling sorts the whole user collection by confidence-weighted trust with no client-side recompute. A 60-second-per-uid in-memory throttle keeps that extra read cheap on hot callables, and a try/catch around the whole thing means it can never throw or block the caller, whatever the App Check SDK returns.

Where we observe, and where we don't

We first instrumented the bonus and check-in callables, because they fire on signup. That was the wrong place: they fire earliest in the app lifecycle, often in the same second as auth state change, before App Attest has minted a token, so real users produced provider: none on every cold start and a valid provider two seconds later. We moved observation onto the generation callables (image, text-to-image, upscale, video) for three reasons: App Attest is reliably provisioned by then, so a none is a clean signal; generation is where each call costs real Civitai or Fal money, which is exactly when we want to know who's real; and heavy users hit those callables hundreds of times, so their score converges fast.

The trust score in admin tooling

The score renders as a column in the user table, read straight off _sort.appCheckTrust. Because it's the Wilson lower bound, the column is readable on its own: brand-new accounts sit low and climb as evidence accumulates, with no separate "insufficient" state to special-case. Real hardware climbs toward the mid-0.90s within a few dozen generations. The known-bad pattern from the grant-abuse post (Apple Hide My Email cycling against a device with its attestation bit already burned) sits near zero and stays there.

What a red flag would have looked like

Since we never saw a real faker, the fair question is what one would look like, and what we'd do. We wrote that down before we needed it, because writing the playbook while you're calm is cheap insurance. A red flag isn't a single bad observation; it's a pattern with three parts together: a cohort well past the bootstrap window whose Wilson score is pinned near zero (many calls, almost none secure), correlating with an independent signup signal (the Apple Hide My Email account-cycling, a burned device bit, a shared payment instrument), and spending real money on the generation callables. One account at 0/30 is a question. Fifty at 0/30 that trace to one device and hammer the video callable is the red flag.

What we'd do is a graded ladder, each rung costing more and reversing less than the last:

1. Stop extending rope. Exclude the cohort from any trust-based relaxation of friction. Instant, invisible to good users, free to reverse.
2. Keep baseline new-account friction on. Don't relax the concurrency cap, velocity cooldown, or moderation restocking fee (the knobs from the sister post). Still no enforcement.
3. Selectively enforce on the money callables. Only if it persists and costs real compute: flip enforceAppCheck to true on the generation callables for that cohort. First user-visible step, narrowest surface, still reversible.
4. Human-confirmed hard action. A block, a clawback, an account hold, only after a person looks. Never automated off the score, which is a confidence signal, not a verdict.

The Slack signal, gated and kept

The original Slack-alert path is still in the code, gated behind a Remote Config flag (simulatorObservationAlerts) we keep flipped off. The data still flows to Firestore and the admin UI; we've just stopped paging the team on a low-confidence signal. A noisy per-call signal isn't steady-state paging material, but it's a fine investigation tool when you turn it on for a day on purpose. That's a local instance of the discipline we wrote up in observe before you enforce. When the flag is on, the first alert looks like this:

🕵️ Suspicious App Check signal observed
UID: `utdp9HNX2iTH5OPqcsZKpNzWbuF2`
Callable: claimStarterGrant
Provider: `device_check_app_attest` (expected app_attest / play_integrity / device_check)
App ID: 1:26011306565:ios:d8289745f6013f47ca712c

Trust is a budget you can spend

A user with 60 of 60 calls returning a hardware provider has demonstrated more about the legitimacy of their account than any email-verification flow would catch. That track record lets us spend some of the trust we've watched them build: lift the concurrency cap, ease the velocity cooldown, waive the moderation restocking fee earlier in their account life. None are big alone, but together they stop us treating someone we've watched produce real attestation sixty times like they just walked in the door.

The asymmetry is what makes it safe. Being wrong about relaxing friction is cheap and reversible in seconds; faking a high score means running a real device and the real signed app, an expensive way to abuse a starter grant. Being wrong about applying friction to a real customer is a durable brand cost. So we relax liberally and apply conservatively. We're not acting on this yet (the distribution is still building) but it's the direction the signal wants to grow into, and Diffusitron is better for having written it down.

What we'd take with us

• The same observations you collect to defend a thing can become the telemetry that tells you who to trust with it. We built two counters to catch abusers and got a trust score we didn't plan for.
• When you're scoring a ratio you'll sort on, reach for the Wilson lower bound before a hard floor. It discounts low-evidence scores continuously, so 1/1 and 100/100 don't collide.
• Persist the derived score, not just the raw counts. Writing Wilson to _sort.appCheckTrust on every observation lets the admin tool sort with no client-side recompute.
• Keep a noisy signal gated and silent, as an on-demand tool rather than steady-state paging. The first alert teaches you about your rules, not your users.