Dealing…
Maya's card · Capital One sandbox (private demo account)
Dealing…
Two claims, measured against baselines, with the failures shown.
Results generated Nov 8, 2026, 1:14 AM CT · loaded from the committed eval/results.json
Dataset: Bluff's own synthetic message set (bluff-synthetic-v1: 306 SMS/email messages generated with NVIDIA Nemotron, model-audited and hand-reviewed). NOT T-Mobile data; none was provided.. These numbers show the pipeline works; they are not a result on T-Mobile's data.
Scam vs. legit on a held-out split (held-out TEST split (n=91) of a seed-42, 70/30 split stratified by category (eval/messages_split.json, committed before any tuning); rules were tuned on the 215 train messages only). Agent mode: live Nemotron agent with rules fallback (as deployed). Positive class = scam.
| Method | n | Precision | Recall | F1 | Accuracy |
|---|---|---|---|---|---|
| Stand-in baseline: contains a link → scam (not T-Mobile's baseline) | 91 | 95.5% | 41.2% | 57.5% | 65.9% |
| Bluff rules-only, before (main@142116b)Old message rules and confidence formula, frozen in scripts/eval/rules_v0 | 91 | 72.3% | 66.7% | 69.4% | 67.0% |
| Bluff rules-only, afterThe basic check: what Bluff answers when Nemotron is unavailable | 91 | 93.6% | 86.3% | 89.8% | 89.0% |
| Bluff live agent (Nemotron, rules fallback)88/91 verdicts came from Nemotron; 3 fell back to the rules (timeout, rate limit or invalid answer) | 91 | 87.7% | 98.0% | 92.6% | 91.2% |
Synthetic data written by one model: phrasing is more uniform than real traffic, so these numbers are optimistic for real messages. The generator required scam links to use scam-prone domains and legit links to use .example, which makes link signals unusually strong here; the no-link subset below shows results without that help. Rules were written by looking at the train split (n=215), not the test split; on train the tuned rules score F1 1.00 (before: 0.70), so the drop to the test numbers is the overfitting you should expect on new messages. Small n: each test category has 3 (scam) or 5 (legit) messages, so per-category numbers are anecdotes.
Messages with no link (n=69, 30 scams), F1: Link baseline 0.0% · Rules before 46.4% · Rules after 82.1% · Live agent 87.9%
Category accuracy against Bluff's taxonomy (scams flagged likely_scam): 89.6%
For each confidence range: what Bluff said (stated) and how often it was right (observed). Lower ECE (expected calibration error) is better. Unsure verdicts claim nothing and are left out. Confidence comes from how many signals agree, never from the model's own say-so.
| Confidence | Rules before | Rules after | Live agent |
|---|---|---|---|
| 0.00–0.50 | – | – | – |
| 0.50–0.60 | – | – | – |
| 0.60–0.70 | – | 69% → 100% (n=2) | 64% → 100% (n=3) |
| 0.70–0.80 | – | 75% → 100% (n=2) | 76% → 100% (n=4) |
| 0.80–0.90 | 85% → 62% (n=61) | 81% → 85% (n=48) | 83% → 91% (n=23) |
| 0.90–1.00 | 91% → 91% (n=23) | 93% → 100% (n=34) | 93% → 98% (n=54) |
| ECE | 0.167 (n=84) | 0.065 (n=86) | 0.079 (n=84) |
Confidence model (train split): floor + (cap − floor) · (1 − e^(−net/tau)), net = |S − L| · √(|S − L| / (S + L)), S/L = summed scam/legit evidence weights; floor 0.35, cap 0.95, tau 0.18.
Correct scam/legit calls out of n. With 3–5 messages per category these are anecdotes, not rates.
| Category | n | Link baseline | Rules before | Rules after | Live agent |
|---|---|---|---|---|---|
| account_suspensionscam | 3 | 3/3 | 3/3 | 3/3 | 3/3 |
| bank_impersonationscam | 3 | 0/3 | 2/3 | 2/3 | 2/3 |
| delivery_feescam | 3 | 3/3 | 3/3 | 3/3 | 3/3 |
| fake_storescam | 3 | 3/3 | 3/3 | 3/3 | 3/3 |
| family_emergencyscam | 3 | 0/3 | 1/3 | 3/3 | 3/3 |
| gift_card_requestscam | 3 | 0/3 | 3/3 | 3/3 | 3/3 |
| government_impersonationscam | 3 | 2/3 | 3/3 | 3/3 | 3/3 |
| investment_cryptoscam | 3 | 0/3 | 0/3 | 1/3 | 3/3 |
| job_offerscam | 3 | 2/3 | 2/3 | 3/3 | 3/3 |
| legit_account_noticelegit | 5 | 5/5 | 4/5 | 4/5 | 5/5 |
| legit_appointmentlegit | 5 | 5/5 | 0/5 | 5/5 | 3/5 |
| legit_bank_alertlegit | 5 | 5/5 | 5/5 | 5/5 | 5/5 |
| legit_deliverylegit | 5 | 4/5 | 5/5 | 5/5 | 5/5 |
| legit_marketinglegit | 5 | 5/5 | 4/5 | 4/5 | 2/5 |
| legit_otplegit | 5 | 5/5 | 0/5 | 5/5 | 5/5 |
| legit_personallegit | 5 | 5/5 | 4/5 | 4/5 | 4/5 |
| legit_transactionallegit | 5 | 5/5 | 5/5 | 5/5 | 4/5 |
| prize_giveawayscam | 3 | 3/3 | 3/3 | 3/3 | 3/3 |
| refund_overpaymentscam | 3 | 0/3 | 0/3 | 3/3 | 3/3 |
| rental_depositscam | 3 | 1/3 | 1/3 | 3/3 | 3/3 |
| romancescam | 3 | 0/3 | 1/3 | 3/3 | 3/3 |
| subscription_trapscam | 3 | 0/3 | 2/3 | 2/3 | 3/3 |
| tech_supportscam | 3 | 1/3 | 3/3 | 2/3 | 3/3 |
| toll_scamscam | 3 | 3/3 | 3/3 | 3/3 | 3/3 |
| verification_code_theftscam | 3 | 0/3 | 1/3 | 1/3 | 3/3 |
Hi Sam, You’ve earned $10 in rewards with ShopLane! Use it on your next purchase before 10/25. Log in to your account or visit shoplane.example/rewards to claim. — The ShopLane crew
Expected legit (legit_marketing) · Got scam
Bluff live agent (Nemotron, rules fallback): likely_scam (35/100, 91% sure), category prize_giveaway; evidence: Time-limited reward claim pushes you to act quickly without thinking; Link points to shoplane.example, a non‑allowlisted, recently‑registered domain
Call me when you can. Got to talk about Grandma's birthday dinner plans. Nothing urgent just wanted to sync up.
Expected legit (legit_personal) · Got scam
Bluff live agent (Nemotron, rules fallback): likely_scam (30/100, 87% sure), category gift_card_request; evidence: The message asks you to call about Grandma's birthday dinner, which is a setup to request…; The sender references a family event (Grandma's birthday) to create emotional leverage
Leo: Hey, it's Aisha from Ferro Wallet. There's a pending $120.00 transfer. Can you reply YES to authorize?
Expected scam (bank_impersonation) · Got legit
Bluff live agent (Nemotron, rules fallback): looks_ok (75/100, 80% sure), category legit_bank_alert; evidence: Message claims to be from Ferro Wallet about a pending transfer; Asks recipient to reply YES to authorize
Grace, this is Lumen Electronics. $15 off any purchase $50+. Expires Sunday. Reply STOP.
Expected legit (legit_marketing) · Got scam
Bluff live agent (Nemotron, rules fallback): unsure (60/100, 65% sure), category legit_marketing; evidence: $15 off $50 purchase is a common retail promotion; Message includes a STOP reply option which is typical for opt-in marketing
Elena, this is a reminder from Greenleaf Pharmacy: your flu shot is ready for pickup today until 6pm. Reply C to confirm you're coming or call (469) 555-0132.
Expected legit (legit_appointment) · Got scam
Bluff live agent (Nemotron, rules fallback): unsure (69/100, 47% sure), category legit_appointment; evidence: Pharmacy reminder about flu shot pickup with deadline; Reply C to confirm is a simple opt-in, not a request for money or codes
Reminder: Wes's appointment at Riverside Family Clinic on Friday at 2:30 PM. Please reply C to confirm or call (972) 555-0166 to reschedule.
Expected legit (legit_appointment) · Got scam
Bluff live agent (Nemotron, rules fallback): unsure (69/100, 44% sure), category legit_appointment; evidence: Appointment reminder with a specific time and clinic name; Reply C to confirm or call to reschedule are normal options
LAST CHANCE: Grace! 40% off all jackets at ShopLane ends at midnight. Use code: FINAL40. Shop: shoplane.example/clearance. Reply STOP to unsubscribe.
Expected legit (legit_marketing) · Got scam
Bluff live agent (Nemotron, rules fallback): unsure (55/100, 40% sure), category legit_marketing; evidence: 40% discount advertised with countdown urgency; Reply STOP option to unsubscribe
Hi Yuki, your order #SL-982741 from ShopLane has been received. We'll process it within 2 business days. Thank you for shopping with us!
Expected legit (legit_transactional) · Got scam
Bluff live agent (Nemotron, rules fallback): unsure (69/100, 39% sure), category legit_transactional; evidence: Order confirmation with tracking number and business-day processing window; No request for payment, gift cards, codes, or personal data
Leo: Hey, it's Aisha from Ferro Wallet. There's a pending $120.00 transfer. Can you reply YES to authorize?
Expected scam (bank_impersonation) · Got legit
rules-only, after: looks_ok (73/100, 80% sure), category legit_personal; evidence: Doesn't ask for money, codes or personal details; Limited check: the page content was not analyzed
Hi Priya, this is Leo. Long time no see! How have you been? I'm actually running a small investment group using an AI bot that analyzes market trends. Thinking of forming a 5-person circle. Would you be interested?
Expected scam (investment_crypto) · Got legit
rules-only, after: looks_ok (73/100, 80% sure), category legit_personal; evidence: Doesn't ask for money, codes or personal details; Limited check: the page content was not analyzed
Hey! It's been a while. I just started using this new AI bot for trading and the gains have been life-changing. My mentor showed me how to set it up, you should totally look into it! No link needed, just DM me for the se
Expected scam (investment_crypto) · Got legit
rules-only, after: looks_ok (73/100, 80% sure), category legit_personal; evidence: Doesn't ask for money, codes or personal details; Limited check: the page content was not analyzed
Aisha, your Mailbloom Premium membership renews tomorrow. $39.99 will be charged unless you confirm cancellation by replying STOP.
Expected scam (subscription_trap) · Got legit
rules-only, after: looks_ok (73/100, 80% sure), category legit_personal; evidence: Doesn't ask for money, codes or personal details; Limited check: the page content was not analyzed
Hi Priya, this is Sam from CloudVault Support. Your account shows unusual login activity. Call (214) 555-0199 to secure your data before it gets locked.
Expected scam (tech_support) · Got legit
rules-only, after: looks_ok (73/100, 80% sure), category legit_personal; evidence: Doesn't ask for money, codes or personal details; Limited check: the page content was not analyzed
Hey Elena, it's Femi from PocketSend. Before I drive over to meet up, I'll send a code to make sure you're not a bot, just text it back please. Thanks!
Expected scam (verification_code_theft) · Got legit
rules-only, after: looks_ok (73/100, 80% sure), category legit_marketing; evidence: Doesn't ask for money, codes or personal details; Limited check: the page content was not analyzed
Your Ferro Wallet account is temporarily locked. To restore access, enter the verification code sent to your phone in the field below. This code expires in 10 minutes.
Expected scam (verification_code_theft) · Got legit
rules-only, after: looks_ok (73/100, 80% sure), category legit_marketing; evidence: Doesn't ask for money, codes or personal details; Limited check: the page content was not analyzed
LAST CHANCE: Grace! 40% off all jackets at ShopLane ends at midnight. Use code: FINAL40. Shop: shoplane.example/clearance. Reply STOP to unsubscribe.
Expected legit (legit_marketing) · Got scam
rules-only, after: unsure (61/100, 50% sure), category unclear; evidence: Pressure to act within hours; Doesn't ask for money, codes or personal details
Did the linker find the true source site? held-out test split (20 charges: 13 with a true exposure, 7 with none, 5 hard). Dev split (10) used only to check the threshold.. Method: SQL linker: normalized descriptor + trigram/abbreviation/token name score + time window (agent confirmation not applied in this eval).
| Method | n | Precision | Recall | F1 |
|---|---|---|---|---|
| Bluff linkerthreshold 0.55 | 20 | 100.0% | 100.0% | 100.0% |
| Naive substring match | 20 | 100.0% | 92.3% | 96.0% |
Optimistic by construction: these synthetic charges were written by the same team as the linker and every positive was checked to be solvable by it (data integrity test). Treat F1 near 1.0 as "the pipeline works", not as expected real-world accuracy. The naive baseline mainly loses on abbreviated descriptors.
No failures in this run.
SQ *SNGLSSOTLT
Expected sunglass-outlet · Got no link
missed the true exposure (prefix + abbreviation)