Your phone buzzes: a payment you did not make. Somewhere, a system had about a tenth of a second to say “allow,” “ask for more proof,” or “block,” and it got this one wrong. This article shows how to design that system in an interview and defend each choice. It is written for L5 candidates, with the L4 essentials marked and senior follow-ups at the end.

We use one running example the whole way: Pingpay, a made-up app where people chat and send each other money. Two bad things happen there. Spam is unwanted messages. Fraud is stolen accounts and fake payments. One system can handle both, because both ask the same question: should this event be allowed right now?

The path through this article

We walk in the order you would speak in the room, and we never jump ahead. Each step is numbered the same way in the headings below. A suggested split for a 45-minute round is in brackets; it is a practice aid. The deep dives (step 8) and evaluation (step 10) are longer here than you could speak in the room, because the goal is that you understand them well enough to pick the few points that matter.

1. The problem and the questions to ask (5 min, with step 2)
2. What “good” means, and the limits
3. A simple baseline (10 min, with steps 6 and 7)
4. Workload and capacity (5 min, with step 5)
5. Data and the learning problem
6. The high-level design
7. Model choices
8. Deep dive on the parts most likely to break (15 min)
9. Reliability and security (7 min, with step 10)
10. Evaluation and operations
11. Trade-offs and follow-ups (3 min, with step 12)
12. Spoken recap

1. The problem and the questions to ask first

Begin by narrowing the problem. A good opening is: “Before I draw anything, can I ask a few questions?” Then ask, and say your assumed answer if the interviewer shrugs. For Pingpay, these are our assumptions:

L4 essentials: clarify events, the three outputs, and the latency target. L5 adds: freshness and who sees what. The rest of the article builds on these answers.

2. What “good” means, and the limits

Say early that accuracy is a trap. In Pingpay, about one payment in a thousand is fraud. A system that always says “allow” is 99.9% accurate and useless. So we measure four different things, and keep them apart:

The picture below makes precision and recall concrete. Take one day of Pingpay payments: 10 million payments, 10,000 of them fraud. Say the system catches 8,000 fraud payments and wrongly flags 889 good ones (the full math is in step 4). Every box is a count of payments.

The central tension is simple: every extra bad event we catch risks blocking one more good user. The whole design is a way to choose that trade-off on purpose.

3. Start with a baseline

A strong candidate does not jump to a neural network. Propose the simplest thing that could work, so you can later show what the model adds. For Pingpay, the baseline has no machine learning:

• Rules: block messages with known bad links; challenge a payment over a set amount from a brand-new account.
• Velocity limits: more than 5 payments in 10 minutes from one device triggers a challenge. “Velocity” just means “how fast something is happening.”
• Blocklists: known stolen cards, bad devices, bad IP ranges.

This baseline is fast, easy to explain, and ships in days. Its weakness is also easy to explain: attackers learn the thresholds and stay just under them, and every new attack needs a human to write a new rule. That weakness is the reason to add a model.

The baseline does not go away. We keep the rules after the model arrives, for three jobs: hard policy limits (never pay out to a sanctioned account, whatever the model says), instant response to a brand-new attack while the model catches up, and a safe fallback when the model is unavailable. Two habits keep rules healthy over time. First, every rule has an owner and an expiry date, so old rules do not pile up and quietly block good users. Second, a new rule first runs in shadow mode: it writes down what it would have done, but does not act. After a few days you read the log, count the good users it would have hit, and only then switch it on.

Also say how you will compare later: “I will report the model against this baseline at the same good-user block rate, so we can see what machine learning actually adds.” This one sentence shows interviewers you plan to prove value, not assume it.

4. Workload and capacity

Do the calculations with units, and say what each result means for the design. These numbers are assumptions, not measurements.

Traffic. 50 million daily active users, 10 scored events each: 500 million events per day. Divide by 86,400 seconds: about 5,800 events per second on average. Plan for peaks of about 3 times that: 17,400 per second. Meaning: one machine cannot do this, so the Decision Service must be stateless and scale by adding copies.

Feature reads. Each event looks up history for four things: the user, the device, the network address and the payee. That is about 70,000 key-value reads per second at peak (17,400 × 4). Meaning: the Online Feature Store must be a fast key-value system, not a data warehouse.

Counter memory. To count “payments in the last 10 minutes,” the Stream Processor keeps one count per minute for each device and adds up the last ten. With 100 million active devices, 10 one-minute counts each, and 8 bytes per count, that is 100,000,000 × 10 × 8 bytes = 8 GB. Meaning: this fits in memory across a small cluster, so seconds-fresh counters are affordable.

Log volume. At 2 KB per event (the event, the features used, the decision), 500 million events per day is about 1 TB per day, or about 90 TB for 90 days. Meaning: this goes in cheap bulk storage, not the Online Feature Store. We keep 90 days because labels arrive late (see step 5).

Catch quality. Say 2% of events are payments: 10 million per day. At 0.1% fraud, that is 10,000 fraudulent payments daily. If the model catches 80% of them at 90% precision, it catches 8,000 and flags about 8,889. So about 889 good payments are flagged each day, 0.009% of good payments. That is the number you tell the interviewer you are trading for the 8,000 catches (Figure 1).

The latency budget below shows where the 100 ms goes. The point is not the exact values. The point is that feature reads, not the model, are the biggest cost, so we run them in parallel.

5. Data and the learning problem

A model learns from labels: “this event was bad” or “this was fine.” Labels are the hardest part of fraud, so spend time here. Four sources are common, and each is flawed in its own way:

The delay problem

Last week’s events look “fine” only because their chargebacks have not arrived yet. If you train on them as good, you teach the model that fraud is fine. The timeline below shows what to do: pick a cut-off and train only on events older than it, so their labels have mostly settled. The newest events are used for monitoring, not for training.

Feedback bias and the random holdout

We never learn the truth about events we blocked, because the attacker leaves and no chargeback ever comes. If we train only on what we allowed, the model slowly forgets what blocked attacks look like, and then the old attacks return. The fix is a small random holdout: for a tiny share of events the system would normally allow or challenge (say 0.5%), we record the score but do not use stricter action, so the outcome is observed anyway. Keep the share small, and cap the dollar risk per user, so a stolen account cannot take much. This holdout is also what lets us measure the true miss rate later (step 10).

Leakage, splits, cold start and cadence

• Leakage: never split train and test at random. Fraud changes over time, so a random split lets the model peek at the future. Split by time: train on older weeks, test on newer ones. Also compute every feature “as of” the moment of the event, not as of today. If a feature quietly includes information from after the event (for example “this device was later banned”), the test score looks wonderful and production disappoints.
• Cold start: a brand-new user has no history, so history-based features are empty. Lean on device, network and sign-up signals, apply stricter amount limits to new accounts, and prefer “challenge” over “block.”
• Update cadence: retrain weekly, refresh counters in seconds. Slow learning (models) and fast learning (counters) do different jobs: the model learns what fraud looks like in general, and the counters tell it what is happening right now.

6. The high-level design

Draw two paths and name them. The online path answers one event in milliseconds. The update path runs behind it, keeping features fresh and models current. Keep these names for the rest of the article.

One request. A user taps “Send $50.” The Event API checks the request and gives it an event_id. The Decision Service checks the Decision Store in case this event was already decided, reads features for the four keys in parallel, runs the rules, then scores the model. The score maps to a decision (step 7). The reply goes back, and the event plus its decision are written to the Event Log.

One update. That same log entry is read by the Stream Processor, which bumps counters like “payments from this device in the last 10 minutes” in the Online Feature Store. Seconds later, the next payment from that device sees the new count. Separately, the Offline Store keeps the entry for the next weekly training run.

Say what each store holds, and why it is a different kind of store:

Three things are versioned so you can replay and roll back: the model, the rule set, and the feature definitions. Name that out loud.

7. Model choices

Choose the model after the data and the path are clear. Compare options on the same axes:

Our choice: rules plus a tree model for all events, with a small text model for messages and graph-based features computed offline and stored as ordinary features. This keeps one scoring path and one latency budget.

What the model sees

For a payment, the feature list falls into a few plain groups. Naming the groups in the interview shows you know where the signal is:

• Event itself: amount, currency, time of day, payment method, is this the first payment to this payee.
• Account history: account age, past payments, past complaints, how often the amount is unusual for this user.
• Velocity: payments, failed logins and new payees in the last 1, 10 and 60 minutes, for the user and for the device.
• Device and network: how many accounts this device has touched, whether the address belongs to a data center, whether the device was seen with a banned account.
• Relationship: has the sender paid this payee before, and how many other senders paid this payee today.

For messages, add content features: links and their reputation, a text-model score, and near-duplicate counts. Spam campaigns send thousands of almost identical messages. A simple fingerprint of the text (a hash that stays the same for small edits) lets you count how many senders sent “the same” message in the last hour. A high count is a strong signal even when the words are new.

Fraud rings: why graph features help

Each stolen account may look ordinary. The pattern shows up only when you connect accounts through what they share. Below, five accounts share one device and a card, while a normal account touches neither. A model that sees only one account at a time will miss this; a feature such as “number of accounts on this device” catches it.

From score to decision

The model outputs a score between 0 and 1, but a score is not a decision. First make sure the score means what it says: a calibrated score of 0.8 should be fraud about 80% of the time. Calibration is a small extra step after training (fit a simple mapping from raw scores to real rates on recent held-out data), and without it your thresholds will drift every time the model changes.

Then pick two thresholds from the cost of each mistake. Letting fraud through costs the payment amount. Wrongly blocking costs trust and support time. Between the two thresholds, we ask for proof instead of choosing.

Also keep three levels of “correct” separate: a model metric (PR-AUC), a decision metric (recall at a fixed good-user block rate), and a product metric (dollars lost). A model can improve the first and leave the last unchanged.

8. Deep dive: the parts most likely to break

You will not have time for everything. Choose by risk. For this system, two things fail most often in real life: features that disagree between training and serving, and a slow dependency stalling the decision. We take each one slowly, in the order: what goes wrong, a worked example, the fix, and how you would know it is working. A third, shorter dive covers keeping the counters correct under duplicates.

Deep dive A: features that must match

What goes wrong. The model was trained on a feature such as “payments from this device in the last 10 minutes,” computed from stored history. At serving time, the Stream Processor computes the same feature live. If the two calculations differ even slightly, the model sees different numbers in production than it learned from, and scores drift for no visible reason. This is called training-serving skew. It is common and hard to spot, because nothing crashes: the system just gets a bit worse.

A worked example. Device D7 makes payments at these times. The batch job and the stream job both want “count in the last 10 minutes” at 12:10:00.

The batch job says 4. The stream job says 4 as well, but for the wrong reasons: it missed P3 and double counted P4. Today the totals happen to match; tomorrow they will not. Three separate bugs are hiding here: a different clock (when it happened versus when it arrived), a late event, and a duplicate. Each needs its own fix.

The fix, in four parts.

1. One definition, written once. Every feature lives in a short written spec that both the batch backfill and the stream job read. No one re-implements it by hand. A spec has the entries below.
2. Event time, not arrival time. Count by when the payment happened. The Stream Processor keeps a watermark, a moving line that says “I believe everything before this time has arrived.” It waits a short, fixed time (allowed lateness, for example 2 minutes) before closing a window. A payment later than that is still saved in the Event Log, so the next training run sees it, but the live counter ignores it. You choose the lateness by looking at how late events really arrive (say, 99.9% arrive within 2 minutes).
3. Duplicate-proof updates. Covered in Deep dive C below.
4. Point-in-time training data. When building training rows, compute each feature as of the event’s own time, using only events before it. This is called a point-in-time join. The simplest safe method is to rely on the features we logged at decision time (step 6) rather than recomputing them from today’s data.

How you know it works. Do not trust the design; measure the gap. Every day, take a sample of events, recompute their features offline from the Event Log, and compare with the values that were logged at serving time. Track the share of mismatches per feature. Alert if one feature mismatches more than a small share (for example 1%). Also compare the feature’s average and spread between training data and live traffic; a sudden gap on a single feature points at a pipeline bug rather than a change in user behaviour.

When the numbers are allowed to differ. Perfect equality is not the goal. A late event that the live counter missed but the backfill saw is a known, small, bounded difference. Write it down, measure it, and make sure it is small enough that the model’s accuracy does not move. If it does move, shorten the lateness window or accept a slightly later feature.

Deep dive B: the decision call (low-level design)

What goes wrong. The decision call depends on several other systems. Any one can become slow. With no care, one slow store makes every request slow, requests pile up, and a small slowdown becomes an outage. The aim of this design is that a slow dependency costs a little accuracy, never the whole request.

The diagram shows one request from start to finish. Names match Figure 4. Time limits are the budget from Figure 2.

Walk through it.

1. Deadline. The Event API stamps the request with a deadline, 100 ms from now. Every later step is told how much time is left, and no step is allowed to wait longer than what remains. Without this, a retry deep inside could keep working long after the user has given up.
2. Dedupe first. The Decision Service looks up the event_id in the Decision Store. If the same event was already decided (the phone retried after a bad network), return the saved decision. Without this, a retried payment could be decided twice and could even get two different answers. Two copies of the same event can also arrive at once; to handle that, the first copy writes a “working on it” marker using a set-if-absent operation, and the second copy waits briefly for the result.
3. Parallel reads. The four feature reads go out at the same time, so the wait is the slowest one, not the sum of four. Each has a hard limit of 30 ms. If one read is late, we do not wait: we use the default value from the feature spec and set a flag such as features_partial = 1. The model was trained with such flags present (we simulate missing values in training), so it knows to be more careful when they are set.
4. Rules, then model. Hard-block rules run first. A rule that says “banned device” ends the request immediately, with no model call. Otherwise the model scores. The model is loaded into the Decision Service’s own memory at start-up from the Model Registry, so scoring is a local call with no network hop. The cost is that a new model needs a rolling restart or a careful in-memory swap; we accept that.
5. Decide and save. Map the score to allow, challenge or block, and save it under the event_id. Return it to the client.
6. Log afterwards. The write to the Event Log happens after the reply, so a slow log never slows a user. If the log is down, buffer on local disk for a short time and retry.

Retries, hedging and circuit breakers. Three small tools keep a bad dependency from spreading:

• Retry budget. Retry a failed read only if time remains, and only once. Cap total retries at a small share of traffic (for example 10%). If everyone retries every failure, a store that is struggling gets even more load and falls over; this is a retry storm.
• Hedged reads. If a read has not returned by its usual 95th-percentile time, send a second copy to another replica and use whichever answers first. This trims the slow tail cheaply, but it adds a little load, so use it only for the reads that matter most.
• Circuit breaker. If the Online Feature Store fails or times out on most requests for a few seconds, stop calling it for a short time and use defaults straight away. After a pause, let a few requests through to test; if they succeed, close the breaker again. This saves every request from waiting 30 ms for a store that is down.

A worked failure. Suppose the Online Feature Store’s p99 climbs from 12 ms to 80 ms at 14:00. What happens, step by step?

Deep dive C: counters that survive duplicates

Streams generally promise that each event is delivered at least once, which means that after a failure the same event can come again. For a counter, that means double counting. The fix is to make each update idempotent: applying it twice has the same effect as once.

• Each event has a unique event_id. The Stream Processor keeps a small set of recently seen ids per key, for slightly longer than the lateness window. If it sees one again, it does nothing.
• Counters are stored as per-minute buckets, not a single running number. A bucket can be recomputed from the Event Log if something goes wrong, and old buckets simply expire. A single running total cannot be repaired without replaying everything.
• The state is saved on a regular schedule (a checkpoint). After a crash the processor restarts from the last checkpoint and replays the log after it. Because updates are idempotent, the replay is safe.

Note what this does not cover: writing the counter and sending a message to another service are separate actions, and no setting makes the pair “exactly once” by itself. That is why we deduplicate by id instead of relying on a delivery guarantee. Check your platform’s documentation for the exact guarantees you depend on.

9. Reliability and security

Trace real failures, one at a time, and say what the user sees. A helpful habit: ask of each failure, “is it worse to let a bad event through, or to block a good one?” That answer tells you whether to fail open (allow) or closed (challenge or block).

Attackers are part of the design

• Hide the answer. Return only allow, challenge or block, never a score or a reason. A score is a free test oracle for an attacker who is tuning amounts and timing.
• Limit probing. Cap how many decisions one account, device or address can trigger per minute. Watch for accounts that send many small payments right under a threshold.
• Combine many signals. No single threshold should be worth learning. Rotate rules and retrain regularly so the target moves.
• Protect the labels. Attackers can poison feedback by mass-reporting good users or mass-approving bad ones. Weigh reports by how reliable the reporter has been, and sample reviewer decisions for checking.
• Challenge design. A challenge (code by text, ID check) should be costly for a bot and cheap for a person. Track pass rates by segment; a challenge nobody fails is not telling you anything.

Privacy and access

Store identifiers hashed where you can. Keep personal data in the Offline Store only as long as you need it, and set a retention limit. Restrict who can open the Review Queue, log every access, and keep reviewer tools separate from the production scoring path so a compromise of one does not give the other. If users can ask for deletion, make sure the Offline Store and training sets can honour it.

10. Evaluation and operations

This is where good candidates separate from average ones. Cover three stages: before release (offline), during release (online), and after release (monitoring), and tie them together with one idea: a metric is only useful if you know which decision it will change.

Before release: offline evaluation

Set up the test properly. Use the time-based split from step 5, and only events whose labels have settled. Keep a final “test” period that you look at once; use a separate “validation” period for tuning. If you tune on the test period, your reported score is too good.

Look at the whole curve, not one number. A model has many possible operating points. The precision-recall curve shows all of them, and the thresholds you pick are points on it. The chart below uses made-up numbers to show the shape: as you catch more fraud (move right), precision falls, slowly at first and then quickly.

Pick the operating point from cost. Suppose, as an assumption, a missed fraud payment costs $120 on average, and a wrongly flagged good payment costs $60 (support time, lost trust, some customers leaving). The daily cost for each point is (missed fraud × $120) + (wrongly flagged × $60):

B is cheapest in this example, even though A looks “cleaner.” The lesson is the method, not the numbers: moving from A to B saves more in missed fraud than it costs in extra wrongly flagged users, but moving on to C costs more than it saves. If the cost of a wrong flag were higher, A would win. Say this out loud: the right threshold is where one more catch costs more in wrong flags than it saves in fraud.

Check slices. A model that is good on average can be bad for one group, and that is usually where complaints begin. Report the same metrics for each slice, and compare to the overall number:

Beware of testing on only what you allowed. Your logged data holds labels mostly for events the old system allowed or reviewed. If you test a new model on only those, it looks better than it is, because the events the old system blocked are missing. Two protections: score the random holdout from step 5 (a fair sample), and, as a more advanced fix, weight each reviewed event by one divided by the chance it had of being reviewed (called inverse propensity weighting). You do not need the formula in the room; knowing the problem exists is the senior signal.

Use backtests. Replay last month’s events through the new model and the old one, with features computed as of each event. Compare the two on the same events. Include a stress test: replay a known past attack and check that the model would have caught it.

During release: online evaluation

Offline tests cannot tell you how real traffic, new attackers and real users will react. Roll out in stages, each with a clear exit rule:

• Shadow. The new model scores every event but its answer is ignored. Compare its scores with the live model. This finds crashes, latency problems and feature bugs at no risk to users.
• Canary. Give the new model 1% of traffic, split by account (not by event, so one user always sees one system). Watch the guardrails (below) for a day. Stop automatically if one fails.
• A/B test. Run 50/50 for long enough to see the effect. Fraud is rare, so you need a lot of traffic: with a 0.1% fraud rate, a small test sees only a handful of cases and the result is noise. Plan the sample size ahead of time, and do not stop the test early because the first days look good.
• Keep a holdout. After launch, keep a tiny slice of traffic on the old baseline (or no model at all) so you can still measure how much the whole system adds.

The label delay makes online testing slower than usual. Chargebacks take weeks. While you wait, track early signals: user reports, reviewer-confirmed fraud, challenge failure rate, and account-takeover reports. Treat these as hints, and confirm with chargebacks when they arrive.

Guardrails. A guardrail is a number that must not get worse. For Pingpay (illustrative limits):

Humans in the loop

Reviewers handle the challenge band, and, importantly, a random sample of both the allowed and the blocked. Only the random sample tells you what you are missing, because reviewing only the borderline cases measures only the borderline. Take a few steps to keep the labels trustworthy: write clear review guidelines, have two people label a small share of cases to measure how often they agree, and send disagreements to a senior reviewer. Add a quality dashboard for reviewers, and rotate them so one person’s habits do not become “the truth.” If you use a language model to summarise cases for reviewers, compare its summaries with human judgement on a sample before you trust it, and never let it set the final label alone.

Plan staffing from the numbers: if the challenge band sends 2,000 cases per day to review and a person can finish 150 per day, you need about 13 reviewers, plus sampling work. If you cannot staff that, the thresholds must change, not the staffing assumption.

After release: monitoring

Models do not break loudly; they slowly get worse. Watch these signals, each with a reason and a first action:

When to retrain. On a schedule (weekly), and early when feature or score drift is large, when a new attack is confirmed, or when the realized loss rises above plan. Each retrain goes through the same offline checks and rollout stages as any release. A retrain is a release, not a cron job that pushes straight to production.

Alert wisely. Page a person for things that hurt users now: latency, error rate, a sudden jump in block rate. Open a ticket for slow things: gentle drift, label shifts. Too many pages train people to ignore them.

An incident plan. When block rate jumps: (1) pin the last good model version, which takes effect without a deploy; (2) compare feature drift to find the cause; (3) look at the decision mix by slice to see who was hit; (4) review a sample of wrongly blocked events and restore access to the affected users; (5) write up what broke and add a test or alert so it cannot go unnoticed again.

11. Trade-offs and follow-ups

Follow-up 1: Attackers adapt to our model. What now?

Assume it will happen. Keep the random holdout so new patterns show up in labels. Alert on the share of “near-threshold” scores rising, which often means probing. Add fast rules for a new pattern while the model catches up in the next retrain. Do not reveal scores, and limit how many decisions one account can trigger.

Follow-up 2: Good-user blocks jump an hour after a release. What do you do?

First roll back by pinning the previous version, then investigate. Compare feature distributions before and after: a pipeline change (for example a counter that stopped updating) is more common than a model bug. Look at the decision mix by slice to find who is affected, then check logged feature values for those events. Restore access to the affected users, and add an alert or test so the same failure is caught next time.

Follow-up 3: A brand-new fraud type appears. How do you respond in a day?

Use reviewer labels and the Event Log to find examples, write a targeted rule with a short expiry, and send it through shadow mode. Add a feature that captures the pattern, then include it in the next retrain. The rule buys time; the model makes it durable.

Follow-up 4: Labels arrive in 60 days. How do you ship a model improvement sooner?

Use early signals (reviewer-confirmed fraud, reports, challenge failures) as a proxy to decide on canary and early A/B results, and confirm with real chargebacks later. Backtest on older matured data first. Keep the first rollout small and reversible, so a wrong early read costs little.

12. Spoken recap

You can say this aloud in about a minute:

“Pingpay needs an allow, challenge or block decision on every message and payment, within 100 ms. Because fraud is rare, I measure recall and precision and dollar loss, not accuracy. I start with rules and velocity limits, then add a boosted-tree model on fresh features. Online, the Decision Service checks for a saved decision, reads features in parallel with hard time limits, applies rules, scores the model, and maps the score to a decision. In the background, a Stream Processor keeps counters fresh and a weekly Training Pipeline retrains from time-split data, using a small random holdout so labels are not biased. If a dependency is slow, we fall back to defaults and rules, and risky actions fail closed. I choose thresholds from the cost of each mistake, roll out in shadow mode, then canary, with guardrails and a one-step rollback, and watch drift and matured labels afterwards.”

Split your depth by level:

• L4 essentials: clarify the problem, use a baseline, explain precision and recall, draw the online request path, and name a fallback.
• L5 extensions: label delay and feedback bias, time-based splits, training-serving skew, idempotent counters, thresholds from costs, staged rollout with guardrails.
• Senior extensions: adversarial adaptation, fraud rings and graph signals, evaluating under selection bias, cost and review capacity planning, cross-team ownership of rules versus models, and when to retire a rule.

What is assumed and what is verified

Every number in this article (traffic, fraud rate, latency budget, costs, thresholds, label delay, guardrail limits, curve shapes) is an illustrative assumption chosen to make the math easy to follow. None comes from a real deployment, and nothing here was executed or benchmarked. The precision-recall chart is drawn from made-up points, not from a trained model. The grading rules and loop formats of specific employers were not verified from official sources. Before relying on a product detail, such as stream delivery guarantees or feature store latency, read your platform’s current documentation; for example, Google Cloud describes online feature serving for low-latency reads, though it does not promise the numbers used above.