People who adapt to the model

A nightclub bouncer is told to turn away anyone in trainers. By the second weekend, the people who were turned away have come back in shoes, and the rule is catching no one it was written for while still annoying the regulars. The bouncer’s rule was about something cheap to change, and anyone who hit the rule learned exactly what it was by being refused.

An experienced bouncer watches different things. How someone behaves in the queue, who they arrived with, and whether the same group has tried the door three times tonight are all harder to fake than shoes. Changing them costs the person something, a new set of friends or a quieter evening. The model version of this is to prefer signals the adversary pays to change over signals they can edit in a second.

The bouncer and the spam filter part ways on how cheaply the adversary can test. A person can try the club door a few times a night, while a spammer can send ten thousand messages an hour and watch which ones get through, so each refusal teaches them something and the lesson arrives fast. The running example for the technical part is a messaging platform that must block spam accounts, with bot detection and payment fraud mentioned where they differ.

A content signal is anything in the thing itself, such as the words of a message, the image in a post or the amount of a payment. Content signals are the easiest for a model to use and the cheapest for an adversary to change, because the adversary writes the content. A filter that learns the word ‘free’ is beaten by ‘fr33’ the same afternoon, and the model has to be retrained on examples of ‘fr33’ before it catches up, by which time the spelling has moved on again.

The pattern is a loop, and the loop runs at the adversary’s pace. The model learns from last week’s spam, the spammer tests new messages against it today, keeps the ones that pass, and sends those at volume. Next week’s training data is therefore mostly the messages the model already misses. Assume a spam campaign changes its wording about every two days, and a model retrained weekly spends five days out of seven behind, which is why content-based detection alone has a hard ceiling in these problems.

Behaviour signals describe what an account does over time, such as how many messages it sends in its first hour, how many different recipients it contacts, and whether anyone ever replies. An ordinary new user might message five people on day one and get three replies, while a spam account messages 400 people and gets none. To fake the ordinary pattern the spammer has to send slowly, which cuts the volume they are paid for, as behaving in the bouncer’s queue costs the troublemaker the quick entrance they wanted.

Network signals describe who an account is connected to and what it shares with others. Two hundred accounts created from the same device, paying with the same card, or all following each other in a ring are a structure that one person cannot produce without owning two hundred devices and cards. The structure itself is the evidence, in the way that arriving in the same taxi is evidence to the bouncer. In payment fraud the same idea appears as one card used across dozens of new accounts, and in bot detection as thousands of accounts that like the same post within the same minute.

Timing signals describe when things happen, and machines keep time differently from people. A human types a message in a few seconds and pauses to read replies, while a script posts every 1.5 seconds, around the clock, from the moment the account was created, and slowing a script to human pace reduces its output a hundredfold. Timing is also the hardest to see in a single event, so it is one of the reasons these systems keep running counts per account rather than scoring each message on its own.

Because the adversary adapts, these systems are retrained more frequently than most, and the training data is chosen with the loop in mind. Random audit samples, where a small slice of traffic is let through unfiltered and labelled by hand, are the only view of what the model is missing, because everything else in the data was chosen by the model. Assume one in a thousand messages goes to audit on a platform sending 100 million messages a day, which is 100,000 messages a day to label, so audit is a cost that is budgeted for rather than hoped for.

The response to a detection also matters, because a hard block is the clearest signal you can give the adversary. Softer actions such as a review delay, a limit on how many recipients a new account can message in its first day, or a hidden limit on how many messages an account may send an hour cost the real user little and leave the spammer unsure whether they were caught. The design trade is that softer actions let some spam through for longer, in exchange for the model staying useful for longer.

Measurement changes too. Offline precision and recall are measured against last week’s spam, which the spammer has already moved on from. The metrics that matter live are the number of user reports per million messages and the share of audited messages that turn out to be spam. The payment fraud version of this is the chargeback rate on payments the model allowed, which arrives late for the reasons the labels topic explains.

People design a spam filter as if it were a model that reads the mood of a review, with the words of the message as the only input. It works in the first week and decays from then on, and the fix is to add signals about the account and its connections, which is a data pipeline decision rather than a modelling one.

Offline precision and recall also get reported as if the test set were the future. In an adversarial problem the test set is the past the adversary has already abandoned, so a live measurement such as user reports or the audit hit rate has to sit beside the offline numbers.

Blocking also gets treated as the only possible action, when it is the most informative thing you can do for the adversary, and a limit or a delay can remove most of the harm while revealing less.

In any fraud, spam, abuse or bot design, interviewers listen for whether you noticed that the other side adapts, and one or two minutes on it is enough. Say it early, for example ‘because spammers adapt, I would lean on behaviour, network and timing signals rather than message content, keep a random audit sample as my only unbiased view, and retrain on a short cycle’. That one sentence usually earns a follow-up on how you would know the model is slipping, and the answer is the audit sample.