Moderation without a panopticon

You can keep people safe without reading everything they write. The way to do it is to be specific about what each layer looks at, when it runs, and what it deliberately does not cover.

The first layer reads text, at send time, before the write. That timing is the whole of its value. A moderation queue only ever reacts: by the time a human opens it, the message has been delivered and read. Running the rules before the message lands means a blocking rule actually blocks, while a flagging rule delivers the message and files it for review. Same code path, two very different promises, and the difference is only available before the write.

Those rules are plain case-insensitive substring matches, cached for a minute, and they are deliberately not regular expressions. A regex written by an operator in a hurry can backtrack catastrophically on an unlucky input, and this code sits on the send path, so the worst case is not a missed match but every message in the product failing to send. If the rule list ever outgrows a handful of terms, the answer is a real matcher, not a more clever regex.

It also fails open. If the rules cannot be read, messages go through. A safety system that turns its own outage into a total messaging outage has chosen the wrong failure.

The second layer looks at pictures, because the first layer is reading the label on the box. A caption is not what makes a photo a problem, and the photos that matter usually carry no caption at all. It runs on the server, after the upload and before anything appears in a feed. It has to be server-side: a check in the app is a check the uploader can switch off, and the one person who would bother is exactly the person it exists for.

One detail from that layer is worth anybody's time who is pointing a model at this problem. We do not ask "is this inappropriate?". Asked a yes-or-no question, a model obliges. It reads the question as a request to find something, and it finds something. Instead it picks from a list of categories that includes an explicit "none", so it has somewhere to put the photograph of a dog. The shape of the question changes the false-positive rate more than the choice of model does.

For the worst category there is no model in the loop at all: known material is matched by digest against known lists and escalated, because that is a lookup and not a judgement, and it should never be subject to a classifier's opinion.

Separately, there is the question of who can reach a child, which is not a content problem and does not have a content answer. Approval happens before a connection exists: nobody follows a child and a child follows nobody until a guardian says so. Filtering what arrives is a poor substitute for controlling whether it can be sent.

Finally, the part we insist on writing down. Video frames are sampled, three per clip, by handing the decode to a service that has a decoder, because a Worker does not. Live streams are not sampled at all: a live has no upload moment to hang a check on, and doing it properly means sampling on an interval against a running broadcast. That is genuinely unbuilt, and it is recorded in the source next to the code that does work, so that "we classify uploads" is never quietly rounded up to "we classify everything".

That last habit is the one we would recommend most. Write the gaps down where the implementers will see them, in the same file, in plain words. A safety system nobody can describe the limits of is one nobody can improve.