Preprint tests human-guided audits to correct AI toxicity measurement bias

Researchers report a method called Threadsafe that uses targeted human labeling to correct biased estimates of conversation toxicity produced by language models. In simulated audits, its uncertainty intervals maintained their intended coverage from 250 human labels onward; with 1,000 to 8,000 labels, intervals were 0.75 times as wide as those from design-based supervised learning. The preprint finds that all seven tested language models excessively flag comments containing "you." Across 1.25 million news comments, Claude Haiku's labels suggested toxicity rose with reply depth, while crowd labels showed a slight decline. Threadsafe uses model scores to select comments for human review and correct predictions. A thousand human labels ruled out Haiku's rising trend.