Reasonable people can disagree about what’s appropriate and what’s not, but Discord’s automated moderation system has decided to draw a very strange hard line: no networks.
according to Report from The VergeDiscord’s security system mistakenly banned more than 8,000 users from images of things like chessboards and Minecraft inventories — a problem the company blamed on a bug.
The issue has been hitting Discord users since at least May, and finally caught the company’s attention after approx. 200 users were banned in one weekend for grid-like images. A user he claimed their account was flagged for posting alleged child sexual exploitation material (CSAM), resulting in their permanent suspension. In reality, the user posted a screenshot from Minecraft.
In response to the incident, Discord tried to explain what happened. a A series of posts on XThe company detailed how its automated system was designed to flag content by matching it to known harmful material. These compliances must be reviewed by the Trust and Security Team before any action is taken against the user’s account. However, a “bug” in the system prevented affected users from clearing their books with human review, and they were banned instead.
Stanislav Vishnevsky, Co-Founder and Chief Technology Officer of Discord, he explained the moderation system apparently trapped at least 8,000 people with false positives, all of whom posted “good images” and were flagged as false. Those affected were subsequently banned.
What’s interesting about the whole debacle is that Discord and Wisniewski deliberately avoided calling their moderation system “AI,” even though most users refer to the system as such. The company previously published blog posts detailing the details how it uses machine learning Discuss how to identify and stop the spread of CSAM uses AI to help curate content shared on the platform.
Discord is far from the only company dealing with excessive moderation from automated systems, whether they’re technically “AI” or not. Both last year Instagram and Facebook hit its users with a wave of seemingly unjustified account bans, which many believed stemmed from the AI systems used to curate content. TikTok also saw increase in prohibitions provided by its automated content filters and is equal to reduce human moderators In favor of an AI alternative. All this to say, be prepared to argue with bots more often.
