Microsoft introduced AI-based content moderation service

Company Microsoft announced a new service Azure AI Content Safety, which is designed to provide security and comfort in online communities. The service uses neural network models trained to recognize “inappropriate” content in images and texts. Models can work with texts in eight languages, including Russian, and assign them a score according to the degree of violation, indicating to moderators which content requires intervention.

“Microsoft has been working on solutions to combat malicious content in online communities for more than two years. We realized that existing systems are not context sensitive or cannot work across languages,” a Microsoft spokesperson said via email. “New [AI] models better understand content and cultural context. They are multilingual from the start…and they provide clear and reasonable explanations, allowing users to understand why content has been flagged or removed.”

At the annual Build conference, Sarah Bird, Head of Responsible AI at Microsoft, said that Azure AI Content Safety is a production version of the security system used for the Bing chatbot and Copilot, GitHub’s AI-powered code generation service.

“Now we’re launching it as a product that third-party customers can use,” Byrd said.

It’s worth noting that the technology behind Azure AI Content Safety has repeatedly let Microsoft down. A chatbot on Bing gave out false information about vaccines and wrote offensive texts on behalf of Adolf Hitler. In addition, a few months ago, Microsoft laid off the ethics and society team in its AI division. This left the company without a dedicated team to ensure that the principles of AI were closely linked to product design.

Azure AI Content Safety protects against biased, sexist, racist, hateful, violent, and self-destructive content, according to Microsoft. The service is integrated with the Azure OpenAI Service, a product for enterprise customers that provides access to OpenAI technologies with additional governance and compliance features. But Azure AI Content Safety can also be applied to non-AI systems such as online communities and gaming platforms.

The price of the service is $1.50 for 1000 images and $0.75 for 1000 text entries.

Azure AI Content Safety is similar to other AI-based toxicity detection services such as Google’s Perspective and Jigsaw. It is the successor to Microsoft’s own Content Moderator tool. (It may have been inspired by Microsoft’s 2021 acquisition of Two Hat, a content moderation provider.) These services, like Azure AI Content Safety, assign a score between zero and 100 to how similar new comments and images are to others. previously identified as toxic.

But there is reason to be skeptical about the technology. In addition to issues with the chatbot at Bing and layoffs at Microsoft, research has shown that AI-based toxicity detection technologies still face challenges, including bias against certain user groups.

A few years ago, a team at Penn State found that social media posts about people with disabilities could be labeled as more negative or toxic by publicly available sentiment and toxicity analysis models. In another study, researchers showed that older versions of Perspective often failed to recognize hate speech that used “rehabilitated” slang words like “queer” or spelling variations such as missing characters.

The problem extends beyond toxicity detection services. This week, a New York Times report revealed that eight years after the scandal that black people were mistakenly identified as gorillas by image analysis software, the tech giants are still afraid to repeat the mistake.

Part of the reason for these failures is that annotators – the people responsible for adding labels to training datasets that serve as examples for models – bring their own biases. For example, there are often differences in annotations between tags who self-identify as African American or members of the LGBT community, compared to annotators who do not belong to either of those two groups.

To combat some of these issues, Microsoft allows you to customize filters in Azure AI Content Safety based on context. Bird explains:

For example, the phrase “run over a hill and attack” used in a game would be considered moderate violence and blocked if the game system is set to block moderate content. Adjusting for acceptability of average levels of violence will allow the model to be more tolerant of the phrase.

“We have a team of linguistic and equity experts who have worked to define guidelines that are culturally, linguistically and contextually aware,” added a Microsoft spokesperson. “We then trained the AI ​​models to reflect these guidelines… AI will always make some mistakes, [однако] for applications that require errors to be almost negligible, we recommend using a person in the loop to check the results.”

One of the early adopters of Azure AI Content Safety is Koo, a blogging platform from Bangalore, India with a user base that speaks over 20 languages. Microsoft says it’s partnering with Koo to address moderation issues, such as parsing memes and learning about conversational nuances in languages ​​other than English.