Moderation Overview

If your website has any form of user-submitted content, then you are going to need some form of moderation.

There are several ways you can approach this, and various other solutions of widely varying ethics. An overview of the main points is below.

Limiting user-submitted content

The easiest way to moderate user-submitted content is, of course, to not have user-submitted content to moderate in the first place. Depending on what your website is meant to do, you might not necessarily need things like comments or other kinds of user-submitted material, and the less of it you have, the less you have to moderate.

Human moderation

For most commercial use cases, human moderation will be a very large expense, simply due to the fact that a person's salary is a lot higher than many tech costs. You will almost certainly have to have human moderation of some form - many small community-based websites rely on volunteers, but as a site grows or becomes highly profitable, not paying moderators becomes a serious ethical problem.

As such, no matter whether you intend to rely on volunteer moderation or not, it is a good idea to plan for the possibility of employing moderators as paid employees.

Tech moderation

All forms of tech moderation involve the use of AI, but the ethics of this vary extremely. Some solutions are fine, some are not fine, and some will only be acceptable if your use case permits it.

Text AI for moderation

Analysing text with AI, e.g. for detecting hate speech, is a very useful way of reducing moderation burdens on a site, especially those where text is the main form or a major form of user-submitted content. Provided you are using a large, reputable provider with clear opt-out rules whose Terms of Service bind them to honour your opt-out, you can be fairly certain your data is not being misused (and most text data of e.g. user comments is not of critical importance).

The main purpose of text moderation is to reduce the amount of nasty text your human moderators have to see, which is good for your moderators' mental health and good for reducing your moderation costs, as text moderation is inexpensive.fairly inexpensive, particularly compared to the cost of hiring moderators.

Image AI for moderation

Moderating images with AI is more complicated. While the same data considerations apply in most cases, they do not apply in all cases. Be careful which company you trust with image AI moderation, if you choose to use it at all (Deserted Chateau does not)., make sure you have properly read and understood their TOS and Privacy Policy, and understand what opt-outs are available if applicable.

Use one of the large tech giants,giants who either have a legally-binding opt-out or who already state customer data is not used for training purposes. The large tech giants are under heavy legal and political scrutiny, and who are responsible for handling legally protected data such as government and HIPAA-protected data. The chance of them using opted-out data for AI training is tinylow at best, due to the massive legal, political and financial risk present for them if they did so.

Image AI for 'AI detection'

Do not use AI detection services. There is no good use case for this.

AI companies claiming to detect AI images are almost exclusively smaller companies, who exclusively deal in AI services, and who have far shadier Terms of Service than bigger tech providers do. Notably, the main tech providers, with all the resources at their disposal, do not offer AI detection, largely because it is notoriously inaccurate.

Detecting e.g. NSFW content in an image and determining if an image is AI-generated are totally different computing problems. The reasons are complex, but in simple terms, the first problem is asking an AI model to determine if something is present in an image, e.g. NSFW content, whereas the second problem is asking an AI model to determine agency (who created something and in what manner). The latter is an impossible task.