Midjourney’s founder, David Holz, says it’s banning these words as a stopgap measure to prevent people from generating shocking or gory content while the company “improves things on the AI side.” Holz says moderators watch how words are being used and what kinds of images are being generated, and adjust the bans periodically. The firm has a community guidelines page that lists the type of content it blocks in this way, including sexual imagery, gore and even the 🍑emoji, which is often used as a symbol for the buttocks.
AI models such as Midjourney, DALL-E 2, and Stable Diffusion are trained on billions of images that have been scraped from the internet. Research by a team at the University of Washington has found that such models learn biases that sexually objectify women, which are then reflected in the images they produce. The massive size of the data set makes it almost impossible to remove unwanted images, such as those of a sexual or violent nature, or those that could produce biased outcomes. The more often something appears in the data set, the stronger the connection the AI model makes, which means it is more likely to appear in images the model generates.
Midjourney’s word bans are a piecemeal attempt to address this problem. Some terms relating to the male reproductive system, such as “sperm” and “testicles,” are blocked too, but the list of banned words seems to skew predominantly female.
The prompt ban was first spotted by Julia Rockwell, a clinical data analyst at Datafy Clinical, and her friend Madeline Keenen, a cell biologist at the University of North Carolina at Chapel Hill. Rockwell used Midjourney to try to generate a fun image of the placenta for Keenen, who studies them. To her surprise, Rockwell found that using “placenta” as a prompt was banned. She then started experimenting with other words related to the human reproductive system, and found the same.
However, the pair also showed how its possible to work around these bans to create sexualized images by using different spellings of words, or other euphemisms for sexual or gory content.
In findings they shared with MIT Technology Review, they found that the prompt “gynaecological exam”—using the British spelling—generated some deeply creepy images: one of two naked women in a doctor’s office, and another of a bald three-limbed person cutting up their own stomach.
JULIA ROCKWELL
Midjourney’s crude banning of prompts relating to reproductive biology highlights how tricky it is to moderate content around generative AI systems. It also demonstrates how the tendency for AI systems to sexualize women extends all the way to their internal organs, says Rockwell.