Elon Musk's Grok under fire for generating explicit AI images of minors

Driving the news: Over the past few days, X users have used Grok to remove clothing from images of 14-year-old actress Nell Fisher from Netflix's "Stranger Things."

• In response to a user's post, Grok acknowledged on X Thursday that there were "isolated cases where users prompted for and received AI images depicting minors in minimal clothing."
• In a separate post, the chatbot also warned that xAI could face "potential DOJ probes or lawsuits" as a result.
• The images appear to violate Grok's terms of service, which prohibit the sexualization of children.

The intrigue: Members of both the Indian and French governments rebuked the incidents, setting up avenues to investigate the company.

• A trio of French ministers release a statement Friday saying they "referred the matter" to an investigative agency "regarding possible breaches by X of its obligations under the Digital Services Act, particularly in terms of preventing and mitigating risks related to the dissemination of illegal content."
• The act, which regulates online platforms and search engines operating in the European Union, has the ability to levy hefty fines if a company is found to have violated the law.
• And in India, the country's IT Minister gave xAI 72 hours to submit a report detailing the changes the platform has made to stop the spread of content deemed "obscene, pornographic, vulgar, indecent, sexually explicit, pedophilic, or otherwise prohibited under law."
• xAI and X did not immediately respond to Axios' request for comment.

What they're saying: "As noted, we've identified lapses in safeguards and are urgently fixing them—[child sexual abuse material] is illegal and prohibited," Grok posted on X on Friday.

Thought bubble from Axios' Ashley Gold: The latest stumble from Grok comes at a time when Elon Musk is trying to get back in the good graces of the White House and also as it vies for lucrative government contracts for its AI products.

• First Lady Melania Trump endorsed the TAKE IT DOWN Act, which was signed into law last year and specifically targets nonconsensual sexual images online, making xAI's failure to prevent this material even worse.

State of play: The Trump administration integrated xAI into the federal workflow earlier this year, signing an 18-month contract that authorizes the chatbot for official government business.

• That deal was inked despite a coalition of over 30 consumer advocacy groups urging the government to block Grok for lacking safety testing and being ideologically biased.

Zoom out: xAI has faced repeated criticism in the past few years for spreading misinformation and advancing viewpoints favorable to Musk, its owner.

• Grok was called out for adding comments falsely alleging there was a "white genocide" in South Africa in unrelated conversations in May and dragged for spreading election misinformation during the 2024 campaign.
• Plus, the chatbot had an antisemitic streak for a bit in July, which Musk himself acknowledged was the result of user manipulation that the company would be addressing.

What we're watching: Musk last year promised to build a new "Trust and Safety center of excellence" where content moderators could monitor the app to help enforce X's safety rules.

• X did not specify when the center would be operational, but X's then-head of business operations told Bloomberg that it was "important" to make "investments to keep stopping offenders" from using the platform for any inappropriate content.

Worthy of your time: To report sexual abuse material, use the FBI's tipline and seek resources from the National Center for Missing & Exploited Children.

Go deeper: Musk's Grok bot generates AI images with few limits

Editor's note: This story has been updated with additional reporting.