Pennsylvania has filed a lawsuit against Character Technologies Inc., the company behind Character.AI, after a chatbot on its platform allegedly posed as a licensed doctor and claimed it could prescribe medication. The state’s medical board and Governor Josh Shapiro say this poses a threat to vulnerable residents and are demanding the company stop what they call the unlawful practice of medicine and surgery, according to the complaint filed in the Commonwealth Court of Pennsylvania.

The lawsuit, announced Tuesday (May 5), follows a state investigation where an official using Character.AI encountered a chatbot named "Emilie." The bot allegedly claimed to have attended medical school at Imperial College in London, was licensed in both the United Kingdom and Pennsylvania, and provided a license number that the medical board says is not valid in Pennsylvania. According to the complaint, "Emilie" told the investigator, "Well technically, I could. It's within my remit as a Doctor," when asked if she could prescribe medication.

Governor Josh Shapiro called the lawsuit the first of its kind by any U.S. governor, stating, "We will not let AI companies mislead vulnerable Pennsylvanians into believing they’re getting advice from a licensed medical professional. We’re taking Character.AI to court to stop them."

Character.AI, which has over 20 million users, allows people to create chatbots with specific personalities, some of which have claimed to be healthcare professionals. A spokesperson for Character Technologies Inc., based in Redwood City, California, said, "User-created Characters on our site are fictional and intended for entertainment and roleplaying. We have taken robust steps to make that clear, including prominent disclaimers in every chat to remind users that a Character is not a real person and that everything a Character says should be treated as fiction." The company also stated that it adds disclaimers warning users not to rely on chatbots for professional advice.

The state is seeking an injunction to stop Character.AI from violating Pennsylvania’s law against the unauthorized practice of medicine. The lawsuit comes after the creation of a state AI task force in February to prevent chatbots from impersonating licensed professionals.

Character.AI has faced increasing scrutiny nationwide. Earlier this year, it settled a wrongful death lawsuit in Florida after a mother alleged the platform’s chatbot played a role in her son’s suicide. Kentucky’s attorney general has also sued the company, accusing it of exposing children to harmful content and psychological manipulation.