LAUSD is probing failed AI chatbot, worried about possible misuse of student info

By Mark Keierleber, LA School Report

Independent Los Angeles school district investigators have opened an inquiry into claims that the district’s $6 million AI chatbot — an unprecedented learning acceleration tool until the company that built it collapsed and LAUSD pulled the plug — put students’ personal information in peril.

Investigators with the Los Angeles Unified School District’s inspector general’s office conducted a video interview with Chris Whiteley, the former senior director of software engineering at AllHere, after he told The 74 education news site that his former employer’s student data security practices violated both industry standards and the district’s own policies.

Whiteley told The 74 he had alerted LAUSD, the IG’s office and state education officials to the data privacy problems with the district’s new learning acceleration tool — an animated sun named “Ed” — but got no response.

He met with district investigators on July 2, a day after The 74 published its story outlining Whiteley’s allegations, including that the chatbot put students’ personally identifiable information at risk of getting hacked by including it in all chatbot prompts, even in those where the data weren’t relevant; sharing it with other third-party companies unnecessarily; and processing prompts on offshore servers in violation of district student privacy rules.

In a July 8 interview with The 74, Whiteley said the officials from LAUSD’s inspector general’s office “were definitely interested in what I had to say,” as speculation swirled about the future of Ed and its creator AllHere, and broader education investments in artificial intelligence.

“It felt like they were after the truth,” Whiteley said, adding, “I’m certain that they were surprised about how bad [students’ personal information] was being handled.”

To generate responses to even mundane prompts, Whiteley said, the chatbot processed the personal information for all students in a household.

If a mother with 10 children asked the chatbot a question about her youngest son’s class schedule, for example, the tool processed data about all of her children to generate a response.

“It’s just sad and crazy,” he said.

The LAUSD inspector general’s office directed The 74’s request for comment to the district spokesperson, who declined to respond to questions or comment on the inquiry.

While the conversation centered primarily on technical aspects related to the company’s data security protocols, Whiteley said investigators probed him on his personal experiences with AllHere — which he described as being abusive — and its finances.

Whiteley was laid off from AllHere in April. Two months later, a notice posted to the company’s website said a majority of its 50 or so employees had been furloughed due to its “current financial position” and an LAUSD spokesperson said company co-founder and CEO Joanna Smith-Griffin had left.

Smith-Griffin, a former Boston teacher and Harvard graduate, raised $12 million in venture capital for AllHere and appeared with LAUSD Superintendent Alberto Carvalho at ed tech conferences and other events throughout the spring touting the highly publicized AI tool they partnered to create.

Carvalho spoke publicly about how the project had put L.A. out in front as school districts and ed tech companies nationally raced to follow the lead of generative artificial intelligence pioneers like ChatGPT.

But Carvalho’s praise of what Ed could do on an individualized basis, with 540,000 students, had some industry observers and AI experts speculating it was destined to fail.

The chatbot was supposed to serve as a “friendly, concise customer support agent” that replied “using simple language a third grader could understand” to help students and parents supplement classroom instruction, find assistance with kids’ academic struggles and navigate attendance, grades, transportation and other key issues.

What they were given, Whiteley charges, was a student privacy nightmare.

Smith-Griffin recently deactivated her LinkedIn page. Attempts to reach AllHere for comment were unsuccessful and parts of the company website had gone dark. LAUSD said earlier that AllHere is for sale and that several companies are interested in acquiring it.

LAUSD paid AllHere $3 million to build the chatbot and “a fully-integrated portal” that gave students and parents access to information and resources in a single location, said a district spokesperson in a statement on July 9,  and “was surprised by the financial disruption to AllHere.”

AllHere’s collapse represents a fall from grace for a company that was named among the world’s top education technology companies by Time Magazine months earlier.

Scrutiny of AllHere intensified when Whiteley became a whistleblower. He said he turned to the press because his concerns, which he shared first with AllHere executives and the school district, had been ignored.

Whitely shared source code with The 74 which showed that students’ information had been processed on offshore servers. Seven out of eight Ed chatbot requests, he said, were sent to places including Japan, Sweden, the United Kingdom, France, Switzerland, Australia and Canada.

What district leaders failed to do as they heralded their new tool, Whiteley said, is conduct sufficient audits. As L.A. — and school systems nationwide — contract with a laundry list of tech vendors, he said it’s imperative that they understand how third-party companies use students’ information.

“If the second-biggest district can’t audit their (personally identifiable information) on new or interesting products and can’t do security audits on external sources, how are smaller districts going to do this?” he asked.

In recent weeks, the district’s official position on Ed has appeared to shift.

In late June when a district spokesperson said that several companies were “interested in acquiring Allhere,” they also said its predecessor would “continue to provide this first-of-its-kind resource to our students and families.” In its initial response to Whiteley’s allegations published by The 74 on July 1, the spokesperson said that education officials would “take any steps necessary to ensure that appropriate privacy and security protections are in place in the Ed platform.”

The district said last month that the chatbot had been unplugged on June 14. The 74 asked the spokesperson to provide documentation showing the tool was disabled but didn’t get a response.

Even after June 14, Carvalho was still commending LAUSD’s foray into generative AI and what he described as its stringent data privacy requirements with third-party vendors.

On September 10, a district spokesperson said that the online portal — even without a chatty, animated sun — “will continue regardless of the outcome with AllHere.” The project could become a source of revenue. Under the contract between AllHere and LAUSD the chatbot is the property of the school district, which was set to receive 2% in royalty payments from AllHere “should other school districts seek to use the tool to benefit their families and students.”

In the September 10 statement, the district spokesperson said that officials chose to “temporarily disable the chatbot” amid AllHere’s uncertainty and that it would “only be restored when the human-in-the-loop aspect is re-established.”

Whiteley agreed that the district could maintain the student information dashboard without the chatbot and, similarly, that another firm could buy what remains of AllHere. He was skeptical, however, that Ed the chatbot would live another day because “it’s broken.”

“The name AllHere,” he said, “I think is dead.”

This article originally appeared on LA School Report, a nonprofit newsroom covering Los Angeles’ education system from early childhood through college and career.