German software developer Andres Freund was running performance tests last month when he noticed strange behavior in a little-known program. He decided to look into it. What he found frightened those in the software world and drew attention from tech executives and government officials.
Freund works for Microsoft in California. He discovered that the latest version of the open-source software program XZ Utils had been sabotaged by one of its developers. The action could have created a secret door to millions of servers across the internet.
Freund noticed the change before the latest version of XZ became widely used. His observation, security experts say, helped save the world from a digital security crisis
The near-miss has re-centered attention on the safety of open-source software. Open-source software is free. Volunteers often maintain the programs. Their openness means they serve as the foundation for the internet economy.
Many such projects depend on a small number of unpaid volunteers working on fixes and improvements.
XZ is a collection of file compression tools for the Linux operating system. It was long maintained by a single person, Lasse Collin.
But in a message published in June 2022, Collin said he was dealing with mental health issues. He suggested he was working privately with a new developer named Jia Tan.
Update logs available through the open-source software site Github show that Tan’s role quickly expanded. By 2023 the logs show Tan was using his code in XZ. It is a sign that he had won a trusted role in the project.
But cybersecurity experts who have studied the logs say that Tan was only acting like a helpful volunteer. Over the next few months, they say, Tan introduced a nearly invisible backdoor into XZ.
Tan did not return messages sent to his email account. Reuters has been unable to find out who Tan is, where he is, or who he was working for. But many people who have examined his updates believe Tan is a pseudonym for an expert hacker or a group of hackers. Experts say Tan was likely working for a powerful intelligence service.
Tan could easily have gotten away with the actions if Freund had not noticed something unusual. He noticed the latest version of XZ sometimes using an unexpected amount of processing power on the system he was testing.
Microsoft did not make Freund available for an interview. But in publicly available emails and posts to social media, Freund said a series of easy-to-miss clues led him to discover the backdoor.
The find “really required a lot of coincidences,” Freund said on the social network Mastodon.
Among those in the open-source community, the discovery has been concerning. The volunteers who maintain the software that supports the internet are used to the idea of little pay or recognition. But the idea that they were now being hunted by well-resourced spies pretending to be volunteers was “incredibly intimidating,” said Omkhar Arasaratnam. He is with the Open Source Security Foundation.
For government officials, the incident has raised concerns about how to protect open-source software. Assistant National Cyber Director Anjana Rajan told the online news organization Politico that “there’s a lot of conversations that we need to have about what we do next” to protect open-source code.
Whatever the solution, almost everyone agrees the XZ incident shows that something must change.
“We got unreasonably lucky here,” said Freund in another Mastodon post. “We can't just bank on that going forward.”
