Recode: Here’s How the FBI Pieced Together Hillary Clinton’s Email

Originally published by Recode.

Forensics experts talk about how investigators were able to put Humpty Dumpty back together.

By Dawn Chmielewski

FBI Director James Comey used a jigsaw puzzle analogy to describe how investigators pieced together Hillary Clinton’s email correspondence over the four years she served as secretary of state.
The process was more complicated than it appeared.
Clinton used several email servers and numerous devices during her tenure with the Obama Administration. As hardware was replaced, the older servers were taken out of service, stored and decommissioned in a variety of ways.
“Piecing all of that back together — to gain as full an understanding as possible of the ways in which personal email was used for government work — has been a painstaking undertaking, requiring thousands of hours of effort,” Comey said.
For example, one of the original servers had its email software removed. That didn’t obliterate the emails, of course. It just left millions of bits of information without any organizational structure, as if someone removed the frame from a giant jigsaw puzzle and dumped all the pieces on the floor, Comey said.
Investigators also found several thousand work-related emails that had not been among the 30,000 Clinton turned over to the State Department — some had been deleted over the years, but traces of them remained on devices.
FBI investigators were able to reassemble the pieces to determine whether the email contained classified information at the time it was sent or received. We talked with forensic experts about the process of putting Humpty Dumpty back together again.
Rene Novoa, an expert in forensics and “eDiscovery” at DriveSavers Data Recovery, said Clinton’s systems administrators appear not to have wiped the email servers — a process of overwriting the data with random characters or zeros to make the information unintelligible.
“One wipe of the system, done correctly, can permanently overwrite [the data],” said Novoa. “If they got that much data back, that tells me it wasn’t a secure wipe.”
Recovering emails involves combing through the server’s hard drive, looking for files with telltale signs (technically speaking, the headers) that identify the bits as email. Deleted files wind up in unallocated space that’s invisible to the user, but still there — until another file writes over it, like a fresh coat of paint.
“You use a technique called file carving, which is a fancy way of saying you’re looking for files inside the haystack,” said Jonathan Zdziarski, an independent security researcher. “You don’t have any directory structure at this point.”
Forensics experts can pull together strings of information to determine whether a particular piece of Clinton’s electronic correspondence deals with confidential information or a personal exchange about flowers at Chelsea’s wedding.
The time required to do this depends on a number of variables — including the capacity of the server, the file system, encryption and the software used.
“Most server software will store all of the messages inside a container, like a mailbox file,” Zdziarski said. “Some spit up headers and body and do funky things with the way it stores the content. That can get trickier.”
The political stakes of this particular forensics exercise gave it a high profile. But corporations increasingly rely on such techniques to monitor employees, for instance in the days or weeks leading up to termination to ensure a disaffected worker isn’t giving confidential information to a competitor.