May 16, 2007 (Computerworld) Sarasota County, Florida's computer database infrastructure was attacked by a notorious Internet worm on the first day of early voting during the 2006 election featuring the now-contested U.S. House race in Florida's 13th Congressional district between Christine Jennings (D) and Vern Buchanan (R).In the early afternoon hours on Monday, October 23, 2006, an Internet worm slammed into the county's database system, breaching its firewall and overwriting the system's administrative password. The havoc brought the county's network, and the electronic voting system which relies on it, to its knees as Internet access was all but lost at voting locations for two hours that afternoon. Voters in one of the nation's most hotly contested Congressional elections were unable to cast ballots during the outage as officials were unable to verify registration data.
Remember Slammer?
An incident report filed by the county explains the intrusion and temporary havoc wrought by the virus.
According to the two-page report ( PDF format), a server on Sarasota County's database system was attacked by "a variant of the SQL Slammer worm." Once infected, as the report details, the server "sent traffic to other database servers on the Internet, and the traffic generated by the infected server rendered the firewall unavailable."
[Note re PDF: The incident occurred on 10/23/06 and the incident report was filed on 10/24/06. The second reference to the incident date as "10/14/06" is a typo, as confirmed by Sarasota County Information Security Analyst, Hal Logan, a member of the team filing the report.]
In a separate document, titled "Conduct of Election Report, Sarasota County General Election, November 7, 2006" there are two different Internet service outages mentioned, though the viral attack described in the incident report from the Sarasota County database security team --- presumably the source of one of those outages --- is not described or even mentioned specifically in that report. It's still unclear what the second incident referred to in that report may be.
The SQL Slammer Worm, commonly known as Slammer, was discovered in 2002. In January of 2003, when it was first triggered, the virus brought Internet systems down across the world in a matter of minutes. Though most systems vulnerable to the attack have since been patched by a fix provided by Microsoft prior to the initial 2003 attack, the machine which was infected in Sarasota and which subsequently overtook the network infrastructure "was completely unpatched. Essentially it was missing five years' worth of security updates," according to the October 24, 2006 incident report.
Effects and disclosures
A network security specialist who works for the county and who was part of the team that authored the incident report explains that the damage was contained once the server where the infection struck was taken offline. He believes that beyond the initial damage and the ensuing two hours during which the system became largely unusable --- temporarily making it next to impossible for elections officials to verify residency of voters --- there was no lasting effect on the voting systems used in the FL-13 election or in other races in Sarasota.
But questions remain about whether the incident was disclosed to the parties challenging the election via discovery. In several previous instances, documents believed relevant to the case were found to have been withheld from the plaintiff's attorneys by the Sarasota Election Supervisors office.
One such document was a bug warning issued by ES&S, the voting machine manufacturer for the touchscreen systems used in Sarasota. That warning, which went unheeded by the county, could well have been a part of the many problems voters had registering votes correctly on the touch-screen machines during the election.
In addition, a set of stipulations made by the company to the county prior to their release of the ES&S iVotronic source code to a state-convened panel of computer scientists was also withheld. The panel was commissioned to investigate the still unexplained, extraordinarily high number of undervotes reported on the touchscreen systems in the District 13 Congressional race in Sarasota.
Delayed reaction
Though the worm intrusion occurred on the first day of early voting, two weeks prior to Election Day on November 7th, major structural changes called for in the wake of the attack were postponed until after Election Day according to both the incident report and an email sent on November 8th to the county Supervisor of Elections, Network Administrator, John Kennedy. That email, written by Hal Logan, an Information Security Analyst at Sarasota County's Suncoast Technology Center, was forwarded to the Supervisor of Elections, Kathy Dent on November 9th.
Dent mentioned nothing about the attack in her state-mandated "Conduct of Election" report, signed on November 18th.
(Though an employee at the Elections Supervisor's office told me that Dent was in the office when I called for comment, after giving my name, I was subsequently put on hold and then told she wasn't available. A request to return the call, along with another voice message the next day, was never returned.)
"We have some configuration changes lined up to prevent this type of incident from happening again, and we will begin implementing them next week," security specialist Logan wrote in his email sent the day after the election which described the outage and included the incident report. "Normally they would have been done sooner but we wanted to wait until after Election Day," he continued.
Logan stated earlier this week that the reason for the delay at the time was due to a "configuration freeze" policy concerning "anything that could affect voting" in the lead-up to Election Day and that the vulnerability was contained after the affected system was taken offline.
He stressed that the network affected was the county's database system, used by elections officials at precincts to "verify residence of voters," but that "the Supervisor of Elections maintains their own network for voting data," which is separate from the network which was attacked.
"Had there been any reason to believe that things could remotely affect elections systems, we would have handled it differently," Logan said in explaining why they felt comfortable waiting until after Election Day to make broader configuration changes beyond taking the infected server offline, resetting admin passwords, and taking other immediate steps -- such as reviewing systems logs and monitoring "traffic leaving the system" -- to assure the damage was contained.
Passwords in peril
According to an advisory posted at the U.S. Computer Emergency Readiness Team (CERT) website, a branch of the U.S. Department of Homeland Security, when the SQL Slammer worm was first discovered:
Compromise by the worm confirms a system is vulnerable to allowing a remote attacker to execute arbitrary code as the local SYSTEM user. It may be possible for an attacker to subsequently leverage a local privilege escalation exploit in order to gain Administrator access to the victim system.
The Sarasota incident report confirms that the attack succeeded in changing the administrator password for the county's database system.
When asked if such a worm sent to the system could be used to mask a more nefarious purpose, such as an attempt to hack into the voting system in some fashion, Logan acknowledge that "it's a possibility".
"That's how hackers would normally work," the security expert explained. "Get access to one machine to test the system to see how the rest of the system works."
But if hacking further into the system or planting a virus elsewhere was the hope, Logan believes that it's unlikely that the attack would have been successful. "Our network doesn't share copper or wire with the Supervisor of Elections' network. That's by design for exactly that reasoning," he told me.
The attorneys from the various groups challenging the election on behalf of voters and Christine Jennings in Sarasota, however, have so far not indicated that they were made aware of the either the issue or the incident report, and whether or not the state or Supervisor Dent has disclosed any of the information to the legal team contesting the election.
A race challenged
The race is being challenged both in Florida state court as well as in the U.S. House of Representatives under the Federal Contested Elections Act.
Republican Vern Buchanan was ultimately certified as the winner over Democrat Christine Jennings by just 369 votes. An ongoing investigation by state officials has been unable to determine the cause of some 18,000 undervotes registered only on Democratic-leaning Sarasota's touch-screen voting systems.
The unusually high undervote rate, approximately 18% of the total, where no candidate was registered as selected by voters, has been the cause of much speculation. Normal undervote rates -- in the neighborhood of 2% -- were reported on other races on the same touch-screen ballots in Sarasota, as well as on the paper-based absentee ballots for the same election in the same county.
The second time around?
I asked Logan if he was unaware of a second "separate occasion" when the county's Internet access went down, affecting the "secure connection through the county's internet service provider to the registration database to verify voter's eligibility," as referred to in the county's "Conduct of Election Report".
"On two separate occasions the county's internet service went down," according to the report describing a "County Level Internet Outage".
When queried about what a second outage could be, Logan told us that when system administrators first became aware of the problem at 12:55pm on October 23rd, they thought it was a hardware issue and rebooted the system while they hurried to the data center to look into it. "We rebooted and that brought us back up," he told us, "but by the time we got to the data center, it was back down again."
"Beyond that, I don't remember anything else during this most recent election," he said. "If anything did happen, I do know that it wasn't anything that involved any security equipment."
As Dent has not returned calls, it hasn't been possible to determine whether the outages referred to in that report are related to the one referred to in the incident report concerning the worm attack or if they are different outages entirely.
Contractors in the mix
Just prior to the November election, in early October, the county contracted with a company named IT Convergence for "upkeep, maintenance and performance" work on their database system, according to Logan. At the time, the older, unpatched server was not accessible to the network, but it was assigned a network address once IT Convergence came on board so they'd be able to monitor all of the county's systems.
Logan says that the older server struck by the worm had previously been set to be removed from the system entirely. "It was a little embarrassing having something that old get on our systems," he told me, "But at same time, it was on an old server scheduled to be decommissioned."
So was the Sarasota system targeted by someone? Or was this just a random worm bouncing around the Net which just happened to hit the newly vulnerable server, by coincidence, on the first day of early voting.
Though he clearly believes nothing untoward came of the attack, Logan agrees the timing was interesting. "It would make somebody raise an eyebrow," he said.
Brad Friedman is an investigative journalist, blogger, proprietor of The BRAD BLOG, and an authority on issues related to American election integrity.