Re: Riverside Voting Machine Verification Process Lacking

[Christian Rudolph <reindeer@xxxxxxxxxx>]

Moon quoted Akin when he wrote:

"The only thing that I've heard about this is that they needed the  
pre-election mode to make sure that the test didn't leave any votes in 
the system that would be counted during the election. 
And this seems  really weird to me ... 
I can see why a company might have a testing mode,
but I do not understand why a company wouldn't test something in  
production mode. And you can ask any QA person about this and they're  
going to tell you the same thing, that it's not a thorough test if it's 
not tested in production mode."

http://www.salon.com/tech/feature/2003/10/15/riverside_voting_machines/print.html
End quote.


Moon Lee, et al.,
In my twenty-three years of experience, eight as a software quality
assurance engineer, I can tell you that if no testing is done in
production mode, then this system has NOT BEEN TESTED.
If Akin's claim is accurate, then Sequoia should not be selected.

-Christian Rudolph









On Wed, 2003-10-15 at 13:09, Moon Lee wrote:
> http://www.salon.com/tech/feature/2003/10/15/riverside_voting_machines/
> print.html
> 
> Bad grades for a voting-machine exam
> 
> Riverside County, Calif., invited citizens to observe a test of its  
> computerized voting systems. One participant was not impressed.
> 
> By Farhad Manjoo
> 
> Oct. 15, 2003  |             Over the past several years, as computer  
> scientists began expressing concerns about the security of touch-screen  
> electronic voting machines, elections officials across the nation have  
> reassured the public with a simple answer: Testing.
> 
> Elections officials maintain that before they are ever used in an  
> election, electronic voting systems are put through a battery of tests,  
> the culmination of which is the "logic and accuracy" test that counties  
> perform a few weeks or days prior to an election. This examination is  
> billed as a simple, straightforward way of telling whether a machine is  
> working as it should. A predetermined number of ballots are fed into  
> the machine, and then the votes are tabulated. If the system spits out  
> the results you expect, the system is deemed fully functional.
> 
> But Jeremiah Akin, a 28-year-old computer programmer who recently  
> observed one of these tests in Riverside County, Calif., says that what  
> he saw did nothing to mitigate his concerns about electronic voting --  
> indeed, the whole thing made him more worried than ever.
> 
> Akin, who observed the test as a representative of the Peace and  
> Freedom Party, says that representatives of other parties who were  
> there signed off on the test without waiting to see the complete  
> results. In fact, he says, nobody else seemed concerned that anything  
> could go wrong with touch-screen machines. In a 22-page report Akin  
> wrote recording his observations of the test, he says that "statements  
> made by the Registrar of Voters indicated to me that she is not  
> qualified to assess the reliability and security of such systems, and  
> that she misunderstands some essentials of computer programming and  
> operation. Her deputies refused to answer some important questions.  
> Some statements made by officials at the Registrar's office, and found  
> on the contractor's Web site, I learned on the test day were misleading  
> or inaccurate. Further research after the test day has turned up  
> several other reasons to doubt the reliability, security and accuracy  
> of the system."
> 
> Riverside County was one of the first places in the nation to employ  
> touch-screen machines -- the county used them in the 2000 election,  
> before anyone had ever heard of the problems with older, punch-card  
> machines. After that race, Mischelle Townsend, the county's registrar  
> of voters, was celebrated in the national media, held up as a visionary  
> who'd seen the promise of voting with computers.
> 
> Since then, however, some of the world's most respected computer  
> scientists have highlighted serious problems with electronic voting  
> machines. In July, scientists at Johns Hopkins and Rice found alarming  
> security holes in voting machines made by Diebold, which provides  
> election systems in 37 states. On its Web site, Sequoia Voting Systems,  
> which makes the machines used in Riverside County, insists that its  
> machines are safer than Diebold's. But technologists say that because  
> Sequoia's systems don't produce a voter-verifiable paper trail -- some  
> physical evidence that the voter's choice has been accurately rendered  
> -- they're no better than most of the other electronic machines on the  
> market.
> 
> Now that such machines are under fire, Akin says that Townsend went out  
> of her way to defend electronic voting during the logic-and-accuracy  
> test, which was held on Sept. 9, in anticipation of California's Oct. 7  
> gubernatorial recall race.
> 
> The story Akin tells of that test indicate serious shortcomings with  
> the machines as well as the process used to verify them. He spoke to  
> Salon on Tuesday from Riverside.
> 
>  >> From what you write about what happened at this test, it seems that  
> the elections officials were trying to reassure everyone about  
> electronic voting machines. Was this whole exercise to prove that these  
> machines work? <<
> 
> Well, before an election there's a legal requirement to run  
> logic-and-accuracy testing, so that's what the purpose of the test was,  
> and we were brought in to observe that. But it did have more of the  
> feel of a sales pitch than of a test.
> 
>  >> You write that the elections officials were specifically addressing  
> some of the concerns that people have with these machines. <<
> 
> Yes, they were, and before and after the meeting Mischelle was going  
> after people who don't support these types of machines, especially Bev  
> Harris and also the computer science community. She was saying computer  
> scientists don't know about how elections are run -- she didn't name  
> any specific computer scientists. But just lumping everybody together  
> and labeling them as ignorant isn't very convincing.
> 
>  >> Can you tell me about the other people who were there? You say that  
> it seemed to you that most of the people who were observing didn't  
> understand or didn't have the same questions you did about the  
> technology. <<
> 
> Yeah, that's true. There was one who said, "This is like 'Star Trek'!"  
> He was talking about how one day voting over the Internet would be  
> possible. He really was not technically proficient. He didn't  
> understand some of the limitations of working with computers. People  
> who work with computers a lot know that they have bugs and know that  
> they crash and there's no real way to get all the software bugs out of  
> a complex system. These people didn't have any experience with that and  
> so they could be easily convinced otherwise.
> 
> There was somebody from the Libertarian party who was there, and  
> somebody from the Republican Party who showed up late and played with  
> his phone a lot. So I don't know how technically proficient he was.
> 
>  >> Describe to me how the test was run. <<
> 
> Well, I was picturing that people would go up and touch the touch  
> screen and verify that what they had pressed was registered as a vote.  
> But the way it's run is, they have a test cartridge that they pop into  
> the back of the machine, and it runs a script -- it runs several  
> hundred different voters, like some type of emulation.
> 
>  >> Sort of a simulation of what would happen during a day of voting. <<
> 
> Yeah. But the touch screens themselves weren't actually pressed. Nobody  
> got to touch those. So we didn't see what was on them, and we didn't  
> see the input that was put into the machine. All that we saw was the  
> output that came out later. And, I mean -- that's like telling somebody  
> that your calculator can add 2 plus 2, then pressing some buttons  
> behind a screen, and then showing them that it says 4.
> 
> The votes were put in and they started to run, and at that point we  
> were told that it would take several hours ... [A couple of hours  
> later] we went back to see if the test scripts had finished running,  
> but they hadn't. So at that point we broke for lunch and we were going  
> to come back to get the results.
> 
> I saw several people standing with Mischelle Townsend, and they were  
> signing this piece of paper. At that point, I didn't know what was on  
> the piece of paper; otherwise I would have said something right then  
> and there. I thought it was a type of roll sheet. [Actually, the sheet  
> certified that the observers had seen the test and believed that  
> everything looked aboveboard. The form, which Akin refused to sign,  
> said: "We the undersigned declare that we observed the process of logic  
> and accuracy testing of voting equipment performed by the Riverside  
> County Registrar of Voters, as required by law and that all tests  
> performed resulted in accurate voting of all units tested, including  
> both touch screen and absentee systems."]
> 
>  >> Mischelle Townsend told Salon that she disagreed with you about  
> people signing off on these tests before they'd seen the results. She  
> said that people saw the results and they had hard copies of the  
> results, and they signed off on it after that. <<
> 
> I saw people signing this paper before that.
> 
>  >> Townsend also said that the document was not any kind of "rigorous  
> legal" form; it's just an official roster to say you were there. <<
> 
> But the document doesn't say it's a roll call. It says that the people  
> watched the entire test and observed the results.
> 
> So at that point I left, went home for lunch, and came back around  
> 2:30. Nobody else was there. Brian Foss [the county's information  
> technology manager] and I went to the machines to take the cartridges  
> out -- but we found that they had already been taken out and the  
> machines had been sealed shut.
> 
>  >> OK, so you went to the machines, and the cartridges that store the  
> ballots weren't there in the machines? <<
> 
> Yeah. And Brian Foss asked the guy from Sequoia who was there what  
> happened to the cartridges, and he said that they'd been pulled out.
> 
> [Eventually], Brian Foss pulled up one -- but I'm not sure if it was  
> part of the test or not. He went over to the software that tallies the  
> votes. He puts the card into the card-reading slot and starts up the  
> software, which I think is called WinEDS. And I immediately notice it's  
> running on Windows XP. This caught me off guard -- I'm like, hold on a  
> second, because throughout the day we were told by Brian Foss and  
> Mischelle Townsend that Sequoia Voting Systems did not use Microsoft  
> software.
> 
> But it turns out that if you go to Sequoia's Web site they're very  
> careful about how they phrase it: They say that the Sequoia voting  
> kiosks don't use Microsoft Windows but the tallying machine does. But  
> the way Mischelle Townsend and Brian Foss said it, they said the whole  
> thing didn't use Microsoft Windows. So I asked Brian Foss how come he  
> didn't mention that the software uses Microsoft Windows, when earlier  
> in the day they talked about how systems based on Microsoft Windows are  
> not secure.
> 
> And he didn't answer me. I asked him several times, and each time I  
> asked him I tried to make eye contact with him and he wouldn't make eye  
> contact with me.
> 
> Now, since the WinEDS program runs on top of Microsoft, there's room  
> for -- well, basically it calls some Microsoft APIs [application  
> programming interfaces], and if those are modified in any way a  
> modification of the system could happen that wouldn't be detectable in  
> the type of code review and security tests that Sequoia's software is  
> subject to.
> 
>  >> So you mean people could just make changes to the computer on which  
> the voting software is running -- and the Sequoia system could be  
> altered in some way? <<
> 
> Yes ... Well, so he printed out the results from one card that he'd put  
> in there, and he also printed out results from four other cartridges  
> that he said had been part of the test, but I don't even know where  
> they came from.
> 
>  >> He gave you this printout of what came out of the machines, but did  
> he show you what was on the script that was run on the machines? So you  
> could sort of compare what votes were put in to what came out? <<
> 
> No, at the beginning we were given sheets of paper and we were told,  
> this is what the script is going to test for. But we have no idea what  
> was on the card that they stuck in.
> 
> Also, this is very important -- I noticed that the vote-tallying  
> software has three different modes. A pre-election mode, an election  
> mode, and what they call the post-election "verification" mode. These  
> tests were only run in the pre-election mode. And to a programmer who's  
> had software go through QA testing that seems really weird, because the  
> software wasn't tested in production mode.
> 
> There are times when you need a testing mode for software. For example,  
> if you're doing credit-card-processing software you need a testing  
> mode, because you don't want to constantly use your actual card to test  
> the system. But testing always goes through a production test, too [in  
> which real cards are used].
> 
>  >> Is there any indication to you what the difference is between the  
> different modes? <<
> 
> The only thing that I've heard about this is that they needed the  
> pre-election mode to make sure that the test didn't leave any votes in  
> the system that would be counted during the election. And this seems  
> really weird to me ... I can see why a company might have a testing  
> mode, but I do not understand why a company wouldn't test something in  
> production mode. And you can ask any QA person about this and they're  
> going to tell you the same thing, that it's not a thorough test if it's  
> not tested in production mode.
> 
>  >> I also wanted to talk to you about this paper-trail issue. The  
> elections officials told you that the machines they used produced a  
> paper trail, but they didn't mean the same thing that the critics mean  
> when they ask for a paper trail.  <<
> 
> Yeah. The way Mischelle said, it was that the computer science  
> community asked for paper trails but they don't understand that there  
> already are paper trails. But what people in the computer science  
> community asked for was a voter-verified paper trail. What's demanded  
> is a piece of physical evidence that the voter verifies after the vote.  
> [The Sequoia machines used in Riverside print out a record of all the  
> votes cast on each machine during an election. This "paper trail"  
> doesn't address concerns that the machines might incorrectly record the  
> votes in the first place.]
> 
> Mischelle also stated that there's no real reason for a voter-verified  
> paper trail. She said that the paper trail wasn't possible because  
> printers would jam up during an election. The woman from the  
> Libertarian party said that she gets receipts all the time when she  
> goes to stores so she didn't understand why they couldn't make a  
> printer that would work reliably. Then Mischelle said there's no reason  
> to make a paper trail because it would do exactly what the machines are  
> doing. It would be wasted effort. Which is a kind of a silly argument.  
> There's a bunch of different reasons why you need a paper trail.
> 
>  >> Did you vote in Riverside County on Oct. 7? <<
> 
> Yeah, I voted absentee. Because even though no system is perfect at  
> least absentee has a paper trail.
> 
>  >> Do you have any thoughts on how the election went that day? <<
> 
> Well, apparently there was a pretty large margin, so I don't think  
> people are going to pay attention to it. [In Riverside County 70  
> percent of the voters were in favor of recalling Gray Davis, and 61  
> percent voted for Arnold Schwarzenegger.]
> 
>  >> But if something bad happened, people might not know about it? <<
> 
> Yeah, exactly: Without a paper trail there's no way to know that. They  
> say there's no known instance of fraud, but they make it so you can't  
> tell if there's an instance of fraud, so that claim doesn't say very  
> much.
> 
>  >> Mischelle Townsend told Salon that you were "a young man who had a  
> chip on his shoulder when he came in here." She said that you came into  
> the test with a "closed-minded" attitude and that you didn't want to  
> "listen to the facts." <<
> 
> This is exactly what I expect from her. Instead of responding to my  
> arguments she'll just try to paint me in a bad light. I'm sure she  
> didn't say anything about the fact that the voting kiosks rely on a  
> Windows operating system in order for the results to be read, even  
> though Sequoia and Mischelle Townsend said that one of the benefits of  
> their system is that the results don't rely on Windows.
> 
> I fully expect Mischelle to not really address anything I say in my  
> report. I expect her to say I have a chip on my shoulder or I don't  
> listen to facts. I've never seen her deal with anyone who's technically  
> literate in any other way.
>