Improving the LAT (Re: Bad ballot markings)

[Neal McBurnett <neal@xxxxxxxxxxxxxxxxx>]

On Fri, Oct 29, 2004 at 04:59:21AM -0600, Some Guy wrote:
> Here's today's Times-Call story on the scanner errors. The
> imperceptible dot must have been Neal.
> -paul
[the rest of the story is below]

Yup - those were my dots.

The delay in finishing the Logic and Accuracy Test (LAT) was more
related to how it was conducted than to the efforts of the testers to
do a good test.  Given the Florida debacle, it is important to figure
out how the system handles marginal votes ahead-of-time.  The machines
did very well on properly marked ballots.  But not everything that
should have been tested was tested.  Here is a summary of my
perceptions, as one who was significantly involved.

The setting

I felt the staff were generally friendly and helpful and sincere,
though sometimes frustrated or anxious about the high stakes,
sometimes embattled by some of Al's passion and tactics, sometimes
tired due to long hours, and sometimes improperly restricting the
rights of the participants.  I think Al is a sincere, savvy and
generally kind professional who just can't tolerate a sloppy job or
lack of openness.  The personality battles have made all of our roles
harder.

I was a close observer of the logic and accuracy test (perhaps more
than anyone but Don Hayden, a representative from the Democratic party
and a good guy!).  Unfortunately, there was a conflict between a court
case in which Al was an expert witness and the revised schedule for
the test so he missed much of it.

The Libertarian and Republican parties were kind enough to let me vote
about one ballot each.

Given the enormous problems in 2000 in Florida when people debated how
to define a vote (remember "hanging chads" and "pregnant chads"), I
thought it was critial to get a good sense of how our new system would
interpret various kinds of marks on the ballot.  So I made what appear
to have been the most comprehensive series of tests on that score,
trying big and little lines, dots, and marks adjacent to and outside
the boxes.

I and others (thank you especially to Paul Walmsley) have been trying
to get ballot-by-ballot tallies for months.  But it was impossible to
get a simple clear answer to the basic, simple question "how did the
computer interpret this vote on this one specific ballot", even as the
hours dragged on and other approaches were explored during the test, .
The staff preferred to have people re-count by hand a stack of 22
ballots, with perhaps 30 races on each ballot, two times rather than
generate a precinct report to help narrow in on of the ballots in
question.

Then, rather than testing how the system will really be used on
election day (in which each ballot will go thru only one of the eight
scanning systems), they ran the whole lot of 222 ballots thru each of
the eight systems.  And for each of the eight systems they had two
resolution judges evaluate each of about 45 ballots which each had one
or more issues (typically an overvote, where both sides of a race were
marked).

That is what took most of the time: 8-fold scanning and resolving and
multiple hand recounts..  That also generated a large volume of
potential conflicts among the different tally results.


I think a better test would have involved only two scans and two
resolutions per ballot.  Split the 222 ballots up into 4 piles, and
run one pile thru each of machines A B C and D, and combine the
results into a precinct tally.  Then run the ballots thru E F G and H,
generate another tally, and compare those tallies.  Producing the
tally in a format which can be easily compared (by computer, not by
hand....) would help enormously.  Finally, chase down the errors by
looking at the handful of ballots from the precincts that have
differences.

Unfortunately, as we've complained for months, The Hart InterCivic
system does not have very helpful reporting in this regard.  We were
working with huge PDF reports on paper or on computer.  A simple text
format output, or HTML output, would have been more suitable for
processing and comparison.  At one point, the vice president of Hart
was there to help, and it was noted that there used to be a way to get
results in spreadsheet form, but that capability was not present in
our system, despite repeated requests by Hart's own field staff.  I
think Boulder County should have made such capabilities a requirement.

I was also disturbed by the fact that the clerks office demanded that
I stop taking pictures of the ballots in question.  They said that it
would be unfair, since they didn't let other parties take pictures.  I
asked the democrats if they wanted pictures, and they declined.  I
pointed out that the Longmont Times-Call took pictures, and they said
that those weren't at quite such close range.  I can't imagine a
reason why during an LAT, I shouldn't be able to take a picture of a
ballot that I marked and put into the test....


I'm glad that we now have much better information on what sorts of marks
are counted by the equipment, and I accept the fact that some
marginal marks are interpreted differently by different scanners (as
they would be by different resolution judges).  Perfection is
impossible when dealing with the vagarities of people and hardware.

Voters should know that the machine will not count circled boxes as
votes.  And that resolution judges, when evaluating the "intent of the
voter", _will_ generally count circled boxes as votes.  As long as
machines aren't Artifially Intelligent, that is the sort of thing we
should expect from time to time.  So as the clerk's staff keeps saying
- you should mark the boxes according to the instructions....

But we do need to know that the system doesn't do crazy things, or
crash, when given unusual ballots, and the staff needs training in how
to handle them.  So I'm glad that we finally have some minimal
experience now with how such ballots are handled.


Some things haven't been tested properly, but overall I trust the
system in Boulder, with its paper ballots, more than the vast majority
of systems in the US because of active citizen involvement here.  But
that is mainly to say that I have heard that clerks and citizens in
other jurisdictions have let even more problems go unnoticed than we
have....

DREs are a problem for many Americans.  And in the US I think absentee
ballots are more dangerous even than DREs.

Neal McBurnett                 http://bcn.boulder.co.us/~neal/
Signed and/or sealed mail encouraged.  GPG/PGP Keyid: 2C9EBA60


On Fri, Oct 29, 2004 at 04:59:21AM -0600, Some Guy wrote:
> Here's today's Times-Call story on the scanner errors. The imperceptible dot
> must have been Neal.
> -paul
> <><><><><>
> Bad ballot markings bog down first election test run
> By Pierrette J. Shields
> The Daily Times-Call
> 
> BOULDER - The machines counted properly marked sample ballots correctly, but
> it took three days to determine the final result of the county's
> ballot-counting test because improperly marked ballots caused problems.
> 
> The Boulder County clerk's office spent hours trying to detect why eight
> optical scanning machines were having trouble getting the same results in
> two races in a test deck of sample ballots.
> 
> "It all had to do with non-uniform vote marking," Boulder County elections
> manager Tom Halicki said about the delay in wrapping up the standard test of
> the elections equipment.
> 
> By law, counties must perform logic and accuracy tests, or LATs, of
> ballot-counting equipment. The first test must be completed within 10 days
> of the election. Two more are done on Election Day before live ballots are
> scanned and tallied.
> 
> The first test, which started last week, uses decks of 25 sample ballots
> marked by representatives of local political parties and jurisdictions. The
> markers keep a tally of the way they marked the ballots, and when the deck
> is run through the ballot scanners, results are compared. If the results do
> not match, election officials try to find where the discrepancy came in and
> whether it was a problem with the human tally or the computer tally.
> 
> Overvotes, undervotes and blank ballots are kicked out of the machine for a
> bipartisan resolution team to examine to determine the intent of the voter.
> When a ballot is resolved, the vote is programmed into the tally.
> 
> During last week's session to mark the test decks, party and jurisdictional
> representatives were told how to properly mark ballots, but markers for the
> Republican and Libertarian parties deviated from the instructions
> considerably.
> 
> Ultimately, those test ballots caused difficulties for the machines.
> 
> On a properly marked ballot, a blue or black mark is placed inside of a box
> next to the selection. According to the Colorado secretary of state's
> election rules, a ballot that does not have a mark inside of the box as
> instructed is considered blank and shouldn't be counted. For instance, if
> the box is circled, the scanner won't read it and the vote shouldn't be
> counted.
> 
> The scanners are also extremely sensitive. If a voter puts a checkmark in a
> box, the vote will be tallied by the machine, according to the tests. But if
> the tail of the check mark leaves one box and extends into another, the
> scanner will see two votes in one race and kick out the ballot as an
> overvote for judges to resolve.
> 
> Halicki said a tiny dot on the line of a box on one of the sample ballots
> was being read by some of the eight scanners in the test and not others. It
> took hours to discover the dot with the naked eye.
> 
> "It would vary between machines," Halicki said.
> 
> Repeating these tests on Election Day and tracking down similar problems
> could cause considerable delays in running the day's ballots through the
> machine.
> 
> Linda Herod - project manager for Hart InterCivic, the company that makes
> the scanners used by Boulder County - said the machines performed exactly as
> they should: They tallied properly marked ballots the same way. However,
> they were tripped up by improperly marked ballots.
> 
> "If you want your vote to count properly, mark it properly," she said.
> 
> Al Kolwicz, representative for the Republican Party during the test and
> critic of ballot-counting processes, has sent a letter of complaint to the
> county regarding the tests. He does not believe the machines are accurate
> enough to interpret voter intent every time. Other parties and jurisdictions
> involved in the test have signed off on the results, Halicki said.
> 
> Pierrette J. Shields can be reached at 303776-2244, Ext. 273, or by e-mail
> at pshields@xxxxxxxxxxxxxxx