Correction to my earlier comments on Rule 45 (fwd)

[Paul Walmsley <paul@xxxxxxxxxxx>]

Hello,

My earlier comments on Rule 45 contained a transcription error in Section 
2.  Fortunately, the error did not affect the substance of the comment, as 
it was based on the Rule 45 language.  Attached is a note about what 
changed, plus the corrected comments.

Al Kolwicz passed the original comments on to Harvie Branscomb, whose 
sharp eyes found the error - thanks, Al and Harvie.

Also, I have posted my original comments, along with the corrected 
version, on the web at:

    http://www.booyaka.com/~paul/ea/cosos-20051029/


Regards,

- Paul

---------- Forwarded message ----------
Date: Sun, 30 Oct 2005 18:46:06 -0700 (MST)
From: Paul Walmsley <paul@xxxxxxxxxxx>
To: john.gardner@xxxxxxxxxxxxxxx
Subject: Correction to my earlier comments on Rule 45


Mr. Gardner:

My earlier comments on Election Rule 45 contained a transcription error in some 
quoted text from Rule 45.  In section 2 of my comments, I quote Rule 45 as 
requiring scanning performance of "100 ballots per minute" -- this should read 
"100 ballots per hour," as Rule 45 reads.  Since the surrounding text was 
written based on the 100 ballot per hour rate, no further modification of the 
section's statements are necessary.

I request that the corrected comments, attached below, be entered into the 
public record for the October 31st hearing in place of my original comments, 
or, that this correction be attached to my original comments.


Regards,


- Paul


Comments on Colorado Secretary of State Election Rule 45 (Corrected)
29 October 2005
Paul Walmsley <paul@xxxxxxxxx>


A brief reading of Colorado Secretary of State Election Rule 45
regarding Colorado's Voting System Certification Standards reveals
several significant problems that should be resolved.  The comments
below describe some of these problems and proposes solutions in the
form of an improved standards drafting process.

Several different versions of Rule 45 exist on the Secretary of
State's web site.  The following comments refer to what is apparently
the most recent version of Rule 45, labeled "RULE 45 - 10-03-05" [1].
Furthermore, the comments below should not be considered a
comprehensive review of Rule 45.


1  The State's voting system certification standards must be
precisely written, and many sections of Rule 45 are not.  This
imprecision make it impossible to convert those into verifiable,
testable metrics.

1.1 One example of this imprecision is section 45.5.2.1.1, which
states that "the voting system shall exhibit an evolution towards new
technologies," an abstract requirement which is never concretely
elaborated.  Another is section 45.5.2.3.8, which states that "the
environment in which all databases in the subsystem are maintained
shall include all necessary provisions for security and access
control," without describing the actual security and access control
provisions required.  The practical requirements of these and other
sections are unclear to the point of being meaningless.  While
statements of general guiding principles are important, they must be
concretely defined with specific, testable criteria in standards
documents.

1.2 Rule 45's performance standards also are very imprecisely defined.
They make references to operational terms which have no
publicly-accepted meaning, and which are not defined in the Rule's
glossary, and extend the meanings of other terms, apparently to
include other processes.  For example, section 45.5.2.2.1 refers to
"counting ballots", but does not define whether this process includes
the process of casting ballots, as implied by the inclusion of the DRE
requirement in subsection (b), or whether it includes the process of
ballot scanning, as implied by the optical scan subsections (a) and
(c), or whether it includes the process of vote tabulation, which on
many systems is separate from the vote casting and ballot scanning
processes.  Section 45.5.2.2.2 refers to terms like "election media
download" and "ballot style assignment" which are not defined - it's
not clear whether these terms refer to pre-election preparations, or
to the actual ballot casting process.  It is also unclear whether
these performance requirements refer to individual DRE or vote
scanning machines, as implied by section 45.5.2.2.1; or whether these
metrics refer to the voting system as a whole, including all of the
DRE or vote scanning machines.  Testing conformance to the performance
requirements in Rule 45 is practically impossible unless the processes
under test are rigorously defined.

1.3 The above examples represent only a portion of unclear
requirements in Rule 45.  These types of imprecision in a
certification standards document is undesirable. It creates loopholes
by which vendors may certify equipment to the letter of the Rule that
does not meet the intention of the Rule.  It also fosters confusion as
to the Rule's true requirements.

2 The State's voting system certification standards should define
performance requirements that specify the maximum amount of time
required to perform real-world tasks required in an election, rather
than requiring metrics which are not useful for Colorado elections, as
are defined in Rule 45. For example, section 45.5.2.2.1 requires a
minimum ballot counting rate requirement of 100 ballots per hour for
central count optical scan ballots.  Such a minimum requirement may be
useful for Hinsdale County, Colorado, with only a few hundred voters;
but is not useful for Denver County, Colorado, with hundreds of
thousands of potential voters.  Vendors should instead be required to
demonstrate that their system is capable of completing the specified
election action in a reasonable election time frame.  For example, a
more meaningful version of section 45.5.2.2.1 would require voting
systems to complete the vote count in a specified maximum number of
hours.

3 The parts of the State's voting system certification standards that
pertain to computer security and cryptography should be written and
reviewed by experts in those fields. This is not the case for at least
the information security sections of Rule 45, which were clearly not
written or reviewed by experts in the field.  For example, section
45.5.2.7.2 refers to "a minimum encryption requirement of 40-bit
encryption."  Presumably Rule 45's authors are trying to ensure the
privacy and integrity of election records that are transmitted over
public networks.  But section 45.5.2.7.2 does not do this - in fact,
it is technically meaningless.  For it to be meaningful, the vendor
must provide full cryptographic protocol and implementation details,
including details on key generation, entropy collection, message
authentication, and sender and recipient authentication.  These
details must be reviewed by experts in cryptography and information
security.

4 The State's voting system certification standards should require
mandatory compliance with the EAC's Voluntary Voting System Standards
[2] - Rule 45 does not require this. These standards are intended to
represent the state of the art in voting system standards, and unlike
Rule 45, have received nationwide public review.  If Colorado's
standards are to live up to the Secretary of State's description of
them as "one of the most challenging and thorough programs in the
country," they should at least require compliance with the latest
standards guidance developed by the Federal Government.

5 The State's voting system certification standards should explicitly
forbid vendors from submitting "confidential" or "trade secret"
information for certification as documentation or application
responses to ensure maximum openness and transparency, and Rule 45
does not do this.  A vendor could conceivably claim "trade secret"
status for significant portions of the documentation required to
comply to the rule.  This would prevent the public from engaging in
any meaningful review of the voting systems, and would conflict with
the Secretary of State's commitment to an "open and transparent"
voting system certification process [3].  The Colorado voting system
standards should forbid the vendor from claiming "trade secret" status
for any documentation provided to the State examiners, given the
special public trust requirements for voting systems.

6 The State's voting system certification standards should include a
glossary which includes all of the terms in the standards which do not
have clear public usages, and should build on glossaries already
created in the federal voting system standards.  But as noted
previously, Rule 45's glossary does not contain entries for terms like
"election media download," which do not have commonly-accepted
unambiguous meanings in elections.  Additionally, terms like "Ballot
Image" are ambiguously defined.  For example, it is unclear whether a
"ballot image" refers to cast vote records, or to graphics files
depicting the scanned paper ballots from optical scan systems.

7 The State's standards should require vendor systems to support live
auditing procedures [4], which Rule 45 does not require.  Live
auditing is the process of continuously evaluating the accuracy and
functionality of election systems during an election using live
ballots.  Without mandating vendor support for live audit techniques,
the accuracy of the election system can only be tested before or after
an election, and such testing is inadequate to assure election accuracy.

Many of the principles that motivate Colorado's voting system
standards seem well-intentioned.  However, the specific manifestation
of these standards in the present Rule 45 does not measure up to these
principles.  It is my belief from a cursory examination of Rule 45
that the problems cited above are not simply endemic to the sections
quoted above, but exist throughout the document. This suggests that
the public would best be served by committing to a different
standards-making process than was used for the Rule under discussion.
A better standards-making process would include the following principles:

- It should be continuously edited by staff experienced with the
   precise language necessary to define standards.

- It should involve independent subject matter experts outside the
   Secretary of State's area of expertise.

- All drafts should be released to the public for ongoing comment
   throughout the drafting process.

- The document should strive to avoid duplicating existing Federal
   standards, and should draw on existing work, both at the Federal
   level and standards work by other states.  In particular, the State
   should mandate compliance with the existing EAC Voluntary Voting
   System Guidelines.

- It should first be articulated in a general set of requirements for
   voting system performance that are then translated into specific,
   testable, concrete requirements.  For example, a general requirement
   that "election voting system records and data must be secure from
   tampering and unauthorized interception" can be translated into
   specific technical requirements that serve that general principle.
   Similar general principles and concrete requirements should be
   articulated for accessibility, reliability, live auditability,
   speed, and other desirable aspects for Colorado's voting systems.

It is also troubling that Rule 45 was adopted in emergency, before any
opportunity for comment or review by independent subject matter
experts or the public. Rule 45 seems rushed, and it is a matter of
some concern that voting systems may be certified under the emergency
rule that would not be certifiable under a more deliberate Rule,
either revised from the current rule or rewritten.  Therefore, I
encourage the Colorado Secretary of State to:

- Revoke the emergency adoption of Rule 45,

- Commit to an open public comment and revision process for a revised
   voting system standard before it is adopted, and for all future
   revisions, and

- When revised standards are complete, require immediate
   re-certification of all equipment certified under the emergency
   adoption of Rule 45.


Sincerely,



Paul Walmsley
Boulder, Colo.


Footnotes

1. 
http://www.elections.colorado.gov/DesktopModules/Downloads/download.aspx?tid=501&_iid=193

2. http://guidelines.kennesaw.edu/vvsg/guide_toc.asp

3. http://www.elections.colorado.gov/DDefault.aspx?tid=501

4. http://www.booyaka.com/~paul/ea/eac-20050930/interpretation-live-audit.txt
    http://www.booyaka.com/~paul/ea/eac-20050930/tabulation-live-audit.txt
    http://www.booyaka.com/~paul/ea/eac-20050930/live-audit-overview.txt