Votetest: Voting system logic testing for the VVSG-NI

This document and associated files have been prepared by the National Institute of Standards and Technology (NIST) and represent draft test materials for the Election Assistance Commission’s next iteration of the VVSG. It is a preliminary draft and does not represent a consensus view or recommendation from NIST, nor does it represent any policy positions of NIST.

Software disclaimer

This software was developed at the National Institute of Standards and Technology by employees of the Federal Government in the course of their official duties. Pursuant to Title 17 Section 105 of the United States Code this software is not subject to copyright protection and is in the public domain. This software is an experimental system. NIST assumes no responsibility whatsoever for its use by other parties, and makes no guarantees, expressed or implied, about its quality, reliability, or any other characteristic. We would appreciate acknowledgement if the software is used.

This software can be redistributed and/or modified freely provided that any derivative works bear some notice that they are derived from it, and any modified versions bear some notice that they have been modified.

Commercial products disclaimer

Specific computer hardware and software products are identified in this paper to support reproducibility of results. Such identification does not imply recommendation or endorsement by the National Institute of Standards and Technology, nor does it imply that the products identified are necessarily the best available for the purpose.

List of Figures

List of Tables

1 Introduction

1.1 What Votetest Is

The Votetest distribution, or simply “Votetest” for short, is a package of public domain data, software, and documentation that the National Institute of Standards and Technology (NIST) is developing. Its purpose is to provide conformance test materials for use by Voting System Testing Laboratories. These test materials are used to assess conformity to the Voluntary Voting System Guidelines (VVSG) [1].

Votetest defines abstract tests that exercise every phase of the voting process from election definition through report generation. The scope of functionality that they cover to some extent is therefore quite broad. The abstract tests are “realized” according to the specifics of the voting system being tested.

In addition to the abstract tests, Votetest includes the expected results for each test, a means to derive the expected results for newly developed tests, a test generator, and detailed documentation.

1.2 What Votetest Is Not

Votetest CANNOT simply be plugged into a voting system to obtain a verdict on conformity. The tests that are defined abstractly by Votetest must be translated into concrete tests that can actually be run on a given voting system. This task requires the same expertise and diligence that test labs have employed in the past when developing their own tests from scratch.

Votetest, by itself, DOES NOT provide complete coverage of the VVSG. Votetest is only one tool that is used in one part of the conformity assessment process. It is designed not to be used in isolation, but rather to complement the other testing activities, which include the physical configuration audit, documentation and design reviews, electromagnetic compatibility and environmental testing, security reviews, usability and accessibility assessments, the volume test (mock election), and the evaluation of reliability, accuracy, and misfeed rates. Test labs should consult Part 3 of the VVSG [1] regarding the full scope of testing to be performed.

Votetest DOES NOT address “fitness for use.” Its scope is strictly limited to conformity assessment. It is not a substitute for Voting System Testing Laboratories’ diligent assessments of the full scope of voting system functionality (including vendor-specific functionality), nor does it obviate the need for acceptance testing by jurisdictions.

1.3 Structure of this document

Section 2 provides additional background of general interest. Section 3 provides detailed technical documentation on the portions of Votetest that are essential to performing conformity assessment. Section 4 and Section 5 provide detailed technical documentation on the portions of Votetest that could be used to develop additional test cases. Finally, Section 6 walks through the process of defining a new test case based on an example election.

2 Background

By authorization of the 2002 Help America Vote Act (HAVA), NIST is assisting the Election Assistance Commission (EAC) with the implementation of Voluntary Voting System Guidelines (VVSG) for states and local governments conducting Federal elections. The EAC’s Technical Guidelines Development Committee (TGDC) in collaboration with NIST researchers has developed a draft of the next iteration of the VVSG. The draft document is a set of detailed technical requirements addressing core requirements, human factors, privacy, security, and transparency of the next generation of voting systems. The EAC plans to issue the next VVSG after receiving and reviewing public comments.

NIST is developing a set of uniform public test suites to be used as part of the EAC’s Testing and Certification Program. Test labs will be able to use these freely available test suites to help determine that VVSG requirements are met by voting systems. The test suites address human factors, security and core functionality requirements for voting systems as specified in the VVSG. Use of the public test suites will produce consistent results and promote transparency of the testing process. The test suites can also assist manufacturers in the development of conforming products by providing precise test specifications. Also, they will help reduce the cost of testing since each test lab would no longer need to develop its own test suites. Finally, a uniform set of public test suites can increase election officials’ and voters’ confidence that voting systems conform to VVSG requirements.

3 Test materials

3.1 Overview of test materials

This is the documentation for the test materials portion of the Votetest distribution. Its intended audience is test labs accredited by the Election Assistance Commission to perform voting system certification testing. The test lab is assumed to have competence with the following:

While Votetest includes both basic and advanced materials, only the data model (Section 3.5), basic schema (Section 3.6), basic test suite (Section 3.7) and expected results (Section 3.3.4.3) are essential to the conformance testing process. Section 3.3 explains in more detail how Votetest is used in that process.

3.2 File listing

The Votetest distribution is provided as a zip file named votetest-YYYYMMDD.zip and alternately as a compressed tar file named votetest-YYYYMMDD.tar.bz2, where YYYYMMDD is a sequence of digits indicating the date of the release. The files contained in the distribution are described in Table 1.

3.3 How to use Votetest

The steps listed in the following subsections fit within the context of the overall conformity assessment process that is described in [3, Part 3 Chapter 2]. Thus, in practice, the steps will not necessarily follow directly one to the next as is implied here, because there may be other conformity assessment activities occurring in between or parallel to them.

3.3.1 Determine relevant voting variation classes—system level

The Technical Data Package received from the manufacturer is required to include an implementation statement, which is specified by the Conformance Clause of the VVSG [3, Part 1 Section 2.4]. This implementation statement includes the manufacturer’s classification of the voting system as a whole and of the different devices of which it is comprised. For Votetest, the relevant voting variation classes are determined by examining the implementation statement. (The completeness and correctness of the manufacturer’s classification should have been established in a preceding documentation and design review; see [3, Part 3 Req. 4.2-C].) The system and device classes that the manufacturer may reference are defined in the Conformance Clause [3, Part 1 Section 2.5].

Usually, multiple classes will apply to a given voting system or voting device. For example, if a DRE supports both N-of-M and ranked order voting variations, it will be classified as both an N-of-M voting device and a Ranked order voting device in addition to being classified as a DRE. This multiple classification allows multiple sets of tests to apply so that all of the supported functionality will be tested.

At this stage, what is relevant are the system-level classes. If the system as a whole does not support a particular voting variation, then support for that variation in isolated devices is irrelevant.

3.3.2 Determine applicable tests

Table 2 (coming up in Section 3.7) lists the tests in the basic Votetest test suite and indicates the voting variations that are used by each test. The set of tests applicable to a given voting system is the set of all tests that do not use any voting variations that the voting system does not support. In other words, every test is applicable unless it is specifically excluded, and a test is excluded only if it uses a voting variation that, according to the reviewed and accepted implementation statement, the system does not implement.

For example, if a voting system did not support ranked order voting, the following tests would be excluded:

There is a special case regarding absentee voting that will be addressed in Section 3.7.

3.3.3 Multiply tests for different interfaces and devices

At this point the test lab has the beginnings of a test plan with a list of the tests that are applicable to the voting system as a whole. That plan must now be expanded down to the device level.

When a voting system offers multiple paths by which voting can occur, such as a touchscreen interface and an audio ballot on one device plus optical scanning of paper ballots by another device, each of these paths must be tested. It is entirely possible for a failure to be specific to a particular user interface. For example, some controls that can be activated using a touchscreen might be incorrectly skipped over if an input device with a navigational model (next option, previous option) is used.

Several different strategies could be used to divide or replicate the testing among the available interfaces and devices. However, since most tests in the basic test suite are small by design, with relatively few ballots, the recommended approach is to repeat each test on each device and interface separately. For large tests such as the volume test (mock election), one would instead divide the ballots among the available devices and interfaces using a distribution that imitates a realistic election.

Absentee voting is again a special case. In systems where absentee ballots follow a separate path from other ballots (e.g., a dedicated central count optical scanner), it makes sense to repeat tests over the absentee interface. However, the basic test suite also contains tests specifically designated for absentee voting, that mix absentee ballots with non-absentee ballots. These designated tests would be executed using both the absentee and non-absentee paths at the same time, not repeated on each path separately.

As noted in [3, Part 1 Section 6.2], it is not necessarily the case that every device in the system would support every voting variation claimed at the system level. Thus there may be cases where it is not possible to multiply certain tests onto certain devices. However, other devices that do pass those tests must be present to enable satisfaction of the system-level claim.

3.3.4 Perform testing

Once the test plan has been approved, work on implementing the plan can begin. The steps that the test lab must do for each applicable Votetest test are summarized in Figure 1 and explained in more detail in the following subsections.

Translate test from Votetest model to voting system Since interfaces to voting systems vary widely, the test lab must use the basic test suite as input from which to generate system-specific tests. For each applicable test, the test lab translates the abstract, SQL version included in the basic test suite (Section 3.7) into a concrete test for the voting system under test, thereby “realizing” the test. The data model (Section 3.5) and basic schema (Section 3.6) are essential to understanding the abstract tests so that they can be realized correctly.

Figure 2 shows an example of a Votetest test expressed in SQL. It specifies both the election definition and the ballots and votes to be cast.

Execute test For each test case, the test lab executes the “realized” test case within the voting system under test following the general test template specified in [3, Part 3 Section 5.2.2.1]:

This template essentially puts the voting system through the technical paces of a very small election for each test. Although the election definition is simple and the number of ballots small, the process from beginning to end exercises all of the voting system functions that would be used in an election. The surrounding procedures and environment are, of course, quite different, and vastly simplified in the laboratory testing environment, so it is not a mock election in the sense that the volume test is.

Also see [3, Part 3 Section 2.5] regarding the test conditions, test fixtures, test data requirements, and test practices that apply.

Compare observed and expected results As part of the test execution, the voting system under test is instructed to generate vote data reports. These contain the “observed results” with respect to ballot counts and vote totals. Figure 6 shows an example report from the hypothetical voting system.

The “expected results” for all of the test cases are saved as text files in two subdirectories of the Votetest distribution. The subdirectory sample_output contains the results expected from the test cases as written, while the subdirectory sample_output_kill-overvotes contains the results expected from the test cases with overvotes changed to undervotes.

An example is shown in Figure 7. Comparing the “observed results” in Figure 6 and the “expected results” in Figure 7, it is clear that the quantities reported for candidate vote totals are identical, even though the form of the reports is somewhat different. The test lab would find no discrepancy in those results. However, the observed results shown in Figure 6 are not in and of themselves sufficient to satisfy all of the VVSG requirements on reporting. For example, [3, Part 1 Req. 7.8.3.2-A through D] specify additional information that the voting system under test must provide in some report if the system is to be found conforming.

The other reports generated as part of the test (pre-election reports, etc.) would also be inspected to look for anomalies and to assess conformity to the L&A testing and auditability requirements of the VVSG. Similarly, any unexpected behavior of the voting system observed by the tester would be assessed by the resident expert (possibly but not necessarily the same person) to determine if a nonconformity was demonstrated.

3.4 Applicability to VVSG 2005

The 2005 Voluntary Voting System Guidelines (VVSG 2005) [4] have several important differences from the next VVSG [1] that impact the application of Votetest.

There is no technical incompatibility that would outright prevent the use of any of the Votetest test cases in a conformity assessment for VVSG 2005, provided that the draft next VVSG accurately reflects the intent of the requirements of VVSG 2005. A focused effort to develop test cases for VVSG 2005 instead of the next VVSG would not necessarily arrive at a different set of test cases.

3.5 Data model

The data model for voting system core requirements is described in Figure 8 by a Unified Modeling Language (UML) class diagram [5]. Following sections explain the diagram.

3.5.1 Assumptions

All entities in this data model are implicitly scoped by an election. It is assumed that different elections are stored in different databases, and any reuse of definitions from one election to another is accomplished by copying over the relevant data.

This data model is constructed from an integrated, top-level viewpoint. In practice, different portions of the system will deal with only a portion of the data at any given time. It is expected that test labs will project and extract data from the integrated schema as needed to support these limited viewpoints for testing.

The results of tabulation and reporting are derived from the content of the data model, but those results are themselves outside the scope of the model.

3.5.2 POD (Plain Old Data) types

3.5.3 Classes

BallotStyle Set of Contests and ReportingContexts that is inherited by all Ballots of that style. Depending on the type of election and local practices, a jurisdiction would define a separate BallotStyle for each precinct, each split within a precinct, and/or for each political party.

[1] defines both “ballot style” and “ballot configuration.” This model assumes that the association between styles and configurations is 1-to-1 and represents the combined concept with BallotStyle.

Choice One of the things you can vote on in a Contest, such as a candidate, a political party, or yes or no. Choice is scoped by Contest, so even if the same person runs as a candidate in two or more Contests, those separate candidacies are represented by separate Choices. Choices do not map 1:1 with ballot positions—a Choice uniquely identifies a candidate, while a given ballot position might just be a generic write-in slot.

Contest Subdivision of a Ballot corresponding to a single question being put before the voters, consisting of header text, a discrete set of Choices, and possibly write-in opportunities. It is possible for a Contest to have zero Choices, e.g., if there are no registered candidates but write-ins are being accepted. Choices corresponding to the candidates written in would be added later.

ElectionDistrict Surrogate for real-world entity that may have associated ReportingContexts. Any relationship between ElectionDistricts and Contests appearing on the ballot in those districts is implemented by BallotStyles.

ReportingContext Particular scope within which the system must be capable of generating reports. E.g., to support reporting at the precinct level, there must be a ReportingContext for each precinct. The association between ReportingContexts and individual tabulators, precincts, election districts, political parties, ballot categories, or other arbitrary scopes of reporting is jurisdiction-defined and jurisdiction-managed, mostly using BallotStyles. The ways in which ReportingContexts overlap or include one another is entirely determined by the assignment of multiple ReportingContexts to BallotStyles and Ballots.

Tabulator Surrogate for real-world entity, e.g., a DRE or optical scanner, that may have associated ReportingContexts.

3.5.4 Named associations

Affiliation Identifies the Party to which a candidate claims allegiance. Does not necessarily have anything to do with Endorsements.

Alias Identifies an alternative Choice that for tabulation purposes is considered equivalent to a particular canonical Choice. Aliases will normally be variant spellings of a candidate’s name that appeared in write-in positions.

Endorsement Identifies a voter response that would be implied by a straight party vote for the endorsing Party. Does not necessarily have anything to do with Affiliation.

In a Cumulative Contest, Value may take on values greater than 1. For example, if the Party recommended that voters cast two votes for the first Choice and one vote for the second, an Endorsement with Value = 2 would exist for the first Choice and an Endorsement with Value = 1 would exist for the second Choice.

In a Ranked order Contest, Value contains the ranking that the Party recommends that voters assign to each Choice.

VoterInput The response that a particular Ballot provides for a particular Choice.

In a 1-of-M or N-of-M Contest, a VoterInput with Value = 1 would exist for the single Choice or for each of the Choices for which the voter voted.

In a Cumulative Contest, Value may take on values greater than 1. For example, if a voter cast two votes for the first Choice and one vote for the second, a VoterInput with Value = 2 would exist for the first Choice and a VoterInput with Value = 1 would exist for the second Choice.

In a Ranked order Contest, Value contains the ranking that the voter assigns to each Choice.

3.5.5 Constraints

3.5.6 Usage for all standard voting variations

Absentee voting Absentee voting is implemented in several different ways in practice, and it can be implemented in several different ways using this model.

While the first option is the least invasive, absentee Ballots are in practice sometimes processed as a separate precinct, which usually means both a separate ReportingContext and a separate BallotStyle.

Review-required ballots Use Categories and Accepted attributes of Ballot as needed.

Write-ins The number of write-ins permitted is an attribute of the Contest. If the write-in is new, a new Choice is created for it (with IsWriteIn = true). Votes are then associated with that Choice. Alias associations are created as applicable during write-in reconciliation.

Straight party voting A single Contest is created with CountingLogic = Straight party selection and Choice Names being equal to the Names of the available Parties. In every other Contest that is straight-party-votable, the straight party behaviors are configured by creating Endorsement associations between the Choices and the Parties.

Cross-party endorsement See straight party voting. Create additional Endorsement associations as needed for multiply endorsed Choices.

Ballot rotation Rotate is a Boolean attribute of Contest. The implementation of variable mapping between Choices and ballot positions is out of scope because ballot positions are abstracted out of the model. However, in paper-based systems, rotation may involve a proliferation of BallotStyles that would have to be added.

Closed primaries Assignment of BallotStyles to voters is procedural and out of scope.

Open primaries Assignment of BallotStyles to voters is procedural and out of scope.

Provisional / challenged ballots Use Categories and Accepted attributes of Ballot as needed.

Cumulative voting Set ContestCountingLogic = Cumulative and set N appropriately.

Ranked order voting Set ContestCountingLogic = Ranked order and set N appropriately. VoterInput Values specify the rankings as provided on each Ballot.

3.6 Basic schema

The classes Tabulator, Precinct and ElectionDistrict are not represented. They were modelled only to clarify how the more general concept ReportingContext relates to the real world and are not needed by the test suite.

3.7 Basic test suite

The basic test suite contains 92 tests that exercise different voting variations in small, simple scenarios to isolate the conditions under which failures occur. The basic test suite is complemented by the volume test specified in Part 3 of the VVSG [1], which exercises all features together in a large, complex scenario where a significant volume of ballots is processed.

The variations identified in the test suite as AbsenteeVoting and AbsenteeByCategories both conform to the voting variation defined in the VVSG as Absentee voting. Tests tagged as AbsenteeByCategories require support for a capability that might not be present in all systems, while AbsenteeVoting tests use a more procedural approach to achieve the same goal. The documented assumption attached to test case 1-basic-AbsenteeByCategory.sql and other tests using the absentee-ballots-by-categories approach means that it is not applicable to Absentee voting systems that are limited to the procedural approach. See Section 3.5.6.2 for further discussion on this issue.

Table 2:

Basic test suite

Test case ID / file name

Applies to

Description

1-basic-1ofM.sql

Voting system

Small 1-of-M contest, no write-ins, no rejected ballots.

1-basic-AbsenteeByCategory.sql

Absentee voting²

Small 1-of-M contest with absentee ballots via categories.

1-basic-AbsenteeByCategory-CrossPartyEndorsement.sql

Absentee voting² ∧ Cross-party endorsement

Small straight party + 1-of-M contest with cross-party endorsement, three absentee ballots via categories.

1-basic-AbsenteeByCategory-Cumulative.sql

Absentee voting² ∧ Cumulative voting

Small Cumulative contest, N=3, no write-ins, no rejected ballots, three absentee ballots via categories.

1-basic-AbsenteeByCategory-NofM.sql

Absentee voting² ∧ N of M voting

Small 2-of-M contest, no write-ins, no rejected ballots, three absentee ballots via categories.

1-basic-AbsenteeByCategory-Provisional.sql

Absentee voting² ∧ Provisional / challenged ballots

Small 1-of-M contest with absentee ballots via categories, accepted and rejected provisionals.

1-basic-AbsenteeByCategory-RankedOrder.sql

Absentee voting² ∧ Ranked order voting

Small ranked order contest with absentee ballots via categories.

1-basic-AbsenteeByCategory-StraightParty.sql

Absentee voting² ∧ Straight party voting

Small straight party + 1-of-M contest, absentee ballots via categories.

1-basic-AbsenteeByCategory-WriteIns.sql

Absentee voting² ∧ Write-ins

Small 1-of-M contest with absentee ballots via categories and write-ins, no aliasing.

1-basic-AbsenteeBySpecialPrecinct.sql

Absentee voting

Small 1-of-M contest with absentee ballots via a special precinct and ballot style.

1-basic-AbsenteeBySpecialPrecinct-CrossPartyEndorsement.sql

Absentee voting ∧ Cross-party endorsement

Small straight party + 1-of-M contest with cross-party endorsement, absentee ballots via a special precinct and ballot style.

1-basic-AbsenteeBySpecialPrecinct-Cumulative.sql

Absentee voting ∧ Cumulative voting

Small Cumulative contest, N=3, with absentee ballots via a special precinct and ballot style, no write-in, no rejected ballots.

1-basic-AbsenteeBySpecialPrecinct-NofM.sql

Absentee voting ∧ N of M voting

Small 3-of-M contest with absentee ballots via a special precinct and ballot style, no write-in, no rejected ballots.

1-basic-AbsenteeBySpecialPrecinct-NofM-Provisional.sql

Absentee voting ∧ N of M voting ∧ Provisional / challenged ballots

Small 3-of-M contest with absentee ballots via a special precinct, accepted and rejected provisionals, no write-ins.

1-basic-AbsenteeBySpecialPrecinct-Provisional.sql

Absentee voting ∧ Provisional / challenged ballots

Small 1-of-M contest with absentee ballots via a special precinct and ballot style, accepted and rejected provisionals.

1-basic-AbsenteeBySpecialPrecinct-RankedOrder.sql

Absentee voting ∧ Ranked order voting

Small ranked order contest with absentee ballots via a special precinct and ballot style.

1-basic-AbsenteeBySpecialPrecinct-SplitPrecinct.sql

Absentee voting ∧ Split precincts

Small 1-of-M contest with absentee ballots via a special precinct and a split precinct.

1-basic-AbsenteeBySpecialPrecinct-StraightParty.sql

Absentee voting ∧ Straight party voting

Small straight party + 1-of-M contest, absentee ballots via a special precinct and ballot style.

1-basic-AbsenteeBySpecialPrecinct-WriteIns.sql

Absentee voting ∧ Write-ins

Small 1-of-M contest with absentee ballots via a special precinct and write-ins.

1-basic-AbsenteeBySpecialPrecinct-Yes-or-No.sql

Absentee voting

Small Yes-or-No contest with absentee ballots via a special precinct and ballot style.

1-basic-BallotRotation.sql

Ballot rotation

Small 1-of-M contest with ballot rotation.

1-basic-BallotRotation-AbsenteeByCategory.sql

Ballot rotation ∧ Absentee voting²

Small 1-of-M contest with ballot rotation and absentee ballots via categories.

1-basic-BallotRotation-AbsenteeBySpecialPrecinct.sql

Ballot rotation ∧ Absentee voting

Small 1-of-M contest with ballot rotation, absentee ballots via a special precinct.

1-basic-BallotRotation-CrossPartyEndorsement.sql

Ballot rotation ∧ Cross-party endorsement

Small straight party + 1-of-M contest with cross-party endorsement and ballot rotation.

1-basic-BallotRotation-Cumulative.sql

Ballot rotation ∧ Cumulative voting

Small cumulative voting contest, N=2, with ballot rotation, no write-ins, no rejected ballots.

1-basic-BallotRotation-NofM.sql

Ballot rotation ∧ N of M voting

Small 2-of-M contest, with ballot rotation.

1-basic-BallotRotation-Provisional.sql

Ballot rotation ∧ Provisional / challenged ballots

Small 1-of-M contest with ballot rotation, accepted and rejected provisionals.

1-basic-BallotRotation-RankedOrder.sql

Ballot rotation ∧ Ranked order voting

Small ranked order contest with ballot rotation.

1-basic-BallotRotation-StraightParty.sql

Ballot rotation ∧ Straight party voting

Small straight party + 1-of-M contest with ballot rotation.

1-basic-BallotRotation-WriteInsAliases.sql

Ballot rotation ∧ Write-ins

Small 1-of-M contest with write-ins and aliases and ballot rotation.

1-basic-CrossPartyEndorsement.sql

Cross-party endorsement

Small straight party + 1-of-M contest with cross-party endorsement.

1-basic-CrossPartyEndorsement-Cumulative.sql

Cross-party endorsement ∧ Cumulative voting

Small straight party + cumulative contest with cross-party endorsement, no write-ins, no rejected ballots.

1-basic-CrossPartyEndorsement-NofM.sql

Cross-party endorsement ∧ N of M voting

Small straight party + 2-of-M contest with cross-party endorsement.

1-basic-CrossPartyEndorsement-Provisional.sql

Cross-party endorsement ∧ Provisional / challenged ballots

Small straight party + 1-of-M contest with cross-party endorsement, accepted and rejected provisionals.

1-basic-CrossPartyEndorsement-WriteIns.sql

Cross-party endorsement ∧ Write-ins

Small straight party + 1-of-M contest with cross-party endorsement and write-ins.

1-basic-Cumulative.sql

Cumulative voting

Small cumulative voting contest, no write-ins, no rejected ballots.

1-basic-Cumulative-NofM.sql

Cumulative voting ∧ N of M voting

Small cumulative voting contest plus small N-of-M contest, no write-ins, no rejected ballots.

1-basic-Cumulative-Provisional.sql

Cumulative voting ∧ Provisional / challenged ballots

Small cumulative voting contest with accepted and rejected provisionals.

1-basic-Cumulative-RankedOrder.sql

Cumulative voting ∧ Ranked order voting

Small ranked order contest plus small cumulative contest, no special cases.

1-basic-NoBallots-1ofM.sql

Voting system

Small 1-of-M contest with no ballots cast.

1-basic-NoBallots-RankedOrder.sql

Ranked order voting

Small ranked order contest with no ballots cast.

1-basic-NoChoices-1ofM.sql

Write-ins

Small 1-of-M contest with no choices (write-ins only).

1-basic-NoChoicesNoBallots-1ofM.sql

Write-ins

Small 1-of-M contest with no choices (write-ins only) and no ballots.

1-basic-NofM.sql

N of M voting

Small 2-of-M contest, no write-ins, no rejected ballots.

1-basic-NofM-Provisional.sql

N of M voting ∧ Provisional / challenged ballots

Small 2-of-M contest with accepted and rejected provisionals.

1-basic-NofM-WriteIns.sql

N of M voting ∧ Write-ins

Small 2-of-M contest with write-ins, no aliasing.

1-basic-Primary.sql

Primary elections

Small primary election, no write-ins, no rejected ballots.

1-basic-Primary-AbsenteeByCategory.sql

Primary elections ∧ Absentee voting²

Small primary election with absentee ballots via categories, no write-ins, no rejected ballots.

1-basic-Primary-AbsenteeBySpecialPrecinct.sql

Primary elections ∧ Absentee voting

Small primary election with absentee ballots via a special precinct and ballot style, no write-ins, no rejected ballots.

1-basic-Primary-BallotRotation.sql

Primary elections ∧ Ballot rotation

Small primary election with ballot rotations, no write-ins, no rejected ballots.

1-basic-Primary-CrossPartyEndorsement.sql

Primary elections ∧ Cross-party endorsement

Small primary election with straight party voting and cross-party endorsement in a non-party-specific contest.

1-basic-Primary-Cumulative.sql

Primary elections ∧ Cumulative voting

Small primary election with cumulative voting, no write-ins, no rejected ballots.

1-basic-Primary-NofM.sql

Primary elections ∧ N of M voting

Small primary election with N-of-M voting, no write-ins, no rejected ballots.

1-basic-Primary-Provisional.sql

Primary elections ∧ Provisional / challenged ballots

Small primary election, no write-ins, with accepted and rejected provisionals.

1-basic-Primary-RankedOrder.sql

Primary elections ∧ Ranked order voting

Small primary election with ranked order voting, no write-ins, no rejected ballots.

1-basic-Primary-SplitPrecinct.sql

Primary elections ∧ Split precincts

Small primary election with a split precinct, no write-ins, no rejected ballots.

1-basic-Primary-StraightParty.sql

Primary elections ∧ Straight party voting

Small primary election with straight party voting in a non-party-specific contest.

1-basic-Primary-WriteIns.sql

Primary elections ∧ Write-ins

Small primary election with write-ins, no rejected ballots.

1-basic-Provisional.sql

Provisional / challenged ballots

Small 1-of-M contest with accepted and rejected provisionals.

1-basic-RankedOrder-1.sql

Ranked order voting

Small ranked order contest, N=1, M=4, no special cases.

1-basic-RankedOrder-2.sql

Ranked order voting

Small ranked order contest, N=2, M=4, no special cases.

1-basic-RankedOrder-NofM.sql

Ranked order voting ∧ N of M voting

Small ranked order contest plus small N-of-M contest, no special cases.

1-basic-RankedOrder-Provisional.sql

Ranked order voting ∧ Provisional / challenged ballots

Small ranked order contest, N=1, M=4, with accepted and rejected provisionals.

1-basic-RankedOrder-WriteIns.sql

Ranked order voting ∧ Write-ins

Small ranked order contest, N=1, M=4, with write-in.

1-basic-SplitPrecinct-1.sql

Split precincts

Small 1-of-M contest with a split precinct.

1-basic-SplitPrecinct-2.sql

Split precincts

Two districts, three precincts (one split), three contests, four ballot styles, forty ballots.

1-basic-SplitPrecinct-AbsenteeByCategory.sql

Split precincts ∧ Absentee voting²

Small 1-of-M contest with a split precinct and absentee ballots via categories.

1-basic-SplitPrecinct-BallotRotation.sql

Split precincts ∧ Ballot rotation

Like SplitPrecinct-2 except with ballot rotation.

1-basic-SplitPrecinct-CrossPartyEndorsement.sql

Split precincts ∧ Cross-party endorsement

Like SplitPrecinct-2 except with straight party voting and cross-party endorsement.

1-basic-SplitPrecinct-Cumulative.sql

Split precincts ∧ Cumulative voting

Small Cumulative contest, N=2, with a split precinct.

1-basic-SplitPrecinct-NofM.sql

Split precincts ∧ N of M voting

Small 3-of-M contest with a split precinct, no write-ins, no rejected ballots.

1-basic-SplitPrecinct-Provisional.sql

Split precincts ∧ Provisional / challenged ballots

Small 1-of-M contest with a split precinct and accepted and rejected provisionals.

1-basic-SplitPrecinct-RankedOrder.sql

Split precincts ∧ Ranked order voting

Like SplitPrecinct-2 except with ranked order contests at the district level.

1-basic-SplitPrecinct-StraightParty.sql

Split precincts ∧ Straight party voting

Like SplitPrecinct-2 except with straight party voting.

1-basic-SplitPrecinct-WriteIns.sql

Split precincts ∧ Write-ins

Small 1-of-M contest with split precinct and write-in.

1-basic-StraightParty.sql

Straight party voting

Small straight party + 1-of-M contest, no write-ins, no rejected ballots.

1-basic-StraightParty-Cumulative.sql

Straight party voting ∧ Cumulative voting

Small straight party + cumulative contest, no write-ins, no rejected ballots.

1-basic-StraightParty-NofM.sql

Straight party voting ∧ N of M voting

Small straight party + 2-of-M contest, no write-ins, no rejected ballots.

1-basic-StraightParty-Provisional.sql

Straight party voting ∧ Provisional / challenged ballots

Small straight party + 1-of-M contest with accepted and rejected provisionals.

1-basic-StraightParty-RankedOrder.sql

Straight party voting ∧ Ranked order voting

Small ranked order contest, N=1, M=4, with straight-party voting.

1-basic-StraightParty-WriteIns.sql

Straight party voting ∧ Write-ins

Small straight party + 1-of-M contest with write-ins.

1-basic-WriteIns.sql

Write-ins

Small 1-of-M contest with write-ins, no aliasing.

1-basic-WriteInsAliases.sql

Write-ins

Small 1-of-M contest with write-ins and aliases.

1-basic-WriteInsAliases-AbsenteeByCategory.sql

Write-ins ∧ Absentee voting²

Small 1-of-M contest with write-ins, aliases and absentee ballots via categories.

1-basic-WriteInsAliases-AbsenteeBySpecialPrecinct.sql

Write-ins ∧ Absentee voting

Small 1-of-M contest with write-ins, aliases, absentee ballots via a special precinct and ballot style.

1-basic-WriteInsAliases-Cumulative.sql

Write-ins ∧ Cumulative voting

Small Cumulative contest, N=2, with write-ins and aliases, no rejected ballots.

1-basic-WriteInsAliases-NofM.sql

Write-ins ∧ N of M voting

Small 2-of-M contest with write-ins and aliases, no rejected ballots.

1-basic-WriteInsAliases-Provisional.sql

Write-ins ∧ Provisional / challenged ballots

Small 1-of-M contest with write-ins, aliases, accepted and rejected provisionals.

1-basic-Yes-or-No.sql

Voting system

Small Yes-or-No contest, no rejected ballots.

1-basic-samples-AlleganyGeneral2004.sql

Write-ins ∧ N of M voting

Test spec based on Allegany County, MD sample ballot for congressional district 6, general election, 2004-11-02.

1-basic-samples-CecilRPrimary1998.sql

N of M voting

Test spec based on Cecil County, MD sample ballot for Republican primary election, 1st congressional district, legislative district 35, 1998-09-15. The generated test does not include anything specific to primary elections.

1-basic-samples-FairfaxGeneral2004.sql

Write-ins

Test spec based on Fairfax County, VA sample ballot for 8th district, general election, 2004-11-02.

3.8 Required test cases not included in the basic Votetest test suite

Some requirements that might logically be tested during the phase when Votetest is used do not have associated test cases because the behaviors in question are orthogonal to the Votetest data model and/or so dependent on the specifics of the implementation that only an abstract test script could be provided. Those requirements and descriptions of the needed test cases are shown in Table 3.

Table 3:

Additional required test cases

Requirements

Test description

Realistic ballot styles [3, Part 1 Req. 7.2-A.1, A.4 and A.5]

Test a variety of sample ballot styles, such as those available on the NIST web site [6], to ensure that the voting system can handle the text of long ballot questions, complicated choice labels (such as for primary presidential delegation nominations), miscellaneous ballot text, and overall formatting and layout of ballot styles as used in practice.

Optical scan accuracy [3, Part 1 Req. 6.3.2-B and Section 7.7.5]

When test cases are realized for optical scan devices, the marks used should exercise the range of reliably detectable marks as defined by the vendor, from ideal marks to the standard mark specified in [3, Part 1 Req. 7.7.5-D]. However, to ensure that the tests are defensible, the relative frequency with which the different types of marks appear should be realistic. While both ideal marks and standard marks should be tested repeatedly, it would not be realistic for either extreme to dominate the input. The average mark should be typical of what an average voter would make. The ability of the scanner to ignore extraneous marks should also be tested ([3, Part 1 Req. 7.7.5-E]).

Marginal marks [3, Part 1 Req. 3.2.2.2-E and 7.7.3-C]

Test marks that are clearly within the marginal zone as defined by the vendor to verify that the behavior on marginal marks is as specified. Do not test marks that are near the boundaries, as the uncertainty of the boundaries is of no consequence (see [3, Part 1 Section 7.7.5.1]).

Respecting limits [3, Part 1 Req. 2.4-A.e, Req. 6.4.1.8-A.1 and Section 7.5.6]³ ³Note on pending erratum. [3, Part 1 Section 7.1] is missing a requirement saying that the central EMS shall prevent the defining of any election that would cause the voting system to exceed design limits such as maximum number of choices in a contest, maximum number of contests on a ballot, etc. That requirement should be cited here if and when it is added.

Construct test cases as needed to satisfy [3, Part 3 Req. 5.2.3-B and C]. The test generator (Section 5.7) may be of use.

Definition reuse [3, Part 1 Req. 7.2-C]

Choose two test cases that use similar ballot styles. Execute the first as usual, but instead of clearing out all definitions between tests, attempt to construct the ballot styles for the second test based on the definitions retained from the first.

Ballot style mismatches [3, Part 1 Req. 7.7.1-A]

Attempt to execute a test case with mismatched ballot styles loaded on different devices and verify that the system prevents incorrect reports from being generated. (It may prevent mismatched ballot styles from being loaded in the first place.)

Ballot orientation [3, Part 1 Req. 7.7.1-B]

If the vendor specifies that the tabulator is capable of scanning ballots in more than one orientation, allow the orientation of fed ballots to vary during the execution of all test cases. If the vendor specifies one or more orientations as being incorrect, attempt to feed ballots in the unsupported orientations and verify that the ballots are rejected.

Validation of input [3, Part 1 Req. 6.4.1.8-A and A.1]

Since poor validation of input often creates an attack vector, this should be covered as part of security testing and evaluation.

Miscellaneous capabilities [3, Part 1 Req. 7.2-A.1, 7.2-A.5, 7.3.1-B, 7.7.3-A.3, 7.8.3.3-B.1 and many others]

The VVSG contains numerous requirements for miscellaneous voting system capabilities that are orthogonal to the Votetest data model. Since the general test template specified in [3, Part 3 Section 5.2.1.1] treats each test case as an entire election, many such capabilities will be tested incidentally during the execution of the basic test suite. Use the Voting Equipment User Documentation to determine how to exercise any capabilities that are not tested incidentally and then verify that they satisfy the applicable VVSG requirements.

System-specific functions

These must be tested in accordance with [3, Part 3 Req. 5.2.3-G].

3.9 Requirements trace

This section lists the tests that are traceable to each requirement in Chapter 7 of the VVSG [3].

1-basic-1ofM.sql 1-basic-NoBallots-1ofM.sql 1-basic-Yes-or-No.sql 1-basic-samples-AlleganyGeneral2004.sql 1-basic-samples-FairfaxGeneral2004.sql 1-basic-StraightParty.sql 1-basic-Primary-StraightParty.sql 1-basic-Primary.sql 1-basic-BallotRotation-StraightParty.sql 1-basic-AbsenteeByCategory-StraightParty.sql 1-basic-SplitPrecinct-StraightParty.sql 1-basic-BallotRotation.sql 1-basic-Primary-BallotRotation.sql 1-basic-StraightParty-WriteIns.sql 1-basic-AbsenteeByCategory.sql 1-basic-SplitPrecinct-1.sql 1-basic-SplitPrecinct-2.sql 1-basic-StraightParty-Provisional.sql 1-basic-CrossPartyEndorsement.sql 1-basic-Primary-SplitPrecinct.sql 1-basic-SplitPrecinct-BallotRotation.sql 1-basic-NoChoices-1ofM.sql 1-basic-NoChoicesNoBallots-1ofM.sql 1-basic-AbsenteeByCategory-CrossPartyEndorsement.sql 1-basic-WriteIns.sql 1-basic-Provisional.sql 1-basic-WriteInsAliases.sql 1-basic-Primary-WriteIns.sql 1-basic-BallotRotation-WriteInsAliases.sql 1-basic-AbsenteeByCategory-WriteIns.sql 1-basic-SplitPrecinct-WriteIns.sql 1-basic-Primary-CrossPartyEndorsement.sql 1-basic-BallotRotation-CrossPartyEndorsement.sql 1-basic-AbsenteeBySpecialPrecinct-CrossPartyEndorsement.sql 1-basic-Primary-Provisional.sql 1-basic-BallotRotation-Provisional.sql 1-basic-AbsenteeByCategory-Provisional.sql 1-basic-SplitPrecinct-Provisional.sql 1-basic-WriteInsAliases-Provisional.sql 1-basic-SplitPrecinct-CrossPartyEndorsement.sql 1-basic-CrossPartyEndorsement-WriteIns.sql 1-basic-CrossPartyEndorsement-Provisional.sql 1-basic-AbsenteeBySpecialPrecinct-Yes-or-No.sql 1-basic-AbsenteeBySpecialPrecinct.sql 1-basic-Primary-AbsenteeBySpecialPrecinct.sql 1-basic-Primary-AbsenteeByCategory.sql 1-basic-AbsenteeBySpecialPrecinct-SplitPrecinct.sql 1-basic-SplitPrecinct-AbsenteeByCategory.sql 1-basic-BallotRotation-AbsenteeBySpecialPrecinct.sql 1-basic-BallotRotation-AbsenteeByCategory.sql 1-basic-AbsenteeBySpecialPrecinct-WriteIns.sql 1-basic-WriteInsAliases-AbsenteeBySpecialPrecinct.sql 1-basic-WriteInsAliases-AbsenteeByCategory.sql 1-basic-AbsenteeBySpecialPrecinct-Provisional.sql 1-basic-AbsenteeBySpecialPrecinct-StraightParty.sql 1-basic-samples-CecilRPrimary1998.sql

1-basic-Yes-or-No.sql 1-basic-samples-AlleganyGeneral2004.sql 1-basic-samples-FairfaxGeneral2004.sql 1-basic-AbsenteeBySpecialPrecinct-Yes-or-No.sql

1-basic-StraightParty.sql 1-basic-Primary-StraightParty.sql 1-basic-StraightParty-RankedOrder.sql 1-basic-BallotRotation-StraightParty.sql 1-basic-AbsenteeByCategory-StraightParty.sql 1-basic-SplitPrecinct-StraightParty.sql 1-basic-StraightParty-Cumulative.sql 1-basic-StraightParty-WriteIns.sql 1-basic-StraightParty-NofM.sql 1-basic-StraightParty-Provisional.sql 1-basic-CrossPartyEndorsement.sql 1-basic-AbsenteeByCategory-CrossPartyEndorsement.sql 1-basic-CrossPartyEndorsement-Cumulative.sql 1-basic-Primary-CrossPartyEndorsement.sql 1-basic-BallotRotation-CrossPartyEndorsement.sql 1-basic-AbsenteeBySpecialPrecinct-CrossPartyEndorsement.sql 1-basic-SplitPrecinct-CrossPartyEndorsement.sql 1-basic-CrossPartyEndorsement-WriteIns.sql 1-basic-CrossPartyEndorsement-Provisional.sql 1-basic-CrossPartyEndorsement-NofM.sql 1-basic-AbsenteeBySpecialPrecinct-StraightParty.sql

1-basic-Primary-StraightParty.sql 1-basic-Primary.sql 1-basic-Primary-RankedOrder.sql 1-basic-Primary-BallotRotation.sql 1-basic-Primary-SplitPrecinct.sql 1-basic-Primary-WriteIns.sql 1-basic-Primary-Cumulative.sql 1-basic-Primary-CrossPartyEndorsement.sql 1-basic-Primary-NofM.sql 1-basic-Primary-Provisional.sql 1-basic-Primary-AbsenteeBySpecialPrecinct.sql 1-basic-Primary-AbsenteeByCategory.sql 1-basic-samples-CecilRPrimary1998.sql

1-basic-samples-AlleganyGeneral2004.sql 1-basic-samples-FairfaxGeneral2004.sql 1-basic-StraightParty-WriteIns.sql 1-basic-RankedOrder-WriteIns.sql 1-basic-NoChoices-1ofM.sql 1-basic-NoChoicesNoBallots-1ofM.sql 1-basic-WriteIns.sql 1-basic-WriteInsAliases.sql 1-basic-Primary-WriteIns.sql 1-basic-BallotRotation-WriteInsAliases.sql 1-basic-AbsenteeByCategory-WriteIns.sql 1-basic-SplitPrecinct-WriteIns.sql 1-basic-WriteInsAliases-Cumulative.sql 1-basic-NofM-WriteIns.sql 1-basic-WriteInsAliases-NofM.sql 1-basic-WriteInsAliases-Provisional.sql 1-basic-CrossPartyEndorsement-WriteIns.sql 1-basic-AbsenteeBySpecialPrecinct-WriteIns.sql 1-basic-WriteInsAliases-AbsenteeBySpecialPrecinct.sql 1-basic-WriteInsAliases-AbsenteeByCategory.sql

1-basic-CrossPartyEndorsement.sql 1-basic-AbsenteeByCategory-CrossPartyEndorsement.sql 1-basic-CrossPartyEndorsement-Cumulative.sql 1-basic-Primary-CrossPartyEndorsement.sql 1-basic-BallotRotation-CrossPartyEndorsement.sql 1-basic-AbsenteeBySpecialPrecinct-CrossPartyEndorsement.sql 1-basic-SplitPrecinct-CrossPartyEndorsement.sql 1-basic-CrossPartyEndorsement-WriteIns.sql 1-basic-CrossPartyEndorsement-Provisional.sql 1-basic-CrossPartyEndorsement-NofM.sql

1-basic-SplitPrecinct-StraightParty.sql 1-basic-SplitPrecinct-1.sql 1-basic-SplitPrecinct-2.sql 1-basic-SplitPrecinct-RankedOrder.sql 1-basic-Primary-SplitPrecinct.sql 1-basic-SplitPrecinct-BallotRotation.sql 1-basic-SplitPrecinct-WriteIns.sql 1-basic-SplitPrecinct-Cumulative.sql 1-basic-SplitPrecinct-NofM.sql 1-basic-SplitPrecinct-Provisional.sql 1-basic-SplitPrecinct-CrossPartyEndorsement.sql 1-basic-AbsenteeBySpecialPrecinct-SplitPrecinct.sql 1-basic-SplitPrecinct-AbsenteeByCategory.sql

1-basic-samples-AlleganyGeneral2004.sql 1-basic-StraightParty-NofM.sql 1-basic-RankedOrder-NofM.sql 1-basic-NofM.sql 1-basic-Primary-NofM.sql 1-basic-Cumulative-NofM.sql 1-basic-BallotRotation-NofM.sql 1-basic-AbsenteeByCategory-NofM.sql 1-basic-SplitPrecinct-NofM.sql 1-basic-NofM-WriteIns.sql 1-basic-WriteInsAliases-NofM.sql 1-basic-NofM-Provisional.sql 1-basic-AbsenteeBySpecialPrecinct-NofM-Provisional.sql 1-basic-CrossPartyEndorsement-NofM.sql 1-basic-AbsenteeBySpecialPrecinct-NofM.sql 1-basic-samples-CecilRPrimary1998.sql

1-basic-Cumulative-RankedOrder.sql 1-basic-StraightParty-Cumulative.sql 1-basic-Cumulative.sql 1-basic-CrossPartyEndorsement-Cumulative.sql 1-basic-AbsenteeByCategory-Cumulative.sql 1-basic-Primary-Cumulative.sql 1-basic-BallotRotation-Cumulative.sql 1-basic-AbsenteeBySpecialPrecinct-Cumulative.sql 1-basic-SplitPrecinct-Cumulative.sql 1-basic-WriteInsAliases-Cumulative.sql 1-basic-Cumulative-NofM.sql 1-basic-Cumulative-Provisional.sql

1-basic-NoBallots-RankedOrder.sql 1-basic-RankedOrder-1.sql 1-basic-RankedOrder-2.sql 1-basic-StraightParty-RankedOrder.sql 1-basic-Primary-RankedOrder.sql 1-basic-BallotRotation-RankedOrder.sql 1-basic-AbsenteeByCategory-RankedOrder.sql 1-basic-Cumulative-RankedOrder.sql 1-basic-SplitPrecinct-RankedOrder.sql 1-basic-RankedOrder-WriteIns.sql 1-basic-RankedOrder-NofM.sql 1-basic-RankedOrder-Provisional.sql 1-basic-AbsenteeBySpecialPrecinct-RankedOrder.sql

This requirement does not specify functional testing as an applicable test method, so it is out of scope for Votetest.

1-basic-BallotRotation-StraightParty.sql 1-basic-BallotRotation.sql 1-basic-Primary-BallotRotation.sql 1-basic-BallotRotation-RankedOrder.sql 1-basic-SplitPrecinct-BallotRotation.sql 1-basic-BallotRotation-WriteInsAliases.sql 1-basic-BallotRotation-Cumulative.sql 1-basic-BallotRotation-CrossPartyEndorsement.sql 1-basic-BallotRotation-NofM.sql 1-basic-BallotRotation-Provisional.sql 1-basic-BallotRotation-AbsenteeBySpecialPrecinct.sql 1-basic-BallotRotation-AbsenteeByCategory.sql

This requirement falls within the scope of security assessment, so it is out of scope for Votetest.

7.5.1.1-C.2 Activation device, DRE, EBP, select ballot configuration for party in primary elections

This requirement falls within the scope of security assessment, so it is out of scope for Votetest.

7.5.1.3-A.4 Activation device, integrity and authenticity of ballot activation information

This requirement falls within the scope of security assessment, so it is out of scope for Votetest.

7.5.1.4-A.2 Activation device, access to remote registration database configurable

This requirement falls within the scope of security assessment, so it is out of scope for Votetest.

7.5.1.4-A.3 Activation device, notification of access to remote registration database

This requirement falls within the scope of security assessment, so it is out of scope for Votetest.

1-basic-Yes-or-No.sql 1-basic-samples-AlleganyGeneral2004.sql 1-basic-samples-FairfaxGeneral2004.sql 1-basic-AbsenteeBySpecialPrecinct-Yes-or-No.sql

1-basic-StraightParty-Provisional.sql 1-basic-Provisional.sql 1-basic-Primary-Provisional.sql 1-basic-RankedOrder-Provisional.sql 1-basic-BallotRotation-Provisional.sql 1-basic-Cumulative-Provisional.sql 1-basic-AbsenteeByCategory-Provisional.sql 1-basic-SplitPrecinct-Provisional.sql 1-basic-WriteInsAliases-Provisional.sql 1-basic-NofM-Provisional.sql 1-basic-AbsenteeBySpecialPrecinct-NofM-Provisional.sql 1-basic-CrossPartyEndorsement-Provisional.sql 1-basic-AbsenteeBySpecialPrecinct-Provisional.sql

Since the reasons for which a ballot might be flagged or separated for review are jurisdiction-dependent, the list of relevant tests will vary.

This requirement does not specify functional testing as an applicable test method, so it is out of scope for Votetest.

This requirement falls within the scope of security assessment, so it is out of scope for Votetest.

This requirement does not specify functional testing as an applicable test method, so it is out of scope for Votetest.

This requirement falls within the scope of security assessment, so it is out of scope for Votetest.

This requirement does not specify functional testing as an applicable test method, so it is out of scope for Votetest.

1-basic-Yes-or-No.sql 1-basic-samples-AlleganyGeneral2004.sql 1-basic-samples-FairfaxGeneral2004.sql 1-basic-AbsenteeBySpecialPrecinct-Yes-or-No.sql

1-basic-AbsenteeByCategory-StraightParty.sql 1-basic-AbsenteeByCategory-RankedOrder.sql 1-basic-AbsenteeByCategory.sql 1-basic-AbsenteeByCategory-CrossPartyEndorsement.sql 1-basic-AbsenteeByCategory-WriteIns.sql 1-basic-AbsenteeByCategory-Cumulative.sql 1-basic-AbsenteeBySpecialPrecinct-Cumulative.sql 1-basic-AbsenteeBySpecialPrecinct-CrossPartyEndorsement.sql 1-basic-AbsenteeByCategory-NofM.sql 1-basic-AbsenteeByCategory-Provisional.sql 1-basic-AbsenteeBySpecialPrecinct-NofM-Provisional.sql 1-basic-AbsenteeBySpecialPrecinct-Yes-or-No.sql 1-basic-AbsenteeBySpecialPrecinct.sql 1-basic-Primary-AbsenteeBySpecialPrecinct.sql 1-basic-Primary-AbsenteeByCategory.sql 1-basic-AbsenteeBySpecialPrecinct-SplitPrecinct.sql 1-basic-SplitPrecinct-AbsenteeByCategory.sql 1-basic-BallotRotation-AbsenteeBySpecialPrecinct.sql 1-basic-BallotRotation-AbsenteeByCategory.sql 1-basic-AbsenteeBySpecialPrecinct-WriteIns.sql 1-basic-WriteInsAliases-AbsenteeBySpecialPrecinct.sql 1-basic-WriteInsAliases-AbsenteeByCategory.sql 1-basic-AbsenteeBySpecialPrecinct-NofM.sql 1-basic-AbsenteeBySpecialPrecinct-RankedOrder.sql 1-basic-AbsenteeBySpecialPrecinct-Provisional.sql 1-basic-AbsenteeBySpecialPrecinct-StraightParty.sql

7.7.2-A.5 Tabulator, accept or reject provisional-challenged ballots individually

7.7.2-A.6 Tabulator, accept or reject provisional-challenged ballots by category

This requirement specifies a “fail safe” behavior for a condition that, ideally, would not be reproducible by a test lab. The method to reproduce it, if there were one, would be specific to a particular voting system. Thus there is no test in Votetest that attempts to create the conditions under which the requirement would apply. For more information, see the discussion of Part 3 Req. 5.2.3-A in the VVSG.

Since the reasons for which a ballot might be flagged or separated for review are jurisdiction-dependent, the list of relevant tests will vary.

This requirement does not specify functional testing as an applicable test method, so it is out of scope for Votetest.

This requirement falls within the scope of security assessment, so it is out of scope for Votetest.

Since the reasons for which a ballot might be flagged or separated for review are jurisdiction-dependent, the list of relevant tests will vary.

4 Advanced schema

The schema for core requirements is designed only for testing. It does not respond to the security, privacy, accessibility, or usability requirements of the VVSG.

The schema was developed with PostgreSQL [7] running on a GNU/Linux operating system. It uses extensions to the SQL standard [2] that might not function as intended with other databases.

Layer 1, the translation of the data model, was covered in Section 3.6. The following subsections cover the other four layers.

4.1 Conveniences

All assertions have as a precondition the assumption that all of the constraints are satisfied (see Section 3.5.5).

4.1.1 FilteredBallotContestAssociation

The view FilteredBallotContestAssociation identifies all Contests that appear on a given Ballot, excluding ranked order Contests.

If BallotStyleContestAssociation contains no rows with a StyleId matching Ballot.StyleId, meaning that the BallotStyle contains no Contests, then the natural join of Ballot and BallotStyleContestAssociation produces no rows for that Ballot. The subsequent natural join with Contest cannot add back a BallotId that was dropped, so FilteredBallotContestAssociation correctly contains no rows for that Ballot.

4.1.2 ReportingContextAssociationMerge

4.1.3 VotableChoices

The view VotableChoices identifies all canonical Choices for which a valid VoterInput could exist (those contained in the applicable BallotStyles), excluding Aliases.

The where clause eliminates rows pertaining to non-canonical Choices (those appearing in the AliasId column of table Alias).

If BallotStyleContestAssociation contains no rows with a StyleId matching Ballot.StyleId, meaning that the BallotStyle contains no Contests, then the natural join of Ballot and BallotStyleContestAssociation produces no rows for that Ballot. The subsequent natural join with Choice cannot add back a BallotId that was dropped, so VotableChoices correctly contains no rows for that Ballot.

If the BallotStyle of a given Ballot contains only Contests having zero Choices, then the natural join with Choice will eliminate all rows containing that BallotId.

4.1.4 ReportingContextContestAssociation

4.1.5 FilteredContextContestAssociation

The view FilteredContextContestAssociation is the same as ReportingContextContestAssociation except it excludes ranked order Contests.

Per Assertion 9, for each ReportingContext, ReportingContextContestAssociation contains exactly one row for each Contest that is relevant in that ReportingContext. Constraints ensure that the natural join with Contest will not drop rows. The only difference from ReportingContextContestAssociation therefore is the elimination of ranked order Contests by the where clause.

4.1.6 FilteredContextChoiceAssociation

Per Assertion 10, for each ReportingContext, FilteredContextContestAssociation contains exactly one row for each non-ranked-order Contest that is relevant in that ReportingContext. For each ReportingContext, the result of FilteredContextContestAssociation natural join Choice contains exactly one row for each Choice in each non-ranked-order Contest that is relevant in that ReportingContext. The where clause eliminates rows pertaining to non-canonical Choices (Aliases).

4.1.7 BallotCounts

The view BallotCounts produces the count of the number of read and counted Ballots for each ReportingContext.

4.1.8 BallotCountsByConfiguration

The natural join of Ballot with ReportingContextAssociationMerge cannot generate a row with a particular combination of ReportingContext and BallotStyle unless there exists a Ballot of that BallotStyle that is relevant in that ReportingContext.

4.1.9 BallotCountsByCategory

The natural join of (Ballot natural join ReportingContextAssociationMerge) with BallotCategoryAssociation cannot generate a row with a particular combination of ReportingContext and BallotCategory unless there exists a Ballot of that BallotCategory that is relevant in that ReportingContext.

4.1.10 BallotCountsByCategoryAndConfiguration

4.1.11 BlankBallot

Ballot natural left outer join VoterInput produces one row with a non-null Value column for each vote (possibly many such rows with the same BallotId), and exactly one row with a null Value column for each Ballot having no associated votes. The where clause selects only the latter rows.

4.1.12 BlankBallotCounts

The assertion and discussion for BlankBallotCounts are parallel to those of BallotCounts, substituting BlankBallot for Ballot.

4.1.13 BlankBallotCountsByConfiguration

The assertions and discussion for BlankBallotCountsByConfiguration are parallel to those of BallotCountsByConfiguration, substituting BlankBallot for Ballot.

4.2 Adaptation

Converting the raw voter inputs into the effective voter inputs required by the logic model involves Alias reconciliation, implementation of straight party voting, and generation of default (0) values for ballot positions that were not voted.

The VoterInput table has a primary key on (BallotId, ChoiceId), so there is at most one row for any given ballot position on any given Ballot. Deliberately, the adaptation views do not preserve this constraint in the event that double votes result from Alias reconciliation or straight party voting. Both of these cases are treated as errors for testing purposes, and the errors are most easily located by looking for duplicate keys. This is done by the integrity view DoubleVotes (see Section 4.3).

All assertions have as a precondition the assumption that all of the constraints are satisfied (see Section 3.5.5).

4.2.1 AntiAliasedVoterInput

AntiAliasedVoterInput provides a view of VoterInput in which all Choices have been “canonicalized.”

4.2.2 VoterInputMerge

VoterInputMerge provides a view over AntiAliasedVoterInput in which the side-effects implied by straight party votes have been incorporated. If a straight party selection Contest is overvoted, it has no side-effects.

VoterInputMerge depends on two intervening views, ValidStraightPartyVotes and ImpliedStraightPartyVotes.

Note that ValidStraightPartyVotes does not filter out Ballots that are not accepted.

Note that ImpliedStraightPartyVotes does not do anything about the possibility that endorsements may be ill-formed (e.g., containing overvotes).

VoterInputMerge simply concatenates the contents of the previous two views. VoterInputMerge does not inherently prevent there from being more than one row with the same (BallotId, ChoiceId). This case violates Constraint XII and is detected by the integrity view StraightPartyOverrides (see Section 4.3).

The assertion follows from Assertion 18, Assertion 20 and the definition of the union all operator.

4.2.3 EffectiveInput

Finally, the view EffectiveInput generates zeroes for ballot positions that were not voted. This is appropriate for all Contest types except ranked order. Ranked order tabulators should instead access VoterInputMerge directly.

4.3 Integrity checks

For those integrity constraints that are too complex to code directly as SQL constraints within the tables, a series of views exists to look for problems. All of the integrity checking views should always be empty. If data appear in any of the views, the input was invalid and the results of the model will be invalid. The integrity views are listed in Table 4 but their definitions have been elided. Interested readers can find them in the Votetest distribution, in the file Infrastructure-VoteSchema.sql.

4.4 Translation of logic model

The following subsections first quote relevant portions of the logic model, then describe their analogs in the schema. Some terms from the logic model are elided from this discussion. For complete information on the logic model, please refer to the VVSG [1].

All assertions have as a precondition the assumption that all of the constraints are satisfied (see Section 3.5.5).

4.4.1 S(c,r,t,v)

Follows from Assertion 23. Rows are not generated for the irrelevant case where S(c,r,t,v) is defined to be zero. Note that EffectiveInput does not eliminate inputs for ranked order Contests.

4.4.2 S(r,t,v)

The ternary function S is implemented by the view S. The current value of S(r,t,v) is obtained by selecting S_val where ContestId = r and BallotId = v. The view S contains rows only for Contests that actually appear on the Ballot according to its BallotStyle. All others are defined to be 0.

VotesByContestAndContext is a convenience to retrieve all of the S_val vote counts for each relevant combination of ReportingContext and Contest. For each relevant combination of ReportingContext and Contest that contains no Ballots, there is a single row with nulls in the last three columns.

4.4.3 S′(c,r,t,v)

The quaternary function S′ is implemented by the view SPrime. The current value of S′(c,r,t,v) is obtained by selecting SPrime_val where ChoiceId = c and BallotId = v.

4.4.4 T(c,j,r,t)

The quaternary function T is implemented by the view T. The current value of T(c,j,r,t) is obtained by selecting T_val where ChoiceId = c and ReportingContext = j.

TSum is a convenience that sums T_val by Contest. Note that TSum does not eliminate ranked order contests but rather provides a value of zero for them in the TSum_val column.

4.4.5 O(j,r,t)

The ternary function O is implemented by the view O. The current value of O(j,r,t) is obtained by selecting O_val where ContestId = r and ReportingContext = j.

4.4.6 U(j,r,t)

The ternary function U is implemented by the view U. The current value of U(j,r,t) is obtained by selecting U_val where ContestId = r and ReportingContext = j.

4.4.7 K(j,r,t)

The ternary function K is implemented by the view K. The current value of K(j,r,t) is obtained by selecting K_val where ContestId = r and ReportingContext = j.

4.4.8 Balance

A check for this assertion is implemented by the view Balance. The current difference between ∑ _cC(r,t)T(c,j,r,t) + O(j,r,t) + U(j,r,t) and K(j,r,t) ×N(r) is obtained by selecting Discrepancy where ContestId = r and ReportingContext = j. Discrepancy should always be zero.

5 Advanced test development environment

5.1 Software prerequisites

All software, tools and materials were developed on a GNU/Linux operating system.

The following packages are required in order to build and run the programs in Votetest:

Votetest uses extensions to the SQL standard [2] and the C++ standard [12] that might not function as intended with other databases and compilers.

5.2 Hardware prerequisites

Votetest was developed on a PC having a 3.6 GHz Pentium 4 processor, 1 GiB of RAM, and an 80 GB SATA hard drive. The resources of this PC were more than adequate for all tests in the basic test suite, and if necessary a lesser configuration should be usable. Performance limitations became obvious only with the large and complex scenarios generated for the scalability testing described in Section 5.8.

5.3 File listing

Files pertaining to the advanced test development environment are described in Table 5.

Table 5:

Advanced files in the Votetest distribution

Files

Description

Details

runTest

Shell script to execute a test case.

Section 5.5.1

runAllTests

Shell script to execute all test cases and save output.

Section 5.5.2

output/

Empty subdirectory used to store the output of runAllTests.

Section 5.5.2

output_kill-overvotes/

Empty subdirectory used to store the output of runAllTests when the –kill-overvotes option is used.

Section 5.5.2

0-integrity-description.sql

SQL, test suite integrity checks (15 files).

Section 5.6

Infrastructure-Features.sql

SQL, create table mapping test cases to system capabilities.

Section 5.5.4

Infrastructure-IntegrityChecks.sql

SQL, show contents of all integrity views. This is invoked automatically by test cases and need not be used directly.

N/A

Infrastructure-KillOvervotes.sql

SQL, transform a test case into one that is executable on a system that prevents overvoting. This is invoked automatically by test cases and need not be used directly.

N/A

Infrastructure-PairsCoverage.pgcc

Source code of coverage checking utility.

Section 5.5.5

Infrastructure-PairsCoverage.sql

SQL, data needed by coverage checking utility.

Section 5.5.5

Infrastructure-TestFooter.sql

SQL, print “END TEST CASE OUTPUT” footer. This is invoked automatically by test cases and need not be used directly.

N/A

Infrastructure-TestHeader.sql

SQL, print “BEGIN TEST CASE OUTPUT” and timestamp and configure verbosity of output for all test cases. This is invoked automatically by test cases and need not be used directly.

N/A

Infrastructure-TestHook.sql

SQL, support the optional invocation of a test transformation script (e.g., Infrastructure-KillOvervotes.sql). This is invoked automatically by test cases and need not be used directly.

N/A

Infrastructure-VoteSchema.sql

SQL, create the schema. This is invoked automatically by test cases and need not be used directly.

Section 3.6
Section 4

ReportGenerator/

Subdirectory containing the source code of the advanced test development environment’s report generator. This is invoked automatically by test cases and need not be used directly.

Section 5.5.3

TestGenerator/

Subdirectory containing the source code of the advanced test development environment’s test generator.

Section 5.7

test_specs/

Subdirectory containing example input files for the test generator.

Section 5.7.3

Makefile.am
Makefile.in
aclocal.m4
configure
configure.ac
depcomp
install-sh
missing

Files having to do with the automake build process.

Section 5.4

INSTALL

Text file containing generic instructions on the use of the configure script.

Section 5.4

AUTHORS
NEWS
README

Unused files required by GNU standard.

N/A

5.4 Installation

Votetest is packaged with GNU automake [13], so all usual GNU tricks should work. Help on configuration options can be found in the INSTALL file or obtained by entering ./configure --help.

However, in the event that PostgreSQL and/or CLN are installed in nonstandard locations, an invocation such as the following might be required.

5.5 Infrastructure

5.5.1 runTest

The script runTest is used to run an SQL test case against the database and report the results from the Votetest environment.

A test case is executed by changing the current working directory to the directory containing the test suite and invoking the runTest script with the file name of the test case. The runTest script resets the database to an initial state and then feeds the test case to the SQL interpreter. No database named votetest other than the one created by the test suite should exist or it will be destroyed without warning.

Some test cases involve overvoting, which means they cannot be executed as-is on a system that prevents overvoting. If the –kill-overvotes switch is used, the test case is transformed into one that can be executed on a system that prevents overvoting. Otherwise, overvotes are processed and reported as they would be in a system that supports overvoting.

If a ballot overvotes a contest, –kill-overvotes removes all of that ballot’s votes in that contest, effectively converting overvotes into undervotes. This of course changes the expected results of the test case. The expected results in the normal configuration and with –kill-overvotes enabled are saved in the subdirectories sample_output and sample_output_kill-overvotes respectively.

5.5.2 runAllTests

The script runAllTests invokes runTest for each test case and directs the output into a file in the subdirectory named “output.” It then invokes runTest –kill-overvotes for each test case and directs the output into a file in the subdirectory named “output_kill-overvotes.” These results may then be compared with the contents of the sample_output and sample_output_kill-overvotes subdirectories to ensure that the test suite is operating as expected.

5.5.3 ReportGenerator

Usage: ReportGenerator [-v] context-name [context-name...]. A no-frills, plain-ASCII post-voting report for each specified ReportingContext is sent to standard output. As a convenience to test labs, the report total volume needed for the accuracy test protocol of the VVSG is also calculated and reported. A sample report is shown in Figure 9.

The verbose flag (-v) is only useful in ranked order contests. It causes the state of the ranked order logic (with many ballot images) to be output for each round of voting.

Name	Short name	Version tested	Source

PostgreSQL	postgres	8.3.7⁴	[7]
Class Library for Numbers	CLN	1.2.2	[8]
GNU Compiler Collection	g++	4.3.3	[9]
Flex: Fast Lexical Analyzer	flex	2.5.35	[10]
Bison: GNU parser generator	bison	2.3	[11]