Test Development for Voting Systems

  Why Test Suites for the VVSG

In the marketplace, testing provides a vehicle for exchanging information between a buyer and a seller. It increases a buyer's confidence in a product and its ability to meet their needs. For sellers (e.g., manufacturers), testing can help to substantiate claims that a product meets a given specification. For the voting systems program, testing is used to provide confidence both to election officials and to the entire community that they serve that voting systems meet the Voluntary Voting System Guidelines (VVSG).

There are many different types of testing and testing programs. The type of testing program used by the election community was established in the 2002 Help America Vote Act (HAVA) and consists of nationally accredited laboratories providing testing services and a national certification performed by the Election Assistance Commission (EAC). The question remains, though, what kind of tests should the accredited laboratories perform? How can the EAC and their customers know that every test lab is testing the best and most efficient way possible in the most transparent way possible?

The primary answer to these questions is the development of public test suites that are used by all accredited laboratories. Public test suites serve several purposes:

• They establish for manufacturers exactly what it means to meet a given specification. This helps manufacturers develop better products.
• It reduces the cost of testing since each test lab does not need to invent the same work. It reduces the amount of time spent testing systems that do not meet the VVSG specification, since manufacturers can already have run the tests in house before they submit a product for testing.
• It helps ensure consistency between test labs. In some circumstances manufacturers have been able to "shop" for easier labs. Public test suites help prevent this.
• It promotes transparency of the testing process since all interested parties can see and comment on the test suites.

The NIST is developing test suites for three different areas: core requirements, security, and human factors. There may be several test suites within each area. For example, core requirements include test suites for logic testing, environmental testing, electromagnetic testing, and volume testing.

• Why so many test suites? The VVSG encompasses a wide range of technologies, each demanding its own test suite. Each type of test suite has to be uniquely designed. For example, the human factors test suites will use humans. There is no substitute for testing how humans will interact with a system.
• Why are test suites so expensive to design and build? In order to build a test suite, each and every requirement in the standard must be scrutinized and unique tests developed. Often times, a single requirement results in many tests. Moreover, the tests require an in-depth understanding of the technology being used. Each test needs to be robust, self-contained, and correct and usable. It is necessary to fully understand both the VVSG and what techniques are available at a reasonable cost. The tests must address every aspect of the VVSG or manufacturers will be able to ignore requirements that are not tested.