Test Development for Voting Systems

  Testing FAQs
HAVA, Certification and Testing

What are NIST's roles related to voting systems certification and testing?
HAVA directs NIST to conduct an evaluation of independent non-Federal laboratories to carry out the testing of voting systems and to submit recommendations of qualified laboratories to the EAC for accreditation.

In order to ensure that all test labs use the same, comprehensive test suites, NIST is developing several conformance test suites to be used as part of the EAC's Testing and Certification Program, to determine that Voluntary Voting System Guidelines (VVSG) requirements are met by voting systems.

Does NIST certify voting systems?
No. The EAC, not NIST, certifies voting systems for use in elections. Information about Voting System Certification is available from the EAC.

Does NIST test voting systems?
No. Testing is performed by EAC accredited test laboratories.

What is the relationship between the VVSG, test suites, test lab conformance testing, and EAC certification?
Think of the relationship between the VVSG, test suites, conformance testing and certification as a set of building blocks with each layer depending on the layer beneath it. None of the higher-level blocks can be performed unless the box beneath it has been completed. Thus, conformance testing cannot be performed unless test suites exist that are built from the standard. Certification can only be accomplished when all the lower levels building blocks are in place.

Why test?
In the marketplace, testing provides a vehicle for exchanging information between a buyer and a seller. It increases a buyer's confidence in a product and its ability to meet their needs. For sellers (e.g., manufacturers), testing can help to substantiate claims that a product meets a given specification. For the voting systems program, testing is used to provide confidence both to election officials and to the entire community that they serve that voting systems meet the requirements in the VVSG.

What is the difference between conformity assessment and conformance testing?
Good question. These terms are often used interchangeably, although there are differences. Conformity assessment is the more general term. It includes all activities concerned with determining directly or indirectly that relevant requirements in standards or regulations are fulfilled, such as: sampling and testing; inspection; conformance testing; certification; management system assessment and registration; accreditation of the competence of those activities and recognition of an accreditation program's capability. Conformance testing is conformity assessment by means of testing. It provides a way to determine whether or not an implementation (e.g., voting system) deviates from the specification (e.g., VVSG).

Where can I find more information about standards, testing and conformance?
Please check out the tutorial: Standards 101

Are there tutorials for the VVSG Recommendations to the EAC (Aug 20007)?
Yes. A companion document, VVSG Companion Document for the Election Official Community for the TGDC's VVSG Recommendations to the EAC of August 31, 2007, provides summary and background information. Additionally, a set of tutorials help explain fundamental concepts found in the VVSG Recommendations.

Test Suites for the VVSG

What is a VVSG test suite?
A VVSG test suite is a set of tests, test procedures, and test documentation, used to check a voting system for conformance to the requirements in the VVSG.

Why is NIST developing test suites?
NIST has extensive experience in developing test suites for a variety of different standards and technologies. NIST-developed test suites will be publicly available and free to use. They will be used by EAC-accredited test labs as part of the EAC's testing and certification program. Test suites are being developed for the human factors, security, and core functionality requirements as specified in the next VVSG.

How can the EAC, the election community, and the public know that every test lab is testing in the best, most efficient and transparent way possible?
Publicly available test suites that are used by all accredited laboratories serve several purposes:

• They reduce the cost of testing since each test lab does not need to invent the same work.
• They help ensure consistency among test labs. In some circumstances manufacturers have been able to "shop" for easier labs. Uniform, public test suites help prevent this.
• They promote transparency of the testing process since all interested parties can see and comment on the test suites.
• They enable manufacturers to run the tests in house before they submit their voting system for testing, thus improving the quality of the system prior to formal testing.
  

What test suites are NIST scientists developing?
NIST scientists are developing test suites for three different areas in the VVSG: core requirements, security, and human factors. There may be several test suites within each area. For example, core requirements include test suites for logic testing, environmental testing, electromagnetic testing, and volume testing. Security includes test suites for setup, software installation, integrity management, system event logging, physical security and communication security. Human factors include test suites for usability and accessibility of voting systems. A list of the NIST test suites is available at: http://vote.nist.gov/SystemTesting/testsuitereview.htm

Why are there so many test suites?
The VVSG encompasses a wide range of technologies, each demanding its own test suite. Each type of test suite has to be uniquely designed. For example, the human factors test suites will use humans. There is no substitute for testing how humans will interact with a system.

Why are test suites expensive to design and build?
In order to build a test suite, each and every requirement in the standard must be scrutinized and unique tests developed. Often times, a single requirement results in many tests. Moreover, the tests require an in-depth understanding of the technology being used. Each test needs to be robust, self-contained, and correct and usable. It is necessary to fully understand both the VVSG and what techniques are available at a reasonable cost. The tests must address every aspect of the VVSG or manufacturers will be able to ignore requirements that are not tested.