October 28, 2008
Mark Phillips
Vice President of Compliance Services
SysTest Labs, Incorporated 216 16th Street, Suite 700 Denver, CO 80202-5115
NVLAP Lab Code 200733-0
Dear Mr. Phillips,
On behalf of the
National Voluntary Laboratory Accreditation Program (NVLAP), I write to notify
of you of NVLAP's decision to suspend its accreditation of SysTest's electronic
voting testing program pursuant to NIST Handbook 150, NVLAP Procedures and General
Requirements, 2006 Edition, section 3.10. This letter provides an explanation
of NVLAP's decision and describes the steps SysTest can take to reinstate its
accreditation.
This action pertains
to voting systems under review by SysTest to be recommended for certification
by the Election Assistance Commission for future elections and is not pertinent
to systems already deployed for the 2008 election which were certified under
alternate systems.
Background Discussion
SysTest Labs,
Incorporated is currently accredited by the National Voluntary Laboratory Accreditation
Program (NVLAP), a program within the National Institute of Standards and Technology
(NIST), to perform testing to federal standards in accordance with the Help
America Vote Act of 2002 (HAVA). These standards are the 2002 Voting System
Standards (VSS-2002) and the 2005 Voluntary Voting System Guidelines (VVSG-2005).
On August 8, 2008, NVLAP sent SysTest Labs a letter outlining specific concerns
with respect to SysTest's NVLAP-accredited testing of voting systems, including
voting system test campaigns submitted to the Election Assistance Commission
(EAC) under their voting system certification process. These specific concerns
are documented in the March 2008 NVLAP on-site assessment checklist, produced
as part of the normal reassessment process, and in communications between the
EAC and NIST regarding issues that EAC staff identified with test reports submitted
by SysTest Labs (enclosed). The August 8th letter (also enclosed) outlined three
specific concerns. In short they were:
1) SysTest's lack of properly documented and validated test methods.
2) Testing conducted by unqualified or untrained personnel.
3) Improper assurances made to manufacturers regarding testing outcomes.
NVLAP directed
SysTest to submit information to NVLAP, including a schedule of all accredited
voting systems testing planned, within 14 days of receipt of the August 8th
letter. NVLAP informed SysTest of its intention to conduct on-site monitoring
of the testing of electronic voting machines. SysTest was notified by email
on October 6, 2008 of NVLAP' s intention to visit their lab on October 14th
through 16th to observe testing that had been scheduled during that period.
NVLAP assembled
a team consisting of the NVLAP voting system technical assessor, the NIST/NVLAP
program manager for voting system testing and four members of the NIST Information
Technology Laboratory (ITL) involved in writing the federal voting system standards.
In addition, two EAC staff members were invited to provide their observations.
During the on-site visit this eight-member team witnessed several tests, interviewed
testers, and examined documents related to the areas of concern.
Site Visit Observations
As a result of
this on-site monitoring visit, NVLAP has serious concerns about SysTest's performance
of voting system testing. These concerns were supported by observations of testing
where the test methods being used were not fully developed, validated, mapped
to the requirements of the applicable standards, and controlled under SysTest's
document control policy.
From the team's
observations it was unclear who at SysTest had the ultimate responsibility for
test method development. During the observed tests, it appeared that the testers
were running the tests for the first time. Changes were made to the test procedures
to address items that should have been caught during an initial run-through
of the test. Basic tests, such as the system readiness test, were not conducted
successfully. Three test methods failed due to problems with the procedure,
tester error, or unfamiliarity with the test set-up. Some anomalies or potential
problems during testing were not reported by the testers but were pointed out
by members of the on-site team.
During the team's
visit SysTest personnel stated that their policy was to validate test methods
during the actual testing of voting equipment. This approach is unacceptable.
The lab must validate all test methods separate from actual testing so that
equipment nonconformance can be isolated from test method problems. This validation
must follow set documented procedures and show a clear chain of responsibility
for the process.
SysTest has undergone
numerous changes in personnel since its original accreditation and, in fact,
since the March 2008 NVLAP on-site assessment. SysTest staff conducting testing
during the monitoring visit demonstrated a lack of familiarity with the test
equipment and procedures. Some personnel who participated in past on-site assessments
were no longer associated with the NVLAP-accredited testing; they had been reassigned
to work in support of state certification of voting systems. SysTest management's
stated goal was to transfer the expertise and testing approach from their New
York testing
campaign to the NVLAP/EAC accredited testing campaign. SysTest must improve
the level of training of personnel involved in NVLAP/EAC accredited testing
given that SysTest has reassigned experienced testers to other work. SysTest
should consider bringing in outside instructors to train laboratory personnel.
SysTest was advised
that an appearance of impropriety had occurred in a case where personnel had
given a client an indication that their equipment would successfully pass testing.
SysTest's response was that this was an isolated incident and the person involved
had not intended to give this impression. SysTest further stated that their
employees were given a quiz which they felt covered training in this situation.
It is NVLAP's position that this quiz is insufficient and SysTest must provide
specific training to their employees on professional ethics and document the
employees' intent to adhere to SysTest's stated policy.
NVLAP's Decision
Pursuant to NIST
Handbook 150, NVLAP Procedures and General Requirements, 2006 Edition, section
3.10, NVLAP hereby suspends SysTest's accreditation effective as of the date
of this letter. SysTest Labs, Incorporated is prohibited from using the NVLAP
symbol on its test reports, correspondences, and advertising during the suspension
period for all voting system testing. Accreditation may be reinstated only after
such time that SysTest can demonstrate voting system testing in accordance with
the requirements of the applicable voting system standards and NIST Handbook
150. This demonstration must be achieved through an on-site visit to SysTest
to witness testing, review documentation, interview personnel, and any other
means necessary to gather objective evidence in support of a decision regarding
reinstatement.
This on-site visit
will occur only after NVLAP is convinced, through the submission of documentation,
that SysTest has taken the necessary steps to correct the areas of nonconformance
herein addressed. This documentation will include, but is not limited to: procedures
for test method development; procedures for test method validation; revised
document control procedures that specifically address technical procedures;
fully developed test methods showing validation, document control, and mapping
to the federal voting system standards; and, procedures or policies that address
methods by which SysTest will control statements or assurances to their clients
regarding the outcome of voting system testing.
SysTest was accredited
by NVLAP based on its ability to develop and perform competent testing within
the framework of an effective management system. SysTest now needs to revise
its management system to correct the nonconformances found during this visit
and implement these system changes. NVLAP believes that the current SysTest
management team is committed to accomplishing this goal and will work with them
to that end.
Sincerely,
Jon Crickenberger
NIST/NVLAP Program Manager
Enclosures
Cc: Brian Hancock,
Election Assistance Commission