|
NIST
GAITHERSBURG, MARYLAND
TECHNICAL GUIDELINES DEVELOPMENT COMMITTEE
(TGDC) MEETING
FRIDAY, AUGUST 17, 2007
(START
OF AUDIOTAPE 1, SIDE A)
MR.
EUSTIS: Well, good morning everybody. This is Allan Eustis
with the NIST Information Technology Laboratory Voting Team.
Welcome to I believe the tenth plenary session of the Technical
Guidelines Development Committee.
You
will notice some different faces from our end because except for
Dr. Jeffrey, all of the TGDC members are thankfully participating
and they are doing so at their locations remotely. So this I
think is an efficient way to do the meeting but it's a little
different then we normally do.
I
just want to quickly go through a few of the safety issues for
our EAC and non NIST attendees at the meeting, and I'll just
go real quickly because we've held this meeting in the employee
lounge many times and if we have an emergency such as a fire drill
or a real fire, you'll exit, take a right and there's the
first exit, (unintelligible) on both sides, glass doors.
And
we're going to take a break everybody at 1:30 Eastern Daylight
Time for a half hour lunch, and come back and hopefully be very
efficient and finish this meeting up as quickly as we can in the
afternoon. We do have the potential to go to 5:30 p.m. if necessary.
A
few preliminary matters, and this is for all the NIST folks too,
please turn off your cell phones and pagers.
I
mentioned the half hour break at 1:30 p.m. for lunch.
If
you are calling in and you're on a cell phone especially, or
you're in an office area, if you could mute your phone until
you're recognized or unless you want to make a comment, and
we're hoping that most of the members will use our electronic
hand raising tool.
And
this is again for all NIST as well as TGDC members who have gotten
very good at this, please identify yourself as you ask a question
or whenever you're called on to answer a question. This will
help us when we transcribe this public meeting and make it available
for the public.
With
that I will hand the meeting over to Dr. Jeffrey to open up the
plenary session. Thank you.
DR.
JEFFREY: Well, good morning everyone. Welcome to the tenth
plenary session of the TGDC.
I
definitely appreciate everybody's attendance in either real
space or virtual space, and to see exactly who's here I'm
going to actually turn it over to the parliamentarian Thelma Allen
to do the roll call.
MS.
ALLEN: Good morning. Williams.
DR.
WILLIAMS: Here.
MS.
ALLEN: Williams is present. Wagner.
MR.
WAGNER: Here. Wagner is present. Paul Miller.
MR.
MILLER: Here.
MS.
ALLEN: Paul Miller is present. Gale.
SECRETARY
GALE: Here.
MS.
ALLEN: Gale is present. Mason.
MS.
MASON: Here.
MS.
ALLEN: Mason is present. Gannon. Gannon. Gannon is
not attending. Pearce.
MR.
PEARCE: Here.
MS.
ALLEN: Pearce is present. Alice Miller.
MS.
MILLER: Here.
MS.
ALLEN: Alice Miller is present. Purcell.
MS.
PURCELL: Here.
MS.
ALLEN: Purcell is present. Quesenbery.
MS.
QUESENBERY: Here.
MS.
ALLEN: Quesenbery is present. Rivest.
DR.
RIVEST: Here.
MS.
ALLEN: Rivest is present. Schutzer. Schutzer. Schutzer
is not responding. Jeffrey.
DR.
JEFFREY: Here.
MS.
ALLENG: Jeffrey is present. We have 1l in attendance.
We have enough for a quorum.
DR.
JEFFREY: Thank you very much, Thelma.
So
one of the things, I'll start off by saying that I'm actually
very excited for a number of reasons. One, I'm hoping that
at the end of today's session that we will have an approved
package that we can forward to the EAC.
I'm
also excited, both happy and sad at the same time, that I've
announced that I am resigning from the U.S. government effective
the beginning of September.
I've
absolutely thoroughly enjoyed being both the NIST Director as
well as being the Chair of the TGDC.
I
will tell you with all honesty that when I look back on the time
that I've spent at NIST and the number of important things going
on, there are a few issues that bubble up to the level of importance
that I think that the TGDC has been working on. When you talk
about affecting the core of our own democracy, there's few better
examples then developing a system.
So
I want to thank all the TGDC members for helping to put this package
together, and also from a personal standpoint, for educating me
as somebody who knew nothing about voting systems and making this
a very enjoyable process.
So
with that I will actually get the meeting going and just remind
people that if you have access to the Internet that we're using
a software package that allows you to just click on Raise My Hand
and we'll then be calling on people in the queue.
If
you are on a cell phone, or if the package is not working, or
if you just care not to use it, just please feel free to just
speak up.
And
also we are very, very pleased to welcome the U.S. Elections Systems
Commissioner, Deneta Davidson here today, who will e speaking
to us in just a few minutes. And it's always a pleasure to
have Deneta here. Another aspect of this job I'm going to
seriously miss.
I
would also like to welcome a potential new member to the committee,
Dr. Semcaner, who has been nominated by IEEE, and while the EAC
is completing the required vetting process we've invited him
to listen in and participate as an observer for this teleconference,
so look forward to him formally joining.
So
at this time I would like to entertain a motion to adopt the August
17, 2007 TGDC agenda. Is there a motion to accept the agenda?
SECRETARY
GALE: Dr. Jeffrey, this John Gale of Nebraska. I move
to accept the agenda.
DR.
JEFFREY: Excellent. Is there a second?
MR.
PEARCE: Philip Pearce, I'll second it.
DR.
JEFFREY: Okay, there is a motion to accept and second. Is
there any discussion? Okay, hearing none, is there any objection
to unanimous consent?
Hearing
no objection, the agenda is adopted by unanimous consent.
At
this time I would also like to entertain a motion to accept the
summary minutes of the last meeting of the TGDC. Is there a
proposal for that?
DR.
RIVEST: Ronald Rivest, so moved.
DR.
JEFFREY: Okay, a second?
MS.
PURCELL: Helen Purcell, second.
DR.
JEFFREY: Great. Any discussion? Okay, hearing no discussion,
is there any objection to adoption of the meeting minutes by unanimous
consent? Okay, this is going to go great today. Hearing no
objection, they are approved by unanimous consent.
Since
the May meeting, I must say when I looked at the number of teleconferences
that we held, three subcommittees have put in an incredible amount
of time and effort to dot the i's, cross the t's, and resolve
all the issues, and it's amazing what you can extract out of
volunteers. So again, thank you on that, and I'm very pleased
to see the amount of progress that's been made.
So
today we're going to start going through those and at the end
of each of the briefings we'll entertain motions to adopt those
sections and to approve those sections as part of the final report
to the EAC.
Now
I'll just also remind any of the public that they are free to
make comments and position statements regarding this work on the
NIST, at the votenist.gov site where they will be posted, and
all of the previous comments are already posted there.
You
can go directly to votenist.gov, which has all of the information
from the TGDC including the public comments that have been provided
and I recommend everyone look at those. They're actually quite
interesting.
So
at this time I would very much like to welcome Chairwoman Davidson
for some comments?
COMMISSIONER
DAVIDSON: Thank you. Good morning everybody.
And
I want to begin by expressing my sincere appreciation to the TGDC
and to NIST for their tireless efforts in developing the NIST
set of voluntary system and standards.
The rewrite of the VVSG was no small task but we're pleased
to know that the work that you have done on this document will
serve to improve the competence in elections and create a stronger
democracy for all.
Also
I want to take this occasion to thank Dr. Jeffrey for his leadership
of this committee. Under Dr. Jeffrey's chairmanship this committee
has written the most comprehensive and thorough set of voting
system standards ever.
Dr.
Jeffrey, your leadership and guidance will be missed. I thank
you for your dedication and service to this community and this
committee, and I wish you the very best in the future.
I'd
like to take time to go through once we get the VVSG at the EAC,
when it's delivered, and I especially think this is important.
Now
what I'm really talking about is a draft that we have put together
because the NIST iteration is a complete rewrite of the 2005 VVSG.
We
do recognize at the EAC that it's the responsibility to conduct
a deliberative and thorough review of the document, providing
as much time as necessary to receive comments and testimony from
all major stakeholders before finalizing the NIST iteration.
Keeping that in mind, the EAC has developed this draft and as
I said, this draft can change, so definitely this is a draft.
It could take longer then what we have planed as sometimes things
do, but I'm going to go through it as well as I can.
The
EAC will receive the TGDC recommendations from NIST and the TGDC
we anticipate in September of this year. Obviously this is next
month.
From
that time we feel it will take us about 14 days to put it in the
Federal Register, and we are going to put that in the Federal
Register for 120 day comment period, not 90 days as the law requires.
That time will take us from October we anticipate, somewhere into
probably next year, about February, and there's a lot that we
feel that has to be done during the present time.
First
of all, the TGDC has agreed that they will give training to a
select group of individuals that's been chosen by the Board
of Advisors and the Standards Board to receive education, thorough
education, from the NIST individuals in October. October is
when that's been set up so that is an ongoing training session
for them.
Also during that time we plan on holding public meetings and in
these public meetings we feel that we need to do several things.
We
need to try to get a handle on when we really talk about it, when
we get into it, the cost of actually what the laboratories cost
will be in testing. NIST has said they would assist in this
process.
As
we go through everything that we are gong to tell you about, not
only the laboratories but then we also need to hear from the manufacturers,
field them and talk to them about in a public meeting obviously
also, how long is this going to take once you know what the VVSG
is. How long is it going to take you to design your new systems,
to build them, to have them tested?
You
know, we may not know the exact dollar amount in any of these
areas, whether it's testing or building the equipment, but what
we will hopefully know is how much more time because of the increased
testing will it take the laboratories to actually get the documents
through or the equipment through for testing.
Also
we have to meet with the advocacy groups. We feel that we need
to bring academics in to vet what has been done from the TGDC
and we need to educate and we need to hear from our election officials.
So
during these public reviews we feel that that information will
be very useful and will also go into the comments in the comment
period.
At
the December meeting we will have a Standard Board meeting and
that is a public meeting, Standard Board and Board of Advisors
again where NIST will do an education to the whole group and hopefully
they will have comments to add to the review period.
At the close of the review period we hope to have some input from
all these different groups that will be very important, and so
we're going to turn around and return that back to the TGDC
for their comments and discussions on it.
We
feel that that will be done sometime for the TGDC recommendations
in probably May to July of next year.
After
that then the EAC will receive the document back from the TGDC,
we'll say in October or any time up to July to October, and
after that period then we are going to put it back in the Federal
Register for another 120 days. That 120 days is the second period
that people can make comments after the TGDC has reviewed it after
the first comment period.
Again
we probably will be meeting with some of our laboratories, the
manufacturers advocacy groups, election officials, and also the
academic community.
This is a very important document. We're trying to give it
the time that it really needs to be put in place to vet the document.
We
feel that the close of the period of the comments of the last
time will be sometime in February of '09, so you can see that
it's taking a great deal of time to actually put this through
and get it vetted properly as we feel it should. It deserves
that, and as we said, with this being a complete rewrite we have
to do it and do it right.
The Commission then reviews it, we feel like in May through June,
and then the guidelines would be published in the Register the
final time as the law requires, and that would be somewhere in
the neighborhood of June or July of '09.
So
hopefully you understand that we heard from our election officials
that they really feel that they've got an election upcoming,
they've got presidential elections that are coming on in January
and February so '08 wasn't the best time to really vet the
document as much as they would like.
So
we wanted to put it out initially upfront as soon as possible
for that 120 days. We feel it's important to hear from the
stakeholders, bring it back to the TGDC, and obviously meeting
with all the individuals so that we can get more input on how
long is it going to take to design this equipment, any type of
a handle of what the cost will be.
So in moving forward we are trying to associate everything that
is in detail with this document to try to come to some type of
conclusion to have the very best document to move forward with
our voluntary voting guidelines.
So
if there are any questions I would be more then happy to try to
answer them but I know you want to get on with the meeting, but
I do want you to know that we do see this as a very important
start. This is the beginning. We see once we receive the document,
we have to follow through the process very carefully. Thank
you.
DR. JEFFREY: Thank you very much, Deneta. Are there any questions
from any of the TGDC members? Okay, thank you very much.
I
think we all recognize that this is just the beginning of the
path and obviously I speak for the TGDC, that as comments come
in, if there are things that we can do to help assess and understand
the implications, I think there's a lot of interest both within
the NIST staff and the TGDC members to help make sure that this
is the best product possible. And it's easy for me to say
since I'm leaving, so I get to volunteer everyone else.
COMMISSIONER
DAVIDSON: We're going to look you up wherever you go.
DR.
JEFFREY: So at this time with that threat, I would like to call
on Mark Skall to provide us an update as to, what have you done
since we last talked to you?
MR.
SKALL: Boy, thank God he's leaving. I'd like to start
really by expressing my appreciation and giving thanks to the
TGDC as a whole, especially to our Chair, Bill.
I
think those of us who work here know how much the NIST Director
has on his plate. If you ever look at his calendar, we are all
just amazed that he actually can make it to work the next day.
Sometimes
it can look overwhelming, and for Bill to stay up to date on the
voting activities, and when we brief him he's just right there,
he asks such good questions and he knows exactly what we've
been doing. I don't know, he must do it in his sleep. It
must be one of those machines that goes on.
So
I certainly want to thank him and I want to thank the whole TGDC.
I know that this is a difficult task. I know there are some
contentious issues and there were very short time constraints
for much of this.
Additionally,
I know almost all of you, if not all of you, have full time jobs
and you do this really in your spare time so this truly is a labor
of love.
And
I really do appreciate getting to know all of you and I certainly
want to thank you for your technical insights, your dedication,
and your patience in working with many disparate groups, including
those crazy NIST scientists. So thank you.
Okay,
I'd like to now speak about the voting activities and talk about
what we've done since the May meeting.
I'd
like to preface it by saying this is really a historic day I believe
for the election community and the public.
In
the next few hours the NIST staff will summarize various sections
of the voluntary voting system guidelines and answer any questions
you have for a document that we believe is essentially complete.
This
document is the NIST iteration of the voluntary systems guidelines.
All that are left are some editorial corrections we need to make,
especially to non-normative sections like the introduction.
So
you will be voting as Bill said on this document before you, and
the document I believe really will have a profound affect on impacting
the next generation of voting systems in reliability, in security,
in accessibility, and usability. So I think this is really an
historic day.
Okay,
so I will speak about the activities since the last meeting in
May, which of course included continued research and the drafting
of the VVSG in coordination with the TGDC, and I'll talk a little
bit about the focus of the meeting and the strategy and the agenda.
So
in general we responded to all of the TGDC issues and comments,
and Human Factors and Privacy, better known as HFP.
We
completed the research for the usability performance benchmarks.
Although it's completed with respect to the delivery of the
document to EAC, we will continue this research to validate many
of the performance benchmarks we put in. This is a completely
new area, we're breaking new ground, and we want to make sure
that we continue to do research in this area.
We've
also made final updates to usability and accessibility requirements
in Core Requirements and Testing or CRT. We've completed the
reliability and accuracy benchmarks research and made final updates
to EAC, and quality requirements.
Security and Transparency or STS, we probably made more updates
then the rest of the sections and it's really, really in good
shape now. I think it reads really well but it has probably
some more changes then the other two sections.
We
re-drafted the E-poll book and externally networked kind of activity
requirements, and we've made updates to open-ended vulnerability
and independent voter verifiable record related requirements.
As
far as coordination with the TGDC, we've had 46 teleconferences
since the last meeting in May. We've made numerous revisions
and updates to the draft material and of course we've had many
individual discussion with people on the TGDC, either in groups
or individually.
The
current draft bill as of August 7th I think is an incredibly impressive
document. There are right now 1170 requirements, 570 pages,
and a new format for improved readability.
I
have been speaking at some conferences over the last few weeks
with voting officials who had some concerns about plain language
and readability and I have urged them to actually read this document.
Many have then looked at it and have been really surprised at
how well it reads.
It
is a technical document. At the end of the day these need to
be precise, testable, unambiguous requirements and those are very
detailed, but certainly all the introductory material and all
the material up until that point is really very readable and I
want to commend the staff and the TGDC for what I consider to
be really an extraordinary document.
The
final technical editing we believe after this meeting will take
approximately two to three weeks to get it to the EAC.
So
again the aims of the meeting, this is the final TGDC meeting
before the VVSG delivery to EAC in September, so we would like
to get approval for the VVSG and have final editing instructions
for NIST staff, and again it will take a few weeks to do these
final edits.
So
we're going to talk about the changes we've made since the
last meeting. These will be high-level summaries. Then of
course we will discuss whatever remaining issues that you would
like us to discuss, and then ask for final approval and a resolution
at the end of each presentation.
And
the order will be STS first, Security and Transparency, Core Requirements
and Testing, and Human Factors and Privacy. I think your agenda
probably lists Nelson as speaking on STS. Barbara Guttman will
speak on that in his stead. That's all I have. Thank you.
DR.
JEFFREY: Thank you, Mark. Are there any questions for Mark?
Okay, great.
At this point I would like to ask John Wack to then present an
overview of the VVSG document structure. So, John.
MR.
WACK: Thank you. I just have a few slides to present
and what I'll do essentially is just go over some of the major
changes and describe at a high level what we have been doing over
the past couple of months and some of the final document production
plans.
And
we also received a couple of comments, a couple of questions from
Secretary Gale that we thought would be good to discuss as well.
And
before I start I would like to add my thanks also to Commissioner
Davidson. Working with her and the EAC has been very much a
pleasure.
And
I think almost exactly 23 months ago we actually started working
on what we call VVSG 2007, and that was Dr. Jeffrey's first
meeting out in Boulder. That was a memorable meeting in many
ways. So again I think his support has been vital and this project
is a success because of him.
Okay,
with that, in a nutshell everybody here has been working very
hard and very diligently. We received a number of comments from
you at the last TGDC meeting, directly at the meeting. We also
received a number of comments as well in the several weeks thereafter.
Everybody
took those comments very seriously, responded to all of them,
and then we had a number of telecons afterwards so we feel that
we have addressed those in the document before you. We've
made the changes you have requested.
We
did a number of other edits simply just to make the document more
readable, less daunting. Many people who looked at this we think
may have been put off by some of the previous versions because
perhaps it was a little too complex looking, looked a little bit
too complicated. We hope the new version looks better. We
put a few other things out there on the website as well, spreadsheets
of requirements cross-referenced.
So after today, presuming that we have just an absolutely fabulous
meeting and walk out of here with smiles on our faces, we expect
that we've got a few more weeks of changes and reviews.
Commissioner
Davidson talked to you a little bit about the public review process
and in a way I think it's great that the TGDC version of the
document is the version that's going out for the public review.
At
the same time it puts more pressure on us at NIST because everybody
is going to be looking at something that we directly produced,
and it takes a long time and it is very difficult to go through
the whole thing and make sure everything reads correctly, but
it behooves us to do that so we'll be very busy after that.
And
we plan to put out the two versions. One is the final PDF.
We'll have that tagged for accessibility. The PDF version
before you has a lot of hypertext links in it but I think everybody
probably agrees that PDF is best for printing.
And
so for onscreen viewing we're going to do an HGML version and
we'll write other introductory material and so on and so forth.
We
would also like to put out a small database of requirements with
requirement language and fields in there that essentially would
make it easier to find a number of different requirements that
apply to different devices and so on and so forth.
With
that, that is all I have there. I wanted to bring up a couple
of things here. You know, you at home or wherever don't necessarily
need to look at this, but I am bringing up a requirements matrix.
We had three of these out there on the website underneath the
PDF for the VVSG itself.
And
Secretary Gale sent in a couple of questions. What external
resources such as ISO, IEEE standards, so on and so forth were
generally used as a basis for the requirements developed for the
subcommittee report?
And,
you know, apart from the telecons, several public hearings at
NIST, we went to a cost of testing meeting with the EAC, a number
of conversations with vendors, comments to the TGDC that we posted
on the website. We've had a lot of consultation with various
different groups.
A
lot has happened over these past two years. Voting has been
pretty controversial and we have tried as much as possible to
stay in the middle and talk with as many different groups as we
can, and get as many different sides of the story, and learn as
much as we can. So we feel that we have done our work.
And
up here you'll see that each of the requirements we have in
these matrices are cross-referenced
to a number of different sources. I'm scrolling down here
rather quickly. You'll see VVSG 2005 appears prominently,
but there are a number of other documents, a number of other standards
that we've gone into as well.
Has
there been adequate opportunity for the usual peer review of the
underlying assumptions used to develop the requirements? I guess
I kind of answered that in the first answer. We have consulted
quite a bit with a number of different groups and we feel that
we have listened to many different sides of the story so I guess
the answer to that is yes.
With
that I think unless there are any questions, I would like to go
to another document which is the VVSG itself, and so what I'm
displaying up on the screen is the PDF of the VVSG, and the people
talking to you from now on will probably be talking about certain
pages or certain requirements so I want to make sure we are all
looking at the same documents.
So
the TGDC members, you've got some documents mailed out to you
on CDs. There is the PDF on the website. You can also download,
or if you're looking at the printed copy you should all make
sure that the first page says draft, August 7th.
DR.
JEFFREY: John, there's a question. Secretary Gale, you have
a question?
SECRETARY
GALE: Yes, I do, Dr. Jeffrey. With regard to the new document
John,
I want to compliment you and your team. The latest revisions
have vastly improved the readability of the document and we're
all very grateful for the hard work to accomplish that.
Of
course I grew up in a paper era so in going through 580 pages
with the Table of Contents it's great but of course I regretted
the lack of an index. And my staff who are much more digitally
based then I am said well, that's an issue of searchability
and on a digital document it's not an issue of a paper index.
And so I obviously understand that now.
And
I would like to encourage you at NIST to continue to develop additional
tools such as the searchable database, which will certainly increase
the accessibility to that document if you're looking for specific
subjects or topics. That would of course serve the purpose of
an index but would aid the election community and the public in
finding specific things that they're looking for. Thank you.
MR.
WACK: Okay, thank you, and we'll take your comments
very seriously.
To make sure we're starting off all on the same foot, the same
page, again I just want to make sure we're all talking about
the draft, August 7, 2007.
And with that I will turn it back to Dr. Jeffrey and talk to you
all later. Thank you.
DR.
JEFFREY: Thank you very much, John. With that, it's time
to roll up our sleeves and start getting into the meat of the
issues. So first Barbara Guttman will be talking about the Security
and Transparency sections. Barbara, over to you.
MS.
GUTTMAN: Thank you very much. I'll roll up my sleeves here
because my jacket actually is lightweight and rolls up.
So
the first thing you'll notice of course, I hope you notice,
is that I'm not Nelson Hastings, I'm Barbara Guttman, and
truthfully I'm not even an electronics engineer.
So
that said, let me present what STS has been doing on its portion
of the VVSG, the portions it's responsible for.
I'm
going to try to give out as I'm talking either slide numbers,
although I'm not sure if the version you all have is page numbers.
I'll also try to give out some slide names to help to follow
along, but if you get lost please speak up.
So
the first two slides that are called agenda just list all the
sections for which there were changes. Just one thing I did
want to point out is that there was some reorganization of the
document and that each security topic used to be its own chapter
and now there are two chapters devoted to security.
The
first is called Security and Audit Architecture and it contains
three security topics, Security and Audit, Electronic Records,
and IVVR, which if you're not quite familiar with the term IVVR,
wait, I'll get to that. And the second one, which is General
Security topics, that includes all the other security topics.
So I just wanted you to make sure you were aware of that before
I went over them.
So
that's what I'm going to start with, a very easy topic first,
which was some general modifications we did throughout all the
security sections, which was primarily harmonizing language.
There
was a lot of harmonizing language to make sure it was consistent
with the definitions, and we also moved all the documentation
material to the part of the standard related to the documentation
so that was really pretty straightforward.
Now I'm going to move on to Security and Audit Architecture,
and once again this one also had general changes that were primarily
scoping it to fit into these new chapters, which I'll talk about
as I go through them.
The
first thing we did was to eliminate a lot of duplicate material.
There was a lot of duplicate material as we were developing things.
When you're writing it it's a lot easier to kind of put it
in more the once everywhere you need it and then when you're
done scope it back out.
There
was a lot of explanatory information about election administration
issues, which we had used to explain the requirements but we now
deleted that material as well.
Also in the audit section, I'm now in Audit, Steps Removed and
Retained, as you recall in the Audit Architecture section we describe
what kind of audits people might do so we could tell you what
kind of requirements the system needed.
But
some of these didn't actually result in actual requirements
for either the system or the documentation so we went ahead and
removed those also.
So
the next section is Electronic Records. This one actually had
probably the most extensive set of changes having to do with harmonization,
clarification, and removing duplication, although actually the
actual content didn't really change that much but it looks really
quite different from how it looked beforehand. But that was
primarily to better coordinate with CRT and the work they do in
requirements for reports versus audit records.
Now
I'm on to a section that really is -- there is sort of kind
of a major difference from how the May draft looked, which was
independent voter verifiable records.
In the May draft we used the term voter verified paper records
and what we did is we abstracted this concept up one level to
recognize that it's really about an independent voting record.
And
so we looked through the requirements we had for IVVR and we rewrote
them to address any kind of independent kind of record, and in
this way you could have non-paper based systems that could conform
to the VVSG. And this is consistent with TGDC Resolution 66.
The (unintelligible) resolution did call for independent voting
records. So that's kind of an important change.
This
change itself did not actually then result in changes to the VV-PAT
or the P-Call section. But if you go to the next slide, the
second slide called independent voter verifiable records, I do
want to point out because we were working on this this summer,
STS has one change that is beyond what is in the draft you have
in front of you, which if you look at the slide, it's slide
10, the second IVVR slide, we want to add one additional requirement
to clarify that all these requirements actually have to refer
to the same record.
So
it doesn't particularly change the intent but I wanted to call
that specifically because it's different from what's in the
document you're voting on.
We also did make some clarification to the VV-Pat section which
we clarified that paper records have to use OCR fonts, that you
have t use a code book to interpret the paper record, that we
tried to make some of the perimeters more tunable by election
officials, and hopping over to the next slide, to clarify how
you do verification for cut sheets VV-Pats.
We
also made one change to the precinct optical scan, which was before
we had a should requirement to support batching of paper records,
but STS decided to remove that because it was actually a little
bit too complicated.
So
that's what I have to say on IVVR.
And
now moving on to the slide called Crypto, which is cryptography,
which is slide 13. In Crypto we actually only made editorial
changes but there was an important question that was asked and
we wanted to make sure everyone knew the answer to this, which
was if you have a machine like a DRE that's actually supporting
multiple precincts, are you going to end up needing more the one
election key, and the answer is no, you do not.
So
we wanted to make sure that was shared with you, and if any of
you have other questions about the cryptography chapter we are
pleased to answer them here at NIST, and feel free to just send
us an e-mail and ask for a little tutorial because it is a little
bit complicated.
Another
important change was in the set up and inspection chapter which
is, as the architecture for the VVSG evolved, and as we developed
the requirements to support software independence, and as we developed
the system integrity management chapter, we realized that we do
not need the requirement for a trusted interface anymore, that
the security goals that that requirement was designed to meet
are already being met by other methods.
So
STS decided to remove it and that's a sort of important change.
And when we made that change we were then able to refocus this
chapter more on software inspection, and then also actually address
software installation which used to include software distribution
and the distribution were really requirements that were primarily
for the test labs. So we moved that to the section that addresses
test labs and that helped us to just refocus that chapter.
Now
let me get to access control. There was a technical issue with
access controls which is, most people are familiar with operating
systems that are really fairly robust but often in election systems
you also have systems that are built with what are constrained
operating systems and we wanted to make sure that we addressed
those within the access control chapter.
So
we were working on this and we came up with some ideas for how
to best address this, to make sure that -- basically election
management systems need to have robust access control, but capture
devices can actually do what we call role base control.
But
here's another place where we continued working on it after
this draft was done so we actually have a change that we want
to make to this draft.
It
is on what is called slide 17, which is the second access control
slide, list the language we want to add. This further clarifies
what I just explained but this is the actual language we wanted
to use and I wanted to call that specifically to your attention.
And there is actually a third access control slide, which actually
says we actually did a lot of work in making sure the scoping
and language was correct.
Moving on to the system and activity management chapter, I already
extolled the virtues of it. Mostly in this chapter we actually
just did some re-scoping and harmonization.
In communications we actually harmonized this with the access
control chapter so that neither chapter allowed for remote administration
of systems. We had a little disconnect there so we fixed that.
System event logging, we also addressed this issue of what happens
when you have a constrained operating system and we added some
material to address that.
In physical security we implemented the TGDC decision about locks
and you can see the language. It's actually off of slide 22
called physical security. You can see the language we used for
that.
And then in the pre and post test slides we clarified when labs
do what we call the Test Lab Bill which was previously known as
the Witness Bill, and we also harmonized with CRT about how the
test labs treated unmodified software which is they have to acquire
an independent copy of it to be used in their testing. And that's
pre and post testing.
So
I'm now on to open ended vulnerability testing, which was an
area that was not terribly flushed out in the May draft so we've
added a lot of material about open ended vulnerability testing
to address the scope, focus, and priorities of the team, the team
competition, the rules of engagement, flip over to the next slide,
and the level of effort and the reporting requirements.
And that's the end of my presentation so I'll pause for questions.
DR. JEFFREY: Are there any comments or questions for Barbara
on the STS section?
SECRETARY GALE: Dr. Jeffrey, this is John Gale.
DR. JEFFREY: Yes. Go ahead Secretary Gale.
SECRETARY GALE: I stepped out for a minute so I didn't
know how to que in so I hope this is acceptable.
DR. JEFFREY: Absolutely.
SECRETARY GALE: With regard to the cryptography, the imbedding
of a chip, I understand that that is current technology. The
military uses it, the banking industry uses it, but it's a whole
new concept for any election equipment hardware in terms of imbedding
an encrypted chip and what I'm concerned about I guess are what
are some of the consequences if those embedded chips fail.
So
I'm interested in knowing the failure rate of such chips in
terms of embedding that in equipment, either the small equipment
like VREs and optical scans, precinct equipment. Do you have
any sense of what the possible failure rate is?
MS. GUTTMAN: Let me address that first by addressing one of
your assumptions which is you described how embedded chips are
becoming pervasive in various fields like (unintelligible).
They have actually just become pervasive in all IT. This is
how all PCs are going to come in the future. This is part of
just where the entire industry is moving.
I don't have specific failure rates but I suspect it's quite
low. I am looking over at the Crypto team.
DR. RIVEST: This is Ron Rivest. Could I --
MS. GUTTMAN: Well, why don't I look over at the Crypto team
on the phone.
DR. RIVEST: This is Ron Rivest. Secretary Gale, this is
a great question, but in terms of technology this is absolutely
routine technology. This is a digital integrated circuit just
like all the other integrated digital circuits on the motherboard
there and there is no reason whatsoever to expect that there should
be any impact on reliability.
Nonetheless, that said, of course this will go through the usual
testing and if vendors have any particular insights into those
issues that I don't or others do, it would be good to hear that,
but I don't expect any issues on the reliability side.
I think issues that arise here are more just the management side,
and making sure that all these chips are (unintelligible) where
they are and what keys they're using.
SECRETARY GALE: Thank you, Ron. I appreciate that. This
is Secretary Gale. And you understand my concern because with
the embedded chips it may not be cost effective if there is a
failure rate such that there can be reasonable expectation for
a precinct to have a piece of equipment fail, and then the need
to replace that equipment, and the flexibility needed of election
officials to be able to move equipment once it is precinct specific
with an embedded chip. That failure rate has a big impact on
election administration.
Secondly,
it also has a big impact on cost and that's who's responsible
for repairing that or replacing it if it even could be replaced
in either precinct optical scan equipment or in the central scan
equipment which is much more expensive to buy.
DR. RIVEST: This is Ron Rivest again. Yes, those are good
points and we'll have to see how this sorts out, but my belief
is there is absolutely no reason to believe that this should have
any impact on the reliability of this equipment. As I said,
these are standard integrated circuit parts.
SECRETARY GALE: Thank you.
MS. GUTTMAN: Any other questions?
MS. DAVIDSON: This is Deneta Davidson with the EAC. When you
were going over the independent voter verification record, and
it kind of went a little fast, and this is one of the areas our
election community has really been very interested in.
Right now what we have is paper, and I know that we've discussed
in the IVVR, does that still require paper -- but I know it doesn't
quite require paper, but can you give us some examples of what
-- I mean I know there are other things that could be developed
in the future but is there something that you can kind of go over
and just --
(Tape
Interrupted While Changing Sides)
(END
OF AUDIOTAPE 1, SIDE A)
(START OF AUDIOTAPE 1, SIDE B)
MS. GUTTMAN: There really isn't something right now
that's an obvious solution to can you have a non-paper IVVR
system, but just because there isn't one today doesn't mean
the cleaver folks out in industry and academia aren't thinking
of cleaver things and that's what the opening is for. It's
for things that people haven't really figured out yet.
And
there's such opportunity. There are just so many great ideas
out there. And I know some of the people from your office went
to like Vote.com and places. People are thinking creatively
out there about better ways to do things and I personally find
that very exciting.
DR. JEFFREY: Any other comments, questions? Okay, hearing
no other comments or questions, I would actually --
MS. GUTTMAN: It goes back to John Wack.
DR. JEFFREY: Okay, the Chairman is being corrected. It goes
back to John Wack. John.
MR. WACK: Thank you, Barbara. It's back to Wack.
If I can grab that from you, just to break things up a little
bit, because I have a copy of my slides and other people don't.
I'm
just going to go through parts of the document here and quickly
I'm just going to add to Barbara's answer by going to a particular
page in the introduction. I'm proud of this picture because
I drew it myself and I'll try to blow it up here. I'm on
page nine of the introduction.
So basically the way things work, the conformance clause section,
chapter two of part one, the conformance clause chapter describes
in a sense what's needed for voting systems to conform to the
VVSG, and to make it clear in that section we say that voting
systems --
DR. JEFFREY: I'm sorry. Whitney, I believe you've got
a comment or question.
MS. QUESENBERY: I do have a question. Chapter nine, page
nine -- I'm sorry what page is that?
DR. JEFFREY: Whitney, could you hold on just a second while
we turn up the volume? We're not hearing you. Okay, could
you try again, Whitney? Sorry about that.
MS. QUESENBERY: This is Whitney Quesenbery. I asked that
when we (unintelligible) documents that we give not only the internal
page reference but the PDF file page so that those of us who are
behind the (unintelligible) without any visual reference at all
can keep up with you. You turn to it, then you tell us where
you are, and you immediately begin speaking.
DR. JEFFREY: So you're looking for just enough time to catch
up, is that what you said?
MS. QUESENBERY: Yes, that's (unintelligible).
DR. JEFFREY: Okay, could you repeat what page you're on and
then after that let's add a 15, 20 second pause to let people
actually flip through to get to the right page.
MR. WACK: Okay. On the PDF file I'm looking at page
number 59, and in the document itself if you're just looking
at the page numbers at the bottom,
I'm
looking at the introduction on page nine.
And the reason I'm looking at this particular slide is just
in a graphical way I wanted to describe the changes and the clarifications
we made to the conformance clause, which is simply to make clear
that systems that meet the definition of software independence
can conform to the VVSG, the draft VVSG, and that we have two
methods for that.
And
to make it clear, one for systems that use records, independent
voter verifiable records, and the other the innovation class.
And
to iterate what Barbara was saying, new types of systems using
new forms of independent voter verifiable records have requirements
already in VVSG, therefore they could conform to the VVSG. They
would not have to use the innovation class.
So the innovation class is really for new innovative voting systems.
Could be for end-to-end cryptographic systems, things of that
sort. Systems that use independent voter verifiable records
do conform to the requirement in the VVSG. So that is the clarification
we made there.
I think I will go back to the slides at this point because I think
probably it's easier for people to follow along there, and I'll
talk about two other changes and these are really in the core
requirements area, not in the STS chapters.
So the first one has to do with data export, and in chapter six
we used to have a section called integratability. We expanded
this to talk about data export as well.
Integratability in a sense kind of means no huge barriers to making
two components integrate with each other and sort of on the road
to being interoperable.
So
we have general requirements in there basically saying that voting
systems shall be integratable, voting devices shall be integratable,
let me correct myself.
But also we needed to handle the question of what format electronic
records when they're exported should be in, and we made it clear
that they should be in a publicly documented open non-proprietary
format.
Vendors
shall include a source code program that shows how these records
can be read and we do have a requirement in there that essentially
is a strong recommendation that a common consensus based format
should be used, that we have followed some proposal by an IEEE
subcommittee, and also the oasis election markup language are
two common consensus based formats that could be used. So these
are the requirements there based on comments from the last meeting.
And then in the other area, which is chapter seven, which is requirements
by voting activity, there we for the first time introduced requirements
for electronic poll books.
We had a subsection there that was on ballot activation and in
an effort to more clearly address privacy related requirements
and requirements related to networking for electronic poll books,
we broke these out into two separate areas and added requirements
essentially to strengthen privacy because on an electronic poll
book it's basically a voter registration database and we want
to make sure that voter information does not somehow leak over
to a DRE, or a VD-Pad system, or whatever system is being used,
and we wanted to insure that records can't be aggregated also
to violate privacy.
In
other words we don't want records from a DRE combined with records
from an electronic poll book put together to show how people voted.
The other thing is that we added requirements to permit electronic
poll books to make external connections to remote voter registration
databases. We added some requirements to improve security.
In
other words there must be some sort of a firewall, things of that
sort. So those are the two areas that we added outside of the
STS chapters. And that is it. With that, are there any questions?
DR. JEFFREY: Any questions or comments? This would wrap up
the discussion the Security and Transparency. Yes, Secretary
Gale, go ahead.
SECRETARY GALE: I'm sorry. The lighting I guess for
getting ourselves up on the que --
DR. JEFFREY: You just popped up.
SECRETARY GALE: Okay. May I proceed?
DR. JEFFREY: Yes, sir, please do.
SECRETARY GALE: Thank you. I had a couple questions for
John.
Obviously your team and subcommittee has worked very, very hard
to develop a voting system that is essentially impervious to attack
and it's called the Gold Standard by people in the election
community, that this is kind of the ultimate set of security standards.
Do
you consider it to be impervious to attack by just current standards
or do you consider it to be impervious to attack from methods
of attack developed in the future? In other words, say maybe
now under the standards that are being proposed is it safe for
(unintelligible) of the future under computer industry standards?
MR. WACK: Well, it's sort of a short and a long answer.
The short answer is no, security you can never guarantee that
it will be impervious to attack and security in a sense always
plays a catch up game and you tend to know about known threats,
threats out in the future, you can take a guess at.
To a certain extent in security, and I think what we've tried
to do is where it seems appropriate to do so, we've tried to
over build.
So
for example, the designers of the Brooklyn Bridge didn't let's
say know enough about engineering and had the materials to build
a modern day bridge, so they over designed to a certain extent
and today it carries lots of traffic and it's a strong bridge.
And in the same way for example with electronic poll books, our
original proposal was that it be best they not connect up to external
networks. We know that external networks are difficult to secure.
There will be new sorts of problems, new sorts of vulnerabilities,
things of that sort.
So given that there was a need to actually hook up to external
networks, we then throw other things in there such as a firewall
that we hope would block out most traffic, most threats, and only
allow the information in that needs to come in.
But the answer is no, security is always a catch up game. At
the same time we do think we've done a good job. We're very
familiar with many of the threats out there affecting IT technology,
especially in the network area
so --
DR. JEFFREY: And John, I think Ron Rivest would like to add
something. Go ahead, Ron.
DR. RIVEST: Yes, thank you. Ron Rivest. Secretary Gale,
that's an excellent question and John's response is a good
one.
I'm
hearing echoes, can you hear okay?
DR. JEFFREY: Yes, you're coming across fine.
DR. RIVEST: Okay. I'd like to amplify that a bit. I'd
like to stress that the steps that we've taken here in the Security
and Transparency subcommittee in no way cab be viewed as the last
word or building a impervious system.
These
are reasonable security steps to mitigate many of the known risks
and to move us forward in the security catch up game as John calls
it, but there are many threats that are not answered by this.
There
are things that need to be answered by procedures, and there are
on the horizon, new classes of voting systems that we hope the
innovation class will encourage our vendors to come forward with.
Some
of those known as the end-to-end category of voting systems have
properties that are not exhibited by any current systems and allow
voters to achieve even higher levels of competence.
So
this is a major step forward in the security of voting systems
but it's in no way a gold plated solution or one that should
be considered impervious. It's a set of reasonable measures
to achieve a significant improvement in security.
SECRETARY GALE: Thank you, Dr. Rivest. If I could ask
a follow-up question. In light of the fact that these standards
will become official standards for this industry to design and
build, again until maybe 2009, 2010 or later, is there enough
I guess what John Wack was saying, enough design built into these
standards to carry it to that date and beyond that date? In
other words, is there kind of a predicable future of this covered
technically for the next five or ten years in terms of security?
MR. WACK: I'm pausing. I wasn't sure whether Ron
wanted to answer that.
DR. RIVEST: Why don't you go ahead, John.
MR. WACK: Well, we think yes. One of the main reasons
is because we have moved towards software independence in this
particular version, primarily because of the difficulty of testing
systems.
As
time moves on and we get a better handle on how to test very large
complex systems such as voting systems, we think that we will
be able to keep pace with new threats, new problems.
We
expect to have a very good set of test sweeps out, roughly 2009,
2010, that will work with the VVSG, and in good testing that's
where we think we will catch most of the problems. So yeah,
we do think that we are moving along, keeping pace.
DR. JEFFREY: Secretary Gale, I believe you've got some additional
comments.
SECRETARY GALE: Yes, Dr. Jeffrey. I don't know if that's
my system or somebody else's.
DR. JEFFREY: It may be your system but we're fine with it.
SECRETARY GALE: Okay. Well, thank you. I appreciate
you addressing that issue.
I
also had some questions on innovation class. It's been a very
exciting addition to this set of iteration and I compliment Dr.
Rivest and the committee for incorporating that, but I was really
concerned in looking over not only the short resolution but the
broader one as well because the standards that the review committee
is supposed to use seem very broad, very subjective, and then
there's really nothing that defines that review committee or
any suggestion on that.
Now
maybe that's totally left to the EAC but with such a subjective
and broad group of criteria it becomes highly critical that the
review committee be a committee of almost a blue ribbon panel
that are very carefully selected and that are going to be able
to take those broad standards and convert them into objective
criteria. So that was one of my concerns.
I
like the idea, but however that review committee is defined is
going to be critically important, or whether the process is stifled
before it begins, or whether it's going to be open enough to
allow vendors to move toward the prototype stage. Does that
make sense?
DR. JEFFREY: Whitney, do you have a comment on the question
or do you want John to answer the question first?
MS. QUESENBERY: No, let John answer the question.
DR. JEFFREY: Okay, thanks.
MR. WACK: If I may, I was actually looking over at Commissioner
Davidson who wanted to add a few things.
COMMISSIONER DAVIDSON: I just felt like I needed to say that
the EAC really understands the concerns, and as we move forward
in this we feel that definitely we didn't want criteria put
in to place that might keep a manufacturer from designing something
in the future that might be the answer. Not knowing what technology
is going to bring.
Obviously
we know there has to be procedures and everything put into place,
but if it's not in the VVSG we felt that it would be better
being in a procedural document that has to be vetted obviously.
We
do very open processing in our office but we can change that as
we move forward because this is a brand new idea. We know that
it's going to take vetting. We know it's going to have to
have that blue ribbon committee, but I wouldn't want to pick
that today because there might be somebody to step forward in
the future that we think would be great.
So
we just felt that it shouldn't be in the VVSG, that it gives
us more ability and flexibility in the future to work and hope
they come up with a great product that everybody will like.
I
didn't really give you a lot of answer, but that's kind of
how we feel at the EAC, is I think we really want that flexibility
right now and being able to change the document without having
to come back to the TGDC and go through a vetting of a document
that would take possibly years to change something to keep up
with technology.
SECRETARY GALE: Thank you, Commissioner Davidson. This
is John Gale, Nebraska. I think that answers my question.
I
think what you're saying is the EAC will address this as a procedural
issue, maybe in a separate set of criteria other than the technical
guidelines.
COMMISSIONER DAVIDSON: Deneta Davidson again. And yes, you're
exactly right. I think if you look at what we have done with
our laboratories in setting up procedures that were vetted in
public meetings, you'll understand that we'll do exactly the
same thing with this type of moving forward with definitely the
innovation class.
SECRETARY GALE: This is John Gale, Nebraska. Thank you,
Commissioner.
DR. JEFFREY: Whitney, I believe you have a question or a comment.
MS. QUESENBERY: Yes, this is Whitney Quesenbery. I have
much the same questions about the OAVT, which we went over I think
in about 30 seconds. I know there's some questions about how
we --
MR. WACK: Well, fire away.
MS. QUESENBERY: Well, how the standards to which the OAVT
will be concepted, will be created, and I suppose the other thing
is (unintelligible) weeks is a long time and does anyone have
any idea what that costs?
MR. WACK: If I may , I'll ask Alyse Clay-Jones to address
that, who is really the primary author there.
MS. CLAY-JONES: Hi, I think that you had two questions for
us, one about standardizing the open ended vulnerability testing,
and the other about the length of time, is that correct, Whitney?
MS. QUESENBERY: (Off microphone). Yes, this is Whitney.
I mean the open endedness of one (unintelligible) brings up some
interesting curricular questions about the (unintelligible), and
the question that we talked about the last one is how do we decide
that something is sufficient (unintelligible).
MS. CLAY-JONES: I don't know if you had a chance to look
at the specific requirements that we've enumerated but we have
made attempts to set up some boundary conditions, some rules of
engagement for the OAVT team so that we get as specific as possible
about what it would take in order to actually fail based on the
open ended vulnerability test.
So if you take a look at the specific requirements you'll see
that there are some boundaries on the team.
MS. QUESENBERY: So this is new material that's been added?
MS. CLAY-JONES: Yes, ma'am.
MR. WACK: I'll just add my two cents to that too.
Sometimes I think of open ended vulnerability testing as a little
bit like network penetration testing, and network penetration
testing is something that evolves as various vulnerabilities are
discovered, and new threats and new techniques.
And
I think the VVSG has good requirements in it for the basis of
open ended vulnerability testing that deal with staffing and a
number of basic items that are kind of fair game for open ended
vulnerability testing, but it will probably be something that
will evolve as labs become better at it.
And
we would expect that labs would cooperate, share information to
a certain extent on vulnerabilities with voting systems to insure
that as much as possible one lab will do basically the same job
as another lab will do.
DR. JEFFREY: Are there any other questions or comments on Security
and Transparency? Okay, hearing none, I believe all of you have
copies of the draft resolution that outlines the specific chapters
and sections of the areas that we're talking about for Security
and Transparency. It's labeled as Resolution 6-07, STS VVSG
Sections, Final Approval.
And
for no other reason then perhaps symbolically, I'd ask whether
the co-chairs, Ron Rivest and Helen Purcell of the STS would be
interested in proposing this resolution.
DR. RIVEST: Ron Rivest speaking. Yes, so moved. I'd
like to cosponsor this resolution.
DR. JEFFREY: Helen, would you like to second it?
MS. PURCELL: Yes, sir I would.
DR. JEFFREY: Excellent. So let me read it for the record for
those who are listening in but not having access to the written
material.
There's been a proposal and it has been seconded. It says
the TGDC grants final approval for the Security and Transparency
sections, part one, chapter two, section 2.7, chapters four and
five all, chapter six, section 6.6, chapter seven, section 7.5.1,
part two, chapter three, section 3.5, chapter four, section 4.3,
part three, chapter three, section 3.4, chapter four, section
5.4 as part of its second set of VVSG recommendations to the Executive
Director of the EAC, subject to editing as instructed by the TGDC
at this meeting, and final review by the Chair of the TGDC.
I now recognize Secretary Gale. Have you a comment or question?
SECRETARY GALE: Dr. Jeffrey, I do have one and I guess it
concerns you. This resolution talks about final review by the
Chair of the TGDC. Obviously that currently would be you and
I'm hoping that would be accomplished before you were to retire
from government service.
I
certainly have fantastic confidence in your ability to bring it
all together and complete that final review but you start talking
about interim chairs or temporary chairs, that concerns me a little
bit.
DR. JEFFREY: I know that my staff, and we've chatted about
this as recently as yesterday, will be working as hard as possible
to have everything to me before I officially punch out and we're
certainly committed to do that.
Certainly what I would plan to do, and I would encourage that
if by some chance they don't get this to me by the time I punch
out, I think that the process would entail that any changes even
minor ones will be in change mode, I mean even happy to glad,
commas replaced with semi-colons, will be in change mode and would
be posted on the website prior to the signature, the Chair of
the TGDC signature so that everyone would get to see that.
Obviously
if there is anything more then a simple non-substantive change
that would obviously go back to make sure that everybody is on
board with that.
So
all I can do is say that the staff is committed as much as possible
to get to the end of the race before I do, and I certainly encourage
that.
SECRETARY GALE: Thank you.
DR. JEFFREY: Any other comments or questions? Okay, if not,
is there any objection to adopting this by unanimous consent?
Okay, hearing no objections then this is adopted. Resolution
06-07 is adopted by unanimous consent and my heartiest congratulations
to the Security and Transparency Subcommittee for a job well done.
With that, we are a little bit ahead of schedule. I would like
to actually if the Core Requirements team is prepared, I think
CRT is next, right? Yes.
So
I'll ask David Flater after he puts his jacket on, if he could
give us a review of the CRT section. So as soon as we get the
correct presentation up on the screen David will start. So for
those who are listening in, sorry for the time delay.
MR. FLATER: Sorry for the delay. This is David Flater of
NIST going to present about changes for the Core Requirements
and Testing Subcommittee.
In a previous presentation John Wack discussed two sections that
were previously tagged as CRT sections. This was the integratability
section, part one, section 6.6, and the ballot activation section,
which is part one, section 7.5.1. Those sections do show changes
that were made by the STO subcommittee.
Apart from those sections, all of the other material tagged as
CRT reflects neither changes in technology direction nor the addition
of any significant amounts of brand new material relative to what
was discussed at the May meeting and previous.
So what I'm looking at instead is a large number of minor changes.
In fact, because some members of the TGDC made a line by line
review of the spec, there were in fact 282 issues within the CRT
material.
However
as I said, these were all relatively minor and I can say that
the vast majority of these were in fact addressed head on or forwarded
to the those who can address them and the ones that remain are
sort of loose change if you will.
They
are sufficiently minor that if in fact we never get back to them
it won't be a tremendous loss but we do hope to clean up those
loose strings in the time remaining to us.
Next slide. Now I'm going to provide some highlights, the
most significant of the relatively insignificant issues if you
will.
I've
been advised both to site part and chapter numbers and also to
wait for people to be able to find those, so I am going to address
these bullets in the order that they appear in the document and
give people time to find those sections.
The first one in document order would be software engineering
practices. This is part one, section 6.4.1. This section benefited
from a very careful review in which folks went requirement by
requirement looking for issues and it was found that some of these
requirements were over specified, some were under specified.
There
were other minor adjustments and polishing that needed to be done
to these requirements so this section reflects a fair number of
those kinds of changes, but again, no major changes in technical
direction or huge surprises.
The next highlighted issue would be back in part three, section
5.1. As of May the section having to do with hardware testing,
otherwise known as shake and bake tests, have been unintentionally
omitted from the draft. That omission has now been corrected.
The significant portions of the material from VVSG '05 that
were to be carried over, plus modifications to that material that
had been previously discussed in CRT and with the full committee
now appear in the draft. Removed were some details that have
been considered more in the scope of test methods work to proceed
afterwards.
The next highlighted change would be part three, section 5.2.3.
This is a functional testing section in which the requirements
on volume testing appear.
The change here was to provide more perimeters for the volume
testing of optical scanners. There was some question left by
the requirements as they were previously drafted regarding the
practice of re-circulating paper ballots through an optical scanner
as part of the volume test.
Neither
extremes seemed appropriate here, neither requiring that all of
the ballots be unique for a huge volume test nor allowing unrestricted
re-circulation of ballots.
So a compromise was struck based on the number of paper ballots
that it was thought feasible to produce as part of a volume test
of the scope that was envisioned, and corresponding perimeters
now appear in those requirements.
Fourth highlight appears in part three, section 5.3.1. This
is the explanation of the test method used for benchmarking of
reliability and accuracy of voting equipment.
The test method itself has not changed since it was discussed
in May. What has changed is the explanation of that method.
The previous draft was found to be confusing so it's been rewritten
in different language to attempt to clarify.
Finally there's a highlight that I suppose is most applicable
to Appendix A, the terminology section, but also applies throughout
the draft, which is that frequently misconstrued terms have been
replaced with more explanatory wording.
We had numerous discussions with the TGDC telecons to clarify
these problematic terms and substitute terms that were found to
be more transparent.
Finally throughout the specification, as I said there were 282
issues, throughout the specification there were many minor changes
for those many issues to change this word to that, things of that
form to polish the requirements that had previously been drafted.
And with that I will conclude and ask if there are any questions.
DR. JEFFREY: Any comments or questions for the CRT or David?
Whitney, I'm sorry you just popped up in my machine. Whitney.
MS. QUESENBERY: (Off microphone). This is Whitney. David
the last slide, which is open issues and the notion about (unintelligible),
I just wondered (unintelligible).
MR. FLATER: We were going to skip that.
MS. QUESENBERY: Okay. Are these all final edit things?
MR. FLATER: Yes, those are the minor inconsequential issues
that I mentioned.
MS. QUESENBERY: Thank you very much.
DR. JEFFREY: Any other comments or questions? Okay, hearing
none I am going to again for symbolic reasons ask if the co-chairs
of the CRT, Dan Schutzer and Paul Miller would like to propose
a motion. There's a draft resolution 07-07 that's in the
handout, that if you want I'd be happy to read, but either Dan
or Paul, would you like to propose that?
MR. MILLER: This is Paul. Yes, I would like to propose that
resolution.
DR. JEFFREY: Thank you. So motion has been proposed. Dan
or anyone else would you like to second? Dan may have dropped
off. Would anyone else like to second?
MR. PEARCE: Philip Pearce, I'll second it.
DR. JEFFREY: Okay, great. So there's a proposal, Resolution
07-07 that has been proposed and seconded. Let me read it for
the record.
The TGDC grants final approval for the Core Requirements and Testing
sections part one, chapters one and eight all, chapter two except
section 2.7, chapter six, all, except section 6.6, chapter seven
all, except section 7.5.1, part two, all except chapter three,
section 3.5, and chapter four, section 4.3, part three, all except
sections 3.4 and 5.4, as part of its second set of VVSG recommendations
to the Executive Director of EAC subject to editing as instructed
by the TGDC at this meeting and final review by the Chair of the
TGDC.
So there is a motion and it has been seconded. Are there any
questions or comments? Secretary Gale.
SECRETARY GALE: Dr. Jeffrey, I do have a point of order.
On the first subcommittee report, I would simply request whether
or not anyone had any objections to the unanimous consent in looking
then at who was present.
It's
a fairly slim quorum that we have and people could easily step
aside for any number of reasons and be away from the screen when
that question is asked. I would ask that we do a roll call vote
so we can insure that we do indeed have a record of quorum voting
on the issue with the majority.
DR. JEFFREY: Okay, noted and so we will do the roll call vote.
So there is a motion on the table and a second. Are there any
comments or questions on the resolution 07-07? Whitney.
MS. QUESENBERY: I have a question that just is a coverage
question, which is I don't see where in all of this unless it
just gets covered in the final overall document, we cover things
like the vocabulary or does that just get covered in the final
--
DR. JEFFREY: It's in the appendix that I think we'll cover
at the end.
MS. QUESENBERY: Thank you.
DR. JEFFREY: Okay, unless there's any additional comments
or questions I'll ask the parliamentarian to do a roll call
vote. This is proposal 07-07.
MS. ALLEN: Roll call for Resolution 07-07. Williams.
DR. WILLIAMS: Abstain.
MS.
ALLEN: Wagner.
MR. WAGNER: Abstain.
MS. ALLEN: Paul Miller.
MR. MILLER: Yes.
MS. ALLEN: Frank Gale. I'm sorry, Gale.
SECRETARY GALE: Yes.
MS. ALLEN: My apologies. Mason.
MS. MASON: Yes.
MS. ALLEN: Gannon.
MR. GANNON: Yes.
MS. ALLEN: Pearce.
MR. PEARCE: Yes.
MS. ALLEN: Alice Miller.
MS. MILLER: Yes.
MS. ALLEN: Purcell.
MS. PURCELL: Yes.
MS. ALLEN: Quesenbery.
MS. QUESENBERY: Yes.
MS. ALLEN: Rivest.
DR. RIVEST: Yes.
MS. ALLEN: Schutzer, Schutzer. Jeffrey.
DR. JEFFREY: Abstain.
MS. ALLEN: We have eight yes'es. We have enough for
a quorum.
DR. JEFFREY: Wait, we have nine. I count nine as well. The
parliamentarian is still counting fingers. We've got nine.
So that was nine yes'es and three abstentions, and zero no's.
And with that, Resolution 07-07 passes and again my heartiest
congratulations to the Core Requirements and Testing team. Job
well done. Thank you.
With that, we are still ahead of schedule. Let me ask a question
to Secretary Gale. Do you feel that we need to go back on the
STS for a roll call vote? Are you satisfied?
SECRETARY GALE: Well, thank you for raising that question,
Dr. Jeffrey. I regret not having raised this when we voted on
the first subcommittee, but this is so extraordinarily important
that it's probably not a bad thought.
Procedurally
I guess someone who was on the prevailing side would make a motion
for reconsideration and then we would take a new vote on that
resolution if we were to do it. It may be that when we get to
the final vote it incorporates all the subcommittee votes anyway,
but I think it makes a clearer record if we do have roll call
vote.
DR. JEFFREY: Okay, is there a motion to reconsider the vote
and do a roll call vote for the Security and Transparency section?
MS. PURCELL: Mr. Chairman, this is Helen Purcell. I'll make
that motion.
DR. JEFFREY: There is a motion. Is there a second?
DR. RIVEST: Ron Rivest, I'll second it.
DR. JEFFREY: Okay, there's a motion and a second. Are there
any comments or questions? Okay, hearing none, there is actually
a proposal on the table to do a roll call vote and so the actual
question, just for the record I will state, and if there is any
objection, that the proposal is that we do a roll call vote for
the Security and Transparency section.
SECRETARY GALE: Dr. Jeffrey, Secretary Gale, point of order.
We'll be voting first on the motion to reconsider before we
vote on the resolution.
DR. JEFFREY: Yes, sir. The only vote on the table right now
is to reconsider and to do a roll call vote. So it's not on
the STS section, it's just whether we have another vote on the
STS.
If there are no comments or questions, I'll ask the parliamentarian
-- and again the motion is whether to reconsider the vote.
MS. ALLEN: Williams.
DR. WILLIAMS: Yes.
MS. ALLEN: Wagner.
MR. WAGNER: Abstain.
MS. ALLEN: Paul Miller.
MR. MILLER: Yes.
MS. ALLEN: Gale.
SECRETARY GALE: Yes.
MS. ALLEN: Mason.
MS. MASON: Yes.
MS. ALLEN: Gannon.
MR. GANNON: Yes.
MS. ALLEN: Pearce.
MR. PEARCE: Yes.
MS. ALLEN: Alice Miller.
MS. MILLER: Yes.
MS. ALLEN: Purcell.
MS. PURCELL: Yes.
MS. ALLEN: Quesenbery.
MS. QUESENBERY: Yes.
MS. ALLEN: Rivest.
DR. RIVEST: Yes.
MS. ALLEN: Schutzer, Schutzer. Jeffrey.
DR. JEFFREY: Abstain.
MS. ALLEN: That would be ten yes'es, two abstains, so
we have enough for a quorum to continue.
DR. JEFFREY: Yes, and thank you. I glad we didn't exceed
the fingers. So the motion to reconsider the STS vote, I'll
ask if there are any additional comments or questions on the STS
section. If not we'll go directly to a roll call vote.
Okay,
if you will allow me not to reread the entire proposal, this is
Resolution 06-07, which is on the screen, and if it's in front
of you it's titled STS VVSG Sections, final approval. And
with that I'll ask the parliamentarian for a roll call vote,
and you're got all the fingers ready.
MS. ALLEN: I need a motion and a second please.
DR. JEFFREY: I'm sorry, I guess we actually need the motion
now. I thought the motion to reconsider was --
MS. ALLEN: The motion to reconsider, but I need a motion
to pass it.
DR. JEFFREY: Okay, I'm sorry. So much for my Roberts Rules.
My apologies to the members. We need a motion to actually consider
Resolution 06-07. Again, if the co-chairs would like to put
the motion on the table and second it.
DR. RIVEST: Yes, so moved. This is Ron Rivest.
MS. PURCELL: And second.
DR. JEFFREY: Okay, so Resolution 06-07 is back on the table.
It has been seconded, and with that if there are no comments or
questions I'll ask the parliamentarian to call the vote by roll
call.
MS. ALLEN: Williams.
DR. WILLIAMS: Abstain.
MS. ALLEN: Wagner.
MR. WAGNER: Abstain.
MS. ALLEN: Paul Miller.
MR. MILLER: Yes.
MS. ALLEN: Gale.
SECRETARY GALE: Yes.
MS. ALLEN: Mason.
MS. MASON: Yes.
MS. ALLEN: Gannon.
MR. GANNON: Yes.
MS. ALLEN: Pearce.
MR. PEARCE: Yes.
MS. ALLEN: Alice Miller.
MS. MILLER: Yes.
MS. ALLEN: Purcell.
MS. PURCELL: Yes.
MS. ALLEN: Quesenbery.
MS. QUESENBERY: Yes.
MS. ALLEN: Rivest.
DR. RIVEST: Yes.
MS. ALLEN: Schutzer, Schutzer. Jeffrey.
DR. JEFFREY: Abstain.
MS. ALLEN: We have nine yes'es and we have three abstains.
We have enough for a quorum.
DR. JEFFREY: And deja vous, congratulations again to the STS
--
(Tape
Interrupted While Changing Sides)
(END
OF AUDIOTAPE 1, SIDE B)
(START OF AUDIOTAPE 2, SIDE A)
DR. JEFFREY: -- Taking a break in the middle of the session
and then come back to that, so I'd like to ask Sharon to come
up and talk about the Human Factors and Privacy.
And
while she is setting up I'd like to say that actually this is
probably one of the sections that's the most exciting because
of the tremendous change that's really occurred in this versus
VVSG 2005, and the original research has actually been folded
into this to really make it much more accessible and usable.
And
so I'd like to thank the entire subcommittee for really going
beyond what was state of the art. So with that, Sharon.
DR. LASKOWSKI: Thank you Dr. Jeffrey, now that the microphone
is turned on.
This is Sharon Laskowski speaking, reporting on the Human Factors
and Privacy Subcommittee work on the final draft of the VVSG.
I do want to thank the entire TGDC for providing so much thoughtful
input as the HFP subcommittee worked to pull this next draft together,
and I really enjoyed working with every single one of you.
I do have a natural stopping point so I think that given the time,
I think that will work fairly well.
So everyone should have the Human Factors and Privacy report on
final draft of the VVSG pulled up. Title slide is number one.
I'm
moving on to slide number two which is the overview. The outline
of my talk is that I will summarize the significant changes from
the VVSG '05, HFP work, then I will review the HFP changes from
the previous draft that we discussed in May, and then I will go
over the new material that is the usability performance pass/fail
benchmarks.
On to slide number three, the significant changes from the VVSG
'05. I'm not going to read through them. I will just note
the ones that were the most dramatic changes.
Of course the performance benchmarks are quite new and different
from anything we'd seen in a usability related standard and
I will be talking about that extensively. We have added coworker
usability requirements and we've also added plain language guidance,
which support different cognitive requirements. Easier the language,
easier for everyone to understand. So on this slide I think
those are the highlights of the usability.
The
highlights of the accessibility would be a requirement that accessibility
throughout the voting session be tested as a requirement.
On to slide number four, continuing significant HFP changes from
the VVSG
'05.
Here I think that the major point is that we've addressed low
vision more fully and we've moved it to the general usability
section, things like font size, and contrast and different choices,
not just for the accessible voting station but for all the voting
stations.
That's
because especially with an aging population, we have people with
varying degrees of vision that will not want to use the accessible
station and will be using the regular voting station, including
people that sometimes forget their glasses.
On to slide number five. There are 14 significant changes since
the May plenary. Some of these, and again I will just summarize
fairly quickly, we've updated some terminology and refined some
definitions.
I think a big significant change especially is that we've clarified
the interaction of section 3.2, which was the usability section,
and section 3.3, which is the accessibility section.
There
was a lot of confusion and in particular we wanted to make it
very clear to everyone that all the vote editable ballot device
requirements for usability also apply to the accessible voting
stations, all of which are in the VEDB class, and we put in clarifying
language in both of those sections so that if people are just
looking at say the accessibility section they will know to go
back and look at these other usability requirements.
Of course the new metrics and proposed performance benchmarks
are significant as well.
On to slide number six, continuing with the changes since the
May plenary. Here we have an additional requirement about making
sure that failure to actually cast the ballot, have some notification
that the voter could see, and we've also clarified discussion
about privacy as well.
Continuing on to slide number seven, changes since the May plenary.
I think the major issue on this slide is that we clarified what
we meant by poor reading vision. Again, discussed a little bit
earlier in terms of this, is really also the placement of the
poor vision, low vision requirements in the usability section
from the accessibility section were also quite important related
to this.
On to slide number eight, continuing the changes since the May
plenary. We upgraded the legibility of paper such as verification
of the VD-Pat to a shall, and we specified two sufficient techniques
that are variable font size and magnification for those with low
vision, and we refined our various response and activity time
and alert time requirements.
And finally slide nine, changes since the May plenary. If you're
looking for the section on maintenance, we've moved that to
section 6.4.5. It seemed better placed there than in the usability
section, and some other minor changes.
And that concludes the changes since the May plenary. So this
is kind of the natural stopping point. I can do a little quick
intro, or first let me ask if there are questions on the first
half so we can kind of go through those questions before we look
at performance requirements. We might want to consider whether
we want to continue here or wait until after lunch.
DR. JEFFREY: Any questions on the first section, which are the
changes? Sharon, roughly how long do you think the rest of your
session will take?
DR. LASKOWSKI: Hard to say. Probably more then 15 minutes.
I'm thinking 20 to 35 minutes.
DR. JEFFREY: Okay, well what I might suggest at this point is
that we take a slightly early break. Would there be any objections
from any of the members to try to reassemble at 1:45 p.m. instead
of two o'clock, still keep to a 30 minute break?
Okay, hearing no objections, thank you all for your patience in
using the high tech system that we've set up. It actually
seems to be working pretty well so my compliments to anyone who
set this up. So let's take a break right now and reconvene
at 1:45 Eastern time, and we'll see you soon.
(BREAK)
DR. JEFFREY: -- Meeting, and I'd like to ask the
parliamentarian to first insure that we still have a quorum, so
Thelma.
MS. ALLEN: Roll call. Williams, Williams. Williams is
not responding. Wagner.
MR. WAGNER: Here.
MS. ALLEN: Wagner is present. Paul Miller.
MR. MILLER: Present.
MS. ALLEN: Paul Miller is present. Gale.
SECRETARY GALE: Here.
MS. ALLEN: Gale is present. Mason, Mason. Mason is not
responding.
Gannon, Gannon. Gannon is not responding. Pearce.
MR. PEARCE: Here.
MS. ALLEN: Pearce is here. Alice Miller, Alice Miller.
Alice Miller is not responding. Purcell.
MS. PURCELL: Here.
MS.
ALLEN: Purcell is present. Quesenbery.
MS. QUESENBERY: Here.
MS. ALLEN: Quesenbery is present. Rivest.
DR. RIVEST: Here.
MS. ALLEN: Rivest is present. Schutzer, Schutzer. Schutzer
is not responding.
Jeffrey.
DR. JEFFREY: Here.
MS. ALLEN: Jeffrey is present. Williams, Williams is not
responding. Mason, Mason is not responding. We have nine.
We have enough for a quorum.
DR. JEFFREY: Okay, thank you. Okay with that, Sharon, if you
could continue. I think you were on page ten.
DR. LASKOWSKI: Thank you, Dr. Jeffrey. So we're on
slide number ten of the HFP report.
So in the second half of my talk, because this is new material,
I am going to be talking about the usability performance requirements.
I am going to give a little bit of review material from last time
just to refresh everyone's memory and then go directly into
the pass/fail benchmarks.
So with usability performance requirements, our goal here is to
develop a test method to distinguish systems with poor usability
from those with good usability.
It's based on the performance not evaluation of the design,
so as you see, for most of the requirement in the VVSG, there
are things about font size or scrolling, very much design oriented.
But
we're interested in a test that detects all the types of errors
one might see when voters interact with the voting system because
guidelines on the design by themselves is not sufficient to detect
those that occur in the interaction and we would like this test
to be reliable so that we can repeat it in a test laboratory.
And part of the reliability is that it is reproducible by the
test laboratory so you trust the results.
One of the benefits of course is that it is technology independent.
You can apply this kind of test method to any type of voting system.
So given such a test method you can calculate benchmarks based
on measurements of the interaction during your test.
So a system meeting the benchmarks has good usability and passes
the test. So the values chosen for the benchmarks, these pass/fail
benchmarks, become the performance requirements.
Slide 11. So what we're really talking about is a test method,
a usability test method for certification of a voting system in
an accredited test laboratory where measuring the performance
of the system in the test lab so we've got to control as much,
as many of the other variables as we can, including the test participants.
So we've got a test ballot that we designed to detect different
types of usability errors and be typical of many types of ballots.
Remember this is a national test so we want it to capture many
sorts of errors, not just the kinds of errors you'd see in one
particular type of ballot and one particular state for example.
The test is done in a lab and the environment is tightly controlled
for lighting, set up, the instructions. We do not allow assistance
to the test participants and the test participants are chosen
so that they reliably detect the same performance on the same
system. And they are told exactly how to vote so we can measure
the errors.
Now the test results measure relative degree of usability between
systems. They are not intended to predict nor can they predict
performance in a specific election because the ballot is different,
the environment is very different.
People
go in knowing who they want to vote for. They can readily obtain
assistance from coworkers. They may have received brochures
in the mail about what the ballot looks like.
And
for each election the voter demographics have got to be different,
and a general sample of the U.S. voting population across the
board can never be truly representative in such a lab test because
all elections are local.
The key is we're creating a measurement device to detect usability
errors in the interaction of the voting system.
Slide number 12, components of the test method. So obviously
the test protocol you're using has to control lots of things
and has to be very precise. It's got to be well defined.
It describes the number and characteristics of the voters participating
in the test, how to conduct the test. We've designed the test
ballot to be relatively complex to insure that we evaluate the
entire voting system and detect significant errors.
The instructions to the voters are exactly how to vote so we can
count those errors. This test method protocol, which we call
the voting performance protocol, has a precise description of
the test environment and also a method for analyzing and reporting
the results so that each test lab could report those consistently,
and performance benchmarks and their associated pass/fail values.
So I'm on slide number 13 now. And let me recap the research
that you heard about in May. We did some initial testing just
to test the validity of our test protocol. Did we detect differences
between systems and did it produce the errors we expected, and
we did that on two different systems and it did. So we were
very encouraged by this.
Was it repeatable, that is do we show reliability? We did four
tests on the same system, different test participants each time,
for a total of 195 participants, and we've got similar results
on that same system for those four tests.
And the demographics we used were sufficient to detect all the
possible errors that we had. Imagine what one could see or could
do with such a system.
So on to the next slide, slide 14. I talked about the benchmark
tests, which are the new tests that I reported on the planning
of in May.
So
we selected four different systems. It was a selection of DREs,
EDMs and P-COSS. We ran through 187 test participants. We
took five measurements, three of these measurements I'll be
talking about. These are our pass/fail benchmarks, and two measurements
that the HFP subcommittee suggests that we report on only and
do not use them as pass/fail criteria.
FEMALE SPEAKER: Sharon, if I could interrupt. We actually
raised this question at the last TGDC and discussed with the entire
TGDC, who I believe collectively came to the decision that two
values should be only reported.
DR. LASKOWSKI: That is correct. So I'm on slide 15
now. So what about the performance measures? The names of
three of the performance measures have been altered slightly and
we've also included a base accuracy score that feeds into our
measures.
So let me first talk about this base accuracy score. That is,
how do we count the number of errors. Well, we've got a test
ballot that has 28 voting opportunities. We give instructions
to the test participants, vote this way, and then we count how
many were correct for each participant so we've got an error
rate.
And to calculate the base accuracy score, we basically look at
the scores for each of the test participants and take the average,
the mean percentage of ballot choices that were correctly cast
and this gives us a base accuracy score, which I'll talk about
how we use it in a couple slides from now.
So we're going to calculate three effective measures that we
then put benchmarks on for pass/fail and effectiveness in this
situation. In fact, it's kind of a usability term, effectiveness
here, it means really the accuracy with which the voters were
able to make their choices.
So first we have a total completion score. This basically is
the percentage of test participants who are able to complete the
process of voting and having their ballot choices recorded by
the system.
So examples of people that wouldn't complete might be someone
who gives up in the middle or someone who forgets to hit the cast
ballot button.
Our second measure of voting accuracy we call now the voter inclusion
index. It uses the base accuracy score and the standard deviation,
in other words the variability that you see across the test participants.
Okay,
so why do we like this measure? Well if two systems have the
same base accuracy score but you see a system that has a large
variability -- there are some voters that do really poorly, some
voters that do really well to offset that, that would be a large
variability. That's not as good as a system that's producing
consistently good results with that same base accuracy score.
I put the formula for people that like to look at this and run
some numbers through, but basically the idea is you divide the
standard deviation by the variability, so higher variability gives
you a lower voter inclusion index.
Our third measure we call the perfect ballot index. So you look
at the number of cast ballots that contain no errors and you look
at the number of ballots that contain one error or more and you
take the ratio. So the perfect ballot index is simply the ratio
of the number of cast ballots containing no errors over those
that contain at least one error.
Now this does deliberately magnify the effect of even a single
error so why are we interested in this, why is it useful?
Well, given a system that has a high base accuracy score, it might
still have a common error that everyone or quite a number of people
are making. So this suggests some design flaw that everyone
is having trouble with. So this measure basically captures that
characteristic.
And as I said, we have two other measures that we're not doing
a pass/fail criteria. We're just reporting on those, and those
are the average voting session time and the average voter competence.
These
are interesting and good information to have. It's not as
important as getting the accuracy of the vote and in fact neither
of these measures correlates with effectiveness.
You
can have people that take a long time or take a short amount of
time and they may or may not have done well with respect to the
accuracy of their vote for example. So the HFP subcommittee
suggests and we discussed this in May that we're just going
to report on those.
So I've got a couple of tables here. I need to show you a
little bit of data but I'll just point out specific items from
these tables so that you get a general idea of how the HFP decided
to make at least an initial recommendation of what the benchmarks
ought to be and to lead into the discussion of that.
We tested four systems. These were systems certified to the
VSS'02, selection of VREs, DBMs (unintelligible). So I've
called them system A,B,C, and D.
In the second column we have total completion scores. Notice
that these are reported
as an interval. When you take a sampling you always have some
uncertainty so you report it with a competence interval.
These
are 95 percent competence intervals so for example, System B has
a completion score interval of 92.8 percent to 100 percent.
So the way to interpret a competence interval is basically to
say if I repeated this test a 100 times, 95 times the true total
completion score of that system would be in that interval.
And the third column is our base accuracy score. Notice for
example System D is similar to System C, 92.4 percent, but it's
got a larger standard deviation of 19. So when we calculate
the voter inclusion index again as an interval, we get quite an
interesting range. For example, system B has inclusion index
in the range of .49 to .85 where system D is much lower, .03 to
.22.
When we determine the pass/fail benchmarks we look to see whether
that benchmark is included in that interval or above it.
Okay, that next table on slide 21, if you look at the second column
you'll see the perfect ballot index competence intervals.
Again a large variety of values here. System D from about 1
to .352. System C totally below, an interval totally below System
D. They don't even overlap.
So going on to slide 22. So the HFP has proposed some benchmarks
but I have a couple of slides to discuss this more fully because
there's a couple ways to view these.
So if we look at the data that I just showed you and we picked
a total completion score of 98 percent, voter inclusion index
of .35 and perfect ballot index of 2.33, two of the systems fail.
These are VSSO 2002 systems.
What we've done in the current draft that you'll see for each
of these three benchmarks, we've put in placeholders, that is
for example, total completion performance says the system shall
achieve a total completion score of at least XXX percent as measured
by the voter performance protocol.
So let me go on to my next slide, 23. But the questions on the
table, which is really a policy issue, is how tough should the
benchmark thresholds be. So we'll point out first the benchmark
data here used -- some interference.
DR. JEFFREY: If anybody does not have their mute on, could you
just check? Thank you.
DR. LASKOWSKI: All right, so in this initial benchmark
data testing that we did, we used 50 test participants which we
showed we could get repeatability so this was sufficient. However,
for the actual test protocol for the labs we will use 100 test
participants.
This
is for statistical reasons. In order to compute the competence
intervals easily we need to assume a normal distribution and best
practice using a voter inclusion index or an index of a nature
of the voter inclusion index, and for other kinds of process,
benchmarks suggest that if you have 100 participants or more you
can make those assumptions.
Okay, when you have more participants you are more sure that you've
captured the true value of that benchmark and you can narrow the
competence intervals, so the test will be tougher because you'll
have smaller competence intervals then you saw in the two tables
I showed you with 100 participants.
But the two points of view, our proposed benchmarks, do weed out
poorly performing systems so this by itself is a big step in terms
of these kinds of requirements and it is relatively easy to raise
the thresholds.
But
these were VSS 2002 systems. And in the standards world and
conformance testing to a standard tend to be in general on the
conservative side. However on the other side, there should be
a forward looking standard.
New
systems are coming down the pike very soon. For example, for
VVSG '05 certification. So how much higher can we put these,
and also recall because people are in the loop here, there is
always some upper bound of how high you can go because humans
always make mistakes.
So in the next slide. I think to make some policy here we do
need some additional data, one to say how much flexibility do
we have with the test laboratories to get good reproducible results.
And
we do propose to do some future, very soon in fact, testing with
voters with difference experiences and different geographic regions,
looking at older populations, such educated populations.
I
should note, benchmark thresholds are always tied to the demographics
of the test participants. We will be collecting also for existing
demographics, some additional data, which will tell us more about
in terms of exploring the statistics for looking at 100 users
and we also will be looking at accessible systems to see how they
perform with this test.
So I think with that, I believe Whiney probably has some
additional discussion and some policy direction for us in terms
of the benchmarks.
DR. JEFFREY: Okay, Whitney, over to you.
MS. QUESENBERY: This is Whitney. Sharon, how did you guess?
I'd like to speak in favor of the notion that this should be
a forward looking standard with somewhat higher levels of benchmarks
but I'm not going to propose any specific benchmarks here.
I
know that we are going to do probably if I have the numbers right,
at least as much data collection in the next round of testing
that Sharon just mentioned, as we have done to date and that will
give us some geographical distribution, a look at some broader
demographics, and will give us a much richer dataset.
What I'd like us to do at the TGDC today is to provide some
direction to NIST as they do the data analysis of those tests
with where they should be looking.
So for instance if we look back at the three benchmarks that will
be the pass/fail benchmarks, right now the level that we've
been talking about are kind of pegged at a median.
They
would disqualify half the systems and half the systems that were
tested would pass, but these are VSS 2002 systems and we know
that there has been improvements in systems since then and we
can only hope that changes in new systems have added to the usability
of those systems as well as improving other aspects like security
and reliability.
So
one way to look at it is to simply say we want to raise the threshold
and just increase it, but in the long conversations on HFP we
have been given a good education in statistics by the NIST scientists
and statisticians that you can't just say I wish it could be
100 and therefore I'll make the level 100.
So
these have to be numbers that are reasonable and I think we've
also been concerned that we not create a standard that is so high
that no system can meet it.
However, one way to look a this is to say as we look at the scores
that the systems we have tested so far and that we will test in
the future in setting these benchmarks, we could say what is the
highest level they have actually achieved.
So
for instance if we look back on slide 20 at the benchmark test
results that include total completion score and voter inclusion
in that, one of the things you see is that one system has a voter
inclusion index range interval that is higher, that is the bottom
level of System B at .49 is actually completely above any of the
other three systems.
So I have to ask the question, especially on a standard that won't
even be approved until 2009, why aren't we aiming at the highest
level of each of those benchmarks?
Now I'm going to say some numbers and I don't mean
them as a formal technical proposal. They're just as an example.
So maybe 98 percent, maybe we'd like to see 98.5 percent.
I'm not sure if 99 is technically possible on the total complete
score.
On
the perfect ballot index we're currently proposing 2.33, but
one system achieves a 1.07 to 352 range, again completely above
-- well not completely above the others but above several of the
others.
Why
not go for something like 3.0 as pulling a number out of my hat,
and on the voter inclusion index which is essentially accuracy
plus variability, why not look for something like 5.0 that gets
us into the range of the top systems.
Now
I know that we can't just sit here as people who neither worked
on the data or are statistical experts and say this is the value
you should pick.
We
really do look to the NIST scientists to give us recommendations
and I believe that based on our initial request of them that the
values that we have in front of us are good values when our direction
was to create benchmarks that weeded out systems with poor usability.
The
question I'd like to raise for everyone is should we be raising
our sights and be looking to try to create benchmarks that exemplify
the best usability that we can reasonably achieve. Thank you.
DR.
JEFFREY: Thank you, Whitney. This is Bill Jeffrey. This
is addressed to you Whitney as a question.
What
you're describing, actually the way I interpret it, is a goal
or an aspiration for this as opposed to a hard requirement.
Is that fair?
Because
certainly in the text of the requirement it should clearly state
which direction -- your higher numbers in certain cases is better
and would indicate perhaps a more robust system.
Putting
an actual stretch goal at the moment as a requirement I think
would be difficult to actually embed as a requirement.
Are
you happy with specifying specific goals and emphasizing in the
text that literally higher is better, in this case lower is better,
and that may result in a more robust system?
MS.
QUESENBERY: Well, I think certainly if the intro doesn't
now, and I think it does say that higher is better, I think we've
constructed all of these numbers so that higher is better.
I
guess what I'm saying is, and I don't want us to set numbers
because I do think that this should happen after the next round
of testing.
One
of the things that Dr. Laskowski alerted us to is that when we
do larger numbers of participants that competence intervals will
be tightened and therefore the numbers will be tightened.
So
I don't want to just say let's change 2.33 to 3. What I
would like and I think we can do this as (unintelligible) of the
meeting is to say perhaps we adopt benchmark levels that have
been proposed now but that we know that those will be revisited
after the next round of data collection, and that we set the direction
for NIST to come back after the next round of data collection
with a set of benchmarks that are not unachievable by any system
but they reflect the best scores by any system.
DR.
JEFFREY: Thank you. Are there any other comments or questions
on this?
MR.
PEARCE: Yes, this is Philip Pearce. I'm sorry, I tried
to raise my hand and it didn't give me the opportunity to do
that for some reason. Anyway, is it okay if I do it at this
time?
DR.
JEFFREY: Absolutely, please do, sir.
MR.
PEARCE: Okay. Again the question or the comment that I've
got is that I would like to in addition to looking at the next
set of tests that are going to be conducted, that have been identified
in the slides and in the discussion so far, is to also look at
future tests because again things are changing and I would be
pretty disappointed if it looked like we even tried to set standards
based on the next set of tests and didn't look three years,
or five years, or ten years down the road and to be able to conduct
these tests again and to see where those values maybe would sit
at that future time.
So
some way or another I would like to see us factor that into the
discussion and also into the recommendations that we make.
MS.
QUESENBERY: This is Whitney. I completely agree with Philip.
I know that this puts a burden on to the EAC for how these things
are updated but I think very much like the innovation process,
hopefully the new requirement in whatever version this becomes
on this whole thing that we're voting on now will improve systems
and will improve the usability of them.
I
think one area where we're all particularly concerned is the
usability of the accessible voting system where we've had some
antidotal reports in the field that are not very encouraging and
hopefully these will also encourage, but we don't want to say
well, we set a level back in 2008, and in 2020 we're still accepting
that as the proper level, even if new systems have come along
that might have improved it.
DR.
JEFFREY: Chairman Davidson.
COMMISSIONER
DAVIDSON: Underneath the law of HAVA, it requires us to go back
every four years, the EAC, to review to see, not only in this
area but any other area, if there is a need to bring the TGDC
back together to revisit writing a new iteration, or especially
one area, arena.
It
doesn't say everything has to be done but that is really up
to the EAC, and as we move forward obviously we have no idea what
four years will be down the road. There may be a lot of changes
or maybe really not that many changes. It's the unknown.
But the law does require that the EAC review it before making
a determination whether they bring back the NIST and TGDC to move
forward.
DR.
JEFFREY: Thank you. This is Bill Jeffrey. I would also like
to just add that I think that this also shows, and I just want
to reemphasize something mentioned earlier, what an exciting area
that this is because this is the first time that we've actually
got even an inkling of what the members would look like.
And
I really do want to applaud the subcommittee for doing this research
and also to emphasized that this is essentially original research
that actually did go out for formal, technical peer review which
also I think adds a lot of credibility to it.
I'd like to recognize Secretary Gale for a comment or question.
SECRETARY
GALE: Thank you, Dr. Jeffrey. Well first of all I want
to congratulate Sharon. She has done a very impressive and a
very outstanding job developing a relatively simple formula with
very real data to back it up.
I
understand Whitney and Phil's position about possibly raising
the threshold to encourage manufacturers to increase simplicity
and usability, but in looking at the test data it does appear
the subjects were probably a bit more educated then average and
perhaps different then other test labs we might recruit so I think
that the standard that Sharon has set maybe be a higher standard
then you might realize and we won't know until there is further
testing.
So
it seems to me that it would be appropriate to approve this and
wait for later testing in other locations to see if there is a
different educational level and maybe a little different skill
set maybe.
DR.
JEFFREY: Whitney.
MS.
QUESENBERY: This is Whitney. I was actually reminded just
looking at my notes from Commissioner Davidson's original talk,
that the schedule actually has a slot in there in between the
two public comment periods for the TGDC to review and look at
the comments.
And
maybe that's an appropriate time to re-look at this, and what
we should do is pass the benchmarks as they've been proposed
and then by then we'll know whether there's been new data
that suggests that we should change it in any way.
DR.
JEFFREY: Thank you. Any other comments or questions? Okay,
if there are no other comments or questions, what I'd like to
do is again for symbolic reasons ask if Whitney as Chair of the
HFP subcommittee would like to propose Resolution 08-07.
MS.
QUESENBERY: Absolutely.
DR.
JEFFREY: Is there a second?
MS.
MILLER: This is Alice. I'll second it.
DR.
JEFFREY: Hi, Alice. I didn't know you were on. I'm sorry.
Okay, so thank you. So good, we've got the symbolism complete.
Okay,
let me read Resolution 08-07, which has been proposed and seconded.
This is the HFP VVSG Sections, Final Approval, and fortunately
it's a little easier to read then some of the other ones.
The
TGDC grants final approval for the Human Factors and Privacy sections
part one, chapter three all, part three, chapter four, section
4.5 as part of its second set of VVSG recommendations to the Executive
Director of the EAC subject to editing as instructed by the TGDC
at this meeting and final review by the Chair of the TGDC.
Are
there any comments or questions on this proposal? Okay, hearing
none I will ask Thelma to call a roll call vote.
MS.
ALLEN: Roll call for 08-07. Williams.
DR.
WILLIAMS: Abstain.
MS.
ALLEN: Wagner.
MR.
WAGNER: Abstain.
MS.
ALLEN: Paul Miller.
MR.
MILLER: Yes.
MS.
ALLEN: Gale.
SECRETARY
GALE: Yes.
MS.
ALLEN: Mason.
MS.
MASON: Yes.
MS.
ALLEN: Gannon.
MR.
GANNON: Yes.
MS.
ALLEN: Pearce.
MR.
PEARCE: Yes.
MS.
ALLEN: Alice Miller.
MS.
MILLER: Yes.
MS.
ALLEN: Purcell.
MS.
PURCELL: Yes.
MS.
ALLEN: Quesenbery.
MS.
QUESENBERY: Yes.
MS.
ALLEN: Rivest.
DR.
RIVEST: Yes.
MS.
ALLEN: Schutzer, Schutzer. Jeffrey.
DR.
JEFFREY: Abstain.
MS.
ALLEN: We have nine yes'es. We have enough for a quorum.
DR.
JEFFREY: Excellent. It's my fingers too. So congratulations
to the Human Factors. I think you just made history actually.
First time I think we ever got requirements like this, so congratulations.
Looking
at my agenda there is also a resolution, which would be the final
approval for the entire document, which includes the appendices.
It has not been formally introduced. Would somebody like to
introduce it? I'm talking about Resolution 09-07 and I'll
read it if somebody wants to introduce it.
MS.
QUESENBERY: I'll be happy to introduce it. This is Whitney.
DR.
JEFFREY: Hey Whitney, thanks. Is there a second? This is
Bill Jeffrey, I'll second it.
Let
me read it. This has been proposed and seconded. VVSG document
final approval reads the TGDC grants final approval for the document
Draft Voluntary Voting System Guidelines Next Iteration, August
7, 2007, in its entirety as the second set of VVSG recommendations
to the Executive Director of the EAC subject to editing as instructed
by the TGDC at this meeting and final review by the Chair of the
TGDC.
Resolution
09-07 has been offered and seconded. Are there any comments
or questions on it? Hearing none, I will ask Thelma to do a
roll call.
MS.
ALLEN: Roll call for 09-07. Williams.
DR.
WILLIAMS: Yes.
MS.
ALLEN: Wagner.
MR.
WAGNER: Abstain.
MS.
ALLEN: Paul Miller.
MR.
MILLER: Yes.
MS.
ALLEN: Gale.
SECRETARY
GALE: Yes.
MS.
ALLEN: Mason.
MS.
MASON: Yes.
MS.
ALLEN: Gannon.
MR.
GANNON: Yes.
MS.
ALLEN: Pearce.
MR.
PEARCE: Yes.
MS.
ALLEN: Alice Miller.
MS.
MILLER: Yes.
MS.
ALLEN: Purcell.
MS.
PURCELL: Yes.
MS.
ALLEN: Quesenbery.
MS.
QUESENBERY: Yes.
MS.
ALLEN: Rivest.
DR.
RIVEST: Yes.
MS.
ALLEN: Schutzer. Jeffrey.
DR.
JEFFREY: Abstain.
MS.
ALLEN: We have ten yes'es. We have enough for a quorum.
DR.
JEFFREY: Congratulations team. I don't know why there is
so much noise in the background. It's probably the clapping.
My heartiest congratulations to the entire TGDC and the NIST support
team. I think you just produced a document that will be going
to the EAC shortly.
So
with that, I will open it up if there are any additional resolutions,
or comments, or questions that TGDC members have, for example
Ron Rivest.
DR.
RIVEST: Yes, thank you. This is Ron Rivest. We had some
discussion earlier about the innovation class and I did want to
introduce a resolution recognizing that we've really just gotten
started with the innovation class.
We're
handing this process over to the EAC. We've had a number of
discussions with the EAC and within the TGDC and as it was noted
earlier by Secretary Gale that this process has really just begun.
There's a lot of uncertainty here and the resolution really
just sort of recognizes the handoff and the importance we put
upon the innovation class.
Let
me now read the proposed resolution. This is Resolution number
07 --
(Tape
Interrupted While Changing Sides)
(END
OF AUDIOTAPE 2, SIDE A)
(START OF AUDIOTAPE 2, SIDE B)
DR. RIVEST: -- Title encouragement of innovation.
The TGDC recognizes that innovation and voting systems must take
place for voting systems to be usable, accessible, secure, reliable,
and accurate for all voters and voting populations.
For
innovation to occur, the TGDC has directed NIST to create initial
requirements and a general set of procedures for an innovation
class as outlined in TGDC Resolution 03-06.
The TGDC urges the EAC to develop and published detailed plans
and specific procedures for an innovation class program so as
to encourage innovation in voting systems and to make clear to
vendors how they may use the specific procedures, steps of the
innovation class to achieve conformance to the VVSG for their
innovative products.
SECRETARY GALE: This is John Gale, Secretary of State, Nebraska.
I would second that but I think the Resolution number might be
wrong, Dr. Rivest.
DR. JEFFREY: Yes.
DR. RIVEST: I have the wrong number?
DR. JEFFREY: If Dr. Rivest would accept a friendly amendment.
He may be reading an old version. We have Resolution 10-07,
which is a slightly more concise version of what you've said.
DR. RIVEST: Well I've got perhaps the older version, yes.
DR. JEFFREY: Would you like me to read --
DR. RIVIST: Please.
DR. JEFFREY: And please then confirm that this is the one that
you are looking for.
This is Resolution number 10-07. The TGDC recognizes that innovation
in voting systems must take place for voting systems to become
more usable, accessible, secure, reliable, and accurate for all
voters in voting populations.
The
TGDC urges the EAC with technical assistance from NIST, to continue
to develop and publish detailed plans and specific procedures
for an innovation class program so as to encourage innovation
in voting systems and to make clear to manufacturers how they
may use the innovation class to achieve conformance to the VVSG
for their innovative products.
DR. RIVEST: I'm happy with that as a friendly amendment,
yes.
SECRETARY GALE: Secretary John Gale. I also accept that
as a second.
DR. JEFFREY: Excellent, thank you. So there is a resolution
and it has been seconded. This is Resolution 10-07. Are there
any comments or questions?
This is Bill Jeffrey. I'll just add a comment. I heartily
endorse this.
Any other comments or questions? Okay, hearing none I will ask
the parliamentarian, Thelma Allen to please call the roll call
vote. That's hard t say.
MS. ALLEN: Roll call for 10-07. Williams.
DR. WILLIAMS: Yes.
MS. ALLEN: Wagner.
MR. WAGNER: Abstain.
MS. ALLEN: Paul Miller, Paul Miller. Gale.
SECRETARY GALE: Yes.
MS.
ALLEN: Mason.
MS. MASON: Yes.
MS. ALLEN: Gannon.
MR. GANNON: Yes.
MS. ALLEN: Pearce.
MR. PEARCE: Yes.
MS. ALLEN: Alice Miller.
MS. MILLER: Yes.
MS. ALLEN: Purcell.
MS. PURCELL: Yes.
MS. ALLEN: Quesenbery.
MS. QUESENBERY: Yes.
MS. ALLEN: Rivest.
DR. RIVEST: Yes.
MS. ALLEN: Schutzer, Schutzer. Jeffrey.
DR. JEFFREY: Abstain.
MS. ALLEN: We have nine yes'es. We have enough for a
quorum.
DR. JEFFREY: Excellent. Congratulations. Number 10-07 passes.
With that I will ask if there are any other comments, questions,
or resolutions that the TGDC would like to offer.
MS. PURCELL: Mr. Chairman, this is Helen Purcell.
DR. JEFFREY: Yes.
MS. PURCELL: I would like to offer another resolution. We
have all talked this morning, or various people have talked about
the good work that has been done not only by the Chair but by
the staff of NIST so I would like to propose Resolution, I believe
it would be 11-07.
DR. JEFFREY: That is correct.
MS. PURCELL: The TGDC expresses its sincere appreciation for
the exemplary leadership of the Chair, Dr. William Jeffrey, and
the work of this committee to meet the relevant
mandates of the Health America Vote Act.
The
TGDC also recognizes the superior technical efforts of NIST scientists
and support staff in both the drafting of the VVSG recommendations
in organizing the activities of this committee and its working
subcommittees.
DR. JEFFREY: Thank you. Is there a second?
DR. RIVEST: This is Ron Rivest, I would like to --
DR. JEFFREY: Okay, there's a second and a third and a fourth.
There is Resolution number 11-07 that has been proposed, seconded,
third and fourth. Are there any comments or questions on this?
SECRETARY GALE: This is Secretary Gale. I just want to
I guess elaborate a little bit on the tremendous leadership you
have provided Dr. Jeffrey for not only NIST and for TGDC but for
the country.
NIST
provides an incredible academic and research institution that's
very significant to America and its future, and your leadership
as captain of that ship is extraordinary and you'll be missed,
not only by all of us but any of those who believe in the importance
of science and the future development and growth of our economy
and provision for our future. So thank you for your service.
DR. JEFFREY: Thank you very much, sir. And again, if everyone
can make sure that their system is on mute.
There's
a very almost embarrassing resolution on the table and it's
been seconded. I will ask the parliamentarian to call the roll
call vote and I'll be noting who says no.
(LAUGHTER)
MS. ALLEN: Roll call for 11-07. Williams.
DR. WILLIAMS: Yes.
Ms. ALLEN: Wagner.
MR. WAGNER: Abstain.
(LAUGHTER)
MS. ALLEN: Paul Miller, Paul Miller. Gale.
SECRETARY GALE: Yes.
MS. ALLEN: Mason.
MS. MASON: Yes.
MS. ALLEN: Gannon.
MR.
GANNON: Enthusiastically, yes.
MS. ALLEN: Pearce.
MR. PEARCE: Yes.
MS.ALLEN: Alice Miller.
MS. MILLER: Yes.
MS. ALLEN: Purcell.
MS. PURCELL: I guess I'm a yes.
MS. ALLEN: Quesenbery.
MS. QUESENBERY: Yes.
MS. ALLEN: Rivest.
DR. RIVEST: Another enthusiastic yes.
MS. ALLEN: Schutzer, Schutzer. Jeffrey.
DR. JEFFREY: I'm going to vote yes.
(LAUGHTER)
MS ALLEN: We have ten yes'es. We have enough
for a quorum.
DR. JEFFREY: But that was only because of the second half of
that compliment was the NIST scientists and staff. With that,
are there any other comments, questions, or resolutions?
SECRETARY GALE: Dr. Jeffrey, this is John Gale, Secretary
of State, Nebraska.
DR. JEFFREY: Yes, sir.
SECRETARY GALE: This is a final comment. Obviously we're
all very, very proud of NIST and your staff and the work they
have done, as well as all the members of TGDC as voluntary members
of this extraordinary effort.
I think we need to urge the election administration community,
and the election equipment industry, and the public, and all of
the non-profit organizations with an interest to really give this
a very thorough vetting.
Obviously
we've all done the best we can do but we're not saying this
is absolutely without some flaw, or some fault, or some possible
need for modification.
The issues of cost, the issues of the election administration
practicality and the reasonableness are things that really deserve
the same amount of attention that we've given to the technical
development of these guidelines so no one should assume by this
vote or by the adoption of this iteration that it is ultimately
an unalterable record that is irrevocably committed in stone.
It's
certainly the best we can offer and we hope that we have a very
vigorous and robust debate out there as to how to move forward.
In
elections there's nothing more critical of our democracy then
voter confidence in our elections and part of that is equipment,
part of that is election administration, and part of it is in
the usability equipment by the average voter.
So
we have a lot of work left to be done, but we certainly have made
a first bold step forward. Thank you.
DR. JEFFREY: Thank you. Actually that was incredibly well
put. I think that this is an historic moment but it's just
the first of what will be many steps along the way to get to the
final version and I don't envy the EAC for the amount of work
that lies ahead. The public comment period is going to be incredibly
important.
I
completely concur that this is not cast in stone, that as we get
comments and as we understand the implications from the vendors
and from the costing, that, you know, I think the TGDC and hopefully
NIST will be standing right there to provide whatever support
the EAC needs in working through the issues because again I think
everybody's got exactly the same vision and goal in mind.
I'll
also say that I came into this two years ago obviously knowing
nothing about elections other then casting my own ballots, so
with quite a bit of trepidation, but I'd like to thank all the
TGDC members for explaining all these things to me, the NIST staff
for being absolutely phenomenal.
I
said at the beginning, this is one of the most important projects
that we could be working on. It affects the life of every single
American. You should be incredibly proud. I'm incredibly
humbled to be working with you on this and I am very, very proud
of the product that we have delivered. So thank you all very
much.
And
with that if there are not any other last comments or questions,
I officially then adjourn this meeting and I would like to offer
a round of applause for everybody who worked on this. So thank
you.
(APPLAUSE)
And thank you to all the TGDC members as well. This
meeting is adjourned.
(END OF AUDIOTAPE 2, SIDE B)
CERTIFICATE
OF AGENCY
I,
Carol J. Schwartz, President of Carol J. Thomas Stenotype Reporting
Services, Inc., do hereby certify we were authorized to transcribe
the submitted cassette tapes, and that thereafter these proceedings
were transcribed under our supervision, and I further certify
that the forgoing transcription contains a full, true and correct
transcription of the cassettes furnished, to the best of our ability.
_____________________________
CAROL J. SCHWARTZ
PRESIDENT
ON THIS DATE OF:
_____________________________
|
