BOARD OF ADVISORS & STANDARDS BOARD
VVSG TRAINING WORKSHOP
Sharon Laskowski, OCTOBER 16, 2007 (Part 1)
NIST BOULDER

VVSG Tutorial Narration*
Usability, Accessibility, and Privacy Part 1

[Slide 1]

[NARRATOR:] This is Part 1 of the next Voluntary Voting System Guidelines Training Module covering Chapter 3 of the VVSG, Usability, Accessibility and Privacy of Voting Systems. Parts 1 through 3 of this VVSG Training Module are presented by Dr. Sharon Laskowski of the National Institute of Standards and Technology's Information Technology Laboratory. The presentation includes questions and answers from the members of the Election Assistance Commission's Board of Advisors and Standards Board.

[MS. LASKOWSKI:] Good morning, everyone. I will be talking about Part 1, Chapter 3, the Usability, Accessibility, and Privacy Chapter.

[Slide 2]

So this presentation includes key concepts for the HFP requirements. I'm going to do an overview of this chapter. I'm going to go in linear order. However, note that, for the most part, this new version of Chapter 3 has gone under incremental, not global, change from the VVSG05, with the exception of the performance requirements. So I basically plan to step through the different sections pausing for questions, but for things that have not gone through any substantial change from VVSG05, I'm going to gloss over. I have notes and will go over explanations of things that are more complex and that are newer. Initially I'll go through the design requirements. Most of these, as I say, have not changed much. And then since the newest and the most important things to discuss are the usability performance requirements, I'm going to put those at the end, and I have a large chunk of slides. I will provide definitions as needed as we go through.

[Slide 3]

I've listed, just for your convenience, the acronyms that I use in this presentation. ACCVS, accessible voting station; DRE- direct recording electronic; EBM- electronically assisted ballot marker, such as the Automark; MMBP- manually marked paper ballot; VEBD- voter editable ballot device with a -a or -v, it could be an audio VEBD or a video; PCOS- precinct count optical scanner; ATI- we haven't talked about that yet, is the audio tactile interface; CDR is the cast vote record; IVVR- we heard a little bit about that yesterday, actually quite a bit about it, is the independent voter verifiable record. In some parts of the standard, you also see the acronym VVPR which is voter verifiable paper record. In general, I just refer to paper IVVR. It's the same. VVPAT- voter verifiable paper audit trail. CIF is an ISO standard for a format for usability test reports, and VPP is voting performance protocol. So you can refer back to this if you get confused by the acronyms.

[Slide 4]

Okay, so I'm going to go through a few key concepts first. What is the goal of our requirements? The goal is to provide a voting system that voters can use comfortably, efficiently, and with justified confidence that they have cast their vote correctly. The focus is primarily on the voter's interaction with the voting system, but we do have some requirements for poll workers. Again, we are looking at that human interface with the voting system.

[Slide 5]

Another key concept that's important to understand, our philosophy developing these requirements has been the notion of universal usability. We view the voting station as a public kiosk, designed for everyone, not a special purpose. In other words, you have to look at the whole range of users. And keep in mind that many people have some special needs, but they don't necessarily identify themselves as having disabilities, such as an aging population. And so we have tried to move as much accessibility into the general voting station as possible. It's important to note, although we have a usability and an accessibility section in this chapter, all the usability requirements apply to the accessible voting station as well.

[Slide 6]

Next slide: Key Concept- Accessibility. Our goal in designing the accessibility requirements is to make the voting system independently accessible to as many voters as possible. We've tried to harmonize with other accessibility standards, and we've worked closely with the U.S. Access Board in doing this. We made a choice in the organization of Section 3.3 by organizing it by the type of disability being addressed. However, sometimes you'll see features that apply across disabilities, and that's so noted in the explanations- the discussion sections. So, for example, the audio which is intended primarily for those people who are blind, also may assist people who are dyslexic. So, it's not a perfect organization, but we found that this was the simplest organization we could come up with.

[Slide 7]

Another key concept, and you heard a little bit about this yesterday, is voter verification and in particular, accessible voter verification. Let me spend a few minutes on this, because this was new with this VVSG. When software independence for security purposes was introduced in this version of the VVSG, it had implications for accessibility. This has been a very thorny issue for the TGDC, and there's been many, many hours of e-mail and discussion on this. Right now in current systems, software independence can only be addressed by using paper IVVR. We hope in the future there will be other methods, but for now, paper is what we have. So the thorny issue from an accessibility point of view is that paper by itself is not accessible. Someone who is blind can't directly verify paper. We struggled with how to write requirements to make the IVVR accessible, and I'll be talking in more detail as we get to those requirements. Let me at this point introduce the notion of observational testing. Given that voters who can't see the paper still will want to verify that their vote has been recorded correctly must use some assistive technology such as audio read back to verify. So, how do you know that indeed that read back is accurate? One way to do that is to make sure that some sighted people also use the accessible voting station and both listen and look to make sure that the read back is performing accurately. Now note that we also often make sure other folks without disabilities use the accessible work voting station as well as stations that are being used for alternative languages, just to preserve privacy- you want enough people using it. So this is not something that is brand new. The purpose again is to just make sure that this read back is working properly, in this case for the voter verification.

[Slide 8]

The next key concept I want to talk about is design requirements versus performance requirements. Design requirements, for the most part, specify the look and feel of the voter interface, based on best practices for other similar domains. The proper use of color, what's your font, and you typically test that by just inspecting or having experts review to make sure those requirements are met. The problem with design requirements is that they don't necessarily ensure the usability of the system when a voter is actually interacting with it. We know, and we have the documentation of other domains based on best practice for similar interfaces that they do improve usability, and the other part is that they are very specific to different kinds of design, so they can stifle innovation. But the bottom line is you want good usability performance. And so a lot of our work for this new version of the VVSG has been looking at how we can design a test and some usability performance benchmarks using a test method called "Usability Testing with Users," and in our case "Voters," based on best practice of how usability testing is done in industry that actually measures the interaction and counts the accuracy of the voters interacting with that system. [Slide 9]
I see Jim has a question. Yes.

[QUESTIONER:] Sharon, could you just go back to the explanation that you just went through, because I'm not sure I followed it?

[MS. LASKOWSKI:] Okay. Jim wants me to go through the explanation I just went through and that is design versus performance. Okay, design requirements give you very specific things about how an interface looks, what size fonts, what functional characteristics you would want say an audio interface to have, for example. That is that you can repeat, you can skip, etc. These are very specific to the design, the color of the buttons, placement of the buttons, whether or not scrolling is allowed, and these have to be based just on designs that we currently see and that we know in general work well. However, the proof of the pudding is really, can a voter use this system and interact with it, that's performance. What we have tried to do with the new performance requirements is to actually put together a test with voters in a very controlled situation, in a lab situation. I'm going to go into details later on at the end of this talk about that- that we actually measure the usability performance and actually count errors and see how well a voter is doing, and we can measure the inherent usability of a system and compare it to other systems. So, if we can set a benchmark of performance there, we can separate out systems with good usability versus bad usability. In the end, the design doesn't really matter, it's can the voter perform well with that system?

[QUESTIONER:] Does that mean that in the end of this process, voting system X will have a 90 percent usable number and voting system Y will have a 92 percent usable number?

[MS. LASKOWSKI:] Yes, we are going to, and I am going to go over this. There's three benchmarks that are numbers, and the benchmark is a number that you pick that says if the system does better than this, they pass, if they do worse than this, they fail. So, the trick here is that you've got to control your environment very carefully so that this test can be repeated in a lab, and you get the same consistent results every time.

[Slide 10]

Okay, first, as I said, I'm going to go linearly through the standards, and I am going to talk about first Section 3.2, the general usability requirements.

[Slide 11]

To ensure that the manufacturer has a user-centered design and testing process, we have a requirement that says that usability testing by the manufacturer for the general population, and "the manufacturer shall conduct summative usability tests on the voting systems, using individuals who are representative of the general population, and shall report the test using the common industry format as part of the technical data package."

So what we are doing with this requirement, which is Section 3.2.1.2-A and others that require the CIF reporting, is just to make sure that the manufacturers are doing some usability tests with users, that is, voters. I don't care what the results of the tests were, the point of this is that they report that they did this; it means that they have some usability engineers on board, on their staff, and they are looking at that part of the software engineering process. That's kind of a very low-level requirement. It's not new in this VVSG, by the way.

[Slide 12]

Question?

[QUESTIONER:] Matt Masterson, with the EAC. In the 2005 VVSG, the requirement was very similar to that as far as reporting or conducting their own usability test, correct? And that they shall report it as part of their technical data package to the EAC. Do we know who they must report this to?

[MS. LASKOWSKI:] Okay. Matt Masterson asks, VVSG05 had something similar and it's to report to the EAC. In fact, they are identical. The wording, I guess, did confuse some manufacturers. This is part of the technical data package and that is submitted to the test laboratories.

[Slide 13]

Okay, the next slide: 3.2.2 - Functional Capabilities. The functional capabilities are models closely on the HAVA provisions- nothing new here. When you have a voter editable ballot device, voters can easily change their votes, so we have a set of requirements about notification of overvoting, undervoting, the voter can go back and correct the ballot, etc. No change there.

[Slide 14]

Yes.

[QUESTIONER:] So does this mean that on a central count optical scan system, if they are using that kind of system in the polling place, the voter is marking up his scanned ballot for the ballot box, how are voters notified? And I may be misreading this section, but if it applies to all systems?

[MS. LASKOWSKI:] Ann asks how this applies to precinct optical counts, I'm sorry central count optical scan-

[MR. CUGINI:] Let me take this. HAVA was written with that in mind, and there is a clause in HAVA, I'm going to paraphrase very loosely, to the extent that if you sort of post a notice on the side of the voting booth that says, "Here's how you change your ballot." That's good enough. HAVA was written specifically not to preclude central count systems, and so this has the same force as that. That's what central count systems have to do.

[QUESTIONER:] Then there might be an inconsistency between HAVA and this requirement, because if you just read this requirement, it looks like central count optical scan wouldn't pass these guidelines, the notification of overvoting or undervoting.

[MR. CUGINI:] Notice it's notification of the effect of overvoting which is subtly different. That says the system tells you if you overvote, this is what's going to happen. It does not necessarily tell you in real time that you have overvoted, because a central count system cannot do that.

[QUESTIONER:] Right, but I guess that is what I don't understand. You said the voting system shall notify the voter, and a central count system won't notify the voter.

[MR. CUGINI:] Are you looking at 3.2.2-A?

[QUESTIONER:] Yes.

[MR. CUGINI:] Again, this is tricky and a little subtle. It says the voting system shall notify the voter of the effect of this action. Then it says in discussion, in the case of manual systems, this may be achieved through appropriately placed instructions. So, that discussion was put there, again, to, we obviously didn't want to go beyond- If HAVA made it a point to allow central count systems, we certainly don't want the VVSG to outlaw central count systems. This is fully written in a way to do that.

[Slide 15]

[MS. LASKOWSKI:] It's a subtle point. So the point is here you have the most flexibility with the VEBD, with PCOS they have some notification about casting failure so the voter can ask for a new ballot to mark. But then with central count, there's not too much you can do.

[Slide 16]

Slide for Privacy, 3.2.3: The voting process must preclude anyone else from determining the content of a voter's ballot without the voter's cooperation. Privacy ensures that the voter can cast votes based solely on his or her own preferences without intimidation or inhibition.
And this is a section for privacy at the polls. There are requirements for system-supported privacy, making sure there is visual privacy, audio privacy, warnings of errors also remain private, and there are no receipts. No change there from 05.

[Slide 17]

The next slide, 3.2.3.2, is no recording of alternate format usage. This is also related to privacy when voters use a nontypical ballot interface, say with an alternative language or large print, their vote still needs to remain anonymous. So these requirements basically say that in the cast vote record, there is no record of what type of interface they used or any special formats, only the logical content of their ballot.

[Slide 18]

[QUESTIONER:] How do you do that with an optical scan ballot?

[MS. LASKOWSKI:] They apply only to electronic.

[MR. CUGINI:] Yeah, 3.2.3.2. Even if you look in the discussion, the last sentence says, "In the case of paper ballots where the interface is the record, some format information is unavoidably preserved." So the following two are worded, no information shall be kept within an electronic CVR that identifies alternative languages and so on.

[Slide 19]

[MS. LASKOWSKI:] Jim, question?

[QUESTIONER:] Yes. I just want to be clear I understand something regarding paper. The manufacturers of the phone voting system have been saying- right now they print the ballot on regular paper, so it looks different from the optical scanned ballots that other people use. Do I read this as saying that under these regulations, the manufacturers could continue the practice of printing the phone-in ballot on paper stock that is different than what other voters are using?

[MS. LASKOWSKI:] Okay. Jim asked the question what about vote by phone and the printout of the paper ballot as the record of that vote by phone. They are using a different format. I think that would be subject to some interpretation. Off the top of my head here, these requirements apply to the electronic cast vote record. So, I think this would not apply to how they are printing out that paper. One would hope if they are just recording it, they are not recording the special format that was used. I suppose by the mere fact that one used vote by phone does identify you as using an accessible voting station. Presumably if you have enough people using it, that wouldn't identify, that would still maintain privacy of those voters.

[MR. CUGINI:] Yeah, let me generalize that a little. That's an interesting point that if you had a mixed system where you had electronic ballot markers and manually marked paper ballots. I mean the vote by phone is interesting just in because it's an EBM. Right now I don't know of anything that would forbid- that mandates that the manually marked paper ballots be indistinguishable from the paper produced by an EBM if you had such a mixed system. That's kind of an interesting case. I don't think, right now, that we have addressed that specific possibility.

[MS. LASKOWSKI:] Wendy, do you have a question?

[QUESTIONER:] Going further on this, if you are going to require, until something is developed that gives independent verification other than paper, then the electronic devices are going to have to produce a paper record also that is going to be distinguishable.

[MR. CUGINI:] The issue is differences among paper ballots. Ideally, all the paper ballots kind of look the same, but there are cases, obviously, if you have a Spanish paper ballot, it doesn't look like an English paper ballot or a large print paper ballot doesn't look like a regular size paper ballot. So, I guess that's no better or worse in the verification case if you produce a VVPAT that produces Spanish which I think they have to do, that Spanish is still going to look different from English. So I mean yeah. This whole idea of privacy with a cast ballot record, an electronic cast vote record, they can all look the same. The minute you go to paper, invariably you are producing, if it's something the voter is looking directly at, the paper is going to look different. Since the voter never looks directly at the electronic record, you can make all the electronic records look the same. But I don't see a way to make all the paper records look the same.

[MS. LASKOWSKI:] So, let me say the other speaker here is John Cugini from NIST, just for the record. And since the purpose of these requirements is to address privacy, all you can do is to make sure you have, procedurally that you have enough people using that machine or producing that paper so that you can't identify a specific voter.

[QUESTIONER:] I just have a comment. That's not possible, if only in a sense that if you have multiple voting machines in which the voter can vote in English or Spanish or in any other language, they can go pick their own machines, and you are not allowed to ask them what language they are going to vote in. You have no way of assuring how many people vote at a specific machine.

[MS. LASKOWSKI:] Yeah, I mean, Larry points out that there are certain limitations to that, you can't necessarily direct people or know what alternate format they are going to be using. You can mitigate it a little bit, but, yeah, it's not, definitely not a perfect solution.

[Slide 20]

On to Section 3.2.4 - Cognitive Issues: Let me remind you that this is still in the usability section. It applies to all voting stations, accessible and those not designated as accessible. These requirements are intended to minimize kinds of difficulties for voters. To start with, the instructions shall be complete and at any point in the voting process, help should be available from the system.

[Slide 21]

The next slide is plain language. It is also a part of cognitive issues. The use of plain language and clear language helps many people including voters with certain types of cognitive disabilities, but also people who are voting at the end of the day and they are tired and they normally aren't cognitively disabled but they are under stress or they are tired. So, these plain language ones based on best practice are meant to make sure that the language is as clear and simply written as possible. Jim?

[QUESTIONER:] Does that language up there address or ignore the possibility of party icons?

[MS. LASKOWSKI:] We are only talking about the text here. There is another requirement later on that says if there is any iconography used, there should be accompanying text associated with it. And then anyplace there is text, plain language would apply.

[Slide 22]

Okay, next slide: still continuing with cognitive issues. No biases among choices- this is a fairly less precise, more of a goal type requirement here. But certain differences in presentation are mandated, of course, by state law like order in which candidates are listed, but font size, voice balance, speed must be the same for all choices, so that you don't bias a voter towards one particular candidate. And 3.2.4-E ballot design under cognitive issues. Here we can't go into specifics of what the election officials are designing in terms of ballot, but the voting systems do provide certain basic ballot design capability. So, basically we are talking about the voting system itself has the capability to design an easy-to-use ballot.
So for example, placement of instructions is something that often on, say a DRE, is fixed. Matt has a question?

[QUESTIONER:] Matt Masterson with the EAC, 3.2.4-E, I think that it is a nice concept, but how are you going to test it?

[MS. LASKOWSKI:] This is the basic design that you see, the capability that the screen interface provides. So, you can test that by either putting in some sample ballots and looking at it- looking at where instructions are placed, for example- looking at the ability to consistently place, you know, font size. But it is a bit complex. You are going to have to design some standard ballots that you run through and you look at what the system does and doesn't provide this capability. I think it's definitely possible to put together a test method with some sample ballots that look at what the voting system allows you to do.

[QUESTIONER:] Okay, except the phrase "high level of clarity and comprehensibility" is sort of an ambiguous term. The test labs will come back to us and say, what you mean by "high level of clarity and comprehensibility"? What meets the level of "high level of clarity and comprehensibility"?

[MS. LASKOWSKI:] So, Matt asks the question what is a high level of clarity and comprehensibility? I think we are going to struggle with some of those issues as NIST works on test methods in the coming year to try to pin that down a little better. Question here?

[QUESTIONER:] I'm not sure how are you going to prevent a particular issue from spanning columns on the ballot. We routinely deal with pretty extensive statewide constitutional amendments that are virtually a half a ballot in length.

[MS. LASKOWSKI:] Keith asked the question how you prevent, on certain ballots you have to go across pages, etc. Indeed, in some cases, for specific ballots, there have to be exceptions. There's no way around that. There we just have to rely on some of the best practices for ballot design for election officials that the EAC has put out, etc. And I also recommend that when one does such a ballot, that some usability testing on that specific ballot is done, not just kind of eyeballing it. Does it look okay, but run it by a few voters. Do a little bit of usability testing at the state level to ensure that you haven't done something that's going to cause a large error. And it is a 'should.'

[QUESTIONER:] Okay, are you suggesting that we bring voters in prior to Election Day and let them vote?

[MS. LASKOWSKI:] If you have a ballot design, for example, a butterfly ballot to save space, if you bring in five voters and test it. See how they interact. See if they've got some confusion. At least you have some level of- so it's fairly inexpensive to do that kind of testing on a ballot. One often asks for feedback anyway. This gives you a little more precise feedback in terms of how the voters are going to react to a particular design.

[Slide 23]

Question here?

[QUESTIONER:]. It's a great suggestion, but that is beyond the scope of the VVSG.

[MS. LASKOWSKI:] Yes, yes, absolutely. That's right.

[QUESTIONER:] To ensure there is no bias among choices, the font size or voice volume and speed, that's usually in the purview of the person doing the layout of the ballot.

[MS. LASKOWSKI:] Many of those things are controlled by the voting system. So, in other words, you don't want the voting system to allow you to put in different volumes in the ballot itself. So, insofar as it's controlled and managed by the system, it's in the scope. Otherwise it's not, yes, it's a fine line that we struggled with. Question, Jim.

[QUESTIONER:] Yes, there have been several times when you've mentioned, have a few voters to do this or that. Is there anywhere in the VVSG where there's a definition of what a few is?

[MS. LASKOWSKI:] Oh yes, I'm sorry. Jim asks, I've referred to a few voters in a usability test rather informally, and Jim says, is there anywhere in the VVSG where we talk about that? No, we do not, but there are other standards. For example, the ISO Common Industry Format for usability test reports that precisely describe how one would perform such a test. So, there are standards that elections officials could use to run these kinds of usability tests. But it's out of scope of the VVSG.

No, my suggestion, it's not in here. When we were talking about ballot design for a specific election, I was merely pointing out the fact that there are ways to double-check that one has not put in some design, fatal design flaws into the ballot design. Jim has another question.

[QUESTIONER:] I have two questions on the ISO standard. Does that include people with disabilities?

[MS. LASKOWSKI:] It's strictly a format that says how many users and what goes into a good usability test. So, one would imagine a test at the state level but obviously the VVSG can't mandate how the states do their testing. But there certainly is best practice in industry that tells you how to do that.

[QUESTIONER:] For election officials, this system of how to do user testing is a best practice, it's not a 'you've got to do it this way.'

[MS. LASKOWSKI:] That's correct. Jim asks if this is our best practice kind of advice. Yes, so I am just pointing out there is a lot of best practice advice out there. Question, Wendy?

[QUESTIONER:] Okay, both in the design things that came from the EAC and in some of these things, you talk about, you know- a lot of these things will increase, particularly on an optical scan paper ballot, the size of the ballot to the point that you get into two pages of ballots and usability. Is there going to be usability testing between the problems associated with the names versus the problems associated with two-page ballots, such as the straight party vote moving from one page to another, or the judges handing out two of one page and none of the other, something like that?

[MS. LASKOWSKI:] Yeah, it's really out of the scope of the equipment standard. So, but it-

[QUESTIONER:] But it is the design things that impact the usability.

[MS. LASKOWSKI:] Many of these are 'shoulds,' and they are suggestions or they are things that the system itself can encourage.

[QUESTIONER:] I guess, you know we keep hearing the usability and how we aren't making these things usable, but nobody wants to come up with how do you resolve some of these usability issues? You may create some problems with them, and so it keeps getting sent to somebody else, is what I am saying and we end up being the culprits when after an election we-

[MS. LASKOWSKI:] Yeah, Wendy says, the bottom line is election officials have to design their ballot for their election. The equipment standard can't directly help with that. Some of the guidance that has been coming out on designing can aid that, and just like you refer to security experts and best practice in security, there is best practice in usability that election officials can rely upon to help them with that. Again, out of my scope here. Question, back here?

[QUESTIONER:] I think there will probably be quite a few comments on this section from Design for Democracy in which-

[MS. LASKOWSKI:] Yes, yes, absolutely, yeah.

[QUESTIONER:] In fact, I know there will be, because I wrote the questions about the section. One thing I did not understand was just a quick comment you made which is that the equipment design can't assist with these issues. I don't know if I did that correctly or not, but it seems to me that it can very much by providing election officials with options, and the standards could require the vendors to provide election officials a variety of different options in terms of design so that they can meet some of those recommendations that have been made.

[MS. LASKOWSKI:] John says I alluded to saying there are certain things the standard can't assist with, but perhaps there are some other things. I guess what we struggled with and we would certainly welcome comments from Design for Democracy and other such experts and will look at them very carefully, because more voices on this is a good thing. We can put in somewhat best practice, but we are looking at the equipment, not the ballot design software per se, plus there is different election law and regulations about ballots across the states, and we couldn't address that. So, we were rather limited in what we could look at.

[QUESTIONER:] And that is precisely why election officials need to be given a lot of options, because as Wendy was saying yesterday, her state law requires all capital letters of candidate names.

There are a variety of state laws like that, but there are some officials, some jurisdictions that are going to be able to not only have the room on their ballots to accommodate some of these things, but have more generous budgets than others, and so they are able to do some of these design standards, and it's very important that the standards both for software and hardware accommodate as many options as possible in design for election officials.

[MS. LASKOWSKI:] So John says it's important that these standards accommodate as much for design. We have tried to err on some flexibility with the standard. And what we can hope is that, over the years, there is a set of best practices and further research on what sorts of options make sense, what are good practices in ballot design for election officials, but that has been, as I say, out of scope for this equipment standard.

[QUESTIONER:] Excuse me. I'm a little concerned about the effects of this chapter, of this set of issues becoming part of this document. I really don't know that they- his comment was good, I think the system should provide a local election official with various opportunities. For instance, there is an issue here about how the system shall provide the voter with the opportunity to get help at any point in time. I don't know how that occurs with optical scan, and I also don't know what the effect that has ultimately on the time spent at a voting booth. I mean I think there's- when you get somebody who's working their way- in some states, at least Ohio, has a five-minute voting time period, and you know there's a problem. This really keeps somebody in the booth for a tremendous amount of time. I just don't think that this, I don't know that this section is really part of manufacturing specifications. I don't think they can anticipate all the issues that they may encounter in ballot design from election to election at the local level. There's just not the way to do it.

[MS. LASKOWSKI:] Keith says in this section he thinks the scope is too broad. Is that a summary?

[QUESTIONER:] Well, it's too ambiguous, it's too general.

[MS. LASKOWSKI:] Would it-

[QUESTIONER:] I think to some degree it doesn't reflect the real-world use of the voting machine on Election Day.

[MS. LASKOWSKI:] Yeah, I believe your comment about assistance, that particular requirement is for DREs so that one, or electronic ballot markers. Oh, I guess there's discussion, so help on demand is something you would want from electronic interface for optical scan paper stuff, instructions have to be made available, written instructions, and that's in the discussion. I certainly think, sort of based on best practice for user interface design, one would want to have help and descriptions available if a voter gets stuck, although one would hope the usability is good enough that they don't get confused.

[QUESTIONER:] You're not saying not on the machine that may cause more confusion. If they are having a hard time, you know it's not going to help-

[MS. LASKOWSKI:] If it's done properly. So, yeah, we welcome certainly public comments on this section. It's a fine line of what we can specify in the equipment versus what is done at the local and state level. So we welcome public comments.

[MR. CUGINI:] Just let me make a very general point, that a lot of these issues are very valid, and this is why often there is a discussion section, because if you sort of read the naked requirement and, even worse, if you just read the title of the requirement, it's not always so obvious what the requirement is really saying. So, I just sort of urge you, read the title, read the text of the requirement, read the discussion, so at least we are kind of on the same page as to what we think we've covered. I'm not saying we've hit all the, that we've covered all the issues, but sometimes those issues were anticipated. So it's worth doing that, I think.

[Slide 24]

[NARRATOR:] Additional explanatory presentations on the Voluntary Voting System Guidelines can be accessed from the Web site: vote.nist.gov.

*Certain commercial entities, equipment, or materials may be identified in this presentation in order to describe an experimental procedure or concept adequately. Such identification is not intended to imply recommendation or endorsement by the National Institute of Standards and Technology, nor is it intended to imply that the entities, materials, or equipment are necessarily the best available for the purpose.

Page Created: November 28, 2007
Last Updated: July 10, 2008

Privacy policy / security notice / accessibility statement
Disclaimer / FOIA
NIST is an agency of the U.S. Commerce Department