VVSG TRAINING NARRATION
NIST BOULDER Mark Skall Presentation
October 15, 2007

VVSG Tutorial Narration*
Standards 101

[Slide 1]

[MR. SKALL:] Welcome, I'm Mark Skall, and I guess for the first few minutes, I'll just talk about what we're going to be doing today.
The EAC called us, I guess, a few months ago and asked us to do this training for voting officials, which we agree is very important. We know that you all need to really understand the VVSG. So we will tell you everything we know, but I'm going to talk about some of my experience in standards, developing standards, developing conformance, and relate it to the VVSG. So I've entitled it Standards 101.

[Slide 2]

So the objectives of this session are to establish a common understanding of concepts and terminology. We're going to talk about standards, requirements, and conformance. We're going to talk about motivation for the new VVSG and set the stage for the rest of the presentations on the VVSG.

[Slide 3]

So as I'm sure you know, we've been focusing our work over the last few years in three areas: core requirements, security, and human factors.

And as I speak during this presentation, there are some words I'm going to use synonymously. The term 'implementation' is kind of a standards geek word. It basically is, you have a standard and then that standard gets turned into a product. That product, in standards, we call an implementation. It's often times synonymous with the term 'system,' and in the VVSG, that's what we call a voting system. So that's equivalent, implementation, system, and voting system.

The person or persons who develop that implementation are called implementers or developers, and what we call them in the VSSG are manufacturers.

Now the last line, standards specification and guideline, really are not synonymous. A specification is a detailed enumeration of requirements. When that specification becomes publicly available and shared among various constituencies and blessed by a standards-making body, it's usually called a standard.

A guideline is a little bit different. A guideline is typically a document that has recommendations, not requirements. It's kind of a softer document.

The VVSG, Voluntary Voting System Guidelines, has the term G, guideline. In all shape and form, however, that is a standard, that's a specification. So sometimes I use standard, specification, or guideline to talk about that particular document, but really it is a standard, and I'll get into that in a few more minutes.

[Slide 4]

So the outline is what is a standard, what does it mean to conform to standards, and what's the difference between conformance and certification. I'll talk a little bit about conformance testing, and then summarize and end with the improvements that we've made to the previous sets of standards.

[Slide 5]

So what is a standard? Again, VVSG, Voluntary Voting System Guidelines, voluntary means the use is not mandated by law or regulation. If you decide to claim conformance, however, you must adhere to its requirements. So the fact it is voluntary is one thing, but if you conform to it, you are obliged to meet all the requirements.

A standard is established by consensus or authority and prescribes technical requirements to be fulfilled by a product, process, or service, and a requirement is criteria, characteristic, behavior, or functionality that a system must do or have.

[Slide 6]

So good standards are really a key because what we really want are good implementations, good systems, and the only way to get that is to move from detailed requirements in a specification or a standard into the actual product.

So the goal is correct, reliable software or correct reliable systems, and the requirements are captured in the standard, and again those requirements need to be translated into the voting system.

So in order for that to happen, that standard needs to be clear, it needs to be precise, it needs to be unambiguous, it needs to be complete, and it needs to be testable.

Now those of us who communicate through the English language understand how inexact the English language can be. Oftentimes I say one thing, and it's interpreted one way. Someone interprets it a certain way, another person interprets it a third way.
We need these specifications to be precise. We need to have no failure of communication. In an ideal world, that specification wouldn't even be written in English. It would be written in some fancy mathematical notation.

And I know there's been concern about the readability of the VVSG. The other alternative, and this happens in many other forums, is you actually get a specification in pure mathematical notation. That hasn't always worked because it's not very readable, but there are actually standards written in what we call a formal specification or math.

Oftentimes people designing mission-critical systems like aerospace, airplanes, will develop specifications and formal specifications. Of course here we're doing it in English, but again that English has to be precise and we also know it has to be readable.

[Slide 7]

So just an example about imprecision in English, the sentence, "The girl touched the cat with a feather." So I could mean the girl has a feather touching the cat, or some other people may interpret it as the girl is touching the cat, which has the feather.

And these examples have nothing whatsoever to do with specifications, but they abound in the English language, so it is tricky and you have to be very careful when you use words in English to define a very, very detailed specification.

[Slide 8]

So what makes a good standard? One that gets used, used correctly, implemented consistently. One that defines what and who needs to implement the standard, and in this case, it's voting systems and test labs. VSTLs are the people that need to implement the standard.

A good standard distinguishes between normative and informative text. Normative is necessary information that you need in order to conform to the specification; informative is everything else.

So for instance in the VVSG, there are requirement sections labeled as requirements, and then there are discussion fields. Discussion is informative. Everywhere that there is a requirement, that says requirements, and in our conformance clause, it distinguishes and tells you what is normative, what is informative.

A good standard tells you what needs to be implemented, what is mandatory and what is optional, and typically the way this is done is through key words, and that's what we use in the VVSG.

There are three primary key words. The key words are in caps in the VVSG.

So "shall" is mandatory. If there is a "shall" requirement, it has to be implemented. "Should" means the requirement is optional, but it's recommended. "May" means it's optional and permitted; there's no recommendation one way or the other.

A good standard is modular. So, for instance, we have different sections talking about different parts of the VVSG. You don't want to mix things together.

A good standard is adaptable as things change. We do not want standards changing every year or two. A moving target is no good for implementers; it's no good for users. It's no good for voting officials, so we want a standard that's adaptable.

[Slide 9]

One of the ways we've done this in the VVSG is through the innovation class, so that new technology can be built into the standard without starting all over and developing a new standard, and one that is technology- and design-independent.

Now technology-independent means that requirements are not tied to a specific technology. So, for instance, we don't require that voting systems be built on any specific platform.

Design-independent means requirements tell developers what to build, not how to build it, just like a good manager tells their underlings what to do, not how to do it. Same idea with design-independent, you would rather tell people what to do, allow innovation, than tell them how to do it.

[Slide 10]

So there are three types of requirements in general in a standard. There are functional requirements specifying that the object is capable of performing a certain action, and you see an example of that.

Performance requirements specify not only that the object is capable of performing the action but also set a benchmark for how well it performs, so typically you must meet or exceed that benchmark.

And a design standard specifies something about the static structure of the object. For instance, any control buttons on the voting system must be at least one inch apart.

Now that sort of violates the design independence principle that I just mentioned. So, in general, we try to concentrate on functional performance requirements. We do have a few design requirements in the VVSG, primarily in usability, but the intent is to do as little of that as possible, and Sharon and John, I'm sure, will talk about that.

[Slide 11]

So are standards enough? And clearly, clearly they're not enough. Standards are a piece of paper. When this VVSG is finished, it's a piece of paper, and if other stuff doesn't happen, then you're just left with a piece of paper.

So I'm here, I work for NIST, and it's a big S for "Standards," but I'm telling you standards are worthless unless they're implemented. Again, it's just a piece of paper. If you don't have a product, you don't have a voting system. They're worthless and they're useless unless they're implemented correctly.

So how do you get it to be implemented correctly, and that's where conformance and testing come in.

[Slide 12]

So this next diagram shows a triangle with a three-legged stool, is the way that I like to describe this, where you have the standard or the specification, again I'm using those terms synonymously, at one point of the triangle, the voting system which is the thing that needs to be developed from the specification at another point, and conformance tests at another point of the triangle.

So you then run the conformance test against the voting system that purports to conform to this specification at the top, and unfortunately you only have two possible things that can happen, you either find an error somewhere, you find an error in one of the functional requirements, you find an error in the security point, usability, accessibility. You find at least one error, and you know for sure that that voting system does not conform to that standard.

The other choice is you find no errors. You go through the whole testing regimen, and you find absolutely no errors.
Unfortunately, you really don't know anything then. You know either that the system conforms perfectly, and it does all the requirements exactly the way they're supposed to, or perhaps your tests were not comprehensive enough to find the error. Testing can never prove an implementation is correct. That's the state of the art in computer science.

[Slide 13]

Okay, I'm going to talk about a conformance clause. All good standards should have a conformance clause, and by the way, the term 'clause' is really misleading because our conformance clause is about what, 50 pages.

Now some conformance clauses are very small; I'll get to that in a second. Others, unfortunately, like ours, are long.
So conformance clauses address what needs to conform, how to conform and claim conformance. Typically they talk about subdivision of a standard, so if a standard is divided into modules, some are divided into nested levels, a conformance clause will describe how these standards are subdivided.

Conformance clause talks about different ways a specification is allowed to conform differently. So everything is not monolithic. You don't have one way to conform in voting, you have many different systems, DREs, op scans, others. So there are many different ways you can conform obviously to the VVSG.

[Slide 14]

I'm going to define some terminology. This typically comes out of ISO directives and other documents that we found.
Conformance again is the fulfillment of a product, process, or service of specified requirements. These requirements are defined in the conformance clause or elsewhere in the specification.

The reason some conformance clauses are short and some are long is that a conformance clause could be as short as talking about the subdivision of specification and then saying the requirements are defined elsewhere in the specification. It's a pointer to all the other requirements. So conformance clauses get into requirements in detail.

Conformance clauses are a section of the specification that states all the requirements, all criteria that must be satisfied to claim conformance.

[Slide 15]

Conformance testing is a way to determine directly or indirectly that relevant requirements are fulfilled, and even without certification, even without processes, conformance testing is always useful.

You'll have a standard on the marketplace, and you will see invariably people say they conform to this standard. It doesn't matter what domain we're talking about.

Conformance tests help buyers increase their confidence that something actually conforms. If you pass tests, you don't know for sure it conforms, but it increased your level of confidence. And sellers help substantiate their claims if they've been tested.
Typically conformance tests are performed by test labs to determine if the voting system conforms to the VVSG.

[Slide 16]

Conformity assessment is a process necessary to perform conformance testing in accordance with prescribed procedures and a test suite. It ensures that testing can be repeatable and reproducible.

And certification is acknowledgment, typically by a certification body or certification authority, in this case the EAC, that a conformity assessment was completed, and that the criteria established for issuing certificates were met.
So typically the EAC will look at a test report, but they'll have additional criteria as well which certification typically has. They may be looking at conflicts of interest and other things.

So certification usually looks at first conformity assessment, looks at the outcomes of testing, and then has additional criteria as well.

[Slide 17]

One of the things I've just talked about in conformance testing, I sort of want to reemphasize. It may seem obvious, but one can only test for requirements in the standard. That's what conformance testing is.

You have a set of requirements. You can't say we'd like to add this other requirement and test for it. I mean you can, you can test for it, but that's not conformance testing. You can't fail an implementation for failure to conform to a standard unless that requirement is in the standard.

The analogous case is you can only test for the requirements in the standards as they are stated, and if I had a dime for every time someone came to me and said, I wrote a requirement to read as it does in A, but I really mean B.

Well, it doesn't matter what you meant. What it says on the piece of paper is what you test for. If there's a problem, if there's an inconsistency, you need to go back and change the standard. You can't test for what you meant; you can only test for what the standard actually says.

[Slide 18]

Conformance testing is also in a way in the VVSG. The VVSG includes testing requirements for test labs. It not only has requirements for voting systems, it also has requirements for test labs.

The VVSG indicates general testing procedures. We have a list of sort of generic testing procedures in the VVSG like inspection, functional testing, vulnerability testing. Every time we have a requirement, we talk about the testing procedure that should be used to test that requirement. That's in the VVSG.

There are also documentation requirements in the VVSG, not only for voting systems but also for test labs as well. So test labs must have test plans, pretesting, they must have test reports, so there are requirements in the VVSG for how tests labs do conformance testing.

There are also procedural requirements in the VVSG, various procedures for hardware setup, OEVT, open-ended vulnerability testing has procedures. So there are requirements in the VVSG for all of these.

The VVSG, however, does not contain the actual test suite. Right now the actual tests are developed by test labs. They've been developed by test labs throughout the history of standards in voting.

The problem with that is that although those tests may be very good, we don't get to see them because they're proprietary, they're trade secrets if you will of the test labs, so no one can look at them to see how good they are, and perhaps just as importantly, we know they're not uniform, because different test labs develop their own test suites, and it would be very unlikely for those to be exactly identical.

So the problem is if you get tested by one test lab, you're being tested by a different test suite. What we are doing at NIST is developing a comprehensive test suite, we are releasing it incrementally over the next two or three years, and this test suite will be available to all the test labs.

All the test labs, we believe, will work with our procedures, will try to ensure that they use that same set of tests, so that everyone will know it's a comprehensive test suite that tests for the new VVSG, and that test suite will be consistent and uniformly used by all test labs.

[Slide 19]

Okay, now this is a really interesting diagram, and it talks about the things I've just discussed as a series of building blocks, and if you look at the bottom, there's the standard. After that, you get conformance testing, then conformity assessment, then certification.

The interesting thing is everything beneath is a prerequisite to do things above, so unless you have a standard, you can't do conformance testing. Unless you have the standard and conformance testing, you can't have conformity assessment. Unless you have all three, you can't have certification.

The other interesting point about this sort of set of building blocks is, in the standards world, you could stop at any place. In voting, we're going all the way up to certification, but there are many standards that exist just at the bottom level, just the standard.
There are standards out there, a tremendous amount don't have any conformance tests, no conformity assessment, no certification.
There are others where there are just standards and conformance tests but no certification. We do that at NIST in many areas.
For instance, the World Wide Web standards that you see today that allow you to use your browsers correctly, we worked with the W3C to develop a standard. We developed the conformance test.

Those conformance tests are out there. They are used voluntarily by the implementers, by Netscape, by Microsoft, and in fact, they're used by every person that develops browsers for the World Wide Web, because they are free resources. Why wouldn't anyone want to use them?

So there are many ways you can develop this process. We in the voting world go all the way to the top: standard, conformance testing, conformity assessment, and certification.

[Slide 20]

I grouped the improvements to previous standards into five areas. Again, we've defined what it means for voting systems to conform. We've created precise and testable requirements. We've created performance benchmarks, which didn't exist before. We addressed new technological advances and added security, accessibility, and usability requirements.

[Slide 21]

Again, the conformance clause, normative versus informative; conformance, by the way, and I think it is stated in the conformance clause, is 100 percent. It's kind of like being pregnant. You either are or you're not. You're not partially pregnant, and you're not partially conformant.

The conformance clause defines classes. These are groupings of requirements by voting devices, and I won't get into more detail, and I'm sure you have questions about them, and the detailed questions will be answered by the experts in classes.
There's an implementation statement. All that is, is a statement requiring the voting system to talk about what optionality it supports. That's a requirement of the conformance clause.

The next thing is extensions. Extensions in the standards parlance are additional functionality above and beyond what's required in the standard, and we allow, as most standards do, extensions.

The idea is you promote advancement and innovation when you allow extensions, and this, I believe, may also be helpful. States may have additional requirements above and beyond the VVSG, so if you as a state develop a voting system with additional requirements, as long as you don't violate what's in the VVSG, that system will still conform to the VVSG.

Software independence is a new concept that I won't get into in detail because John will, but it is defined in the conformance clause.

[Slide 22]

Again, improvements through previous standards: so we have tried to specify precise testable requirements. The intent and the objective is all requirements should have only one interpretation. Everyone should understand what is meant. That's the goal.
Now we have many hundreds of pages and probably somewhere in there, although this was our goal, it may not be perfectly precise, and if it isn't, we try to find that out and correct it, but the goal again is precision and no ambiguity.

The goal is to make every requirement testable, the ability to determine that that requirement has been met. We feel if the requirement can't be tested, then it's really not a meaningful requirement, with the possible exception of these goal requirements.

[Slide 23]

Now, this was a late edition to my slide set. It was pointed out that we do have a few requirements in the VVSG that are what we would call goal requirements, sort of higher level for the most part, nonspecifically testable requirements, and John, do you have the slide up with the examples of some of those?

So just to summarize, there are just a few of these in the VVSG, and I think different people in the TGDC may have wanted these because they want to have sort of a goal for implementers, even though they know it's not testable, to encourage certain things.

[Slide 24]

So I think we have a couple of those in the accessibility section. I think there is something that says systems shall be end-to-end accessible, which isn't very specific.

[Slide 25]

We have a couple of goal requirements in OEVT and integratability.

[Slide 26]

So those are kind of vague, and they're really exceptions to the rule, and they are really a trade-off of having this high-level sort of essentially non-testable requirement, but that's what was voted, but for the most part, these are exceptions to the rule.

[Slide 27]

One of the things we've done that hadn't been done in the previous standards is benchmark requirements both in usability and in reliability, and accuracy testing.

There are many new technological advances that we had to account for in these sets of voluntary voting system guidelines, both in the 2005 one and the new one.

[Slide 28]

VVPAT, we had to deal with wireless and how it affects the voting scenario, electronic poll books, EBMs and digital signatures, so these are all accounted for in the new VVSG.

[Slide 29]

And just sort of as a summary slide, there are many, many new areas that didn't exist before, many new chapters on security in these different areas, and many new expanded human factors sections.

So that's my introduction. Before we get into John Wack's presentation, questions or comments?

[Slide 30]

[QUESTIONER:] Are all the goal requirements clearly marked in the discussion part that they are goal requirements?

[MR. CUGINI:] The question over there was, are the goal requirements clearly marked, and the answer is no, they're not.
My point I wanted to make, I want to disagree with Mark a little bit. You sort of say the goal requirements aren't testable and leave it at that. I think that's a little severe.

You can test them, but you typically need to test them with expert judgment. I think it's fair to say they're testable, but the test method is more subjective than the test method for other, more specific requirements, but it's not as if they won't be tested at all, so I think they're testable.

[MR. SKALL:] It depends on the requirement.

[MR. CUGINI:] Well, the one I have in mind is testable. Maybe some aren't either.

[MR. SKALL:] I think the one that you guys did is testable. The integratable one, I don't know how you would test that systems shall be integratable.

[MR. CUGINI:] Yeah. At least the testability is more problematic than for a more specific requirement, leave it at that.

[MR. SKALL:] But that's a good point. I think I would be very welcome for public comment to talk about maybe making sure those goal requirements are certainly marked as such and something in the discussion field highlighting that. That's a good point.

[Slide 31]

[NARRATOR:] Additional explanatory presentations on the voluntary voting system guidelines can be accessed from the Web site: vote.nist.gov.

Page Created: November 28, 2007
Last Updated: June 26, 2008

Privacy policy / security notice / accessibility statement
Disclaimer / FOIA
NIST is an agency of the U.S. Commerce Department