Ballot Design Issue Causes Major Under-Vote in King Co. Washington

A blog post at  the horsesass.org blog published on November 11, 2009 brought attention to a surprising result on a ballot measure in King County's election held a week earlier.

David Goldstein of horsesass.org writes that, "forty-some thousand King County voters … were disenfranchised due to our state’s wholly inadequate ballot design and review procedures" because the ballot measure appeared on the ballot below the instructions in the left hand column.

It is a heavily researched and tested best practice in ballot design to put instructions for marking a printed optical scan ballot in the top left column on the first ballot page. Typically, this leaves a blank space beneath. Unfortunately, it's extremely tempting to fill that space, and election officials in King Co. did.

Although Washington State has been among the most progressive in implementing good ballot design practices, and in having local elections officials usability test ballots and other forms, King Co. tested after ballots had been sent out to voters. Their test revealed the problem that otherwise well-trained officials had missed, leaving them to expect a large under-vote on the measure. That’s exactly what happened. By the estimate of the state election director, Nick Handy, the undervote was somewhere nearer 50,000 votes.

The measure failed statewide by a large margin. If the measure had passed by 5,000 or so votes, this burp in King Co. would likely have tripped a recount because the 2-1 vote against in King Co. could have changed the result.

Goldstein of horsesass.org goes on to laud the Brennan Center's recommendation in their Better Ballots report that counties conduct usability testing before ballots are final using the guidelines and kit developed by UPA's Voting and Usability Project.

Trends in design issues in voting and elections

The elections track at IACREOT was geek heaven. I don’t mean just for election wonks like me, but also for anyone who cares about technology at all. The International Association of Clerks, Recorders, Elections Officials, and Treasurers (IACREOT) was in Spokane, Washington July 7-11.

A few themes ran through the panel discussions. I would argue that they all have design and usability components that need attention:

• Voter registration (databases)
• Military and overseas voting (email, Internet, encryption, troop security)
• The current Holt bill in Congress (proposes replacing all direct record electronic (DRE) voting machines with paper or optical scan voting systems)

Trend 1: Voter registration forms and databases

As the elections world chips away at solving problems in local jurisdictions such as security and managing recounts, more difficult problems arise to be solved. There was much discussion about creating a national voter registration database. Creating and maintaining such a database would solve problems with people being registered in multiple states and voting more than once in federal elections. This doesn’t happen a lot, but it would smooth operations in every single state in managing voter registration. In one session, a panel of elections officials from countries outside the US talked about (among other things) how they register voters and where the data comes from to maintain the voter registration databases. For example, in Canada, the Quebec provincial elections authority that is charged with overseeing elections to parliament maintains address and other personal information for voters (in Canada, they’re called electors), by getting feeds from the provincial socialized healthcare databases, which are constantly being updated by citizens themselves.


Trend 2: Military and overseas voting

With hundreds of thousands of troops actively engaged in conflicts, getting ballots to voters in the military and getting those ballots counted is an extremely difficult problem. For military personnel on bases that are not in war zones, it is somewhat easier; electronic means are available, such as faxes, to send ballots back to local jurisdictions, which can then be duplicated and scanned to be counted. But in war zones, some troops are in such sensitive situations, they only know each other’s first or last names. Finding these troops, getting them their ballots, and then finding ways for them to return their voted ballots that is timely, secure, and secret is a huge problem.


Trend 3: Holt bill on replacing voting systems (again)

It was the consensus of consultants and think-tank types that the bill that Rush Holt (D-New Jersey) has proposed will not only not be passed, but will probably never reach the floor for a vote, mainly because there are so many other, very important problems with Congress to deal with, and frankly, because the other Democrats who would back it have got what they wanted: a Democratic president and legislature.



Creativity and innovation abound in the elections world
Though attendance was lower this year than last year at IACREOT, there were plenty of stories of creative approaches to dealing with the complexities of voter registration and military and overseas voting. Many counties are trying combinations of new technologies in small experiments to make administration easier, but also to make it easier for voters to vote, no matter where they are in the world. In the meantime, they're looking for design guidance. What's the best voter registration form? How does that transfer to an online form? Many interesting design problems to be worked on.

Testing ballots: Real names or fictional? Direct how to vote or not?

How do you design a study to learn about ballot and voting system usability without doing the research during an election? The ideal situation would be to watch over voters’ shoulders on Election Day. But because we prize voting privately in the United States, observing voting on Election Day just is not an option. What’s a researcher to do?

It’s a challenging research situation.

What ballot should be used in the study? Should you use a real ballot from a recent election? What are the tradeoffs there? Next, you have to set up a situation that is similar to Election Day but isn’t. Turns out that making a study a lot like Election Day doesn’t really work for research.


Why researchers use constructed ballots with fictional names

There's a lot of research about whether fictional names are okay in voting studies, particularly by the people at Rice University in the ACCURATE project. They found that fictional names are okay as long as they're realistic. The NIST standard ballot for certification testing uses fictional names. Many researchers have picked up that ballot (or subsets of it) to use in their research.

When you use real names locally, it can be jarring if the design or the format looks different from what participants are expecting, and you instantly have artifacts from that. If there's one thing that does not look like the ballot they used, then voters notice and it's an instant distraction. So why not make the whole thing up?

Most researchers have decided not to use a real ballot from a recent election with. Why not? Using a constructed ballot, with fictional contests, names, and amendments or questions:

• avoids asking people to vote in a contest where they might have their own opinions, or where it asks them to reveal their political preferences
  
  
• levels playing field across levels of political interest
  
  
• allows constructing a ballot that can be used to test different usability issues in types of contests or tasks

Why researchers tell participants how to vote

Now, why not let study participants vote the way they want to? Why give them a slate to vote or task scenarios to work from?

In usability tests, researchers often ask participants to carry out pre-determined scenarios. Sometimes this is done to measure specific behavior, sometimes to make sure certain things are tested, sometimes to make sure that the facilitator is ready for the next expected thing. Part of the art of conducting a usability study is knowing when to let participants do what they want to do and knowing when to go back to the test design. (In the ideal world, what the participant wants to do and what you want them to do are the same thing.) Researchers in the elections space make this decision consciously and deliberately to make sure that they can collect the data measures that will prove (or not) a hypothesis.

Instructed voting makes it possible to evaluate error rates without directly observing the participant voting. People who study how people use other kinds of technology try to instrument the system to capture test data or observe directly. These things are difficult to do with voting systems. So, instructed voting versus "just vote as you might" asks the participants to be thoughtfully accurate, and not just randomly mark the ballot.


The special challenges of voting research

Researching how people interact with most technology, a researcher can go into the field, hang around where the person is doing what they want to do and ask questions. Election Day is not the time for that. Most voting research requires that the number of variables be limited and those that remain are controlled.
So far, researchers have found that using the NIST standard ballots and directed tasks is the best way to manage that.

-- By Whitney Quesenbery and Dana Chisnell

Plain language makes a difference: Voters are more likely to vote as they intend when ballot instructions are simple and clear

How well do voters and poll workers understand the language of ballots and voting system instructions?

Today the National Institute of Standards and Technology released its report of a carefully controlled quantitative study in which 45 US adults over a range of ages and education levels each voted two ballots that differed only in the wording and presentation of the ballot instructions. The participants also discussed and compared specific pages from the two ballots. Janice (Ginny) Redish, Ph.D. led the study. She found that:

• Voters voted more accurately on a ballot with plain language instructions than on a ballot with traditional instructions.

• Voters with less education made more errors in voting.

• Voters could tell the difference and preferred the plain language ballot by a wide margin.

See the full report here: http://vote.nist.gov/NISTIR-7556.pdf, or go to vote.nist.gov/docmap.htm.

Technology as a solution in 1957: "Behind the Freedom Curtain"

This 1957 film housed in the Internet Archives in the Prelinger Archives is a tutorial on how mechanical lever machines work -- with a little chauvinistic flag-waving thrown in for good measure. It is very much a period piece (remember HUAC?), but also does a good job of explaining the kinds of mistakes that voters make on paper ballots. The Automatic Voting Machine Co. that made the film makes the case that those mistakes can't be made on the lever machine. See: http://www.archive.org/details/Behindth1957.

Go from that film to the results of a review of the November 2008 election in Florida, where 15 counties switched from direct record electronic (DRE) voting machines to paper, optical scan ballots. Twice as many ballots were rejected in 2008 than were rejected in 2004. But still, the rejection rate was 0.75%, well below the 2.9% in 2000, and below the "expected" residual vote rate of about 1% on average. From the New York Times on February 26, 2009:

The final report sent to state lawmakers showed that 0.75 percent, or 63,680 of the 8.39 million ballots cast in Florida, did not count in the presidential race. Barack Obama defeated John McCain in Florida by more than 236,000 votes. In the presidential race four years earlier, the rate of uncounted ballots was 0.41 percent, equating to about 31,000 votes.

Read the whole New York Times article here: http://www.nytimes.com/2009/02/26/us/26florida.html?nl=pol&emc=pola1.

In 2004, George W. Bush defeated John Kerry in Florida by 380,978 votes. In 2000, Al Gore received 543,895 more individual votes nationally than George W. Bush, but Bush received more electoral votes. In Florida in 2000, the margin of victory was 0.0092%, in favor of Bush. (See: http://en.wikipedia.org/wiki/United_States_presidential_election,_2000 and http://en.wikipedia.org/wiki/United_States_presidential_election,_2004.)

Ever onward: A hope for continued progress in election reform

The Bush Administration spurred major election reform in the US. This reform movement included technology, security, and accessibility. It urged improvements in plain language in public documents, transparency in process, and alternative ways and days for voting.

Election reform has resulted in some great things: ability to vote for many who were unable to vote independently before; examination of steps in the process that hadn’t received much attention before, such as recounts and record-keeping; research-based improvements in ballot design – and much more.

But there is more yet to be done. Many voters are disenfranchised – still – by suboptimal design and the lack of usability in election materials they receive, voting machines they use, ballots they mark. Here’s hoping that the new administration continues to see election reform as a priority and continues and expands funding for research sponsored by the federal government that will result in easier, more accurate voting.

-- Dana Chisnell

Note: This post reflects the views of the person who wrote and posted it. It does not necessarily reflect the views and policies of the Usability Professionals' Association or its individual members.

Come hear: UPAers on ballot usability, plain language, and evaluating doc for poll workers

Several of us on the Usability in Civic Life Project at the Usability Professionals' Association will be speaking at events in 2009 on topics that we hope elections officials, design practitioners, and human factors researchers are interested in. The dates, places, events, topic titles, and speakers are listed below.

Date	Place	Event	Topic title	Speakers
Feb 6	Washington, DC	NASED	Usability Testing Ballots	Dana Chisnell
May 3-6	Atlanta, GA	STC	Rewriting the Voting Experience On Election Day	Susan Becker, Ginny Redish, Whitney Quesenbery, Josie Scott, Sarah Swierenga, Dana Chisnell
June 12	Portland, OR	UPA	Improving the User Experience of Voting	Ginny Redish, Dana Chisnell, Sharon Laskowski, Svetlana Lowry
July 7-11	Spokane, WA	IACREOT	Improving the User Experience of Voting	Dana Chisnell
July 19-24	San Diego, CA	HCI International	User Experience in Elections: Poll Workers	Dana Chisnell, Karen Bachmann