Grading State Disclosure 2004 Logo Graphic

Methodology of Usability Testing

golden bar divider

The usability tests determine if the disclosure information provided on the Web by state disclosure agencies is accessible to the average citizen.  To do this we designed a test to answer the following question: "Can a non-expert find basic, informative data about campaign finances on the Internet in his or her state without undue difficulty or investment of time?" 

Most usability tests compare a handful of web sites, and are concerned with minor differences between them (see Steve Krug's "Don't Make Me Think"(2000)).  Web site designers might be concerned about the location of a task bar on a web page or the use of drop down menus.  They hire testers to sit in front of computers and do simple tasks, and the web designers watch how they navigate around the site.  The Grading State Disclosure usability test is different.  We were not concerned with minor differences but with gross ones.  Dozens of interfaces were compared across 50 states, and the test measured whether the overall design of a state's web site—from architecture to jargon to database—facilitated access to information by the average voter.  The two types of testing do share a common trait, however.  In both types of testing it is not necessary to determine which design is optimal, only which design is better.

Two standard measures of usability were used.  The first was a degree of difficulty measure, on the assumption that difficulty and accessibility are inversely related.  Subjects were given three tasks to perform and the test measured the time and number of mouseclicks it took to perform each task.  The three relatively simple tasks were devised, after some experimentation, to represent the minimum any citizen should expect from a campaign disclosure site.  Subjects were asked to: (a) locate the state’s disclosure web site starting from the state’s homepage; (b) ascertain the total contributions received by the incumbent governor in his or her last campaign (subjects were given a list of incumbent governors that included the year they were last elected); and (c) provide the name and amount contributed by any individual contributor to the incumbent governor’s last campaign. 

The second measure of usability was a survey.  After the third task was completed, each subject was given a short questionnaire and asked to evaluate his or her experiences on each state’s web site.  Subjects were asked to rate their level of confidence in their answers, whether the web site’s disclosure terminology was understandable, and to provide a ranking (one to five) of their overall experience on the site.  Subjects were also asked if any special software or unusual browser plug-ins were required to access the site’s disclosure information. 

Subjects were recruited from the undergraduate student population at UCLA, and the experiments were conducted at the California Social Science Experimental Laboratory (CASSEL) at UCLA. The experiment was administered five times to ten different students, and five different students tested each state. The states were assigned randomly to subjects, and each subject was assigned five states.  Each state was tested by six different subjects.  Limits were imposed on the amount of time a subject could take with each state and each subject was given no fewer than 25 minutes to complete the three tasks for each state.  Each experiment lasted no longer than 150 minutes, and some subjects were finished after 90 minutes.

There were two concerns about the time and mouseclicks data that were collected:  first, subjects might learn during the experiment and become more proficient with the later states than the earlier ones; second, there might be subject effects (level of competency, prior experience with disclosure web sites, etc.).  To address these issues, a fixed-effects ordinary least squares model was constructed to control for subject differences, and included a variable to control for the order in which each state was tested by the subject.  With these controls in place, each state’s average time and number of mouseclicks was estimated for each of the three tasks.  These scores were then combined into two separate indices (Cronbach’s alpha > .75) and ranked.  The survey data were also combined into a single index (alpha = .77) and ranked. 

Each state could receive up to a total of 27 points for the usability test score.  The distribution of scores in the three separate indices (time, clicks and survey) was examined and scores were assigned based upon the apparent thresholds in the distributions.  The top-ranked states received six points each, the medium states received three points, and the lowest ranked states received zero points for each of the time and clicks indices.  The remaining 15 points were assigned according to the survey responses, with a maximum of 15 and a minimum of three points assigned to each state.  These three scores were then added together to create the usability test score for the state.

Back to the Grading State Disclosure home page

Back to Appendices Next to Glossary


This page was first published on October 25, 2004
| Last updated on October 25, 2004
copyright ©
Campaign Disclosure Project. All rights reserved.