The usability
tests determine if the disclosure information provided
on the Web by state disclosure agencies is accessible to
the average citizen. To
do this we designed a test to answer the following question:
“Can a non-expert find basic, informative data about
campaign finances on the Internet in his or her state without
undue difficulty or investment of time?”
Most
usability tests compare a handful of web sites, and are
concerned with minor differences between them (see Steve
Krug's “Don't Make Me Think” (2000)).
Web site designers might be concerned about the location
of a task bar on a web page or the use of drop down menus.
They hire testers to sit in front of computers and do simple
tasks, and the web designers watch how they navigate around
the site. The Grading State Disclosure usability test is
different. We were not concerned with minor differences
but with gross ones. Dozens of interfaces were compared
across 50 states, and the test measured whether the overall
design of a state's web site – from architecture to jargon
to database – facilitated access to information by the
average voter. The two types of testing do share a common
trait, however. In both types of testing it is not necessary
to determine which design is optimal, only which design
is better.
Two standard measures of usability were used. The
first was a degree of difficulty measure, on the assumption
that difficulty and accessibility are inversely related. Subjects
were given three tasks to perform and the test measured the
time and number of mouseclicks it took to perform each task. The
three relatively simple tasks were devised, after some experimentation,
to represent the minimum any citizen should expect from a
campaign disclosure site. Subjects were asked to:
(a) locate the state's disclosure web site starting from
the state's homepage; (b) ascertain the total contributions
received by the incumbent governor in his or her last campaign
(subjects were given a list of incumbent governors that included
the year they were last elected); and (c) provide the name
and amount contributed by any individual contributor to the
incumbent governor's last campaign.
The second measure of usability was a survey. After
the third task was completed, each subject was given a short
questionnaire and asked to evaluate his or her experiences
on each state's web site. Subjects were asked to rate
their level of confidence in their answers, whether the web
site's disclosure terminology was understandable, and to
provide a ranking (one to five) of their overall experience
on the site. Subjects were also asked if any special
software or unusual browser plug-ins were required to access
the site's disclosure information.
Subjects were recruited from the undergraduate student population
at UCLA, and the experiments were conducted at the California
Social Science Experimental Laboratory (CASSEL) at UCLA. The
experiment was administered twice to thirty students each
time, for a total of sixty unique subjects. The states
were assigned randomly to subjects, and each subject was
assigned five states. Each state was tested by six
different subjects. Limits were imposed on the amount
of time a subject could take with each state and each subject
was given no fewer than 25 minutes to complete the three
tasks for each state. Each experiment lasted no longer
than 150 minutes, and some subjects were finished after 90
minutes.
There
were two concerns about the time and mouseclicks data that
were collected: first, subjects might learn during the
experiment and become more proficient with the later states
than the earlier ones; second, there might be subject effects
(level of competency, prior experience with disclosure
web sites, etc.). To address these issues, a fixed-effects
ordinary least squares model was constructed to control
for subject differences, and included a variable to control
for the order in which each state was tested by the subject.
With these controls in place, each state's average time
and number of mouseclicks was estimated for each of the
three tasks. These scores were then combined into two separate
indices (Cronbach's alpha > .75) and ranked. The survey
data were also combined into a single index (alpha = .77)
and ranked.
Each state could receive up to a total of 27 points for
the usability test score. The distribution of scores
in the three separate indices (time, clicks and survey) was
examined and scores were assigned based upon the apparent
thresholds in the distributions. The top-ranked states
received six points each, the medium states received three
points, and the lowest ranked states received zero points
for each of the time and clicks indices. The remaining
15 points were assigned according to the survey responses,
with a maximum of 15 and a minimum of three points assigned
to each state. These three scores were then added
together to create the usability test score for the state. |