|
The usability
tests determine if the disclosure information provided
on the Web by state disclosure agencies is accessible to the
average citizen. To do this we designed a test
to answer the following question: "Can
a non-expert find basic, informative data about campaign
finances on the Internet in his or her state without undue
difficulty or investment of time?"
Most
usability tests compare a handful of web sites, and are
concerned with minor differences between them (see Steve
Krug's "Don't Make Me Think"(2000)). Web
site designers might be concerned about the location of
a task bar on a web page or the use of drop down menus. They
hire testers to sit in front of computers and do simple
tasks, and the web designers watch how they navigate around
the site. The Grading State Disclosure
usability test is different. We were not concerned
with minor differences but with gross ones. Dozens
of interfaces were compared across 50 states, and the test
measured whether the overall design of a state's web site—from
architecture to jargon to database—facilitated access
to information by the average voter. The two types
of testing do share a common trait, however. In both
types of testing it is not necessary to determine which design
is optimal, only which design is better.
Two standard
measures of usability were used. The
first was a degree of difficulty measure, on the assumption
that difficulty and accessibility are inversely related. Subjects
were given three tasks to perform and the test measured the
time and number of mouseclicks it took to perform each task. The
three relatively simple tasks were devised, after some experimentation,
to represent the minimum any citizen should expect from a
campaign disclosure site. Subjects were asked to: (a)
locate the state’s disclosure web site starting from
the state’s homepage; (b) ascertain the total contributions
received by the incumbent governor in his or her last campaign
(subjects were given a list of incumbent governors that included
the year they were last elected); and (c) provide the name
and amount contributed by any individual contributor to the
incumbent governor’s last campaign.
The second
measure of usability was a survey. After
the third task was completed, each subject was given a short
questionnaire and asked to evaluate his or her experiences
on each state’s web site. Subjects were asked
to rate their level of confidence in their answers, whether
the web site’s disclosure terminology was understandable,
and to provide a ranking (one to five) of their overall experience
on the site. Subjects were also asked if any special
software or unusual browser plug-ins were required to access
the site’s disclosure information.
Subjects
were recruited from the undergraduate student population
at UCLA, and the experiments were conducted at the California
Social Science Experimental Laboratory (CASSEL) at UCLA.
The experiment was administered five times to ten different
students, and five different students tested each state.
The states were assigned randomly to subjects, and each subject
was assigned five states. Each state was tested by
six different subjects. Limits were imposed on the
amount of time a subject could take with each state and each
subject was given no fewer than 25 minutes to complete the
three tasks for each state. Each experiment lasted
no longer than 150 minutes, and some subjects were finished
after 90 minutes.
There
were two concerns about the time and mouseclicks data that
were collected: first, subjects might learn during
the experiment and become more proficient with the later
states than the earlier ones; second, there might be subject
effects (level of competency, prior experience with disclosure
web sites, etc.). To address these issues, a fixed-effects
ordinary least squares model was constructed to control for
subject differences, and included a variable to control for
the order in which each state was tested by the subject. With
these controls in place, each state’s average time
and number of mouseclicks was estimated for each of the three
tasks. These scores were then combined into two separate
indices (Cronbach’s alpha > .75) and ranked. The
survey data were also combined into a single index (alpha
= .77) and ranked.
Each
state could receive up to a total of 27 points for the
usability test score. The distribution of scores
in the three separate indices (time, clicks and survey) was
examined and scores were assigned based upon the apparent
thresholds in the distributions. The top-ranked states
received six points each, the medium states received three
points, and the lowest ranked states received zero points
for each of the time and clicks indices. The remaining
15 points were assigned according to the survey responses,
with a maximum of 15 and a minimum of three points assigned
to each state. These three scores were then added together
to create the usability test score for the state. |
