So you want to set up a test that allows you to understand how well your target users can comprehend content designed for their use. How do you design something that provides you with reliable data?
As always, we need the right users, so the very first thing we want to do is make sure that we more or less recruit users for testing that not only meet the general demographic profile, but they also match up in terms of reading ability. What this means is that you need to test the users so you can gain a decent benchmark of their reading comprehension level.
You can do this as a means of filtering during recruitment, or you can do it pre-test, after recruitment, so that you can better correlate user performance with user capability. In other words, if I knew that my book in question was for 7th graders and I needed to test comprehension of content, I’d make sure that no one tested the book unless I could verify they had at least a 7th grade reading level. That way, after testing, when I tried to make conclusions about reading comprehension, I would know I wasn’t going to change the book’s structure, even content, based on what people with non-targeted reading levels told me to do based on their performance or other feedback.
Testing users at the outset then is our first step to generating reliable data for analysis.
Cloze Test for Screening
I would suggest, as a means of finding the right folks for testing, using the Cloze formula. It’s empirical and easy. See how it works here: http://www.useit.com/alertbox/cloze-test.html. Basically, you take a passage of text from your target publication, replace key words, and then have your users as part of screening replace these words. Users that have the right comprehension will be successful in filling in the blanks. Others won’t. The ones that get it right move on to testing, the others don’t.
During Testing
So now that you’ve found the right users, you need to employ some different methods to get the right data to help you make conclusions about reading comprehension. Remember, triangulation. You can’t just observe comprehension. Reading with my mouth open, reading aloud, reading with stops and starts, reading slow…they may indicate comprehension difficulty but they may not. But if I experience things like this as a reader, I make errors in comprehension, I take too long, then I’m doing things that offer more reliable indications of comprehension issues.
So How to Triangulate for Reading Comprehension
These are ideas you want to experiment with, but they all help to get at comprehension issues:
- Time to read
- Errors in comprehension
- Use the Cloze testing for a task during testing
- Ask comprehension questions after the fact
- Verify even before you do all this that your text is at the level you think it is by using different formulas, such as Flesh-Kincaid:
- http://www.standards-schmandards.com/exhibits/rix/
- http://www.online-utility.org/english/readability_test_and_improve.jsp
- Make use of a comprehension rubric, http://specialed.about.com/library/comprehensionrubric.pdf; a rubric like this allows you to make better judgments about what you observe during or after testing
Yes, you can look into design, spacing, font, or other cosmetic features, but it’s best to do comparative testing of these. Give half of your sample an example with say a line spacing or line length different than the example used by the other half of your sample. If your users have comprehension levels equal, then this sort of comparative will be useful.
So, to summarize, use something like Cloze to recruit and filter the population correctly, make sure your actual document content is at the comprehension level you think it is with a formula like Flesch-Kincaid, and then in the end use triangulation to evaluate the data.
That’s a start, and then as you accumulate more data you can begin to create your own rubric based on your knowledge for designing comprehendible content.
Associate Professor, Technical Communication Director, Usability Research Lab Texas Tech University
No comments:
Post a Comment