How do you test what someone doesn't expect to be able to do?

Interpreting the results of a usability test is tricky at the best of times, but in a study I recently conducted I found that it's even more difficult when the feature you're testing is something your users don't expect the software to be able to do. In this type of situation, the interface has to reveal the software's capabilities to the user in addition to making those capabilities easy to use. That's an additional constraint on the design. (Normally the user expects to be able to accomplish something with the software, and the interface's task is to match the user's mental model—their expectation of how the feature should—as well as possible, or to bridge the gap between their mental model and the system's implementation. In this case, the user has no mental model, or their mental model doesn't extend that far.)

So how do you test something the user doesn't expect to be able to do? And how do you interpret the results?

I can't discuss the details of my test but I can use a roughly parallel example. Imagine you were testing the Smart Playlist feature in iTunes (the ability to create a playlist that automatically fills with songs based on criteria you set, like genre, artist, rating, or any combination of attributes). Many iTunes users I know have never heard of this feature. If you were testing it, what would you do?

What we decided to do was give them a scenario that we thought would supply the expectation for them. In the iTunes example, this might be something like: "Create a playlist that automatically includes all of your Rock songs from the 1980s."

Now imagine that all of your test participants immediately tried searching for 1980s Rock songs in their music library and then looked for a way to add these to a playlist. And when you informed them that there was another way to complete the task, they tried creating a playlist and calling it "1980s Rock."

In this case, the scenario seemed to prompt the participants to imagine capabilities the software doesn't have (saving search results as a playlist, filling a playlist based on its name), instead of the one it does have that they didn't expect.

Does that mean the Smart Playlist feature is difficult to use?

I don't think so. It seems like anyone who discovered Smart Playlists would bypass the confusion that was uncovered during the tests, and discovering it in context would do a better job at setting their expectations about it than the scenario did.

In short, it seems like the scenario created the confusion, and thus the difficulties encountered in the task really tell us little to nothing about the usability of the feature we set out to measure.

I'll have to try something different next time.