How did we select items to test?
Since we have a reasonable size of navigation consisting of a few dozens items, we included all of them in the study. We worked with items that came from a series of grooming sessions, as result of improving the old tree structure of Sense/Net ECM.
How many participants are to be invited for a Tree Test?
The sample size question initially comes down to the outcome metric. Because the tree test is basically a mini-usability test, we can use the same metrics in a usability test along with the same procedure to identify sample sizes. In general, the key metric will be whether the user successfully located an item, which is a binary measure like task completion ("found/didn't find" coded as 1 and 0 respectively). The more participants are involved the less the margin error statistically is. We targeted to involve at least 50 participants.
How many tasks should each participant be asked?
Since the complexity of the navigation and difficulty of the items we estimated, that one needs around 1-2 minutes to complete a task. So we run the tree test with 9 tasks, which takes a median time of 13.5 minutes, which does not include the time for users to answer post item questions (personal data and message, and difficulty).
Well-designed SN application
We are absolutely clear about what this test is going to be used for, so by designing the test, we aimed for something that was:
- Quick for an information architect to learn and get going on
- Simple for participants to do the test
- Able to handle a large sample of users
- Able to present clear results
Tree Testing Tool
We built in a couple of useful tools to support evaluating of results. Different visualization tools were applied for different datatypes to support the evaluation of the result of the statistical analysis by the measured parameters.
- Overview: we collect summarized results introduced in an overall chart-bar of
- Number of participants
- Time taken
- Analysis: we introduce the results of analysis on various charts collected in various groups.
- Task results will be shown in collections per tasks. As percentage value by all answers results are introduced on a pie chart and bar-chart as well. Also the directness of answers are marked differently. The average time spent on answering is shown up by task.
- First click logs per task can be followed on the next tab in a table. Also further clicks rate appears in the last column.
- Paths: all path per completion and per task is introduced here, and also marked with a set of colors if they were successful or failed, even directness is readable from this view.
- Destinations: on this tab all labels are listed in a matrix by tasks. Each cell represents the number of completions where the specific label the destination was.
5. Messages: messages can be read here, that are added on the last page of the survey, from participants by completions