Creating a new content structure - UX testing (part 2)

How did we select items to test?

Since we have a reasonable size of navigation consisting of a few dozens items, we included all of them in the study. We worked with items that came from a series of grooming sessions, as result of improving the old tree structure of Sense/Net ECM.

How many participants are to be invited for a Tree Test?

The sample size question initially comes down to the outcome metric. Because the tree test is basically a mini-usability test, we can use the same metrics in a usability test along with the same procedure to identify sample sizes. In general, the key metric will be whether the user successfully located an item, which is a binary measure like task completion ("found/didn't find" coded as 1 and 0 respectively). The more participants are involved the less the margin error statistically is. We targeted to involve at least 50 participants.

How many tasks should each participant be asked?

Since the complexity of the navigation and difficulty of the items we estimated, that one needs around 1-2 minutes to complete a task. So we run the tree test with 9 tasks, which takes a median time of 13.5 minutes, which does not include the time for users to answer post item questions (personal data and message, and difficulty).

Show task 1

Well-designed SN application

We are absolutely clear about what this test is going to be used for, so by designing the test, we aimed for something that was:

  • Quick for an information architect to learn and get going on
  • Simple for participants to do the test
  • Able to handle a large sample of users
  • Able to present clear results

Tree Testing Tool

We’ve made a tool for tree testing in Sense/Net which is Javascript plugin built on Sense/Net ListItem content and OData REST API. There’re a custom Content Type for the task and the survey items. Survey items have only a LongText field for containing the clicked treeitems and some additional information about the steps made by the user (e.g. skipping a task or restart searching for the path with another root element) in JSON. The Javascript tool builds-up a tree based on a hierarchical JSON object and it’s rendered with KendoUI’s tree widget.

Evaluating results

We built in a couple of useful tools to support evaluating of results. Different visualization tools were applied for different datatypes to support the evaluation of the result of the statistical analysis by the measured parameters.

  1. Overview: we collect summarized results introduced in an overall chart-bar of
    1. Number of participants
    2. Success
    3. Directness
    4. Time taken


  2. Analysis: we introduce the results of analysis on various charts collected in various groups.
    1. Task results will be shown in collections per tasks. As percentage value by all answers results are introduced on a pie chart and bar-chart as well. Also the directness of answers are marked differently. The average time spent on answering is shown up by task.


    2. First click logs per task can be followed on the next tab in a table. Also further clicks rate appears in the last column.

      First click

    3. Paths: all path per completion and per task is introduced here, and also marked with a set of colors if they were successful or failed, even directness is readable from this view.


    4. Destinations: on this tab all labels are listed in a matrix by tasks. Each cell represents the number of completions where the specific label the destination was.


5. Messages: messages can be read here, that are added on the last page of the survey, from participants by completions

Comments are closed

Welcome to the blog!

Sense/Net ECM is ever evolving. Community means the world to us so we want to keep you apprised on what’s happening on our side of the woods. Want to make us happy? Add a comment and tell us what you think!

Month List