What we learned from tree testing

To improve the structure of Sense/Net ECM, we decided to do some UX research on the tree structure of the Content Repository before starting the actual work. To ground this study we conducted a test -which we have also written about- with active Sense/Net users. Our participants on average had 4.5 years of experience with Sense/Net, as developers, site builders, or consultants. As the result of these discussions, we developed a completely new system, including a lot of improvements based on the feedback we received.

Objectives

  1. Improve the tree structure of SN7 on ways
  2. Make the Content Repository more intuitive
  3. Test prototype iteratively
  4. Validate our hypothesis with numerous participants
  5. Engage with existing and new customer segments

Methodology

Participants were Sense/Net users, who were invited to take part in the research by email, from all over the word. The test was available online, so people could give a response from the comfort of their homes.
With all questions and tasks, individuals had multiple choices according to their decision: after reading the task, they could skip the task directly, or begin to carry it out. Even after starting, one could decide any time to rather skip question, in case they couldn’t give a relevant answer. This was to prevent any false result, deriving from inconclusive answers when participants do not feel familiar with the specific question.
We practically measured two factors: whether the participant found the target (success) and how easily did they found their way (directness).

Overall result of the research

Completion rate of the whole test was 45%, compared to all invitations. After all a total of 37 persons took part in the research. We received evaluable results from all of them. Research (http://www.measuringu.com/blog/tree-testing-ia.php) suggests, that this amount of data results within a 15-17% margin of error.
The average time, participants spent per task, was 4.16 minutes.
The overall success through all 9 questions was over 57%. 30% of all successful targets were found directly, without reversal.

Overall results of tree-test

Results per tasks

There were two questions related to configuration-issues, that seemed to be out of line: results for these questions dropped out of the standard deviation.
The rest of the results, in general showed 70-78% success with 27-37% directness
We found one task, that showed up with 91% success and 67% directness. We will analyze this item later.

Success and directness ratio

On the figure above, test-results show that directness and success are related to each other.
The higher rate of success and directness by a task, show that it was easier to find the desired path to the key element.

Interpreting results within the standard deviation, we see, that even the lowest directness value was about 30%, and the same measure was observable by success rate. In average, the success showed a value over 70%, and directness resulted close to 40%, which confirmed our presumptions about the new structure, that we assembled on the basis of preliminary studies.

Tasks

We have included tasks in relation with:

  • customizing Pagetemplate
  • creating custom ContentType
  • adding localization string resource
  • updating custom ContentTemplate
  • finding Auditlog
  • setting up user-related tasks
  • creating shared content

In general we ascertained, that more than half of first clicks, in average 72% were executed on the right place.
Finding localization files caused difficulty to users: almost half of first clicks (17 occurs) landed on “System” folder before finding the right place (70.27%).

When we asked participants to find where they can create a shared document, we found that over 70% of first clicks landed on the appropriate “Shared content” folder, with an over 90% success rate.

Task-result chart

The eventual tree structure in SN7

After having analyzed the answers, we finalized the Content Repository’s Root-structure as follows:

New tree-structure in SN7

Feedback

We received several messages at the end of the surveys, with all adherent and forward-thinking proposals. These all help us a lot and keep us going. Your messages confirmed, that our efforts are worthwhile. We understood that reorganizing the structure urges you to rethink the use of the Content Repository, but we also learned that you found the logic and usability of tree composition improving.

Comments are closed

Welcome to the blog!

Sense/Net ECM is ever evolving. Community means the world to us so we want to keep you apprised on what’s happening on our side of the woods. Want to make us happy? Add a comment and tell us what you think!

Month List