Over the summer, usability testing was conducted on the University’s search tool. The main purpose of the test was to set a benchmark and find out how users interacted with the current search system, which will shortly be replaced. This post provides a summary of the test and an overview of the users’ experiences.
Ten participants were used to test the University search: five members of staff and five students.
Before they completed the tasks, users were asked whether they used the current University search at all. This was an opportunity for us to gauge how many people used the search and gain an understanding of what they use it for on a regular basis.
The final usability test script for both audiences included the following tasks:
- Find semester dates.
- Find entry requirements for the Film Studies MLitt.
- Find evening language class information.
Participants were also asked a supplementary question to gauge their opinion on how they found using the University search.
Out of the ten participants asked, approximately 50% stated that they would use Google before using the University search. The reasons they gave for this were that the results were often more accurate and that they didn’t think the University search would show results for specific areas of the site, such as a School website.
Those who relied on the University’s search before Google did so because they were looking for University-specific information and didn’t want any external websites showing up in their search results.
University search results are inaccurate
Users reported that often when they are searching for things, the list of results is inaccurate as it displays irrelevant or out-of-date results.
This was evident when users completed certain tasks in the test. For example, for task 2, if a user searched for ‘Mlitt film studies entry requirements’, the first result that would appear was for the 2016 Film Studies course page, not the most recent year of entry. In task 1, when users tried to find the 2018-2019 semester dates, the 2017-2018 semester dates page was the first result.
Verbal user feedback supported this trend, with some participants stating that inaccuracies such as those listed above resulted in a sense of distrust when relying on the University search. One user summarised this overall finding when they stated, “You have to know the terms the University uses to be able to find what you’re looking for.”
Users rely on the first result listing
One notable pattern seen in the majority of the tests was that once a user submitted their search term, they were likely to select the first search result without hesitation.
This meant that they would not filter the search, scroll down the page to see what other search results were found, nor engage with the search result description.
This finding suggests that there is a certain expectation from users that the first result within a search list will automatically align with their intended destination. However, due to the inaccuracy and unreliability of the University’s search results list, this added to the overall feeling of distrust as they ended up on pages they did not intend to.
Distrust of keymatch boxes
Task 3 presented users with the option to select the keymatch box to take them towards the evening language course pages. Seven out of the ten users clicked on the box without hesitation, whereas the others preferred to find the right result for them within the search results list.
One user who did interact with the keymatch box stated that she used the box because it “gives [the result] status”. Another stated that it simply “looked like the most accurate result”, and one other indicated that they clicked on the keymatch box with the assumption that it would take them to the correct page.
However, users who disregarded the keymatch box made this decision primarily based on experience with Google ads appearing in the Google search results. One user expressed that these boxes often don’t show what she wants it to, so skipped it. Another highlighted that Google had in some way ‘trained’ him to recognise these results as adverts.
Certain participants who did select the keymatch box still expressed uncertainty regarding the box due to their previous experience with Google search results, with two participants stating that within a normal Google search they would always scroll past the adverts listed as the results in these boxes may not be relevant.
No users interacted with the search sort or filter functions
Upon doing an initial search, none of the users relied on the filter or sort functions to find what they were looking for.
Only one user out of the ten attempted to, but this was only after they could not find what they were looking for in the first couple of results. And upon attempting to interact with the filter option, they could not find an appropriate filter, so they instead refined their original search with the intention of finding more accurate results.
Users are more likely to refine their original search than explore additional search results
Once a participant had searched for something, if they couldn’t initially see what they were looking for in the first result, or if they had already clicked on that result and were taken to the wrong page, that user would then refine their original search for the right result.
Refinement was preferred by users instead of looking at and interacting with the additional results which appeared with their original search.
For example, one participant searched for ‘term dates’ and then selected the first result. This took them to the School term pages, which wasn’t what they were looking for. So, they went back to their original search and altered it to ‘Semester dates’. After refining the search, the participant again selected the first link and was taken to what he thought was the correct page.
This further highlights that users place the first one or two search results in higher regard than those further down the list. It is also worth noting that none of the participants during any of the tasks relied on any additional search pages other than the initial search results page.
Overall this test highlights just how little faith the users had in the search system. While all users could complete the tasks, this was overshadowed by the sheer level of distrust in the initial search results (either because they looked like ads or that the user had previous experience of poor results appearing.)
What do you think of these findings? Have you tried the University’s current search? Let us know what you think of it and how you would improve it in the comments below.