Skip to content

Findings from Funnelback search usability testing

Last November I wrote about the initial findings from the control usability test that was conducted on Google Search Appliance (GSA), the University’s search system at the time. The main outcomes from that test were that University search results were inaccurate, there was a distrust of keymatch boxes and there was little interaction with search filters.

Since then, we now have a new search system in place which uses Funnelback. When this was launched I conducted the same usability test again to see if the changes had improved user experience.

Screenshot of new search results page

Key findings from Funnelback search usability testing

  1. Funnelback has not drastically altered the users’ search habits.

Many of the behaviours displayed in the GSA test were repeated in the Funnelback test, highlighting that the new search system has not affected the way users interact with the University search, despite its new design.

This was particularly evident in the way that users continued to rely on the first couple of results shown and refine their original search term rather than scroll through the results, or select another page for their desired result.

There was also the issue that users still had to specifically know what it was they were searching for, otherwise their result was unlikely to appear next to the top of the list. One example of this is when one user searched for “Term dates” for semester dates and was taken to a school term time page. Based on experiences like this, the user stated “I think it’s less effective than using the Current students page or Google search”. They mentioned the Current students page particularly because it had tasks categorised in larger sections, suggesting the removal of any guesswork as to what page the user need at a particular time.

This was also the case when a user searched “Learn German free time” to find the evening degree programme, and no useful results were returned. A reliance on knowing what to search for makes the search tool a page-driven search rather than a task-driven search. Only one participant highlighted the requirement for filters and assets. This mirrors the finding from the GSA test whereby none of the participants used these features of search.

  1. Search functions on separate sections of the website cause major usability issues.

A major usability issue found during the test was when a participant completed their initial search using the main university search, but then became trapped in a loop when then trying to search from the Department of Film Studies website. There was nothing to indicate to the user that this search was different from the main University search. The user attempted to complete a search several times before travelling back to the University homepage to start their search again there.

Header section of Film Studies webpage

  1. Users completed tasks faster using Funnelback.

The average time to complete a task for all participants was 49 seconds. (Staff: 58 seconds, students: 41 seconds). This is faster by approximately 12 seconds than the GSA test participants. Other than average time to complete tasks there were no noticeable difference in the findings from staff and student participants.

Improved time spent completing tasks does not always equal improved usability, as variables such as the user’s experience with the site and the testing environment can skew results.

User feedback

Overall the feedback for the new search was fairly neutral, leaning to positive as all participants were able to complete all tasks without much issue. However, one member of staff provided negative feedback because they were unable to just search an individual School site anymore. Another felt that there were too many results shown at once in the search results list, which made it feel overwhelming. The user stated this as a reason they would prefer to use Google search in certain instances.

Some positive example feedback included:

  • “[The search is] straightforward and intuitive I guess”
  • “I think it’s decently easy – kind of like Google search. You have to know which words to use. […] other than that, it’s fairly simple.”
  • “It’s pretty simple to use, it was pretty easy to find what I was looking for. What I was looking for was at the top of the list.”
  • “It’s a lot better, a lot easier […] the design of the page is a little easier to scroll through and things jump out at me more”
  • “It was fairly clear – it’s fairly similar to Google so people will be used to it.”

Further updates to Funnelback search will be implemented in the future, and after that more usability testing will take place to monitor whether or not the changes affect the usability of the tool. If you have any feedback on the new tool, please email itservicedesk@st-andrews.ac.uk.

Related topics

Share this story


Leave a reply

Your email address will not be published. Required fields are marked *