Redesigning RTÉ Player: Designing the Test Plan (4/5)
This blog is the fourth of five posts of an academic group project. The goal of the project is to iteratively redesign and evaluate the user experience (UX) problems of the RTÉ Player mobile app. In this blog, we will be looking formulated our test plan.
If you haven’t read the previous posts yet, you can find them in the links below:
I. Test Plan
Testing the redesigned RTE Player was done using remote concept testing. To gain quantitative insights and also gauge the success of the redesign, a system usability scale questionnaire was included.
Ideally, conducting a test session requires at least two resources - one who facilitates the meeting itself and another who takes down notes. This wasn’t a feasible scenario considering the challenges brought by the pandemic and by online collaboration. It couldn’t be helped for members of the project to conduct testing in their own free time. This may have lead to inconsistent recording of data.
As a workaround, a note-taking template was devised to help members transcribe consistently. The template was hosted in Miro where all members had access to. The template consisted of the of the test script, instructions, tasks and even probing questions. Notes were already colour-coded representing specific participants which helped simplify synthesis of results.
Before the actual concept testing, a dry run was conducted to catch any errors and help the team practice.
II. Recruitment
Ten participants were successfully recruited. Everyone formally gave their consent to participate by signing a consent form. Initially, it was targeted to recruit participants who used RTÉ Player already in the past. However, there was one participant who never used. It was decided to include that participant, seeing that it would also be a good opportunity to see how the redesign faired in terms of recognition over recall.
III. Results
A. SUS Results — Quantitative
All tests were successful and results were fairly positive. The overall SUS score given by the participants is 89.2%. The redesigned app also received positive feedback:
“It’s so cool! And it’s so cool that you have Pick by Mood, I’d definitely browse with that.”
Interestingly there some comments as well for improvement:
“Better than current app. Live chat is good but have to make sure user set up the profile before using the app. It is lot of work to set up but it’d worth if all working well.”
B. Key Findings — Qualitative
The test results provided rich qualitative information. After synthesis of the results, not only were there insights about the features that were tested, but there were also insights on user choosing behaviour and information architecture.
One interesting insight was finding out that majority of the participants, 7 of 10, didn’t mind setting up an account for personalised content. It was previously assumed that it would be a nuisance to them. It was also discovered that the star rating style encouraged users to rate shows more compared to the like/dislike style. One participant commented that the latter was too binary and didn’t give them much of a choice, leaving them to not rate at all.
