User testing is an essential part of FlixBus’ design process to aim for a seamlessly intuitive product that our customers love. It’s our goal to create a product that is easy to use and helps the company achieve its business goals at the same time. We believe having continuous improvements on existing and new products based on users’ feedback helps us achieve these goals.
Having employees as testers of existing or new features? Why not! By asking our co-workers, we receive immediate feedback. However, there is a downside when it comes to testing employees instead of real users. Their knowledge of the company’s business goals and product could bring up some potential biases and might not always represent the real users’ needs and behavior. Nevertheless, there are some pros to consider when testing employees: it’s cheaper and employees are more accessible to reach than real users. In addition to that they are the right target group to test new confidential features or to detect possible bugs before the feature is released to the end customer. Internal user testing does also get employees more familiar with the design process, which could result in getting people more involved with the UX topic. Apart of the pros and cons, we believe that by applying elaborate selection criteria, FlixBus employees still represent our average users. We are happy that FlixLabs is helping us to facilitate our needs through thoroughly reaching the FlixBus employees in offices around the world to take part in testing and supporting the UX topic. We jumped onto this collaborative platform that was initiated within FlixBus. It introduced us to various methods of user testing within the company. One of them is Unmoderated Remote User Testing (URUT), a new approach we discovered to be useful for the UX researchers to cover for the lack of on-site presence of testers in the multiple FlixBus offices around the world. A new user testing tool was also introduced to help us run the test remotely.
Without further ado, here are some fruitful insights and lessons we’ve gathered throughout the process of running Unmoderated Remote User Testing with FlixBus employees:
Firstly, what is Unmoderated Remote User Testing (URUT)? Unlike on-site moderated user testing, we’re missing interaction between the tester and the facilitator. In an on-site moderated user testing, the facilitator is sitting next to the tester when performing the task to ask questions and observe his/her behavior. URUT works the other way around: the testers need to independently perform the task remotely. In this setting, the testers inarguably have more challenges to master, especially for testers who are not familiar with user testing at all. Why? Because we need them to run the test independently with no guidance from the facilitator even if there’s a need to clarify things. Moreover, it’s missing the real-time feedback and in depth interview between both parties after they finish the test.
The method has proven that it could cover the geographical distribution of the testers. We just need to think of how to handle the challenges mentioned above. We need to make sure we prepare clear instructions to avoid confusion of testers. In addition to that, we also conducted a small “Dry Run Test”. With some friends, we made sure everything makes sense before rolling out the test to the whole company. If it doesn’t work on the first try, then we have to adjust the test plans and try again.
Here are some steps we prepared to make sure things run smoothly:
In general, to have a clear objective and alignment with the development plans, always involve Product Owners to help prioritize the features that need to be tested. However, for this pilot session, we tried to come up with a simple task from our own. The first task we came up with was about “Booking Multiple Trips”. The hypothesis for this task was that users are not sure if they can book more than 2 trips in one single booking. We created the task to book a trip from Berlin to Vienna and then from Vienna to Graz to see how users perceive this task and if they could do it easily in one booking.
We posted the advertisement for user testing with FlixLabs on our intranet that is accessible for all employees in FlixBus. So whoever was interested in this test could participate voluntarily. We also printed some posters and hung them on the walls of offices across Europe. Nevertheless, we are still experimenting on the right way of to reach the people. We did this intentionally as a first try to see how many people would respond and how their reaction is to only then then come up with a better screening with more precise criteria at the next session.
We need to observe how the testers interacted with the feature and observe their behavior remotely. We needed a tool that enables testers to share their screen, camera and audio at the same time, even though we leave at their discretion to choose if they refuse to share their camera and audio. Sharing the screen is the best way to observe their interactions with the product. The tool we’re using is https://lookback.io/.
Again, this document is very important for the testers to be able to leave them independently with no assistance. The documentation of the test consists of test instructions, the task with the scenario and a follow up feedback form. There should also be very precise written instructions on how to run the test from start to finish: from which kind of browser required, to downloading and using extended tools up to recording their interaction. Furthermore, the scenario also formulated to immerse the testers into the task they need to accomplish. The scenario should let the testers picture themselves in a situation. Example:
“You plan to go to Austria for your summer holiday for 3 weeks. You're planning to go from Berlin to Vienna for 2 days to visit your friend, Anna. From Vienna, you and Anna need want to visit Tim who lives in Graz. You don't know where the trip would take the 3 of you from Graz, so you decided not to book anything further, yet. So for this travel plan, you need to:
Please book these two trips in a single booking from the website. The session will end when you have reached the passengers page (no need to fill in the passenger names).”
After they finish the test, we ask them to follow up stating their experience and feedback after performing the task. We asked about what they think of the feature they just tested, what pain points they experienced while performing the task and if they could think of some suggestions for improvements.
In the end, we sent the testers personalized thank you mails with a gift card as a form of appreciation for the time and effort they took to take part in this project.
After running the “Booking Multiple Trips” user test for 2 weeks, we found some interesting insights to support our hypothesis and some other unexpected results. Our hypothesis was that users are not sure if they can book more than 2 trips in one single booking. To answer that, we learned that some of the testers found difficulties completing the task. Some of them are unsure if they were able to book multiple trips in 1 single booking. They felt that they were simply uninformed that it’s possible to do so. On the other hand, some other pain points arose along the way when it came to choosing the right bus stations. Almost half of the testers were confused when choosing a bus station in an unknown city, as it’s important for them to choose the one that was close to the central station or their accommodation.
In addition, unexpected finding came up during the session. We found some testers were choosing the central station as their departure location when booking stations in an unknown city. This behavior resulted due to missing city names in the destination selection when there’s no bus connection to that city. Here’s the example from the task “Book a trip from Vienna to Graz”: the testers were choosing Vienna central bus station as a starting point. Then while they typed Graz in the destination selection, nothing showed up. Apparently, there is no connection from Vienna Central Station to Graz in the bus network. They were frustrated, not knowing that a different station in Vienna went to Graz. One of them even gave up. The other tester struggled a lot, but ended up on "Route Map" page where he was able to see all the possible routes on a map. From there, he found the connection by choosing Vienna and Graz on the search mask without specifying any specific bus station. He managed to list of all connections between all stations from both cities and was able to complete the task.
So, is involving FlixBus employees for user testing a good idea?
As mentioned above, user testing was full of surprises, there are many interesting and unexpected results. It was very *interesting* to reveal a users’ behavior and needs when we took our time to “talk” to them. However, because of their view and knowledge of the company, we obviously shouldn’t let our employees be the only testers of our product. The data we received from them shouldn’t be fully trusted to reflect our real users’ behavior. This project should run in parallel with testing our real users.
In conclusion: Having employees do user testing could point out weaknesses on the planned test. With their feedback we can also adapt and improve the test structure, not only on the product. This project can get more people into the UX topic and get more UX awareness into the company.
Is the whole process for Unmoderated Remote User Testing worth the effort?
The URUT method works well to cover remote testing. However, the on-site moderated user testing is rather highly suggested. During the test we learned that it’s challenging for testers to think out loud (or asking the participants to talk us through what they’re doing as they’re doing it). It can be because they’re in an office environment and feel intimidated to share their thoughts aloud or simply because there’s no one around to remind them to do it. Moreover, we found URUT to be more error prone due to the fact that no facilitator fixes errors or clarifies the tester during a test. As an example: the tester shares the wrong screen or misunderstood the instruction. There will be no one to clarify these mistakes. The written instructions should be explicit and self-explanatory, even though that would not always prevent errors from happening.