Role: User Research, Test Moderator, Test Observer and Presenter (to stalkholders)
Team: Lindsay Blankenbeker, Shatia Blanton, Delphine Diallo (Lead) & Gabriella Hunt
Project Duration: 3 months
Overview
Application: Marta On the Go is an app designed to provide MARTA's real time rail and bus schedules, as well as transit trip planning for individuals living in the city of Atlanta and surrounding areas.
Goals: Our goal was to evaluate the overall usability of the Marta On the Go application and also investigate whether users could find desired information and easily perform common tasks. Lastly, we also wanted to learn how the user's experience differed between the Android and iOS platforms and see how that might affect the user's outcome (A/B testing).
Methodology: Cognitive Walkthrough, Persona, Scenarios, Participants, Testing, Conclusion and Recommendations
Cognitive Walkthrough
We began our study by performing a cognitive walkthrough of the app on our personal mobile devices while sharing thoughts with the group aloud. We discovered that different issues were arising, some similar on each platform, and different on the other. We concluded that the app was performing differently on the two platforms, and therefore, it would be best to conduct testing in a way that would highlight issues both with the app itself and the app's performance on each platform.
Meet the user
We created a potential user for our study named Allie Lyn who fit our user population which included college students living in the state of Georgia who were 18 years or older and who expressed an interest in riding public transportation. We also created a scenario which included two realistic tasks that Allie Lyn and individuals fitting her persona might experience when using the Marta On the Go app. With these details in mind, we conducted a live study on the Kennesaw State University, Marietta Campus.
Participants
We recruited four college students to participate in our study. Two participants were randomly selected at the time of our study, and two were planned in advance by word of mouth. There were both male and female participants who were all over 18 years old. These participants were each asked to fill out a pre- test screener, participant consent forms, and video consent forms before entering the testing lab.
Scenario/Tasks
To provide the context, test participants were given a scenario which required them to complete two tasks.
Scenario
You recently moved to Atlanta. You are a college student living on a budget and you do not own a car. Your only mode of transportation is MARTA. This weekend you have decided that you wanted to visit the city. The first place you want to go to today is the Georgia Aquarium.
TASK
Task 1: Check your MARTA Breeze card balance and see if you have enough funds.
Task 2: After checking your balance and seeing that you are good to go, now plan a trip to the Georgia Aquarium.
Findings/Results
After the user tests, we gathered data from each participant's testing session and compared their results on the Android and iOS phone. From the results we collected, we were able to identify the main issues encountered by our users.
Task completion
Below are the findings for the two tasks completed during the usability test.
OBSERVATION 1
During task 1 all users indicated that the balance on the card was $0.00, although the card indicated 1 trip.
OBSERVATION 2
All users, regardless of the phone platform used proceeded to the train tab and opened the system Map in an effort to locate the name “GA Aquarium” on the map. All participants of the test assumed “GA aquarium” was a stop on the train/bus route.
OBSERVATION 3
Overall task 2 which we assumed would take each participant a long time to complete, did indeed.
OBSERVATION 4
The table below compares the total time completed by each operating system based on which phone the user started with. Regardless of which operating system, iPhone or Android, the user started with the user completed the same task faster the second time. This would seem that there is a learning curve to complete basic tasks within the MARTA app. Even though the initial assumption was that users on Android would find Google Maps integrated to provide an easier experience, the data does not show a difference.
OBSERVATION 5
While all participants completed task 1, only 1 out of 4 completed task 2. Furthermore, 2 users were able to get to the “Trip Planner” screen within Google Maps, but failed to switch from auto to transit to successfully complete the task. One user was unable to find “Trip Planner” altogether, so he clicked throughout the application ultimately finding a bus and train combination he thought would get him to the “GA Aquarium”.
OBSERVATION 6
Below indicates where two of our user scores were inferior to 50 which indicates a score that is not acceptable. The two other users score indicates that they found the usability of the app to be marginally acceptable. Overall the SUS scores of our four participants being lower than 68 indicates that there are problems with the usability of the app that needs to be addressed.
System Usability Scale (SUS)
After testing, participants were asked to answer a set of 10 questions with five responses ranging from "Strongly agree" to "Strongly disagree". The goal was to assess the usability of the app. The figure below depicts the results that we obtained from calculating the SUS score for each of our user.
Recommendations
Based on the findings we outlined the following recommendations, and have assigned a severity rating to each recommendation as it applies to the issue of the Marta on the Go application on both platforms.
1= Cosmetic: Need not be fixed unless extratime is available on project
2= Minor: Fixing this should be given low priority
3= Major: Important to fix, so should be given high priority
4= Catastrophic: Show Stopper/ imperative to fix this
2= Minor: Fixing this should be given low priority
3= Major: Important to fix, so should be given high priority
4= Catastrophic: Show Stopper/ imperative to fix this
Severity (1) When using the Marta mobile app on the iPhone, the user is taken out of the Marta app to Google maps (Figure X) once on the
trip planner screen. Having the maps built into the app would make for a smoother transition and appeal.
trip planner screen. Having the maps built into the app would make for a smoother transition and appeal.
Severity (3) To be successful with the MARTA mobile app you need to be fully trained in Google maps, a 3rd party application. This created a
problem for 2 of the 4 participants, as the users forgot to change the mode of transportation from car to transit. Defaulting the results to
transit would eliminate this problem.
problem for 2 of the 4 participants, as the users forgot to change the mode of transportation from car to transit. Defaulting the results to
transit would eliminate this problem.
Severity (1) The nearby on the main menu should be removed as it is neither functional or a primary feature MARTA users desire.
Severity (3) Replicate the Google Trip Planner, currently under the more options tab to the train tab and bus tab. By doing this it becomes
readily available to the user whose initial action during the test was to click an option on the train tab and select the system map. If the Trip
planner was available on the bus and train tabs, the user would select it and avoid clicking the system map.
readily available to the user whose initial action during the test was to click an option on the train tab and select the system map. If the Trip
planner was available on the bus and train tabs, the user would select it and avoid clicking the system map.
Severity (2) When using the android platform, it would be a good idea to incorporate the More options tab in the menu at the bottom of the
screen, as it is currently placed at the top of the page and users at first had a hard time locating it.
screen, as it is currently placed at the top of the page and users at first had a hard time locating it.
Severity (2) The current card info screen indicates the number of trips available, but it would be a good idea to add the total balance of the breeze card. along with the number of trips. Users during the test indicated that the current balance on the card was $0.00, which was the current stored value amount indicated on the screen.
Click the link below to view the Usability Report.
