Gortbot
GroupMe Usability Testing
Synopsis
Goals
Groupme is an app that allows small to big groups communicate in a chat. GORT Bot was designed to increase the efficiency and security of large GroupMe chats by using a hierarchal user system with specific permissions that would allow administrators to control and restrict certain user actions.
We reviewed GORT Bot’s functionality with the popular GroupMe app. During our analysis, we set goals to test the functionality and usability using participants gathered from a survey taken by students from Kennesaw State University. The test identified several issues with GORT Bot.
1.
2.
3.
4.
Preparation to test the functionality and usability of the bot with the GroupMe app.
Use an Task analysis and Cognitive Walkthrough to document user behavior.
Analyze data to identify issues that GORT Bot developers will find useful.
Find ways to improve the efficiency of managing, organizing and the user experience to have an overall functionality increase through recommendations.
Goal 1
Research Preparation
We are Certified under The IRB in order to conduct minimal-risk Research. From here, we constructed general recruitment emails, consent forms, moderator scripts/checklist and pre-test/pro-test questionaries following best & current practice methods.
Method
Since GORT Bot is a program that is installed into a group chat in GroupMe, we determined that it would be best to test it using a cognitive walkthrough. The walkthrough was based on personas of an average student and based off of an experience that a student would go through.
Perspective Users
Name: Michael Stevenson
Age: 21
Occupation: University Student
He is working towards a degree in Business Management. He would like to use GroupMe to help communicate and monitor large groups with the use of GORT Bot. Michael would be using GroupMe and GORT bot on his phone or computer to communicate with group members. While in school, Michael has been added to groups, but he has never managed a group. The idea of GORT bot sparks interest since using the commands would maximize efficiency and control large groups.
Scenario
After developing a persona, we created a task based and goal-oriented scenario. This scenario encompasses a typical college student’s experience with GroupMe and using it for a group project. This also fits our persona and will help our users understand the intended purpose of testing GORT Bot.
A current college student is in a group project for his class. He has four other students in his group and has been elected as the group leader. He heard from a classmate that GORT Bot would help him manage the group and keep the chat room secure. After he leaves class, he decides to give GORT Bot a chance to see if it would be worth using.
Goal 2
Test Goals
Using the persona and scenario, we determined the goal of our cognitive walkthrough. We then created a task list that would help the user achieve the goal.
The end goal of the test was for the user to successfully install GORT Bot and use it to manage a group within the GroupMe app. The reason why this test goal was chosen is because recently the GORT Bot developers noticed a decline in use of the bot—and they wanted to know why this sudden decrease was happening. The developers thought the overall issue seemed to stem from the usability of the program.
Conducting a usability test, with a set list of tasks, would help identify the disconnect between the user and the bot itself. The tasks for the usability test were developed by following the user manual to determine the common commands that would be used when moderating a group in GroupMe.
Task List
-
Create a Group in GroupMe
-
Users are prompted to login with their GroupMe account and create a new group.
-
-
Google GORT Bot
-
Users Google GORT Bot to find its website.
-
-
Install GORT Bot
-
Install the GORT Bot in GroupMe through the website's activation portal.
-
-
Re-name the Group: Final Project
-
Using GORT commands, re-name the group to “Final Project.”
-
-
Set a Security Level
-
Using GORT, set the security level for the group.
-
-
Kick a Member
-
Kick a member with the GORT command.
-
-
User Hierarchy: Promote and Demote a Member
-
Using GORT commands, promote one member and demote another member.
-
-
Allow Adding Members
-
Use GORT to allow adding members to the group.
-
-
Re-add a Kicked Member
-
Re-add the previous member the user kicked in Step 6.
-
-
Send a Message to the Group
-
Using GORT commands, send a message to the group.
-
-
Uninstall GORT Bot
-
Finally, use a GORT command to uninstall GORT Bot from the group.
-
Cognitive Walkthrough
Each user was warmly welcomed into the lab and introduced to the team. Once they were seated, the moderator let the user know the purpose of the walkthrough and what they would be doing for it. They were informed that they would be recorded both visually and audibly and were given the appropriate forms to sign that documented their consent. No user waived their right to be recorded. Once the forms and the post-test questionnaire were filled out, the user was given a list of group members to add to their group in GroupMe and a list of common commands they would use to moderate their group with. We gave each user the same manual available on the website that included the command syntax they would need to complete the tasks. The moderator explained the importance of the “think aloud protocol.”
Goal 3
The Results
Each user in the test was given the same list of commands to complete. When we created this list, we based it on common commands that a group admin would use to moderate a group with. We assumed that each task would only take a minute or two to complete, and for that time to decrease as each user completed each task and their confidence in the system increased. The graph below illustrates each assigned task, and the average completion time for each user vs. our projected completion time.
Our definition of a success is when the user does not have any difficulty performing the task. This means that they did not ask for assistance and they were able to complete the task in a reasonable amount of time (1-2 minutes).
We defined unsuccessful as when the user has more than one question about the task and if it takes them more than 3-4 minutes to complete it. The purpose of GORT is to be efficient and easy to use, so if the user can not complete common commands in a timely manner, then opportunities for improvement exist.
As seen above, our expected completion time was drastically different than that of the actual task completion time. We developed our expected times based on the typical amount of time to manually perform the task in the GroupMe app, and added 45% as a standard margin of error for our sample size. (Measuring) The actual completion times far exceeded our estimated times for every task. During the test, we observed the users struggling to accomplish various tasks. Whenever questioned by the moderator, they often responded with “I don’t understand what this means”, or “I cannot find where this command prompt is”. Based on their communications while using the think aloud protocol, we were able to determine that the cause of their struggles was due to the structural and formative issues with the manual.
Goal 4
Main Issues
The instruction manual was provided online, and gave all of the commands that could be performed by GORT Bot. We noticed that there were several flaws in the design and layout of the document, including the mis-capitalizations of certain words, and lack of a hierarchy of commands. We noticed that each user struggled to find on the document where to begin, and how they would know if their command prompt was successful. Also, there was no apparent structure of the data, and it lacked page numbers to maintain order and flow. We have used four “issue” categories to rank issues with the software and have listed their definitions below.
-
Catastrophic- Overall system failure
-
Major Problem- Major issues with system functionality
-
Minor Problem- Minor issues with system functionality
-
Cosmetic- Stylistic issues
Recommendations
Reformat and update the user manual with more descriptions/examples
Within the manual, clearly define rules for the commands(syntax)
Make the process of installing GORT Bot clearer and defined
Define what the program is and make it easier to understand
1.
2.
3.
4.
Conclusion
I have thoroughly enjoyed working with the GORT Bot program, and working to improve their solution to group chat management. It is our hope that the information found in this report is well received and implemented, and that the GORT team is able to document more downloads in the upcoming months.