Evaluative Research on Cost Analysis Tool

Problem
The production site had a goal of reducing production costs, which made it critical for the site to have visibility into value erosion. To achieve this, development of a production cost analysis tool started in 2021, but due to changes in the product owner team and subject matter experts (SMEs), the project team lost sight of whether the tool was meeting user needs. The team reached out to me to evaluate the tool's usability and identify gaps between the solution and user needs before the release.
Solution
After gathering initial background information, I conducted stakeholder interviews with the Product Owner and SMEs to understand their vision for the tool and where they thought it was lacking. The interviews revealed a stark difference between the SMEs' vision and that of the project team. To assess the tool, I conducted usability testing to evaluate the existing functionalities, determine whether the product met user needs, identify missing functionalities, and assess the potential impact of introducing the tool. Using the data collected, I provided suggestions on access control, reported on necessary feature enhancements and new features, mapped out the current user workflow, identified potential to-be workflows, perceived potential tool value, and led the business SMEs to prioritize the list of functional pain points. With this information, I was able to help the team populate their backlog and re-establish communication with users. My work also helped the team stay aligned on product value and confirmed they want to move forward with development. Overall, my assessment helped the team gain a better understanding of the tool's usability and user needs, allowing them to make informed decisions and prioritize their back log for future release.
My Role
UX Designer
Team Member
Only UX person working with project team
Employer
ExxonMobil
Duration
January 2023 - March 2023
View: BackgroundStakeholder InterviewHeuristic Evaluation Research ObjectivesUsability Testing Outcome
Background
This cost analysis tool is designed to analyze production cost using 3 logics, one logic for calculating production, one for calculating mined volume, and one for calculating cost. MVP has the cost and production logic built and is on a pause for building rest of the logic due to resource limitations. I joined the project 2 months before release and was asked to gather feedback from the end users on where the project team can make improvements to the interface design. After getting a background overview on the project journey and reviewing existing documentation, it was clear that on top of providing usability support, the project would also benefit from having better understanding of their targeted user groups.
Step 1:
Uncover Hidden Concerns with Stakeholder Interviews
Because the appointed SMEs were still new to the project and the tool, I wanted to conduct stakeholder interviews with them to get more context on project history, better understand user workflows, and see if there is a need for stakeholder alignment on project goal, targeted users, stage, and success criteria. I conducted stakeholder interview with 6 stakeholders and identified several gaps that helped drive my overall research direction. I was also able to map out their vision of user access levels.
Step 2:
Early Assessment and Prep Testing with Heuristic Evaluation
I moved on to conducting heuristic evaluation because of two reasons. First, to get an assessment in regarding tool usability. Second, to get myself familiar with the tool so I can better prepare for usability testing. With heuristic evaluation, I was also trying to establish my credibility by demonstrating I can provide what my stakeholders were looking for and push for formal usability testing next. With heuristic evaluation, I was able to provide early feedback on tool usability issues and eventually convince the team to allow me to do usability testing.
Research Objectives
With a basic understanding of the project and the tool, I was ready to further my research. After aligning with my stakeholders, we agreed to assess the following 4 areas.
  1. Usability of the cost analysis tool MVP.
  2. If the value drivers included in the tool were sufficient for user needs.
  3. The usefulness of the tool for the user.
  4. The desired access level for the tool from user perspective.
Step 3:
Research with Usability Testing and Follow Up Interviews
Since I was brought onto the project to provide interface enhancement feedback, I decided to conduct usability testing with the targeted primary user groups which were people in Role A and Role B. There was a total of 8 business teams, each with their own Role A and Role B. Although it would be great to talk to them all since each business team had their unique characteristics and requirements that the project should know about, it was not possible due to time constraints and not necessary for usability evaluation purpose.
‍
I ended up conducting usability test with 2 users in Role A, 4 users in Role B and 1 new SME. When needed, I scheduled follow up interviews with them to further understand their as-is processes and detailed requirements.

I was able to get a SUS score during the test which helped stakeholders understand how the tool ranked in terms of usability.
Image of SUS scale created.
Figure 1 Image created based on Jeff Sauro's SUS scale. Reference image: Sauro, Jeff. " 5 Ways to Interpret a SUS Score." MeasuringU, September 19, 2018.

I was also able to gather a list of usability issues that were leading to task failure or redundant work, desired feature enhancements for easier use, requests of new features that would make the tool more attractive, and bugs.
Additionally, I mapped out the desired level of access from users’ perspective. Project team is actively working on adjusting access level accordingly.
Lastly, since the project team did not have much documentation on the targeted users, I used the information collected from research to summarize primary user characteristics and the value they see in the tool with how it’s built and its potential. Additionally, I mapped out users’ as-is workflow and potential to-be workflow. Highlighted boxes are steps affected by the tool.
Orange boxes = steps that will be affected by the tool.
Outcome: Provided Project Next Steps