M365 Copilot Research Analysis Study
A year after Blue Shield of California invested in M365 Copilot licenses, the company wanted to assess if the rollout was successful. Blue Shield’s UX team was called upon to evaluate both user sentiment and overall effectiveness of M365 Copilot usage. We wanted to find out which roles were seeing the most value, where adoption gaps existed, what unmet training needs remained and if the hypothesis of saving around 2 hours per week from Copilot usage was accurate.
A year after Blue Shield of California invested in M365 Copilot licenses, the company wanted to assess if the rollout was successful. Blue Shield’s UX team was called upon to evaluate both user sentiment and overall effectiveness of M365 Copilot usage. We wanted to find out which roles were seeing the most value, where adoption gaps existed, what unmet training needs remained and if the hypothesis of saving around 2 hours per week from Copilot usage was accurate.
Over 600 licensed users across BSC and Stellarus responded, generating 13,200+ data points. As the primary researcher (with my manager on PTO and limited team bandwidth), I led the full analysis, uncovering which roles were seeing the most value, where adoption was lagging, and what prevented employees from getting more out of Copilot. The findings were synthesized into a 48-page report presented to senior leadership.
Over 600 licensed users across BSC and Stellarus responded, generating 13,200+ data points. As the primary researcher (with my manager on PTO and limited team bandwidth), I led the full analysis, uncovering which roles were seeing the most value, where adoption was lagging, and what prevented employees from getting more out of Copilot. The findings were synthesized into a 48-page report presented to senior leadership.
The insights I captured from analyzing this survey ensured that licenses could be directed to the right roles and unmet training needs would be addressed, maximizing impact and effectiveness of the tool. My findings served a basis for general AI readiness at BSC, informing future training plans that are created specifically for improving usage of M365 Copilot.
The insights I captured from analyzing this survey ensured that licenses could be directed to the right roles and unmet training needs would be addressed, maximizing impact and effectiveness of the tool. My findings served a basis for general AI readiness at BSC, informing future training plans that are created specifically for improving usage of M365 Copilot.
My Role
My Role
UX Researcher
UX Researcher
Worked with: UX team, product directors, AI leadership
Worked with: UX team, product directors, AI leadership
Timeline
Timeline
June 2025 - August 2025
June 2025 - August 2025
My Tasks
My Tasks
Led UX research
Storytelling
XFN collaboration
Mixed methods analysis
Led UX research
Storytelling
XFN collaboration
Mixed methods analysis
Tools
Tools
Figma
UserTesting
Microsoft Excel
Microsoft Copilot
Viva Insights
Figma
UserTesting
Microsoft Excel
Microsoft Copilot
Viva Insights
The Main Problem
At the end of the M365 Pilot Program at Blue Shield of California, we didn't understand if the license rollout was sucessful.
At the end of the M365 Pilot Program at Blue Shield of California, we didn't understand if the license rollout was sucessful.
My Deliverables
What Did I Do?
1
48 Page Report
The study results and analysis were compiled into a 48 page report which was presented to leadership.
2
600+ Users Analyzed
The pulse check survey was created and sent out on UserZoom. I imported results from UserTesting into Excel for analysis.
3
13,200+ Datapoints
I applied a mixed methods approach to capture both measurable trends and deeper user perspectives.
Pre-Research Process
Initial Thinking and Previous Results
Initial Thinking and Previous Results
Before diving into in depth analysis, I started with some ground rules:
Before diving into in depth analysis, I started with some ground rules:
Study participants: Blue Shield of California employees across all roles with access to Copilot licenses.
Study and analysis deployment: I created a new survey via UserTesting and compiled my results in Figma.
Secondary research: Since we were testing improvement from the pilot program, I gathered all resources used in the initial study. This gave me context on which key areas to explore further and which metrics to pay attention to.
Study participants: Blue Shield of California employees across all roles with access to Copilot licenses.
Study and analysis deployment: I created a new survey via UserTesting and compiled my results in Figma.
Secondary research: Since we were testing improvement from the pilot program, I gathered all resources used in the initial study. This gave me context on which key areas to explore further and which metrics to pay attention to.
Previous Pilot Study Results
Previous Pilot Study Results
Positive Feedback
87% of users reported that they liked using M365 Copilot. Roles that required more technical work reported higher satisfaction.
57% of users felt confident in using M365 Copilot, but others felt that they were only “scratching the surface” and expressed desire to learn more use cases for the tool.
Users reported saving an estimated 2 hours of time per week from using M365 Copilot. This self-reported data may be inaccurate.
Negative Feedback
40% of respondents reported needing to remember to use Copilot. AI tools were not currently a part of daily workflows.
63% of users did not use Copilot for technical use cases like data analysis. They felt Copilot often missed the mark, highlighting a tool issue rather than a user issue.
Users reported frequent unreliable output in Excel and Powerpoint. It especially struggled with slide creation and spreadsheet analysis.
Research Goals
Research Goals
Understand Adoption & Maturity
Understand Adoption & Maturity
Evaluate how widely Copilot was being used, how effectively it fit into daily workflows, and whether performance on technical tasks like Excel and PowerPoint has improved since the previous year.
Evaluate how widely Copilot was being used, how effectively it fit into daily workflows, and whether performance on technical tasks like Excel and PowerPoint has improved since the previous year.
Identify Unmet Training Needs
Identify Unmet Training Needs
Pinpoint where users lacked guidance, support, or confidence that limited their ability to get value from Copilot. Training does exist, however a notable amount of users stated feeling they need more assistance despite that.
Pinpoint where users lacked guidance, support, or confidence that limited their ability to get value from Copilot. Training does exist, however a notable amount of users stated feeling they need more assistance despite that.
Surface High Value Use Cases
Surface High Value Use Cases
Highlight roles, tasks, and scenarios where Copilot delivered the most impact to guide licenses allocation and lay the groundwork for positioning BSC as a more AI-forward company.
Highlight roles, tasks, and scenarios where Copilot delivered the most impact to guide licenses allocation and lay the groundwork for positioning BSC as a more AI-forward company.
Test the 2 Hour Time Saving Hypothesis
Test the 2 Hour Time Saving Hypothesis
Users self-reported saving around 2 hours of work per week by using M365 Copilot. Access to Viva Insights gave me more a in-depth look into how much time users spent using Copilot, allowing me to get a more objective figure.
Users self-reported saving around 2 hours of work per week by using M365 Copilot. Access to Viva Insights gave me more a in-depth look into how much time users spent using Copilot, allowing me to get a more objective figure.
Research Process
My Methodology
My Methodology
My study contained 24 questions, many of which were open ended. As a result, I had 13,200+ data points of qualitative and quantitative data. I used a convergent mixed-methods approach that combined quantitative trends with qualitative depth.
My study contained 24 questions, many of which were open ended. As a result, I had 13,200+ data points of qualitative and quantitative data. I used a convergent mixed-methods approach that combined quantitative trends with qualitative depth.
Quantitative Data: Insights were gathered directly through UserZoom and organized into charts and graphs.
Quantitative Data: Insights were gathered directly through UserZoom and organized into charts and graphs.
Qualitative Data: Write-in responses were exported from UserZoom to Excel. I used thematic analysis, looking for trends in user’s feelings about the tool as well as common benefits and issues they experienced.
Qualitative Data: Write-in responses were exported from UserZoom to Excel. I used thematic analysis, looking for trends in user’s feelings about the tool as well as common benefits and issues they experienced.
*Spreadsheet altered to protect NDA.
*Spreadsheet altered to protect NDA.
Key Study Findings
M365 Copilot Performance and AI Adoption at Blue Shield of California Improved
M365 Copilot Performance and AI Adoption at Blue Shield of California Improved
Copilot Performance Ratings Improved
93% of respondents found M365 Copilot to be a valuable tool (up 6% from previous survey).
89% of users rated M365 Copilot high quality.
80% of users reported that using M365 Copilot provided them with meaningful productivity gains.
Average reported time savings were around 2.6 hours per week.
Users also noted intangible benefits such as reduced burnout, better stress management, and more time for higher-impact work.
Copilot Performance Ratings Improved
93% of respondents found M365 Copilot to be a valuable tool (up 6% from previous survey).
89% of users rated M365 Copilot high quality.
80% of users reported that using M365 Copilot provided them with meaningful productivity gains.
Average reported time savings were around 2.6 hours per week.
Users also noted intangible benefits such as reduced burnout, better stress management, and more time for higher-impact work.
Copilot Performance Ratings Improved
93% of respondents found M365 Copilot to be a valuable tool (up 6% from previous survey).
89% of users rated M365 Copilot high quality.
80% of users reported that using M365 Copilot provided them with meaningful productivity gains.
Average reported time savings were around 2.6 hours per week.
Users also noted intangible benefits such as reduced burnout, better stress management, and more time for higher-impact work.
Adoption & Value Differ By Workflow
70% of all respondents use M365 Copilot daily with the most common applications being in Teams summaries, writing/ editing communications and general summarization.
Knowledge workers and technical users report the highest productivity gains and value from Copilot, indicating strong alignment with their workflows and task types. Administrative assistants reported the lowest value gains.
The 39% of respondents who mainly use Teams meeting summaries reported high tool value and significant productivity improvements.
Adoption & Value Differ By Workflow
70% of all respondents use M365 Copilot daily with the most common applications being in Teams summaries, writing/ editing communications and general summarization.
Knowledge workers and technical users report the highest productivity gains and value from Copilot, indicating strong alignment with their workflows and task types. Administrative assistants reported the lowest value gains.
The 39% of respondents who mainly use Teams meeting summaries reported high tool value and significant productivity improvements.
Adoption & Value Differ By Workflow
70% of all respondents use M365 Copilot daily with the most common applications being in Teams summaries, writing/ editing communications and general summarization.
Knowledge workers and technical users report the highest productivity gains and value from Copilot, indicating strong alignment with their workflows and task types. Administrative assistants reported the lowest value gains.
The 39% of respondents who mainly use Teams meeting summaries reported high tool value and significant productivity improvements.
Low Tool & Training Awareness Persist
Despite training being rolled out to every user with a license, some users were completely unaware of training existing, hindering adoption.
Many users noted that while they would love to learn more about Copilot, they do not have the time to invest into the long training courses.
Adoption is limited when users are unaware of how tools fit into their daily tasks. Bringing awareness to valuable, common use cases can result in greater adoption.
Low Tool & Training Awareness Persist
Despite training being rolled out to every user with a license, some users were completely unaware of training existing, hindering adoption.
Many users noted that while they would love to learn more about Copilot, they do not have the time to invest into the long training courses.
Adoption is limited when users are unaware of how tools fit into their daily tasks. Bringing awareness to valuable, common use cases can result in greater adoption.
Low Tool & Training Awareness Persist
Despite training being rolled out to every user with a license, some users were completely unaware of training existing, hindering adoption.
Many users noted that while they would love to learn more about Copilot, they do not have the time to invest into the long training courses.
Adoption is limited when users are unaware of how tools fit into their daily tasks. Bringing awareness to valuable, common use cases can result in greater adoption.
*Images shown above are excerpts from the full 48-page report — available on request!
*Images shown above are excerpts from the full 48-page report — available on request!
*Images shown above are excerpts from the full 48-page report — available on request!
My Recommendations
1
1
Revise Existing Training
Revise Existing Training
An audit revealed the current training site’s layout is difficult to navigate with a non-linear flow, beginner content is hidden behind advanced modules. Redesigning it for clarity and increasing visibility would help users access training more effectively.
An audit revealed the current training site’s layout is difficult to navigate with a non-linear flow, beginner content is hidden behind advanced modules. Redesigning it for clarity and increasing visibility would help users access training more effectively.
2
2
Create Modular Learning
Create Modular Learning
The most common pain point with training modules is their long duration. Structuring training into shorter modules with defined timeframes as well as creating learning pathways for varying skill levels and roles may improve impact and accessibility.
The most common pain point with training modules is their long duration. Structuring training into shorter modules with defined timeframes as well as creating learning pathways for varying skill levels and roles may improve impact and accessibility.
3
3
Collect More Feedback
Collect More Feedback
Likely due to negativity bias, I found that many users who reported low value gains did not provide reasons why. Conducting 1:1 interviews and creating a dedicated Copilot feedback channel could reveal concrete barriers and add richer individual context.
Likely due to negativity bias, I found that many users who reported low value gains did not provide reasons why. Conducting 1:1 interviews and creating a dedicated Copilot feedback channel could reveal concrete barriers and add richer individual context.
Research Impact
Informing AI Adoption at Blue Shield
Informing AI Adoption at Blue Shield
The M365 Copilot pulse check analysis study validated the investment hypothesis which had encouraged the company to make a sizable investment into new licenses. The investment was successful and resources were not wasted, with a majority of users reporting meaningful boosts to their productivity. Users found that M365 was a valuable tool to have at their disposal, administrative costs were lowered and some users even reported improvements to their mental health because they could now manage their workload and stress better.
The M365 Copilot pulse check analysis study validated the investment hypothesis which had encouraged the company to make a sizable investment into new licenses. The investment was successful and resources were not wasted, with a majority of users reporting meaningful boosts to their productivity. Users found that M365 was a valuable tool to have at their disposal, administrative costs were lowered and some users even reported improvements to their mental health because they could now manage their workload and stress better.
Validated Pilot Hypothesis
Validated Pilot Hypothesis
The findings from my survey analysis proved the pilot hypothesis correct, proving that licensees save around 2.6 hours of work per week. The time savings brought other benefits such as being freed up to focus more on higher impact projects, improving communication quality, and supporting health literacy. I proved continued investment into this tool moves the needle and would be incredibly beneficial for BSC.
The findings from my survey analysis proved the pilot hypothesis correct, proving that licensees save around 2.6 hours of work per week. The time savings brought other benefits such as being freed up to focus more on higher impact projects, improving communication quality, and supporting health literacy. I proved continued investment into this tool moves the needle and would be incredibly beneficial for BSC.
Informed Continued Training
Informed Continued Training
My analysis uncovered the highest value roles, the lowest value roles, and the specific use cases where M365 Copilot either excelled or fell short. A content audit of the current training site revealed additional areas where training could be improved. This knowledge is currently informing the development of new training plans by senior leadership to optimize Copilot usage.
My analysis uncovered the highest value roles, the lowest value roles, and the specific use cases where M365 Copilot either excelled or fell short. A content audit of the current training site revealed additional areas where training could be improved. This knowledge is currently informing the development of new training plans by senior leadership to optimize Copilot usage.
Shaped Adoption Strategy
Shaped Adoption Strategy
I identified where Copilot delivers maximum value and where it does not, giving BSC’s AI leadership clarity on how to prioritize future license distribution. My insights ensure that new licenses are allocated to the roles and teams where they will have the greatest impact, maximizing ROI and driving effective adoption across the company.
I identified where Copilot delivers maximum value and where it does not, giving BSC’s AI leadership clarity on how to prioritize future license distribution. My insights ensure that new licenses are allocated to the roles and teams where they will have the greatest impact, maximizing ROI and driving effective adoption across the company.
Reflection and Learnings
Becoming a Better Researcher
This study was a major milestone in my growth as a UX researcher. I learned how to check my own assumptions, synthesize a massive dataset with mixed methods analysis, and build a research report that clearly communicated actionable insights to leadership. Most importantly, I learned that effective research goes beyond finding insights, it requires asking the right questions, grounding conclusions in evidence, and ensuring the work drives meaningful decisions.
This study was a major milestone in my growth as a UX researcher. I learned how to check my own assumptions, synthesize a massive dataset with mixed methods analysis, and build a research report that clearly communicated actionable insights to leadership. Most importantly, I learned that effective research goes beyond finding insights, it requires asking the right questions, grounding conclusions in evidence, and ensuring the work drives meaningful decisions.
1
1
Avoiding Confirmation Bias
Avoiding Confirmation Bias
My first passes revealed unintentional bias. I overemphasized findings that fit earlier hypotheses and surfaced quotes that reflected only a small portion of users, around 2-5% of the total. Reanalyzing the data with structured methods and under the guidance of our lead researcher helped me deliver more balanced, accurate, richer, data-driven insights.
My first passes revealed unintentional bias. I overemphasized findings that fit earlier hypotheses and surfaced quotes that reflected only a small portion of users, around 2-5% of the total. Reanalyzing the data with structured methods and under the guidance of our lead researcher helped me deliver more balanced, accurate, richer, data-driven insights.
2
2
Staying Positive Through Criticism
Staying Positive Through Criticism
Throughout the project, I received extensive and sometimes conflicting feedback. Staying positive and open-minded to criticism allowed me to refine my analysis, find clearer narratives in the data, and improve each iteration. Treating feedback as an opportunity rather than a setback made the final report much stronger and also allowed me to learn a lot more by working through mistakes.
Throughout the project, I received extensive and sometimes conflicting feedback. Staying positive and open-minded to criticism allowed me to refine my analysis, find clearer narratives in the data, and improve each iteration. Treating feedback as an opportunity rather than a setback made the final report much stronger and also allowed me to learn a lot more by working through mistakes.
3
3
Strengthening My Storytelling
Strengthening My Storytelling
Collaborating with engineering and care management showed me the power of empathy and knowledge-sharing. By exchanging skills through design workshop sessions and technical walkthroughs, we strengthened our partnership as well as improved the clarity, quality and development of the product.
Collaborating with engineering and care management showed me the power of empathy and knowledge-sharing. By exchanging skills through design workshop sessions and technical walkthroughs, we strengthened our partnership as well as improved the clarity, quality and development of the product.
Manager Feedback
POV: Working with Danial
I’m glad we could cross paths!
I’m glad we could cross paths!
I hope you have a warm, wonderful, whimsical day.
Made with love and cappuccinos by Danial Tahir © 2026