Case Study
FTI Technology Provides Assessment, Testing and Governance Framework for Copilot Implementation at Large Financial Services Institution
A financial services institution was seeking to deploy Copilot across its Microsoft 365 environment as part of a broader objective to leverage artificial intelligence to improve employee efficiency and work experience. With a robust risk management program in place, the company understood the importance of testing and evaluating the legal and regulatory risks associated with the new generative AI tool. FTI Technology is a longstanding trusted advisor to the company across information governance, e-discovery and digital risk, and was engaged to conduct a robust governance assessment during a contained project pilot.
Situation
The company was aiming to quickly run the pilot and roll out the tool to the entire enterprise. Given the general excitement about generative AI, FTI Technology and the organization’s risk management team had to moderate the conflicting forces of enthusiasm to move fast and the need to understand and mitigate risks. The assessment also involved many stakeholders across functions, who all had unique perspectives and goals for the project, which needed to be managed and addressed throughout.
In addition to managing internal culture and crossfunctional challenges, the team also faced the prospect of identifying specifically what needed to be tested, given that governance within Copilot is a new and largely uncharted issue. Without a precedent for testing implications for deeply integrated generative AI within the Microsoft 365 environment, FTI Technology needed to closely research and scrutinize the organization’s settings within the Copilot architecture and examine its impacts across every Microsoft application, including Outlook, Word, OneNote, Copilot Chat, Teams, etc. Also, the features and functionality were continually changing as Microsoft added and enhanced the Copilot offering. The environment was changing so quickly that the team had to double back and retest certain features repeatedly to detect updates occurring even before initial reporting had been completed.
Our Role
Working with the organization’s IT, legal, compliance and outside counsel teams, FTI Technology initiated a Copilot evaluation as part of its larger, ongoing program for testing new technology. With deep expertise in Microsoft 365 governance and e-discovery, the team developed a workflow and customized testing plan to deliver a thorough examination of Copilot within an accelerated timeline, and ensured ongoing AI governance could be incorporated into the broader information risk management framework.
The assessment spanned every aspect of Copilot within the organization’s environment, from end-user impacts to administrative considerations and functionality, providing the client with a way to clearly understand the tool’s business value, legal and regulatory risks, and settings needed to mitigate potential issues. The team methodically went through each product within Microsoft 365 to identify specific risk issues within certain applications, so they could be escalated with Microsoft customer support. Throughout the review, FTI Technology identified the following:
- Issues with certain Copilot activity and interactions that were not captured by Microsoft Purview Compliance tools.
- Permissions and access controls for OneDrive and SharePoint content that could help avoid inadvertent or inappropriate user access to sensitive information.
- Copilot artifacts that were unavailable to access or extract for potential e-discovery needs. This finding helped inform steps to establish litigation and investigative readiness for Copilot data.
- Variances in risk between different tools within Microsoft 365, including the types and extent of data Copilot could access within each application, allowing the team to establish customized governance controls to prevent sensitive or protected information from access by unauthorized users.
As issues were identified, FTI Technology worked closely with the client to lead discussions with Microsoft around adjusting functionality and controls to resolve risks and improve capabilities. The team also provided the client with detailed reports of testing results, so they could socialize potential concerns across stakeholders and determine the settings and configurations needed to address them.
Our Impact
- As a trusted advisor to the client, FTI Technology delivered a robust Copilot assessment and technical testing to help the client develop a strategy for using the tool in a way in that matched the company’s risk appetite for implementing an enterprise-wide generative AI tool.
- Recommendations were provided to help the client establish a comprehensive governance framework for Copilot use and to establish an ongoing risk management workflow for testing future Copilot functionality, as well as other AI tools. This framework is already in use within the client organization for additional AI implementations underway, which FTI Technology is also supporting.
- Documentation and reporting to support the client in defending its approach to AI and demonstrate that AI is being used in an informed and balanced manner.
- FTI Technology’s deep understanding of Microsoft 365 and the client’s governance needs served as a foundation to productively collaborate with Microsoft representatives to resolve and mitigate key risks.