Evaluation of a trial of generative AI (Copilot) in The Treasury

The Treasury participated in an Australian Government trial of Microsoft 365 Copilot (Copilot) which was coordinated by the Digital Transformation Agency (DTA), with support from the AI in Government Taskforce.

The trial aimed to understand:

• whether and to what extent Copilot could be used as an example of generative AI
• whether it could be implemented safely
• what benefits and challenges it poses in the short and longer term. 

Further, the trial aimed to understand the effect of Copilot on:

• staff productivity
• efficiency
• process improvements
• employee‑related outcomes. 

This report summarises the outcomes of Treasury’s internal evaluation of the Copilot trial.

Overall, Treasury’s trial found that generative AI can be deployed within the Treasury environment, but there are some limitations.

Almost all trial participants reported using Copilot during the trial period, with 22 per cent reporting using it at least 4 to 5 times per week. Copilot was useful for some purposes, with almost two‑thirds of users finding it appropriate and beneficial for basic administrative tasks and work processes. Access to Copilot also had other unintended benefits, including improvements in accessibility, inclusion and work confidence.

However, Copilot’s value for more complex work tasks was limited – this was due to limitations with Copilot’s functionality, and/or the impact of the privacy and security restrictions that limited the product’s quality.

The report concludes with recommendations for how any future implementation of generative AI products in Treasury could be improved. These recommendations are contingent on whether Treasury decides to rollout new generative AI products, and are applicable to any generative AI product (versus Copilot specifically).