New Report: Challenges to the Monitoring of Deployed AI Systems

The Center for AI Standards and Innovation released the report NIST AI 800-4 on challenges to the monitoring of deployed artificial intelligence systems. The report follows three practitioner workshops and an in-depth literature review conducted by the center in 2025. It identifies monitoring categories and details challenges in the form of gaps, barriers and open questions. The primary contribution of the report is the identification, organization and documentation of monitoring challenges. It also reports views expressed by experts in the field. Six common categories of monitoring were developed through thematic coding. Challenges were organized by monitoring category and by those shared across categories. Open questions on AI system monitoring were sorted into who, what, when, why and how to monitor. The report includes a sampling of post-deployment monitoring challenges in a table. The identified gaps, barriers and open questions point to opportunities for further investigation and innovation. The monitoring categories can offer a common language for describing sub-fields within AI system monitoring. Stakeholders may share comments via email to NISTAI800-4 at nist.gov.