The NIST AI Risk Management Framework and its Generative AI Profile

The NIST AI Risk Management Framework (AI RMF 1.0) is the governance layer of this course. It was released on 26 January 2023 as “a framework to better manage risks to individuals, organizations, and society associated with artificial intelligence”, and NIST says it is “intended for voluntary use.” It is not a technical test plan and it does not list attacks. What it does is tell an organisation how to organise itself so that AI risks are found, owned and treated over time.

Trustworthiness and where security sits

The framework aims to help organisations build trustworthiness into how AI is designed, developed, used and evaluated. It names seven characteristics of trustworthy systems: valid and reliable, safe, secure and resilient, accountable and transparent, explainable and interpretable, privacy-enhanced, and fair with harmful bias managed. Security is one of the seven, and NIST describes a system as secure when it can maintain confidentiality, integrity and availability through protection mechanisms that prevent unauthorised access and use. It lists adversarial examples, data poisoning and exfiltration of models or training data through system endpoints as common security concerns. Those concerns are then traded off against the other characteristics rather than treated alone.

The four functions

The Core of the framework has four functions. NIST is explicit that its actions “do not constitute a checklist, nor are they necessarily an ordered set of steps”, and that risk management should be continuous and performed throughout the AI lifecycle.

Two design points matter for security teams. First, Govern comes first and stays: after establishing it, NIST says most users start with Map and continue to Measure or Manage, iterating and cross-referencing. Second, Map ends in a decision. Once context is understood, organisations should be able to make an initial go or no-go call on whether to design, develop or deploy the system at all. A red team that skips Map tests the wrong thing.

Measure is where testing lives

The Measure function says AI systems “should be tested before their deployment and regularly while in operation.” Measure 2.7 is the security subcategory: system security and resilience, as identified in Map, are evaluated and documented. Measure 1.1 also requires that risks or trustworthiness characteristics that will not or cannot be measured are documented, which is a useful habit for any test report: say what you did not test.

What the Generative AI Profile adds

NIST released the Generative AI Profile (NIST AI 600-1) on 26 July 2024. It is a companion to the framework that “can help organizations identify unique risks posed by generative AI” and proposes actions to manage them. It defines twelve risks that are unique to or exacerbated by generative AI. Several are directly relevant to a security review: Information Security, Data Privacy, Value Chain and Component Integration (untraceable third-party components and poor supplier vetting), Confabulation, and Human-AI Configuration (over-reliance and automation bias).

The profile ties red teaming to the Measure function. Under Measure 2.7 it suggests performing AI red teaming to assess resilience against abuse to facilitate attacks on other systems, generative-AI attacks such as prompt injection, and machine-learning attacks including adversarial examples, data poisoning, membership inference, model extraction and sponge examples. It defines AI red teaming as a structured testing exercise used to probe an AI system to find flaws and vulnerabilities, often in a controlled environment and in collaboration with system developers.

• NIST notes that the quality of red-team output relates to the background and expertise of the team, and that diverse, interdisciplinary teams can find flaws in the varying contexts where the system will be used.
• It describes general-public, expert, combined and human-plus-AI red teaming, and says AI-led red teaming can be more cost effective than human red teamers alone, while human and AI approaches may suit different kinds of harm.
• It says red-team results should get additional analysis before they feed governance decisions and policy updates. A finding is an input to Manage, not the end of the process.

Keep the framework’s status in mind. AI RMF 1.0 is currently being revised as part of the White House AI Action Plan, according to NIST’s framework page, so verify the current text before you cite a subcategory number in a report.