Compliance-ready AI: building audit trails for LLM applications

SOC 2 auditors do not accept probabilistic outputs as evidence. HIPAA does not allow ePHI to be processed without an auditable record of who accessed what and when. The NIST AI Risk Management Framework requires organizations to establish monitoring, auditing, and review processes for AI systems as a core governance function. All three frameworks share the same foundational requirement: AI systems must produce evidence that their decisions were made correctly, on authorized data, following a defensible process, and that evidence must be available on demand.

LLM applications do not produce this evidence automatically. Unlike a traditional system where every database query and function call can be logged deterministically, an LLM operates probabilistically. The same prompt may produce different outputs. The model’s reasoning is not directly observable from its output. Without explicit audit logging infrastructure, the LLM application is a black box that satisfies none of the evidentiary requirements of any compliance framework.

Gartner predicts that by 2028, LLM observability investments will reach 50% of all GenAI deployments, up from 15% today. Gartner states that without both explainability and observability, GenAI cannot mature beyond controlled lab environments. Teams not building this infrastructure now are accumulating compliance debt.

What IBM requires for trustworthy AI in compliance-critical environments

IBM’s guidance on building trustworthy AI agents for compliance documents five categories of audit evidence for compliance-critical AI. These categories apply to any LLM application making decisions that affect regulated data or outcomes, not only agentic systems.

• Input data and context: a record of all information the model received for the decision, including input data, relevant policies, and environmental context. This documents that the decision was based on authorized, appropriate data rather than out-of-scope information
• Decision context: proof that the model used appropriate, authorized data sources. The record must demonstrate scope alignment, not just that data was available, but that it was within the authorized set for this specific decision
• Reasoning chain: a step-by-step record of the logic followed. For rule-based agents this is the deterministic rule trace; for LLM-based agents it requires explicit instrumentation of reasoning steps, not just the final output. IBM notes that recording counterfactuals should be considered
• Alternatives considered: evidence that the system evaluated multiple options. This is especially important for compliance decisions, credit decisions, and clinical recommendations where selection criteria must be demonstrable
• Human oversight trail: documentation that governance structures remained intact, who was accountable for autonomous decisions, and whether any human override occurred

IBM frames the standard precisely: building trustworthy AI for compliance is not about achieving perfect explainability. It is about producing enough evidence to demonstrate that decisions were made through a defensible process, sufficient to reconstruct the reasoning six months after the decision was made.

The NIST AI RMF and its audit requirements

NIST’s AI Risk Management Framework, released January 2023, organizes AI governance into four functions: Govern, Map, Measure, and Manage. For LLM applications, the Measure and Manage functions carry the most direct audit implications.

• The Measure function requires monitoring, auditing, and review processes connected to existing risk controls, for LLM applications, inference logs must feed the same SIEM infrastructure as traditional systems
• The Manage function requires incident response plans for AI system failures and change management processes for model updates, a documented process for when a model is updated, when a user challenges an output, and when a decision requires legal review
• NIST’s Generative AI profile (NIST AI 600-1) adds content provenance as a primary governance consideration: the ability to demonstrate that AI-generated content was produced by an authorized system using authorized data, a direct audit requirement for LLM applications in regulated industries

What SOC 2 auditors look for in LLM systems

SOC 2 Type II applies the AICPA Common Criteria to LLM applications just as it applies them to any system handling customer data. The controls most directly relevant to LLM applications are in the CC7 monitoring series and CC6 logical access series.

• CC7.2 anomaly monitoring: the organization monitors system components for anomalies and addresses monitoring alerts. For LLM applications, model outputs must be monitored for anomalies, alerts generated when the model behaves outside expected parameters, and those alerts reviewed. A system with no output monitoring produces no evidence for this control
• CC6.1 logical access: only authorized users and systems have access to systems containing customer data. For LLM applications, the prompt pipeline must enforce access controls so the LLM cannot receive data from sources the requesting user is not authorized to access, and that enforcement must be logged
• Evidence auditors sample: inference logs (user identity, data source, timestamp), model output logs, access control records, and change management records for model updates

A critical principle: tools produce compliance records, LLMs do not

Microsoft’s Azure architecture documentation states this directly: the tool produces the compliance record. Letting an LLM evaluate whether a rule was followed collapses the audit trail and introduces probabilistic outputs on questions that have regulatory answers.

• Deterministic logging layer: the audit log must be generated by logging infrastructure before or after the LLM call, not by asking the LLM to explain its reasoning. Structured metadata (user identity, data source accessed, tool calls made, model output) is captured deterministically by the application layer
• LLM explanations are presentation, not evidence: an LLM can generate a human-readable explanation of a decision for display purposes, but that explanation cannot serve as audit evidence. The evidence is the deterministic record; the explanation is a layer on top of it
• Compliance verification must be deterministic: for checks that determine whether a data access was authorized or an output was within policy, the verification must be a deterministic function. Routing compliance verification to an LLM produces a probabilistic answer, not legally defensible as an audit record

LLM observability tools and how they map to compliance

Microsoft’s Agent Governance Toolkit integrates with LangSmith, OpenTelemetry, Arize, MLflow, and others, confirming that observability and compliance are the same engineering investment. The toolkit maps to NIST AI RMF and OWASP Agentic Top 10 for automated governance grading.

Microsoft specifically identifies observability as the architectural element that separates teams shipping production agents from teams perpetually in pilot. The audit trail and the production reliability system are not separate engineering investments, they are the same infrastructure.

What to log on every LLM inference call

The minimum viable audit record for an LLM inference call in a regulated environment covers six fields.

• User identity and authorization; data sources accessed (with BAA status for HIPAA); complete prompt context encrypted at rest; model response including any tool call inputs and results; timestamp and session identifier; outcome and escalation status
• These six fields satisfy the evidence requirements across SOC 2 CC6.1 and CC7.2, HIPAA Security Rule audit controls, and IBM’s five-category audit evidence framework. Missing any field creates a gap that compliance reviews will find

WebOsmotic’s compliance architecture practice for clients in fintech and healthcare designs the logging layer before the application layer is built. The six-field audit record is implemented from the first production call, ensuring that compliance evidence is available from day one, not retrofitted when an auditor asks for it.

Frequently asked questions

What does SOC 2 require for LLM applications?

SOC 2 Type II applies its Common Criteria to LLM applications. The most relevant controls are CC7.2, which requires monitoring system components for anomalies and addressing alerts, and CC6.1, which requires that only authorized users and systems access data. For LLM applications, auditors sample inference logs showing user identity and data source per call, model output logs, access control enforcement records, and change management records for model updates. AI-generated summaries cannot serve as audit evidence, compliance records must be produced by deterministic logging infrastructure, not by the model being audited.

What does HIPAA require for LLM logging?

HIPAA’s Security Rule requires audit controls recording activity on systems that access ePHI. For LLM applications receiving prompts containing PHI, every inference call must log user identity, data accessed, timestamp, and clinical or operational purpose. HHS’s cloud computing guidance establishes that any service processing ePHI on behalf of a covered entity is a business associate requiring a BAA. Logs must be stored in encrypted, tamper-evident format and retained for the HIPAA-required minimum of six years. The BAA must be verified for every API endpoint in the call chain, including tool calls the agent makes during inference.

What is the NIST AI RMF and why does it matter for LLM compliance?

The NIST AI Risk Management Framework, released January 2023, provides voluntary governance guidance for AI systems organized into Govern, Map, Measure, and Manage functions. The Measure function requires monitoring, auditing, and review processes connected to existing organizational risk controls. The Manage function requires incident response plans and change management for model updates. NIST AI 600-1 adds content provenance as a primary requirement for generative AI, the ability to demonstrate that AI-generated content was produced by an authorized system using authorized data. Organizations aligning with NIST AI RMF should build these four functions into their governance documentation alongside the technical logging infrastructure.

What is LangSmith and is it sufficient for compliance observability?

LangSmith is LangChain’s observability platform that captures traces of every LLM call, prompt, response, latency, and token usage, with step-by-step execution traces for LangChain and LangGraph applications. It is well-suited to debugging production LLM behavior, regression testing after model updates, and SOC 2 monitoring evidence. For regulated industries with strict data residency requirements, the question of whether LangSmith’s data residency satisfies HIPAA or financial services requirements should be verified before it is used as the primary audit log store for PHI-involving inference calls. Microsoft’s Agent Governance Toolkit also integrates LangSmith into a broader compliance framework including NIST AI RMF mapping.

What is the key architectural principle for LLM audit trails?

Microsoft’s Azure architecture documentation states it directly: the tool produces the compliance record, not the LLM. Letting an LLM evaluate whether a rule was followed collapses the audit trail and introduces probabilistic outputs on questions that have regulatory answers. Deterministic logging infrastructure must capture audit evidence before or after the LLM call. Compliance verification checks, was this access authorized, was this output within policy, must be deterministic functions, not LLM calls. LLM-generated explanations can be provided as user-facing context but cannot serve as legally defensible audit evidence.

How does WebOsmotic build LLM audit logging for regulated industries?

WebOsmotic designs the audit logging layer before the LLM application layer. The logging infrastructure captures six fields for every inference call: user identity and authorization, data sources accessed with BAA status for HIPAA workloads, complete prompt context in encrypted format, model response including tool call details, timestamp and session identifier, and outcome and escalation status. This six-field record is mapped to the specific compliance requirements of the engagement, SOC 2 Common Criteria, HIPAA Security Rule, or NIST AI RMF Measure requirements, before any development begins. We work with fintech, healthcare, and SaaS clients in the US and India.