Confidencial · Whitepaper
A Security Leader's Guide to Governing AI That Acts on Your Data
Based on candid conversations with CISOs, CIOs, CTOs, and technical and business leaders at the forefront of this shift, including an on-the-record panel discussion hosted at RSA Conference, alongside research from Cybersecurity Insiders, LayerX, Kiteworks, Gartner, and the World Economic Forum.
Most organizations believe they have AI under control. Policies are in place. Guardrails defined. But beneath that surface, data is moving faster than security teams can track. The gap between what organizations think is happening and what is actually happening is widening quickly. The shift happened in three stages, and most governance frameworks have coverage for the first, partial coverage for the second, and almost none for the third.
People use AI tools to do existing jobs faster: chatbots, copilots, summarizers.
Governance is tractable.
The human is still in every loop.
Workflows redesigned around AI: agents in the loop, AI-first processes.
Governance is harder.
The human reviews, not controls.
RAG pipelines, autonomous agents, custom deployments. AI acts without being prompted.
Most governance has least coverage here.
"Last year, AI was simple: you asked a question, you got a response. Now we're talking about users asking for something and actions coming out the other side."
RSA CISO Panel · 2026
"Data access is the biggest problem in security, and it feels like a new paradigm is needed to wrangle that."
Sam Altman · Cisco AI Forum, Feb 2025
These aren't projections. They describe what's happening in your organization right now, regardless of what your policies say. Shadow AI is no longer just employees pasting data into chatbots, it includes autonomous agents, IDE extensions, and command-line tools interacting with enterprise systems in ways traditional monitoring was never designed to see.
of employees regularly paste company data into generative AI tools. 57% use personal GenAI accounts for work.
of those AI interactions happen through unmanaged personal accounts, outside your visibility, control, or audit trail.
of files uploaded to generative AI tools contain PII or PCI data. Nearly a quarter of that is GDPR or HIPAA-regulated material.
of AI usage in organizations is invisible to security, happening despite policies, outside governance, off every dashboard.
of organizations have AI agents in pilot or production. 12% have already granted agents privileged access to core enterprise systems.
of organizations planning to deploy agentic AI have a mature governance model. Deloitte names this the most urgent governance challenge of 2026.
Most of this isn't malicious, a developer pastes code into a coding assistant, a researcher uploads a protocol to summarize it. Practitioners describe it as a continuous background leak: individually reasonable actions that collectively represent an ungoverned transfer of sensitive data at scale. Gartner predicts that through 2026, at least 80% of unauthorized AI transactions will come from internal policy violations, not external attackers.
The most common response: "We've locked down permissions." That covers access. It doesn't cover what happens to data once it moves through AI. The point of failure isn't access, it's movement. Machine identities already outnumber human identities at roughly 82:1 (CyberArk 2025). AI agents are the newest, fastest-growing, and least-governed subset of that number.
A user authorized to view a document uploads it to an external AI tool. The source system logs a legitimate access event. The file arrives in an ungoverned environment. Nothing triggered. Authorization to view is not authorization to transfer.
A RAG pipeline with appropriate read permissions retrieves and surfaces content the user was never intended to see. Access controls don't consistently carry through to the retrieval layer. The document was restricted for the human. The model accessed it anyway.
EchoLeak: an attacker embedded hidden instructions in an email. Microsoft 365 Copilot ingested it, retrieved sensitive data from OneDrive, SharePoint, and Teams, and exfiltrated it through trusted Microsoft domains. Zero user interaction. No alert.
"Once AI pulls data and does its own thing, it may have access to more than the actual person who requested it." Access provisioned at configuration. Broader than necessary. Rarely revisited. 47% of CISOs have already seen unauthorized agent behavior as a result.
MCP is rapidly becoming the standard interface for how AI agents connect to data sources and enterprise systems. Where MCP adoption leads, governance frameworks haven't followed, yet. Each new connector is a new potential data exposure surface.
of organizations report AI has access to core enterprise systems
say that access is governed effectively, leaving 84% exposed
of CISOs have already observed unauthorized AI agent behavior
Most AI governance investment has gone into visibility: dashboards, policy advisories, monitoring tools. The controls that actually limit what AI can expose are a different category, and most organizations haven't built them. You can see what the agent is doing. You can't stop it. When it does something it shouldn't, you may not have the lineage to explain what happened, which is exactly what regulators are asking about.
The ability to limit what agents are authorized to do with data, not just what they can access. 63% of organizations have no mechanism for this.
The ability to stop an agent or revoke its access when behavior drifts. Without it, a misconfigured agent operates until someone notices, which 33% of the time, nobody does.
67% have no audit trails for AI data flows at all. 61% have logs too fragmented to produce actionable evidence for a regulator or in incident response.
"Audit trail quality is the single strongest predictor of AI governance maturity, yet 33% of organizations lack audit trails entirely."
Kiteworks 2026 Forecast Report
Machine identities already outnumber human identities. AI agents are the newest, fastest-growing, and least-governed subset of that number. (CyberArk 2025)
At RSA, a pharma security leader described a real FDA inspection: not knowing the data lineage created critical risk. The head of pharmacovigilance was accountable. The pharmaceutical company was accountable. The AI was not. Forrester predicts an agentic AI deployment will cause a public data breach by end of 2026, framed as a cascade of governance gaps that compound until an incident is inevitable.
"If 15 bots acted on a particular record, you need to know what changed the data, how it changed, and what the impact was. The accountability doesn't change just because AI made the decision."
RSA CISO Panel
Standard logs track who opened a file. They don't capture what a RAG pipeline retrieved, what a copilot surfaced, or what an agent accessed across a multi-step workflow. 33% of organizations have no AI-specific audit trails.
"We need continuous authorization, not just one-time authentication. Once authentication is done and data access has been granted, continuous authorization needs to be built in." Access provisioned once and never revisited is the default today.
A pharmacovigilance AI assessing patient causality at 50% accuracy isn't an efficiency gain, it's a liability. Human review at defined, non-negotiable checkpoints is the accountability mechanism, not overhead.
The AI is not liable. The humans who deployed it, configured it, and failed to govern it are. Without audit trails, the chain of accountability breaks, leaving the CISO holding the gap.
Framed not as a single failure, but as a cascade of governance gaps that compound until an incident is inevitable. The breach is the symptom. The gap is the cause. The accountability doesn't change just because AI made the decision, it transfers to the humans who deployed it.
The governance gap you have today is increasingly the compliance gap you'll be asked to close tomorrow. The questions have shifted from "do you have AI policies?" to "can you prove your controls actually work?" Most of these deadlines are not future projections. They are now.
| Regulation | Status | Who It Hits | Key Requirement |
|---|---|---|---|
| EU AI Act | Now | Healthcare, credit, hiring, critical infrastructure | Full application begins Aug 2, 2026. Penalties up to 7% of global annual turnover. Documented risk management, human oversight, and auditability required. |
| DORA (EU) | In Force | Banks, insurers, asset managers, payment institutions | In full force since Jan 17, 2025. Continuous, evidence-based ICT resilience. Third-party AI tools are in scope. |
| HIPAA | Updated | Healthcare, pharma, health insurers and business associates | Final Security Rule updates published early 2026. 22% of files uploaded to gen AI tools contain HIPAA-regulated material. |
| CCPA/CPRA | In Force | Any org using AI decisions affecting California consumers | Automated decision-making regs effective Jan 1, 2026. Consumers may opt out and request explanations. Audit trail requirements apply. |
| SEC Rules | Active | Investment advisers, broker-dealers, private equity, hedge funds | 2026 exam priorities explicitly name AI governance. Documentation and evidence carry more weight than written policy. |
| PQC Orders | New | Federal agencies, defense, life sciences, financial services | Signed June 22, 2026. Post-quantum cryptography requirements extend to federal contractors (2030 deadline). The data leaving your files right now is the point. |
"Regulators are increasingly shifting from AI experimentation to accountability. Organizations may need to demonstrate how sensitive information is accessed, processed, and protected within AI workflows."
NIST AI RMF
Adapted from Trail of Bits' AI maturity framework. Know where you are, close the gaps that create liability, and make improvement measurable rather than aspirational.
| Level 1Aware | Level 2Governed | Level 3Controlled | Level 4Adaptive | |
|---|---|---|---|---|
| Discovery | Know you have sensitive data. Rough idea of where. | Classified data across primary repos. Know what AI can reach. | Continuous discovery including shadow AI pipelines. | Real-time classification that updates as AI workflows evolve. |
| Access Control | Role-based access for humans. No AI agent identity governance. | AI agents have defined identities. Access scoped at configuration. | Continuous authorization. Purpose binding. Least-privilege enforced. | Attribute-based, context-aware access. Agent scope auto-adjusts to task. |
| Protection | System-level encryption. Protection stops when data leaves origin. | Content-level classification. Selective protection before AI ingestion. | Protection persists through chunking, embedding, and retrieval. | Cryptographic enforcement at the data layer. Post-quantum ready. |
| Auditability | File access logs. No AI-specific data flow tracking. | AI use case inventory. Basic logging of approved pipelines. | Span-level audit trails for RAG, copilot outputs, agent actions. | Tamper-proof, real-time lineage for every AI data interaction. |
Developed from the RSA CISO panel. The most prepared security leaders could already answer these. Score yourself, then map your gaps to the maturity model.
Do you know where sensitive data lives across unstructured environments, including every repository your AI systems can reach?
Do you know which AI tools employees are already using, including the ones IT never approved and can't see?
Can you control what data AI systems, including agents, are allowed to access? Is that access continuously enforced, not just granted once at configuration?
Does protection stay with data after it leaves the system where it was created, through downloads, external sharing, and AI ingestion?
Can you govern the full AI data lifecycle: prompts, embeddings, vector databases, generated outputs, and agent workflows, not just source files?
Can you trace where AI-generated outputs came from, what data influenced them, at what point, under whose access?
Can your policies follow data across hybrid and multi-cloud environments without relying on manual controls at each boundary?
Can you audit how sensitive data is used by AI systems in a way that satisfies regulators, AI-specific data flows, not just file access logs?
Can you enable AI innovation without exposing crown-jewel data, supporting adoption without becoming the team that slows everything down?
Is your protection architecture ready for cryptographic change, crypto-agility, customer-controlled keys, and a path toward post-quantum protections?
Strong foundations. Focus on moving from Controlled to Adaptive.
AI adoption is moving faster than your controls. Close gaps before a regulator does.
AI is already outpacing your ability to maintain control. Start with governance.
"Governance has to be first. We've got to understand it before we can start taking action." (RSA Panel). The sequence matters as much as the work. Be brilliant at the basics.
Every security leader on the RSA panel independently arrived at the same conclusion. Sam Altman called it a new paradigm. Controls need to live at the data layer, not the system or network layer. AI hasn't invalidated the fundamentals of security. It has raised the standard for executing them.
of organizations now rank AI-related vulnerabilities as the fastest-growing cyber risk. Data leaks through generative AI have overtaken adversarial capability advancement as the leading AI concern. (WEF Global Cybersecurity Outlook 2026)
Request a data flow assessment → confidencial.io77% paste stat, 82% unmanaged accounts, 40% PII/PCI, 22% regulated data, 89% invisible AI usage.
71% AI has access to core systems; 16% say access is governed effectively; 47% observed unauthorized AI agent behavior.
Purpose binding 37%, kill switch 40%, 33% no audit trails, 61% fragmented logs.
Machine identities outnumber human identities at roughly 82:1.
80% of unauthorized AI transactions through 2026 will come from internal policy violations.
All direct quotes. Hosted by Kevin Daisy; panelists including Kurt Faulkner (CCO) and pharma/enterprise security leaders.
"Data access is the biggest problem in security, and it feels like a new paradigm is needed."
AI taxonomy and maturity framework adapted for the governance maturity model.
87% of organizations rank AI-related vulnerabilities as the fastest-growing cyber risk.
Zero-click prompt injection in Microsoft 365 Copilot, June 2025. Exfiltration via trusted Microsoft domains without user interaction.
Agentic AI breach prediction. OWASP LLM02/LLM06 classifications. NIST AI RMF guidance on AI governance.
Come to the conversation knowing your gaps. Leave with a plan to close them.