Kenneth Ingham Consulting, LLC

Follow the data

2026-09-29T00:00:00-06:00

Ask an organization where its sensitive data lives and the answer usually names a system. Ask where it goes and the answer takes considerably longer, because the honest version involves an export somebody built years ago, a spreadsheet that leaves by email once a month, and a vendor portal nobody in IT provisioned.

That second answer is the one that matters, though not quite for the reason usually given. Two in-scope systems talking over an in-scope network leak nothing: every point on that path already meets the requirements, and the data never leaves the protected environment. The trouble starts where a flow crosses a boundary—into a system nobody assessed, onto a network that is not in scope, out to a provider, or into somebody's mail.

Internal flows still have to be written down. A desktop reading from a file server across an in-scope link is not a risk, but an organization that cannot describe that path cannot show the link is in scope, cannot tell when the path changes, and has no way to notice the day somebody adds a hop that does leave.

What a flow has to capture

A useful data flow follows a category of data through its whole life rather than mapping the network. That presumes the categories exist: an organization that has not decided what counts as sensitive data will draw one enormous flow containing everything, which answers nothing.

In practice the unit is the project rather than the system, because that is how data arrives and leaves. Project A receives data from sources B and C, processes it on systems D and E, and returns a result to the customer by method X. Project F has a different shape even when it runs on the same infrastructure. One flow per project, per category of sensitive data, is the granularity that stays accurate. For each kind of sensitive data:

where it originates, and whether the organization created it or received it from somewhere else;
every system it passes through, including the ones that merely transport or transform it;
everyone who can read it along the way, which includes administrators of each system and staff at any provider involved;
where it comes to rest, along with every copy that resting produces—backups, archives, test environments loaded from production, and the laptop somebody works from;
how it leaves, whether by deletion, media destruction, or the end of a contract with a provider;
which obligation attaches at each point, since the same data can be governed by different rules in different places.

The copies are where this exercise earns its cost. Most organizations can name the primary system immediately and have never enumerated the derivatives.

What happens when the project ends belongs in the flow too, and it is the part most often missing. Sometimes the data is destroyed. Sometimes it has to be kept for a stated number of years because a contract or a regulation says so, which means somebody is storing and protecting it long after the work that produced it finished, and somebody is paying for that. Either way the obligation outlives the project. The flow is where it should be recorded—along with whether the project budgeted for it, which is commonly forgotten and is discovered at the point the cost has already been incurred.

The flow determines the threat

The reason this belongs in a risk assessment rather than in a diagramming tool is that the route changes what can go wrong.

Consider two companies moving identical data between two sites. The first writes it to write-once optical media and sends it by courier. Its risks are physical: the package goes astray, the custody record is incomplete, and the media cannot be revoked once it has left the building, because there is no remote wipe for a disc. Its mitigations are correspondingly physical—encrypt the media, log custody, control who can produce a copy.

The second uploads the data to a cloud service and lets the far site download it. Its risks are entirely different: an account that can be phished, an administrator at the cloud service provider who is outside the organization's control, a sharing setting that quietly permits more than intended, and a copy that persists on someone else's hardware after the contract ends.

Neither arrangement is wrong, and neither is safe in the way the other is. An organization that has not documented which one it is using cannot say which set of risks it has chosen, and a generic control list will not tell it.

That comparison is the argument for doing the work at all. The flow is the input a risk assessment needs in order to identify the risks that actually exist rather than the ones a template suggests. But it does something else too: it is how an organization establishes that every point under its control is doing the right thing with the data.

Protection has to travel with the data, and the obligation does not stop at the edge of the primary system. In the first company, the courier needs terms about custody and loss, and the media needs encrypting—because the control that matters is the one accompanying the disc, not the one on the server it was written from. In the second, the cloud service needs the authentication the policy requires, a sharing configuration somebody has actually checked rather than assumed, and a contractual answer about deletion when the term ends.

Every hop is a place where the organization either does the right thing or finds out later that it did not. Enumerating the hops is what makes that checkable, because a control cannot be applied to a path nobody has written down.

Standards ask for this, in nearly the same words

The requirement recurs across frameworks.

NIST SP 800-171 expects the system security plan to describe the system boundary, the operating environment, and how controlled unclassified information moves through and beyond it. In a CMMC assessment this is inseparable from scoping: the flow is what establishes which systems are in scope, and an assessor with an unconvincing flow diagram will widen the boundary rather than narrow it.

PCI DSS is the most explicit of the group. Version 4.0 requirement 1.2.4 calls for an accurate data-flow diagram showing all account data flows across systems and networks, kept updated as the environment changes. Version 3.2.1 numbered the same obligation 1.1.3, which is worth knowing before quoting a number from memory.

CIS Controls safeguard 3.8 asks for documented data flows including those of service providers, reviewed annually or when significant changes occur.

NIST Cybersecurity Framework ID.AM-03 asks that representations of internal and external network data flows be maintained, including flows to third parties and to infrastructure services.

The convergence is not a coincidence. Each of these frameworks has to define what it applies to, and the honest way to do that is to follow the data.

Why it gets skipped, and what that costs

Documenting flows is unglamorous, and the resistance to it is predictable. In practice it arrives most often from project managers, who are measured on delivery dates and see the exercise as a tax on a schedule that is already tight. The data is moving perfectly well without a diagram. The diagram can be written afterward, when there is time.

There is rarely time afterward, and the argument misjudges what is being deferred. Three things follow from skipping it.

The first is the obvious one. Undocumented data movement is unprotected data movement, because nobody applies controls to a path nobody has described.

The second is that the omission is itself a control failure. The standards above are requirements rather than suggestions, and for a defense contractor the requirement has a contract behind it: implementing NIST SP 800-171 is mandated by DFARS 252.204-7012, a clause the company signed. An organization that has not documented its flows is not merely disorganized. It is failing a requirement it agreed to in writing.

The third follows from the second, and is the one worth raising when a schedule argument is being lost. Failing to meet a contractual security requirement is a breach of that contract. In the federal context the exposure does not stop there: an organization that has affirmed a compliance it does not have—and CMMC requires such an affirmation every year—can face liability under the False Claims Act, which the Department of Justice has pursued specifically against cybersecurity misrepresentation.

That is a considerable distance from a conversation about whether a diagram is worth two days, which is rather the point.

It is worth being honest about that cost, because the objection usually inflates it. Only a genuinely complicated environment takes days. A simple flow can be written in a couple of sentences: the data arrives through DoD SAFE, is stored on server abc, processed on compute server def, visualized on desktop ghi, and returned to the customer through DoD SAFE again; backups go to backup servers jkl and mno; every network between those systems is in scope. That is a complete flow for that project. It takes about a minute to write, and it is enough to reason about. The diagram that takes two days is usually the one nobody broke into projects first.

The flow includes people

A data flow that names only systems is half a flow. Data is not moved solely by integrations; it is read by people, and who can see it at each stop belongs in the description.

For most organizations that list is expressed as group membership. Access is granted by role, the role maps to a group, and the group is what actually determines who can open the file. That is the right design, and it has a well-known decay: groups accumulate. Someone joins a project and is added. The project ends. The person transfers to another department, and nothing in the process that moved them removes them from the group—often because the person who granted it has themselves moved on.

The result is a flow that is accurate about systems and quietly wrong about people. The data goes exactly where the diagram says it goes, and is readable by rather more people than the organization believes.

The remedy is a periodic review of both, and the standards ask for it. NIST SP 800-171 requires that the privileges assigned to roles or classes of users be reviewed at an organization-defined frequency to validate that those privileges are still needed, and for defense contractors the department has set that frequency at least every 12 months.

Two things make the review worth conducting. Compare membership against a current statement of who should have access, rather than against last year's list, because comparing a list to itself confirms nothing. And have the business owner do the confirming rather than IT, since IT can say who has access but not who needs it.

Both reviews—of the flow and of the group membership—have to leave a record. An assessor asking whether access is controlled is not answered by a current screenshot of a group, which describes only today. What answers it is a series of dated reviews showing that someone has been checking, what they found, and what they removed. A history of protecting the data is a different claim from a snapshot of it, and only one of them is evidence.

Keeping it honest

A flow diagram drawn once and filed is worse than none, because it invites confidence in a description that has stopped being true. Three habits keep it honest.

Trigger the review from events that already happen. A calendar review alone will always lag, because whatever invalidated the diagram happened in March and the review is in November. The events worth attaching it to are the ones the organization already has a process for: a new vendor or service provider, a new integration between systems, a department changing how it works, an acquisition, a new product line, a system replaced or retired, a contract ending, and any real change in where people work from.

The practical move is to make the flow review a step inside a process that already runs—change control, procurement approval, vendor onboarding—rather than a separate obligation somebody has to remember. Better still if those are not three processes: where procurement and vendor changes are themselves raised through change control, there is one place a change to the environment is recorded and one place the flow review hangs from. An obligation that depends on memory depends on one particular person still working there.

Validate against observation rather than recollection. The diagram records what the organization believes. Several systems already record what actually happened, and each catches a different kind of omission:

firewall, proxy, and DNS records show destinations nobody described;
the identity provider's application list shows services connected through single sign-on, including ones IT never provisioned;
expense and corporate-card records show subscriptions bought outside procurement, which is where an undocumented service usually surfaces first;
the email gateway shows recurring attachments leaving on a schedule, the classic shape of an export nobody remembers building;
backup catalogs and storage inventories show where copies actually live, as opposed to where the design says they should;
change control records show the interfaces added since the diagram was drawn.

None of these is complete alone. Together they are usually enough to find whatever the diagram is missing.

An organization already running data loss prevention has one more: tagged files can be counted, so the volume moving along each path becomes something measured rather than estimated. That is a real capability and an expensive one—to buy, and more so in the human time good tagging takes, since the shortcuts that make tagging quick are the ones that make it wrong. It also tends to commit an organization to a single vendor from the server through the desktop to the network infrastructure. Worth using where it exists; not worth buying in order to draw a diagram.

Keep it readable, and keep it dated. A diagram that has grown to cover every system at maximum detail stops being maintained, because updating it becomes a project. One flow per category of sensitive data, drawn at a granularity where a reader can actually follow the path, is more useful and survives longer. Record the date of each review, who performed it, and what it was checked against—both because next year's reviewer needs that and because it is the evidence the review happened at all.

The gap between the documented flow and the observed one is the finding. It is also, reliably, where the unregistered vendor and the undocumented export turn up.

Kenneth Ingham Consulting helps organizations map their sensitive data flows as part of scoping and risk work, which is usually where an organization discovers how far its data actually travels. It is not work that can be done from the outside: the people who know where the data really goes are the ones who handle it, and the map is the product of a structured conversation with them.

A POA&M is a commitment, not a parking space

2026-09-22T00:00:00-06:00

Every organization has gaps between what its policies require and what is currently true. The question an assessor, a customer, or a regulator asks is not whether gaps exist. It is whether the organization knows about them and is doing something specific about each one.

That is what a plan of action and milestones is for. It converts a known deficiency into scheduled work with a name attached.

The difference it makes

An unremediated gap with a documented plan, an owner, and a date is a managed problem. The same gap without one is simply a gap that has been written down.

The distinction is not cosmetic. It is close to the whole of what separates an organization that is behind from an organization that is not in control of its own security program, and experienced assessors read the difference quickly. A short POA&M with realistic dates and evidence of prior items closing on time is a good sign. A long one with dates that have all been revised twice is a different signal entirely.

What an entry has to contain

A usable entry answers the questions someone else would ask.

The deficiency, stated specifically enough that its resolution is unambiguous. Not "improve access control" but the requirement that is unmet and where.
Why it is not already done. Cost, a dependency on another project, a vendor limitation, or a decision to sequence it behind something else. An entry with no stated reason usually means nobody has actually looked.
The risk in the meantime, and any compensating measure in place. This is what makes the delay a considered decision rather than an omission.
A named owner. A role, not a department. Work assigned to "IT" is assigned to nobody.
Milestones, where the work is large enough to have intermediate steps worth tracking. Milestones are what allow the organization to notice that an item is slipping before its completion date arrives.
A completion date, chosen because it is achievable rather than because it is comfortably distant.
Resources, where the work needs budget or people that have to be secured before it can start. An item whose funding was never approved will not close, and recording that early is more honest than discovering it at the deadline.

A worked example, from an unusually explicit source

Most standards require milestones and leave the reader to work out what an adequate one looks like. The Department of Homeland Security's POA&M guide is public and does not, and it prints a bad set beside a good one for the same finding. What follows is from version 3.0, dated July 28, 2022; the document is revised periodically, so check the current one before relying on any detail.

The finding is that vulnerability scanning does not cover the whole environment described in the system security plan. The guide's inadequate example is a single milestone: ensure vulnerability scanning covers the entire environment, by a date. Its adequate example is five: schedule a review of the environment inventory; update the system security plan and the scanner to reflect it; run a scan to confirm the inventory is covered; implement an ongoing process to keep the inventory, the plan, and the scans aligned; and run a further scan cross-checked against the inventory to verify.

The difference is not thoroughness for its own sake. The first restates the objective, so it can only ever be reported as done or not done, and nobody learns anything until the deadline arrives. The second describes work, so progress is observable while there is still time to act on it. And the fourth step commits to preventing recurrence rather than only correcting this instance, which is the difference between fixing a finding and fixing the reason it existed.

The guide states the underlying rule plainly: a milestone description should not simply repeat the description of the weakness. It requires at least two milestones on every entry, and asks that each be specific, measurable, assignable, realistic, and time-related.

Three further provisions in that guide are worth borrowing regardless of who the customer is.

The completion date is locked once the entry is approved. Slipping it requires recording a reason and requesting a waiver, which is an act somebody has to approve rather than an edit somebody can quietly make. That single design decision is most of what separates a commitment from a parking space, and no organization needs permission to adopt it.

A waiver is not compliance. The guide is explicit that granting one does not bring the system into compliance; it acknowledges non-compliance and records an accepted plan to remediate within a bounded period, with compensating controls in place. Organizations routinely treat an approved exception as the end of the matter. It is the opposite: a waiver is a promise with a deadline attached.

Closure requires evidence, and the person validating it should not be the person requesting it. The stated reason is separation of duties. The practical reason is that whoever wants an item closed is the worst available judge of whether it should be.

None of this binds a defense contractor, and the tooling and role names are particular to that department. It is quoted because the expectations are written down, which is unusual, and because an organization that met them would have a POA&M process nobody could reasonably criticize.

In the defense context specifically

For organizations pursuing CMMC, the POA&M has a formal role and formal limits, set out in the assessment guides and scoping guides on the DoD CIO's CMMC documentation page and in the rule itself, at 32 CFR 170.21.

A Conditional CMMC Status exists precisely to accommodate an organization that meets most requirements and has a plan for the remainder. At Level 2 it requires an assessment score of at least 0.8 of the total requirements, which is 88 of 110, and the open items must be closed within 180 days of the Conditional CMMC Status Date, verified by a POA&M closeout assessment. Miss that date and the conditional status expires.

The eligibility limits are tighter than most organizations expect. No requirement worth more than one point may go on a POA&M at all, with a single exception: SC.L2-3.13.11, CUI encryption, may be deferred where encryption is employed but is not FIPS-validated, which costs three points rather than five.

Six one-point requirements are excluded outright:

AC.L2-3.1.20, External Connections
AC.L2-3.1.22, Control Public Information
CA.L2-3.12.4, System Security Plan
PE.L2-3.10.3, Escort Visitors
PE.L2-3.10.4, Physical Access Logs
PE.L2-3.10.5, Manage Physical Access

The pattern in that list repays reading. Five of the six are points where a gap means CUI is actually exposed rather than merely undocumented. The sixth is the system security plan, which cannot be deferred because it is the document the assessment is conducted against. There is no assessing an organization that intends to describe itself later.

At Level 1 there is no POA&M at all. Every requirement must be met.

The practical consequence is that a POA&M cannot be a strategy. An organization planning to certify with a substantial POA&M and sort it out afterward has usually misjudged both what is eligible and how much time closing the items takes. The stronger position is full certification without open items, and that is a preparation question rather than an assessment-day one.

How they fail

POA&Ms fail in a small number of recognizable ways.

The perpetual item. Its completion date has moved four times. Nobody has ever decided not to do it, and nobody has ever done it. This is a decision the organization is making without recording that it is making one, and the honest resolution is either to schedule it properly or to move it to an accepted risk with an approver's name on it.

The item with no owner. Assigned to a function rather than a person, noticed by no one, closed by no one.

The date chosen for comfort. Set far enough out that nothing has to happen this quarter, which guarantees nothing happens next quarter either.

The POA&M nobody reads between assessments. Reviewed once a year, the week before someone external asks for it. A POA&M is an operational document or it is a work of fiction produced annually.

Where entries come from

Anything that identifies a gap should be able to deposit work here: the risk assessment, the annual policy review, incident and exercise reports, audit findings, self-assessments, and vulnerability management.

That is the test of whether the surrounding processes are connected to anything. A policy review that identifies operational work and does not produce POA&M entries has ended at the document. An incident report with recommendations that never became tracked items has been filed rather than acted upon.

Kenneth Ingham Consulting helps organizations plan remediation that reflects business priorities and real budgets, which is the difference between a plan that closes and a list that grows.

The risk assessment that should come first

2026-09-15T00:00:00-06:00

Nothing can be protected by someone who does not know what threatens it. That sounds obvious enough to skip past, and it is the step most often skipped.

Years ago, at a client site, an executive one level below the chief executive found Kenneth and said, "I want to be secure." The reply was straightforward: happy to help—so tell me what you are protecting, and what the threats against it are. The executive could answer neither question. He wanted security as a property of the company, the way a roof is weatherproof, without reference to what was underneath it or who might want to get in.

That conversation is more common than it should be, and it is not a criticism of the executive. Wanting to be secure is the correct instinct. It simply is not yet a specification, and nobody can act on it. Spending against it produces purchases rather than protection, because there is no standard by which to judge whether any given purchase helped.

The risk assessment is what converts the instinct into something actionable: these are the assets, these are the people who would want them, this is what they would attempt, and this is what currently stands in the way. Most security governance activities take that knowledge for granted. The risk assessment is where it is supposed to come from, and it is frequently the document produced last, when it should be the one produced first.

What it is required to be

The requirement is not optional for most organizations with obligations. NIST SP 800-53 carries the parent control as RA-3, and NIST SP 800-171 states it in Revision 3 as 03.11.01: assess the risk, including supply chain risk, of unauthorized disclosure resulting from the processing, storage, or transmission of controlled unclassified information, and update that assessment at an organization-defined frequency. Revision 3 maps it to RA-3, RA-3(1), and SR-6, so the supply chain clause is not decoration. For defense contractors the department has already set the frequency, as parameter 03.11.01.b: at least every 12 months, or when there are significant incidents or significant changes to risks. CMMC inherits the requirement along with the rest of 800-171, and other frameworks state it in their own vocabulary.

Read 03.11.01 closely, though, and it is narrower than it first appears. It asks about unauthorized disclosure—which is deliberate, because 800-171 exists to protect the confidentiality of CUI, not to run the contractor's security program. Nothing in it obliges anyone to consider what happens when that same data is altered or cannot be reached. Those risks belong to the organization regardless, and an assessment scoped only to the compliance requirement will not find them.

What the requirement does not do is say the assessment must be elaborate. A risk assessment proportionate to a thirty-person manufacturer is not the document a defense prime would produce, and an assessment padded to look substantial is worse than a short one that is actually used.

Who should do it

The method matters less than the person applying it, and what that person needs is unusual: a working understanding of how the business actually operates, a current picture of what threatens organizations like it, and enough discipline to keep both up to date.

Either half alone fails predictably. Someone who knows the technology but not the business produces a document that is technically defensible and ranks everything wrongly, because nothing in a network diagram indicates which system the company cannot trade without, which customer relationship would not survive a disclosure, or which three days of the year an outage would be ruinous. Someone who knows the business but not the current threat landscape assesses this year's operations against last decade's attacks, and writes controls for the attacker they read about rather than the one presently working through their sector.

Staying current is the part that gets underestimated, because both halves move. Attacker behavior changes—techniques become commodity, a criminal business model appears and displaces another, a class of target becomes fashionable. The business changes at least as fast: a new product line, an acquisition, a department that quietly started using a service, a supplier that became load-bearing.

That combination is why the assessment is not a form for whoever has capacity this quarter. It also does not have to be an outsider. Someone inside the organization who genuinely holds both halves is usually better placed than a consultant holding one, since they know where the awkward workflows are. What an outsider brings is a different thing entirely: no stake in the conclusion, and recent sight of how other organizations in the same position actually got into trouble.

Choosing a method

Several established frameworks exist, and they are not competitors so much as tools shaped for different jobs. Picking one matters less than picking one deliberately, but the differences are real.

NIST SP 800-30, Guide for Conducting Risk Assessments. The default for anyone in the federal orbit. It is qualitative, works at the organization, mission, and system levels, and sits alongside SP 800-39 for organization-wide risk management and SP 800-37 for the Risk Management Framework. Best where: an assessor will read the result, since it is the vocabulary they expect and it maps directly to 800-53 controls. It is also free. Against it: it is long, and its qualitative high/moderate/low scales can feel arbitrary without discipline about what the words mean.

ISO/IEC 27005. Guidance on managing information security risks, built to support an ISO 27001 management system. Process-oriented—identify, analyze, evaluate, treat—and notably firm that risk owners approve treatment plans and accept residual risk. Best where: the organization is pursuing or holds ISO 27001, since the two are designed to fit. Against it: the standard costs money, and it is deliberately non-prescriptive about method, which is freedom if you want it and a blank page if you do not.

ISACA's Risk IT, within COBIT. Frames technology risk as enterprise risk, in the language of business objectives and governance rather than systems. Best where: the audience is a board, an audit committee, or an enterprise risk function that already thinks in those terms. Against it: it is built for scale, and a small organization adopting it wholesale will spend more on the framework than on the risk.

Open FAIR, from The Open Group. Quantitative. It decomposes risk into frequency and magnitude and expresses the result as probable loss in money, using the Risk Taxonomy and Risk Analysis standards. Best where: choices have to be compared or defended financially—which control to fund, whether a mitigation costs less than the loss it avoids, how to answer a chief financial officer. Against it: it needs data and calibrated estimation, and confident numbers derived from weak inputs are more dangerous than an honest qualitative scale.

OCTAVE Allegro, from Carnegie Mellon's Software Engineering Institute. Asset-driven and workshop-based, designed explicitly so an organization can run it itself with a small team and limited time. Best where: a smaller organization wants to conduct its own assessment rather than buy one. Against it: it dates from 2007 and is no longer actively developed, so the method is sound but the examples show their age.

CIS RAM. Ties risk analysis to the CIS Controls and, unusually, to the question of whether a safeguard is reasonable—balancing the burden of a control against the harm it prevents. Best where: the organization needs to show due care to a regulator, an insurer, or a court, since that balancing is the test those audiences apply. Against it: it is bound to the CIS Controls.

Two things that are not risk assessment frameworks are worth having beside whichever one is chosen. MITRE ATT&CK catalogues what attackers actually do once inside, which is the difference between naming a threat and describing one. And threat modeling methods operate a level down, on a single system or design, answering how a particular thing could be attacked rather than what the organization's risks are.

For most small and medium organizations with a compliance obligation, 800-30 qualitatively is the pragmatic answer, with FAIR reserved for the two or three decisions large enough that a number would change what gets funded.

Start from who would attack, and why

A risk assessment that begins with a list of controls and asks which are missing is a gap assessment wearing the wrong label. A risk assessment begins with who would attack this organization and, just as importantly, what they would want.

The second question is the one usually skipped, and it determines everything downstream. Someone who wants money behaves differently from someone who wants a specific document, and both behave differently from someone who wants the organization to stop functioning. The same intrusion, by attackers with different objectives, produces different damage and calls for different defenses.

The common classes are few enough to list.

Criminals want money. That shapes their behavior more than any technical preference: they take the cheapest path to a payment, prefer targets that pay, and move on when a target proves expensive. Ransomware, payment fraud, and business email compromise are business models rather than vendettas.
Industrial spies and competitors want trade secrets, designs, pricing, customer lists, and negotiating positions. Unlike criminals, they want the intrusion never to be discovered, because the value evaporates once the target knows what was taken.
Nation-states want data or effects that serve geopolitical goals: intelligence, positioning inside infrastructure, leverage. They also sometimes want commercial advantage for their own industries, which organizations outside the defense sector tend to discount. It is not a new practice—a 1992 GAO testimony to Congress laid out economic espionage against U.S. industry by allied governments, France's among them, whose first intelligence director was later candid that in economics the two countries were competitors rather than allies.
Political activists want to advance a cause, which usually means visibility. Defacement, leaks timed for maximum attention, and disruption of something symbolically connected to the grievance.
Terrorists want fear. The target is chosen for what its disruption signifies rather than for what it holds.
Insiders want a range of things—money, revenge, advantage at a new employer, or nothing at all. They matter as a separate class not because their motives are unusual but because their position is: they already hold credentials, they know where the valuable material is, and their activity looks like work. Most insider damage is not malicious at all. It is a person doing their job through a shortcut that was easier than the approved path.
Opportunists want whatever is reachable. Mass scanning, credential stuffing, and commodity malware are aimed at no one in particular, and account for a great deal of the actual damage done.

Two organizations rarely face the same mix, and the differences are larger than most expect. A defense subcontractor holding design data has a plausible nation-state interest that a regional accounting firm does not. That accounting firm has a criminal-monetization problem the subcontractor may worry about less. A municipality has both, plus a disruption risk with political motivation attached. All three have insiders, and all three are reachable by opportunists who have never heard of them.

Perspective matters as much as the answer, because the question has an implied subject: an attacker of whom?

Kenneth used to teach an introductory security course at a major IT manufacturer, where the class would build a threat model for a mobile phone. Deliberately a simple phone—one that made calls and, with T9, sent text messages—because a smartphone's threat model is far too large to hold in a classroom.

The instructive part was that the model changed entirely depending on whose it was. The manufacturer worried about counterfeits, warranty fraud, and firmware being replaced. The carrier worried about handsets leaving its network and about revenue leaking away. The owner worried about theft, about the bill, and about what was stored on the handset.

Those three were not merely different. They were opposed. In the years when carriers charged heavily for data, many locked the phone's Bluetooth and Wi-Fi transfer so that moving a photograph off the handset required the metered path. To the carrier that lock was a control protecting revenue; to the owner it was an obstacle to be routed around. Each was, from the other's seat, the attacker. The same tension survives anywhere a handset is locked to a network.

An attacker is therefore not a property of the system. The question becomes answerable only once it is clear who is asking—which is also why an assessment conducted from the wrong seat produces a document that does not fit the organization that paid for it.

Setting those attacker classes and motivations against the organization's existing policies, procedures, and controls gives a first view of where risk actually sits and what already mitigates it.

Disclosure is only one of three questions

Risk conversations drift toward confidentiality. It is the failure that breach-notification laws attach to, it is what reaches the news, and it is what most people mean when they say security incident. For a great many organizations it is also not the failure that would hurt most.

Every asset is worth three separate questions. What happens if this is disclosed? What happens if it is wrong? What happens if it cannot be reached? The answers rank differently, and an assessment that asks only the first will protect the wrong things well.

FIPS 199 builds federal categorization on exactly that structure: a system's security category is three separate judgments, one each for confidentiality, integrity, and availability. Borrowing the structure is useful mainly because it forces the two neglected questions to be asked aloud.

Integrity. The risk is not that someone reads the data but that someone changes it—or that it changes by accident and nobody notices.

Payment fraud is the everyday version. An attacker who alters the bank details on an outgoing payment has disclosed nothing and has taken the money. In engineering and manufacturing the same failure is quieter and worse: a tolerance, a material specification, or a machine toolpath altered slightly produces parts that pass inspection and fail in service, and the discovery arrives years later from the field rather than from a security tool. A municipality's exposure is in its records, since permits, assessments, utility billing, and court dockets are authoritative precisely because people trust they have not been edited.

Integrity failures share an unpleasant property. A confidentiality breach is usually discovered eventually, because the data surfaces somewhere. An integrity breach may never be discovered at all, and the organization goes on operating with the altered data. That is also why backup and log integrity deserve more attention than their dull reputation attracts: a silently corrupted backup is not a backup, and logs an attacker can edit will not explain what happened.

Availability. The question is what the organization cannot do while something is missing, and how long that remains survivable.

Ransomware is the obvious case, and it is worth naming what the damage actually is—almost always an availability loss rather than a disclosure one. The extortion over stolen copies came later and remains secondary to the fact that nothing runs. A manufacturer whose line has stopped is losing money by the hour whether or not a single file left the building. For a municipality the impact is not financial at all: dispatch, water treatment, and the court calendar do not have a revenue figure attached.

Availability risk also has a calendar, which assessments routinely miss. An accounting firm offline for three days in June has a problem. The same firm offline for the same three days in April has a different one. When a loss would land is part of how large it is.

The judgment of what an outage costs belongs in the inventory, recorded once against each system rather than reconstructed during the incident. It also varies far more than a system list suggests. A virtualization host carrying thirty workloads takes all thirty down with it. A DNS server that is replicated, has a secondary already answering queries, and is one of several is an inconvenience whose loss may go unnoticed until the second one fails. Both are servers, both appear on the same inventory line by default, and only one of them stops the company.

The attacker classes map onto the three unevenly, which is worth checking against the list above. Industrial spies want confidentiality to fail and nothing else. Criminals are largely indifferent and take whichever pays: ransomware for availability, payment fraud for integrity, stolen records for confidentiality. Activists and terrorists mostly want availability, because disruption is visible in a way that quiet theft is not. Insiders can reach all three, and the negligent ones damage integrity or availability without intending anything at all.

Refine against how the organization really works

The first pass is built from documents. The refinement comes from working through the ways the organization genuinely uses technology, rather than the ways an inventory or an architecture diagram says it does.

This is where the interesting findings appear: the data that leaves the protected environment because a workflow requires it, the administrative access a vendor holds because a support contract required it three years ago, the process that depends on one person's judgment and has no documented alternative.

Following the data is the most productive version of that exercise, and it leads directly to documenting sensitive data flows: where the data originates, every system it passes through, everyone who can see it on the way, where it comes to rest, and how it is eventually disposed of.

The reason this belongs in a risk assessment rather than in a diagramming exercise is that the flow determines the threat. A company that moves data between sites on write-once optical media has a set of risks involving couriers, physical custody, and media that cannot be revoked once it leaves the building. A company that moves the same data through a cloud service has an entirely different set involving an account that can be phished, an administrator at the provider, a misconfigured share, and a copy that persists after the contract ends. Both are defensible arrangements. Neither is safe in the way the other is, and no generic control list will tell an organization which risks it has actually chosen.

The requirement to document those flows is not unusual, either. NIST SP 800-171 expects the system security plan to describe the boundary, the operating environment, and how CUI moves through and beyond it, which is a data flow by another name. PCI DSS is explicit at requirement 1.2.4 in version 4.0, calling for an accurate data-flow diagram showing all account data flows across systems and networks, updated as the environment changes. The CIS Controls devote safeguard 3.8 to documenting data flows, including those of service providers, reviewed annually or when significant enterprise changes occur. And the NIST Cybersecurity Framework asks under ID.AM-03 that representations of internal and external network data flows be maintained.

What it produces, and what consumes it

The deliverable is usually a risk register: a list of identified risks, each with the mitigations the organization has actually implemented against it, and the residual risk that remains once those mitigations are accounted for. That last column is the one most often left out, and it is the one that carries the meaning. A risk with a control in front of it has not gone away; it has been reduced by some amount, and the organization should be able to say roughly how much and what is left.

The register also has to record the risks nobody is mitigating. Some have been accepted deliberately, which is a legitimate decision when it is made by someone with the authority to make it and is written down with a reason. Others have been transferred, most often to an insurer or by contract to a supplier, which moves the financial consequence without moving the operational one—the outage still happens.

Then there is the residue. Any risk left unmitigated has been accepted, whether or not the organization intended to accept it. Nothing about the absence of a decision makes the exposure smaller. The only difference between deliberate acceptance and accidental acceptance is that in the first case someone can explain the reasoning, and in the second the organization discovers its position after something has already happened. Writing the unmitigated risks down converts the second into the first, which is most of what a register is for.

Its value, beyond that, is measured by what it feeds.

The policy review is one consumer. A policy review informed by a current risk assessment can ask whether the policy addresses the risks the organization actually has; without one, it can only ask whether the document is internally consistent. The sequencing therefore matters, and the assessment should be completed first.

Accepted risks are another. An organization that has accepted a risk should be able to point to where that risk is described, what accepting it costs, and who decided. Exceptions granted against policy are the same category viewed through a different document.

Remediation planning is the third, and it is where most of the register's output goes. Anything that cannot be fixed immediately becomes a plan of action and milestones: the deficiency stated plainly, a named owner, a date chosen because it is achievable, and the interim risk in the meantime. An assessment that identifies work and does not deposit it somewhere it will be tracked has ended at the document, which is the same failure as a policy nobody can verify.

Deciding what to do first

A good risk assessment finds more risks than the organization can address at once. That is the normal result rather than a defect, and it makes prioritization part of the deliverable rather than a step someone improvises afterward.

The workable method is the simple one. Rate each risk twice: on impact, meaning what it costs if it happens, and on likelihood, meaning how probable it is within a stated period. Three levels for each are enough—low, moderate, high—and the federal scheme already uses those three for impact under FIPS 199, so an organization categorizing systems that way has the vocabulary and should rate confidentiality, integrity, and availability separately.

Setting the two against each other produces an order that survives argument.

High impact, high likelihood. These come first, and anything here that cannot be fixed quickly needs an interim compensating measure rather than a place in a queue. If the register has several, that is the finding.
High impact, low likelihood. The ones that end organizations. They are easy to defer precisely because they are improbable, and improbable is not the same as impossible. Their answer is often continuity planning, insurance, or a tested recovery path rather than a preventive control.
Low impact, high likelihood. The steady nuisances. Frequently the cheapest to fix, and worth fixing when the fix can be automated, because the aggregate cost is real and because people stop reporting things that happen constantly.
Low impact, low likelihood. Document, accept explicitly, revisit at the next assessment.

Two cautions keep this from becoming theater.

The ratings are judgments, and what makes them defensible is deciding what each level means before rating anything. "High likelihood" needs a definition—more probable than not within a year, say—rather than being a feeling that varies with who is in the room. The same applies to impact, which should be expressed in whatever the organization actually measures: money, downtime, safety, or obligation.

And the ordering ranks risks, not remediations. A moderate risk with a half-day fix should usually be done before a high risk that needs a six-month project, because the first reduces exposure now. The honest sequencing question is how much risk each unit of effort removes, which is why prioritization has to reflect real budgets rather than an abstract severity score.

Impact, likelihood, and the resulting priority all belong in the register beside the mitigation and the residual risk. That is what turns it from a record of what is wrong into an answer to what happens next—and rerunning the ratings after the work is done is how the organization confirms the residual risk actually moved.

The annual cadence, and the trigger

Annual is the usual interval, and like the policy review it should carry an event trigger alongside it. A new line of business, an acquisition, a significant architecture change, or a serious incident all change the answer enough to justify revisiting it before the anniversary.

An assessment that is repeated annually and reaches the same conclusions without examination is not being repeated; it is being reprinted. The test is whether this year's version reflects anything that changed.

Kenneth Ingham Consulting performs risk assessments that start from who would attack an organization and why, and produce mitigations an organization can actually operate.

What counts as sensitive data

2026-09-08T00:00:00-06:00

Nearly every security requirement is written in terms of sensitive data. Protect it, inventory where it lives, restrict who can reach it, know when it has left. All of that assumes the organization has already answered a question that is rarely asked directly: which data, specifically, and how does anyone know?

Organizations that skip the question tend to land in one of two places. Either sensitivity is decided by instinct in the moment, which cannot be trained, audited, or applied consistently by two people on the same day. Or everything becomes nominally sensitive because nobody decided otherwise, which sounds cautious and generally means nothing receives particular care while staff quietly work around controls that apply to the cafeteria menu.

That second one has to be separated from something it resembles closely and is entirely defensible. An organization can decide, deliberately, to hold everything to the strictest standard it is subject to. That is a real strategy with real advantages, and for some organizations it is the only sane one. It is covered further down. The failure is not treating everything as sensitive. It is arriving there by default, having never established what the strictest standard requires.

The categories are genuinely different

Sensitive data is not one thing with one owner. The common categories differ in where the obligation comes from, who defines the boundary, and what happens when it is wrong.

Regulated personal information, commonly called personally identifiable information (PII), and protected health information (PHI) where health records are involved, carries obligations created by law. The organization did not agree to them and cannot negotiate them, they follow the person rather than any contract, and they frequently reach across state and national borders to wherever the person is. What counts is defined externally and changes without the organization being consulted.

There is no single authority to consult here, which is itself the difficulty. The United States regulates personal information by sector rather than comprehensively, so which law applies depends on what the organization does. Health records fall under HIPAA, whose Privacy and Security Rules sit at 45 CFR Part 164 and whose definition of protected health information is at 45 CFR 160.103. A financial institution holding customer information is under the Gramm-Leach-Bliley Act, which states an affirmative and continuing obligation to protect the security and confidentiality of nonpublic personal information. A federal agency, or a contractor operating a system of records on its behalf, is under the Privacy Act of 1974.

Underneath all of it sits breach notification, which is state law rather than federal, exists in every state, and differs between them on what counts as personal information and how quickly notice must be given. An organization with customers abroad acquires those regimes too. The practical consequence is that this category cannot be settled by reading one document, which is why it more often needs counsel than an engineer.

Controlled Unclassified Information (CUI) carries obligations created by a contract, and its definition belongs to the government rather than to the organization holding it. This is the category most often misunderstood, because the name sounds like a description of importance. It is not. CUI is a defined set of categories with an official registry and marking rules, published by the National Archives, which runs the CUI program and maintains the registry of categories. Information either falls inside a listed category or does not. An organization's belief that something feels sensitive has no bearing on whether it is CUI, and neither does its belief that something is routine. The program itself is established by 32 CFR Part 2002, and defense contractors have a second place to look, since the DoD CUI Program publishes its own category and marking abbreviations.

CUI is also not one obligation. The regulation divides it at 32 CFR 2002.4 into CUI Basic, where the authorizing law sets out no specific handling or dissemination controls and the uniform controls apply, and CUI Specified, where the authorizing law does set out its own handling controls. Specified controls are not merely stricter; they can simply differ. Treating the two as interchangeable is the most common and most expensive mistake in this area, because the consequences are not remotely comparable.

A spill of CUI Basic is a serious matter, handled through contract mechanisms and reporting obligations. Nobody is likely to go to prison over it. Export controlled information, marked with the export control category's abbreviation, is a different proposition. Where the underlying authority is the export control regime, disclosure to a foreign person can be a criminal violation carrying substantial fines and a real prospect of imprisonment—and "foreign person" includes an employee working lawfully in the organization's own office, and a cloud administrator abroad whose employer never told the customer where support is staffed.

The practical consequence is that a single control set applied to everything marked CUI will be simultaneously too heavy for the basic categories and too light for the specified ones.

The marking is not the requirement. The marking names a category, the registry entry for that category names the law or regulation that authorized it, and that authority is what the organization is actually obliged to follow. Looking the category up is the first step and by far the easier one. What follows is reading the authority itself, working out what it requires for this data in this situation, and then complying with it—which for a Specified category can mean obligations that appear nowhere in the general CUI guidance, nowhere in the contract clause that brought the data in, and nowhere in the control set the organization has already built.

That is the work. An organization that has looked up its categories and stopped there knows the names of its obligations and has not yet met any of them.

Company proprietary information has no external authority at all. Nobody outside the organization will define it, and no registry will settle an argument about it. That makes it the category most often left undone, because deciding requires judgment rather than a lookup.

Information belonging to somebody else, held under a nondisclosure agreement (NDA) or equivalent confidentiality terms, carries obligations the organization accepted deliberately and often years ago. Customer data held to deliver a service usually lands here, and the terms are in an agreement rather than in a statute.

The practical consequence is that these cannot be handled by one policy sentence. They have different definitions, different owners, different retention pressures, and different consequences for getting them wrong.

Every contract says something about protecting data

Organizations often assume that data protection obligations arrive only with the obviously regulated work. Reading the whole contract portfolio of a mid-sized company says otherwise. Every single agreement contained a data protection requirement. Not most of them, and not only the government ones.

What varied enormously was what the requirement said. At one end, a single sentence obliging the contractor to protect the customer's information. At the other, a Department of Homeland Security contract that incorporated by reference the department's own reimplementation of the federal control catalog—a policy directive, a handbook, and roughly thirty lettered attachments, one of which tailors the federal controls and another of which carries the department's own baselines and parameter values. The clause in the contract was a few lines long. The obligation it created ran to something on the order of a thousand pages.

Both are binding, and the short one is arguably harder to satisfy. A sentence requiring appropriate protection obliges the organization to decide what appropriate means, apply that consistently, and be able to defend the decision years later to somebody whose view of appropriate may differ.

"Best practices" is not a requirement

A clause requiring best practices, industry standard care, or commercially reasonable security has the same defect as one requiring appropriate protection. It is worth naming separately because the phrase is common enough that it reads as though it means something.

Best for whom? What is right for a four-person non-profit with a donor list is not what is right for a billion-dollar multinational, and neither answer is wrong. Best when? Practice moves, and what was defensible three years ago may not be defensible now. Best against what? An organization facing opportunistic crime and one facing a well-resourced adversary have different correct answers to the same question.

The phrase has no referent, so nothing can be measured against it. Both parties sign believing they know what it means and find out they disagree at the one moment it matters. The organization cannot demonstrate compliance, an assessor cannot test it, and a dispute ends up buying expert testimony to supply a meaning the contract declined to provide.

For anyone in a position to write the clause, the fix is to name a published standard and be specific about which part of it: the CIS Controls at a stated implementation group, NIST SP 800-171 at a stated revision, or whatever fits the work and the size of the organization doing it. That makes the obligation determinable before signature, so both sides can price it, and testable afterwards, so an argument about whether it was met has an answer.

For the organization receiving such a clause rather than writing it, "best practices" is the signal to ask which standard is meant, and to get the answer in writing. Same question, same technical contact, same record.

The conditional contract

Return to that Homeland Security contract, because length was not its real difficulty either. It was written to be usable across the whole department rather than for the work actually being performed, so much of it was conditional: if the system does this, and the data is that, then the following controls apply. Determining which conditions were met was not possible from the contract, because the contract did not say. The answer had to be obtained by having the program manager ask the customer.

A contracts lawyer's description of that drafting is that it is poor practice, and the security consequence is worse than the legal one. An organization cannot categorize data whose obligations it cannot determine, and it cannot determine them by reading.

What it can do is ask, and asking well matters. The question goes to the customer's technical point of contact rather than to the contracting officer. The contracting officer administers the agreement and is often genuinely unable to say which control set applies to which data on this particular effort; the technical contact usually can, or knows who can. Organizations reverse this regularly, get an unsatisfying answer from the person whose name is on the contract, and conclude that nobody knows.

Then write down what was asked, who answered, their role, the date, and what they said, and keep it with the contract. Two reasons, and the second is the important one. The first is that the record is the only place the applicable control set will ever exist in one piece; organizations that skip it spend the next three years re-deriving it, usually during an assessment, usually from somebody who has since changed jobs.

The second is that if a spill happens later, there is an enormous difference between an organization that assumed and one that asked, was told, and can produce the exchange. It costs an email and a file. It is the cheapest insurance available in this entire subject.

The same duty, for a different reason

That was one badly drafted contract. The obligation to ask also arrives in a structural form that has nothing to do with drafting quality, and any defense contractor will recognize it.

The Department of Defense includes DFARS 252.204-7012, "Safeguarding Covered Defense Information and Cyber Incident Reporting", in very nearly every contract as a matter of routine. It requires protections meeting NIST SP 800-171 and reporting of cyber incidents within 72 hours.

Its presence, though, does not establish that any covered defense information is actually involved in the work. Plenty of contracts carry the clause and never touch CUI at all. The clause is in the contract because it is in almost every contract, not because somebody determined that this effort handles protected information.

So the determination falls to the contractor, and it is the contractor's problem in both directions. Markings are not always applied, and when applied are not always applied correctly. The government does not reliably volunteer which categories are in play. Assume CUI is present when it is not, and the organization has built and priced a control set nobody required. Assume it is absent when it is present, and the organization has a spill, a reporting failure, and a contract problem, in that order and on a 72-hour clock.

The answer is the one above, for the same reasons: ask the technical point of contact, get the answer in writing, and keep it beside the contract. A clause that appears in every contract tells an organization nothing about its own work. Only asking does.

The practical lesson is that the question is not which contracts mention security. They all do. The question is what each one actually requires, and incorporation by reference is where that hides: the clause is short, and the obligation is not.

Who decides, and who does not

For regulated information and CUI, the organization does not get a vote on the definition. It only gets to decide where the data is, who touches it, and how it is protected. Arguing about whether something ought to count is time spent on a question that has already been answered elsewhere.

Proprietary information is the opposite: the organization is the only authority that exists, so a decision has to be made rather than discovered. The useful test is to name the harm. What would it cost, and to whom, if a competitor had this, or a customer, or the public? Information that survives that question belongs on the list. Information that does not is ordinary business information, and saying so out loud is what makes the protected list credible.

"Everything is confidential" is the same statement as "nothing is confidential", delivered with more confidence. That is a claim about what the information is, which is a separate question from how much of it to protect. An organization can conclude honestly that very little of its information is proprietary and still choose to handle everything to one standard, for reasons covered below.

A label has to change something

A classification scheme with four levels and one set of handling rules is decoration. Each label has to change what happens: where the data may be stored, who may see it, whether it may leave the organization, what may be done with it after the work is finished, and how it is disposed of.

The test is straightforward. Take any two labels and describe how handling differs between them. If the answer is difficult to produce, there is one label wearing two names, and the scheme should be simplified until every distinction does work.

Short schemes get followed. A scheme with three levels that people can recall without looking is applied far more consistently than a scheme with seven that requires a reference table, and consistency is most of the value.

One level is a legitimate answer

Taken to its conclusion, that reasoning sometimes lands on a single level. Find the strictest requirement the organization is subject to, apply it to everything, and stop classifying. For some organizations this is not laziness but the correct decision, and it deserves stating clearly because the literature tends to treat classification schemes as self-evidently good.

The advantages are substantial. There is one control set to build, document, and be assessed against. Nobody has to classify anything during ordinary work, which means nothing gets misclassified, and misclassification is the failure mode that actually causes spills. Training is one conversation rather than a decision tree. An assessor sampling any system finds the same controls, so evidence from anywhere is evidence about everywhere. For a small organization holding one regulated category, the cost of building and maintaining two handling regimes usually exceeds the cost of over-protecting the rest.

The cost is equally real and it grows with scale. Applying controls designed for the most sensitive category to everything means encryption, access restriction, retention limits, and disposal requirements on the cafeteria menu. That is money, and it is friction, and past a certain size it becomes an obstacle to doing the work. The point where it stops paying is roughly where the protected material becomes a small fraction of the whole, and the overhead on everything else exceeds what maintaining a boundary would cost. That is when a second level earns its keep.

Someone still has to track the floor. Whichever way the organization goes, the strictest applicable standard has to be identified and then watched, because it moves when a new contract is signed or a new line of business opens. Choosing a single level is a decision about handling, not an exemption from the analysis in the rest of this article. An organization that applies one standard everywhere without knowing which standard that is has not simplified anything; it has guessed, and it will find out whether it guessed high enough at the worst possible time.

Where the tooling question comes in, and where it does not

Some of this sounds like a description of data loss prevention (DLP) software, and it is worth being direct about the relationship. DLP tools watch data in motion and at rest—outbound email, uploads, removable media, cloud storage—and act on what they find, either by blocking it, quarantining it, or recording it for review. Some read the labels a classification scheme applies. Others infer sensitivity from the content itself, matching patterns such as account numbers or the markings on a document.

The important point is the order of operations. Such a tool enforces a decision about what is sensitive and where it is permitted to go. It does not make that decision, and it cannot be configured by an organization that has not made it. Buying one first produces either a policy of blocking nothing, or months of false positives that end with the tool being switched to monitoring and quietly ignored.

There is also a category of data these tools handle badly, and source code is the clearest example. Consider an organization holding a large body of source code: some written in-house, some open source under a permissive license, some under a reciprocal license with obligations attached, and some received from a customer or a partner under terms of its own. Those files carry genuinely different restrictions on where they may go and who may see them. Nothing in the text distinguishes them. Sensitivity here is a property of provenance and license, not of content, and content inspection is what these tools do.

The usual answer is that the files should be labeled, and this is where the approach quietly fails. Labeling that volume of code is work nobody has budgeted, it has to be redone as files are added, moved, refactored, and vendored in, and it depends on every developer applying the right label every time. No program manager is willing to spend the team's time on that, and saying so is not a criticism of the program manager. A control that requires sustained voluntary effort from people measured on something else stays accurate for about a quarter, if that long.

Where content inspection cannot categorize the data, the answer is usually to control the container rather than the contents: which repositories exist, where they are hosted, who can clone them, and what may leave. That is an access control problem with a reliable answer, rather than a classification problem with an unreliable one. Provenance and license are worth tracking too, but that is a software composition question answered by different tooling than the kind that watches outbound traffic.

DLP software is also not required. Nothing in this article assumes such a product exists. A small organization whose staff know which categories exist, where each is allowed to live, and who to ask when unsure may be genuinely well controlled with no tooling at all, and an assessor can be shown that through the procedures people actually follow. Tooling earns its place as scale grows, turnover rises, or the consequence of a single mistake becomes large enough that relying on everyone remembering stops being reasonable. That is a judgment about the organization and its people, not a requirement that follows automatically from holding sensitive data.

The categories overlap, and that is normal

A personnel record is both regulated personal information and company proprietary information at the same time. The salary, the performance review, and the disciplinary history are personal data about an identifiable employee, carrying whatever the applicable privacy law requires: access rights, retention limits, and notification duties if it is disclosed. The same file is also information the organization has its own reasons to protect, because a compensation structure is useful to a competitor recruiting from it. Treating the record as purely an HR privacy matter tends to secure it against outsiders while leaving it readable across the company. Treating it as purely proprietary tends to protect it well and ignore the employee's rights over it entirely.

A contract deliverable can be CUI and somebody else's confidential information at once. A report written for a defense customer may contain government information that falls in a CUI category, and alongside it the technical data a subcontractor supplied under a nondisclosure agreement. The CUI obligations arrive through the prime contract and specify how the information is stored, marked, and reported on if spilled. The subcontractor's obligations arrive through a separate agreement that may forbid disclosure to named competitors, require return or destruction at the end of the engagement, and say nothing about marking at all. Neither set is a subset of the other.

When categories overlap, the handling is the union of the obligations rather than an average of them. The strictest storage requirement applies, the shortest retention limit applies, and the broadest notification obligation applies. Organizations sometimes try to resolve overlap by picking the label that fits best, which reliably discards one of the obligations—usually the one that came from the quieter party, because the government sends assessors and the subcontractor does not.

Notification duties travel with the data

Every category above carries some duty to tell somebody when things go wrong, and those duties are the part most often missing from a categorization record. They get left out because they are not about protecting the data. They are about what happens once protecting it has failed, which feels like a different subject and is not: the obligation attaches to the category, arrives with it, and travels with it wherever it goes.

They also multiply in ways that surprise people. Federal agencies do not share one reporting requirement; a defense contract, a health regulator, and a grant-making agency each have their own trigger, their own recipient, and their own clock. Within a single state, different agencies can impose different duties on the same organization for the same event. And a commercial contract can add its own on top of all of it, frequently on a shorter deadline than any statute, because a customer who wants to be told within twenty-four hours can simply write that down and will.

The practical consequence is that "who has to be told, and how quickly" has a different answer for each category, and sometimes several answers for one category held under several agreements. Working that out while an incident is in progress is the worst available time, and it is when most organizations first attempt it.

Breach notification in its own right is a large subject and not this one. The point here is narrower: the duties belong in the categorization record next to the handling rules, because they are a property of the data rather than a property of the incident, and an organization that has categorized its data without recording them has left out the part with a clock attached.

Where it usually goes wrong

The copies are forgotten. The database is identified and protected while the extract someone made for a report, the backup, the test fixture built from production, the email attachment, and the meeting transcript are not. Data does not become less sensitive by being copied somewhere less formal, though it almost always becomes less protected.

Aggregation changes the answer. Fields that carry no obligation individually can identify a person when combined, and a list of ordinary facts about a project can describe a capability the organization would rather not publish. Sensitivity is a property of the collection, not only of the field.

Nobody owns a category. A list of categories with no named owner produces no decisions when an unusual case appears, and unusual cases are exactly when the decision matters.

The list is written once. New obligations arrive with new contracts and new lines of business, and a classification scheme that has not been revisited since a contract was signed is describing the organization that existed then.

That last one is why the major control standards do not treat this as a one-time exercise. They require the categorization to be reviewed on a defined schedule and again when something changes, and they say so in more than one place: in the requirements covering how information is categorized in the first place, in the system inventory requirements that expect the record to stay current, in the media and information handling requirements that determine marking and storage, and in the periodic review of policies and procedures. An organization that categorized its data once and has grown two lines of business since is not partially compliant with those. It is describing a company that no longer exists, and the assessment will be against the one that does.

What this should produce

A record naming each category of sensitive data the organization actually holds, and for each one: where the obligation comes from, who owns decisions about it, and what handling follows. That is enough to train against, enough to audit against, and enough to answer the question that starts every serious security conversation.

The word to resist is "document", because it suggests something written once and filed. This is a live record. Every new contract can change it, and most organizations sign contracts more often than they revise policies. Two fields carry most of the weight and are the ones usually missing.

The contract reference. Each obligation should name the agreement it came from, by number or identifier, because obligations end. When a contract closes, the requirements that arrived with it may lapse, and an organization that cannot trace which requirement came from where will keep applying all of them forever. That is a slow, invisible, permanent cost increase that nobody ever decides to accept.

The effective dates. When the obligation started, and when it ends or comes up for renewal. Without dates there is no way to answer what applied at a given moment, which is exactly the question asked after an incident, during a dispute, and in any assessment covering a period rather than a day.

The notification duties, as described above: who has to be told, on what trigger, within what period. This is the field with a clock attached, and the only one whose absence is discovered under time pressure.

For a small organization a maintained table is entirely adequate. Past a certain number of contracts, or when the same data category arrives under several agreements with different terms, a table stops working and a small database is the honest answer. The trigger is usually not size but overlap: the first time somebody has to determine which of three customers' terms governs a particular file, a spreadsheet has already failed.

It is also the prerequisite for nearly everything else. An inventory cannot record what the organization has not defined, a risk assessment cannot weigh consequences for data nobody has categorized, and an incident response plan cannot tell anyone whether an event is reportable. Those all depend on this record existing, which is why it is worth doing before the work that assumes it.

The control standards agree, in the sense that matters: categorization sits near the front of every one of them, and the requirements that follow are written as though it has already been done. An organization working through a control set from the top will meet the categorization requirement early, and an organization that skipped it will find every later requirement asking a question it cannot answer.

Kenneth Ingham Consulting helps organizations decide what their sensitive data actually is, and what handling each category should carry.

Not all evidence is equally believable

2026-09-01T00:00:00-06:00

Two organizations can hand over evidence for the same requirement and get noticeably different receptions. That is not inconsistency on the assessor's part. Evidence differs in how much weight it can bear, and audit practice has ranked it that way for a very long time.

The ranking is worth understanding before an assessment, because an organization has more control over what kind of evidence it produces than it usually realizes.

Roughly strongest to weakest

What the assessor obtains directly. A configuration the assessor reads from the system, a test they run, a process they watch being performed. Nothing sits between the fact and the person forming the conclusion.

What a system produced while doing its job. Logs, automatically generated reports, ticket histories with system timestamps. These were created for the organization's own operational purposes, not for the assessment, and they are usually inconvenient to alter selectively.

What the organization compiled and handed over. Exports, screenshots, spreadsheets assembled to answer the question that was asked. These might be entirely accurate, and they were produced by someone who knew what answer would be convenient.

That knowledge is the difficulty, and acting on it requires falsifying nothing. Someone who knows what the answer is supposed to be makes a series of small choices while gathering the data—which systems to pull from, which date range to cover, how to word the query, which of several exports to attach—and each of those choices has a defensible rationale and a direction. The result can be truthful in every particular and still not represent the population it appears to describe. It is also the kind of bias that does not feel like bias to the person doing it, which is why an assessor would rather specify the sample than receive one.

What someone says. An explanation of how the process works. Useful for understanding, necessary for context, and the weakest thing on which to rest a conclusion.

The principle underneath

Two things move evidence up or down that list.

The first is distance from the party with an interest in the result. This is not an accusation of dishonesty, and it is worth being clear about that, since organizations sometimes hear it as one. The ranking exists because incentives exist. A method that only works when nobody is tempted is not much of a method, so audit practice assumes the incentive and prefers evidence that does not depend on the auditee's disinterest.

The second is whether the record was created for its own purposes or for the assessment. A report the operations team has produced monthly for three years, and acted on, says something a report generated the week before the assessor arrives does not—even when both contain the same numbers.

Two related factors adjust the weight further. Contemporaneous records beat reconstructed ones: something written when the work happened is stronger than something assembled afterward from memory. And corroboration matters, because two independent sources agreeing is stronger than either alone, particularly when one of them is a system record and the other is a person's account.

The idea is not unique to security

ISACA's CISA material is where many security practitioners first meet this ranking, but it neither originated there nor is peculiar to information systems audit. Financial and internal audit reached the same conclusions, in some cases decades earlier, and phrase them in strikingly similar terms.

The AICPA's audit evidence standard, AU-C 500, long set out generalizations that will look familiar: evidence obtained directly by the auditor is more reliable than evidence obtained indirectly; evidence from independent sources outside the entity is more reliable than evidence from within it; and original documents are more reliable than photocopies or converted electronic copies, whose reliability depends on the controls over the conversion.

Statement on Auditing Standards No. 142 rewrote that section, effective for periods ending on or after December 15, 2022, and replaced the ranking with attributes of the information itself: accuracy, completeness, authenticity, and susceptibility to bias. That is less a reversal than a generalization of the same reasoning. Directness and independence were always proxies for those attributes, and naming the attributes directly travels better to forms of evidence the older wording never anticipated. The PCAOB's AS 1105 makes the same point for public company audits, tying reliability to the nature and source of the evidence and to whether that source is knowledgeable and independent of the company. ISA 500 covers the same ground internationally.

Internal audit frames it as properties rather than a ranking. Under the IIA's 2017 framework, standard 2310 required auditors to identify information that is sufficient, reliable, relevant, and useful, and defined sufficient as factual and convincing enough that a prudent, informed person would reach the same conclusion. The 2024 Global Internal Audit Standards, mandatory since January 2025, carry that requirement into Standard 14.1 on gathering information for analyses and evaluation, framed as relevance, reliability, and sufficiency. The numbering changed; the reasoning did not.

The convergence is the interesting part. Bodies with different mandates, overseeing different work, arrived at the same handful of principles: prefer what the auditor obtained directly, prefer what came from a source with no stake in the answer, prefer the original to the copy, and have enough of it to carry the conclusion. An organization assembling evidence for a CMMC assessment is being measured against reasoning that predates CMMC by a long way, which is also why the reasoning is unlikely to shift under it.

What this means for the organization

The useful consequence is that evidence strength is largely a design decision, made when procedures are written rather than when an assessor asks.

Prefer output a system generates on a schedule over something a person assembles on request. The first is stronger and, after setup, less work.
Keep what makes a record self-describing: the timestamp, the system it came from, the account that produced it, and the period it covers. An undated screenshot with no hostname is a picture of a screen.
Retain records where they cannot be quietly edited, and where the retention period outlasts the assessment window.
Where an assessor can be given read-only access to a live system rather than an export, offer it. It is stronger evidence, and it means the assessor is not carrying a copy of the organization's data.
Keep the record of the work, not only the result. That a review was completed is weaker than the dated record naming what was examined and what changed.

None of these is expensive when chosen at the outset. Retrofitting them under assessment pressure is expensive, and it produces exactly the reconstructed, compiled-on-request evidence that carries the least weight.

Weak evidence is not useless

An organization that can only offer a spreadsheet and an explanation has not failed. But it should understand what it has just set in motion.

The mechanism is simple, and worth stating as plainly as possible. When an assessor receives strong evidence that a requirement is met, the item is settled and they move to the next one. When the evidence is weak, or has to be produced during the assessment because it did not already exist, they do not move on. They ask another question, request another sample, look at a second system, and start wondering what else is in the same condition.

Nothing about that is punitive. An assessor's job is to reach a defensible conclusion, and weak evidence has not given them one yet, so they keep going until something does.

The cost lands in two places. The obvious one is time, and assessment time is expensive. The less obvious one matters more: a longer examination covers more ground than a short one, and findings are discovered in the ground that gets covered. Weak evidence rarely produces a finding by itself. It produces the extended look during which the actual findings turn up.

The organization that finishes quickly is usually not the one with fewer problems. It is the one that could answer each question the first time it was asked.

Kenneth Ingham Consulting reviews what an organization's procedures actually produce and what that evidence will be worth when someone independent examines it.

What an annual security policy review should produce

2026-08-25T00:00:00-06:00

An annual policy review should be more than changing the date and collecting a new approval signature. Technology, contracts, staff, vendors, threats, and business processes change during the year. The review is the point at which the organization checks whether the document still describes what it needs to do and what it actually does.

Why annual, and not merely "regularly"

Organizations sometimes treat the yearly cycle as a convention rather than a requirement. It is a requirement, in nearly every framework an organization is likely to be measured against, and the wording converges to a striking degree.

NIST SP 800-53 puts a review obligation in the first control of every family—AC-1, AT-1, AU-1, CA-1, CM-1, IA-1, IR-1 and the rest—each requiring that the policy and procedures be reviewed and updated at an organization-defined frequency, and following organization-defined events. It names neither the frequency nor the events itself, leaving both to be filled in elsewhere. Note that both halves of the pattern are already present in the most general of these frameworks: a schedule, and a trigger that does not wait for it.

That phrase—organization-defined—is misleading, and worth pausing on. It suggests the organization being assessed chooses the value. Usually it does not. The customer, the regulator, or the framework chooses, and the contractor inherits the result. An organization under no external obligation is genuinely free to pick. Every other organization should find out what has already been picked for it.

NIST SP 800-171 carries the requirement forward for organizations handling controlled unclassified information, and supplies the clearest example of that inheritance. In Revision 3 the requirement is 03.15.01: develop, document, and disseminate the policies and procedures needed to satisfy the requirements, then review and update them at an organization-defined frequency.

Fifty of Revision 3's requirements carry such a parameter, and for defense contractors they are not open questions. In April 2025 the department published DoD Organization-Defined Parameters for NIST SP 800-171 Revision 3, defining the values as policy in preparation for making Revision 3 the minimum requirement for contractors.

For the review requirement, the value is explicit. Parameter 03.15.01.b is assigned:

at least every 12 months, or when there are significant incidents or significant changes to risks

The memorandum defines significant in a footnote, as "having or likely to have influence or effect." Both halves of the pattern are there again: a maximum interval, and an event trigger that does not wait for it. Nor is twelve months a one-off choice—it is the value the department assigns to nearly every frequency parameter in the document.

A contractor reading Revision 3 should therefore read that list beside it. The values are a floor rather than a suggestion, and only in four instances did the department leave guidance in place of a specified value.

CMMC rests for now on Revision 2, where the obligation is distributed across the requirement families rather than stated once, and where the interval arrives through a definition rather than a parameter. The CMMC Assessment Guide for Level 2 defines the term the requirements use:

Periodically: Occurring at a regular interval as determined by the OSA that may not exceed one year.

That is the sentence to cite when someone proposes a longer cycle, and it also marks where the organization's discretion begins and ends. Anything shorter than a year is genuinely its own choice; a year is the outer limit of what it may choose. The assessment and scoping guides for every level are published on the DoD CIO's CMMC documentation page.

CIS Controls state it plainly and repeatedly. In version 8.1.2, twenty-one Safeguards close with the same clause:

Review and update documentation annually, or when significant enterprise changes occur that could impact this Safeguard.

Only the object varies—documentation, content, the inventory, the policy, the classification scheme. The trigger never does. Most of those Safeguards carry the Govern security function that version 8.1 introduced, which is CIS saying outright that this is governance work rather than paperwork.

Two others, 15.4 and 16.6, require the annual review without the event trigger. That is the exception worth noticing: a schedule with no trigger is the weaker of the two halves, and it is rare. The Controls themselves are free; downloading them requires registration.

PCI DSS is the most explicit of the group. In version 4.0, requirement 12.1.2 sets the interval at least once every twelve months, with updates as needed to reflect changes to business objectives or risks to the environment. Requirement 12.1.1, immediately above it, is the separate obligation to establish, publish, maintain, and disseminate the policy in the first place.

Version 3.2.1 numbered the review requirement 12.1.1, which is worth knowing before quoting a number from memory. A citation carried forward from the older standard now lands on the adjacent requirement.

ISO/IEC 27001 is the outlier, and instructively so. Annex A 5.1 asks that policies be reviewed at planned intervals, and additionally if significant changes occur, without naming an interval. Organizations are free to choose, and the interval they choose is almost always a year, because a longer one is difficult to defend to a certification auditor.

The convergence is the point. Two things recur everywhere: a maximum interval, stated outright by PCI DSS and CMMC and left to the organization by the others, and an event trigger that does not wait for it. A review conducted only when something happens has no floor, and a review conducted only on schedule misses the reorganization in March. Both are required.

An assessor's note on dates

Assessors read review dates as a signal about the organization, not only as a compliance fact.

A policy stating that it is reviewed annually, with a history showing intervals of about eleven months, reads well. It says someone owns the calendar and schedules the work with room to spare.

A history showing 366 days between reviews in a non-leap year reads quite differently. It is one day over, and it will rarely fail an assessment by itself. What it says is that the review happened because the deadline arrived, that nobody noticed it had already passed, and that the organization has no margin anywhere in the process. An assessor who sees that will tend to look more carefully at other practices, because the same habit of just-barely tends not to appear in only one place.

The remedy costs nothing: schedule the review at ten or eleven months and treat the anniversary as the deadline rather than the appointment.

Inputs to the review

The reviewer should consider what has changed since the previous approval. Each category below has a place it ought to be found, and the reviewer needing to ask around for it is itself a finding.

Laws, regulations, and contract clauses

New or changed laws, regulations, grants, customer terms, and standards all bear on whether the policy still says the right thing.

Where to look depends on the obligation. Legal or outside counsel should be tracking statutory and regulatory change for the jurisdictions and sectors the organization operates in. Sector regulators and state attorneys general publish changes directly. Trade associations and sector information sharing organizations frequently summarize them earlier and more usefully than the primary sources. For anything federal, the contracting activity is often the first to say so in writing.

Contract clauses deserve separate treatment, because they are the obligations most often missed. A cybersecurity requirement arriving through a contract or a flow-down is as binding as a regulation and considerably easier to overlook, since it enters through the sales or contracts function rather than through legal or IT. Any clause carrying a security requirement should already be tracked before signature—that is the point at which the organization can still decline it, price it, or negotiate it. The review then works from that register rather than from a fresh reading of every contract.

An organization without such a register has found its first piece of work. It is also, usually, the reason a requirement surfaces for the first time during an assessment.

Changes to systems, data, vendors, and roles

Changes to systems, data, locations, vendors, and organizational roles all potentially invalidate policy language.

These should be discoverable from the change control system rather than from memory. If the organization operates change management with any discipline, the year's changes are already enumerated, dated, and attributed, and the reviewer's job is to read the log rather than to reconstruct the year. Vendor and role changes may sit elsewhere—procurement records, the HR system—but the principle holds: the review consumes records that already exist.

Where the change log cannot answer the question, the gap is worth recording. A policy review is a reasonable place to discover that change control is not capturing what it should.

Incidents and exercises

Incidents and tabletop exercises belong together, because they answer the same question from opposite directions: what actually happens when the plan meets events.

Both should arrive with reports. An incident's final report says what occurred, what the response revealed, and what was recommended afterward. A tabletop exercise report does the same for a scenario that was not real, which is the cheaper way to learn it. Either may show that a policy requirement is unworkable in practice, that a responsibility is assigned to a role that cannot discharge it, or that a procedure everyone believed existed does not.

Recommendations from those reports that were never implemented are especially worth surfacing at review time. A recommendation made after an incident and quietly dropped is a decision the organization made without recording that it was making one.

Audits, assessments, and self-assessments

Audit findings should likewise arrive with a report, and the review should consider both the findings and what became of them.

Beyond external audits are self-assessments, whether conducted internally or by an outside firm. For defense contractors these are not optional exercises. CMMC requires an annual affirmation, submitted by a senior official under 32 CFR 170.22, that the organization has implemented and will continue to maintain the applicable requirements. That affirmation is a legal statement, not an administrative one.

The consequences of getting it wrong are worth stating plainly, because they are frequently underestimated. Misrepresenting compliance status to the government can expose the affirming official to criminal prosecution under 18 U.S.C. 1001, which addresses false statements to federal agencies and carries the possibility of imprisonment. Separately, the organization faces civil liability under the False Claims Act, which the Department of Justice has pursued against cybersecurity misrepresentation specifically since establishing its Civil Cyber-Fraud Initiative in 2021. Because the affirmation recurs every year, so does the exposure.

None of that argues for pessimism in the affirmation. It argues for knowing the answer before signing it, which is what a genuine self-assessment produces and what an optimistic one does not. The policy review is one of the places that knowledge is assembled or found to be missing.

Exceptions and accepted risks

Exceptions and accepted risks are the same subject seen from two angles, and both belong in the review.

A documented exception should already describe what it is an exception to, the risk that accepting it creates, the compensating measures taken, who approved it, and when it expires. Reviewing those is straightforward when the documentation is good: are they still needed, are the compensating measures still in place, and has anything expired without being noticed?

The rest of the organization's risk is a larger question, and it is answered by a risk assessment rather than by the policy review. Many standards require one periodically, CMMC and NIST SP 800-171 among them—which, under the CMMC definition above, means at least annually—and the sequencing matters. A policy review informed by a current risk assessment can ask whether the policy addresses the risks the organization actually has. One conducted without it is reduced to asking whether the document is internally consistent.

Measurements and evidence

Measurements and samples showing whether important requirements were actually met are the difference between reviewing a document and reviewing a practice. This is the same question as whether the policy can be verified at all, asked a year later and with a year of evidence available.

The evidence is often not named in the policy at all, and it usually should not be. The policy states the requirement; the procedure beneath it says what performing that requirement produces, where the artifact lands, and who checks it. So the reviewer's question is not whether the policy identifies its evidence, but whether the policy and its procedures together do.

That question is easy or hard depending on one thing. Where the organization has cross-referenced its documents—each policy statement pointing to the procedures that carry it out, each procedure naming the policy it serves—the reviewer follows the requirement to the procedure to the artifact, and the review moves quickly. Where those links do not exist, the reviewer is reconstructing which procedure implements which requirement before any evidence can be examined, and that reconstruction is the bulk of the work.

Where evidence does exist, the review is also the moment to ask how believable it is. A year of operation is long enough to show what a procedure really produces, and a review that never asks how strong those artifacts are will keep renewing its confidence in the weakest of them.

If no procedure produces evidence for a consequential requirement, that omission is the most useful finding the review will generate.

Overlaps and contradictions

Policies and procedures that overlap or contradict one another need to be reconciled, and the review is where that happens. This is also where confusion between the two documents becomes expensive, since a policy says what and a procedure says how, and a requirement that has migrated into a procedure has escaped the approval authority that was supposed to govern it.

Planned changes

Finally, planned changes that will make current language inaccurate. A policy rewritten to match a system being replaced next quarter has been written twice.

Where to look for these is genuinely organization-specific, and size drives the answer more than anything else. A small organization can find them by asking the handful of people who make such decisions, and that conversation is reliable because there are few enough of them to ask. A larger one needs deliberate sources: the IT and capital project portfolios, the procurement pipeline, budget submissions for the coming year, merger and acquisition activity, facility plans, and whatever governance body approves significant initiatives.

The failure mode differs by size too. Small organizations miss planned changes because nobody wrote them down. Large ones miss them because the person doing the review is not in the room where they were decided.

Speaking with the people who perform the work is essential regardless of size. A document owner may know what the process is supposed to be; operators know where it is unclear, impractical, bypassed, or dependent on undocumented knowledge.

What the report should say

A review that produces no record has not been performed, whatever was discussed. The report is the artifact, and next year's reviewer is one of its readers.

The policy identified precisely. Its title, version, effective date, and scope, along with the reviewer's name and the date the review was conducted. A report that does not identify which version was examined cannot be relied on later, and the review interval cannot be calculated from it.

The evidence and people consulted. What was read, which records were sampled, and who was interviewed. This is what allows someone else to judge how thorough the review was, and what lets a subsequent reviewer avoid repeating work or, more importantly, notice that a source consulted last year was skipped this year.

What is working and remains appropriate. Confirming that most of a policy is correct is a finding, not filler. It records that the language was examined and found accurate rather than merely left alone, and it is the part that makes "no change" defensible.

What should change, and why. Each recommendation with its reason, tied to the input that prompted it: a new contract clause, an incident report, a measurement that came back short. A recommendation without a stated cause is difficult for an approver to weigh and impossible for a successor to understand.

Conflicts, gaps, exceptions, and risks found. Including the uncomfortable ones. A review that surfaces nothing is either describing an unusually mature organization or was not really conducted, and assessors know the ratio between those two.

An owner and a target date for every follow-up action. Not "the IT department" and not "as soon as practical." A named role and a date, because work assigned to everyone is assigned to no one, and this is the part of the report that will be checked next year.

The approval decision and the next scheduled review. Who approved the outcome, on what date, and when the next review falls due—scheduled with margin rather than on the anniversary.

Not every review needs to produce a rewritten policy. "No change" can be a valid result when the report shows what was checked and why the existing language remains accurate.

Conversely, approving edits is not enough when the review uncovers operational work. Those actions need owners and follow-up outside the document workflow, and for anything that cannot be fixed immediately, that means a plan of action and milestones rather than an intention. The distinction matters: an unremediated gap with a documented plan, an owner, and a date is a managed problem, and an unremediated gap without one is simply a gap that has now been written down.

Review related documents together

Policies form a system. An AI policy may depend on privacy, data protection, acceptable-use, procurement, incident response, and records-management rules. Reviewing one in isolation can introduce contradictions or leave a new responsibility without a procedure.

The review is also the natural moment to have procedure owners confirm that their procedures are still accurate—that the steps match what is done, that the named roles still exist, and that the tools referenced are the tools in use. A procedure nobody has read in a year is usually wrong in at least one particular, and the owner is the person who will notice.

That confirmation is worth more when it produces something. Asking each owner to generate a current artifact—the report the procedure says it produces, a dated sample, a fresh export—turns an assertion that the procedure works into evidence that it does, and does it once a year at a predictable time rather than under assessment pressure.

Approval has to mean something

A policy review ends in an approval, and who signs matters as much as that someone did.

The approver must hold enough authority to require compliance from everyone the policy's scope covers. That is the whole function of approval: a policy binds people, and only someone with authority over those people can bind them. A security policy approved by the security manager governs the security team. A security policy that reaches every employee, contractor, and vendor with access needs an executive, an owner, or a board—someone whose authority is not questioned by the department that finds the requirement inconvenient.

This is also where scope and approval have to be checked against each other. A policy whose scope has grown during the review—now covering contractors, or a newly acquired division—may have outgrown the person who approved it last year.

An approver who cannot explain what the organization has committed to has been handed the wrong document, which is usually a sign that the policy has drifted into procedural detail.

The review as a maturity test

The annual review is a useful measure of the organization quite apart from the document it produces. If the organization cannot identify current ownership, evidence, exceptions, and prior corrective work, the problem is larger than stale wording. The report should make that visible rather than hiding it behind a renewed approval date.

The reviews that go quickly are the ones where every input already existed: the contract register, the change log, the incident reports, the risk assessment, the evidence the procedures were supposed to produce. The reviews that take weeks are the ones spent assembling those things for the first time. Which kind an organization has is not really a question about its policies.

Kenneth Ingham Consulting reviews existing policies and procedures and delivers a report of this kind: what is working, what should change, and why and how.

The systems inventory: local machines are the easy half

2026-08-18T00:00:00-06:00

Almost every security requirement contains a hidden dependency: the list of systems it applies to. A control can be implemented well, verified regularly, and still leave the organization exposed, because the verification ran against an inventory that was missing something. That makes the inventory worth treating as a deliverable in its own right, rather than as a spreadsheet someone assembles the week before an assessment.

The half that tooling solves

Workstations and servers are comparatively tractable, because the tool that configures them can usually enumerate them. An organization running centralized configuration management—the Unix-oriented tools or their Windows equivalents—can demonstrate that a setting is enforced across the estate and produce the list of machines it was enforced on.

The valuable output of that tool is not the list of managed machines. It is the difference between that list and reality: the laboratory system nobody enrolled, the contractor's laptop, the server stood up for a project that ended two years ago, the machine that has not checked in for months. Those gaps are findable, because the organization owns the network the machines sit on and the directory they authenticate against.

Finding that difference means holding several lists against each other, because each is built from a different vantage point. Configuration management knows what it manages. DHCP logs know what asked for an address, which includes everything the configuration tool never heard of. The vulnerability management system knows what it scanned. A network scan or switch tables know what is actually connected. And the inventory records what the organization believes it has.

In practice those lists rarely agree. That is not by itself alarming: a machine built this week may legitimately appear in some and not yet in others, and those short-term differences resolve as normal onboarding catches up. The signal worth acting on is duration. A system that has appeared in the DHCP logs and the scanner for months, and has never appeared in the inventory, is not a timing artifact. It is a system nobody is accounting for, and it has been running that way long enough for the omission to be the normal state.

Reconciling the lists periodically, and asking about anything that persists in one and not another, turns four partial pictures into something close to a complete one.

Access control at the network layer narrows the problem from the other direction. A small network can require that only known hardware addresses be permitted to connect, which is administratively simple at that scale and makes an unknown device conspicuous. Larger networks need something more robust than a list of hardware addresses, which are straightforward to observe and impersonate; the usual answer is port-based network access control to the IEEE 802.1X standard, authenticating the device or its user before the port carries traffic.

Straightforward is not the same as free. It still takes a decision about who owns enrollment and what happens when something appears unmanaged. But the mechanism exists, and the answer is knowable.

The layer below the machine

Most inventories stop at the machine and its installed applications. One level below that sits software the organization rarely counts and frequently cannot list at all: the extensions loaded into browsers, code editors, and office suites.

They deserve counting because they behave like separate systems. An extension has its own vendor, its own update channel that usually operates automatically and without review, its own permissions, and often its own network connections. A browser extension permitted to read and change data on every site can see everything the person using it sees. An editor extension can read an entire source tree, execute code, and talk to a remote service, and it is typically installed by a developer in a few seconds because it looked useful. Office add-ins sit in the same position with respect to documents, as do plugins in engineering and laboratory applications.

The practical consequence is that an inventory can be accurate about every machine and still be unable to answer what has access to the source code or the contract files. Browser extensions can be enumerated through managed browser policy. Editor extensions are harder, because they live in per-user directories rather than in the system package manager, so this usually means pointing the endpoint software inventory at those locations deliberately.

An organization that has never looked will find the answer surprising, which is the ordinary reason to look.

The half that tooling does not solve

The harder half is the systems the organization pays for and does not run.

Consider three that most organizations have. The outsourced HR platform holds employee records, and the payroll and benefits data around them. The learning management system tracks role-based training completion—which is to say, it holds the evidence for a control the organization has almost certainly committed to somewhere. The external design validation service that a handful of engineers use may hold the most sensitive technical material the company owns.

What those three have in common is more instructive than what they do. None appears in a configuration management console. Each may or may not authenticate through the corporate identity provider. And in each case the department that bought it thinks of it as a subscription rather than as a system, which is exactly why it does not appear on a list of systems.

How much IT knows varies, and the variation is the point. IT was probably consulted about the HR platform, since it had to be connected to something. It may have been consulted about the learning management system. It may know nothing at all about the service the engineers use, which was very likely bought on a card, approved by a manager who saw a reasonable business expense, and never described to anyone as a system.

That last case is shadow IT, and it is not a discipline problem. It is what happens when people who have work to do find a tool that does it faster than the procurement process moves. The organization still owns the consequences.

The design validation service is the instructive one. It is likely the smallest line item of the three, used by the fewest people, purchased with the least ceremony—and holding the material a competitor or a foreign intelligence service would most want.

Finding what was never registered

The methods that surface unregistered systems are the same ones that surface unregistered AI use: expense and corporate-card records, identity provider application lists, web proxy and DNS records, contract and renewal files, and conversations with departments about the work they are actually trying to get done.

No single source is complete. Purchasing records miss anything free. Identity provider records miss anything with its own local accounts, which tends to be the systems that matter most for authentication questions. Network records show that a service was reached, not what was sent to it.

What a record needs to carry

Organizations working to a standard do not have to invent this list. NIST SP 800-53 CM-8 requires an inventory that accurately reflects the system, includes all components within it, avoids double-counting components assigned to another system, sits at a granularity sufficient for tracking and reporting, and carries whatever information the organization defines as necessary for component accountability. The accountability information named in the guidance is concrete: the system name; software owners, version numbers, and license information; hardware inventory specifications; and for anything networked, machine names and addresses across every protocol in use. The inventory specifications are themselves enumerated—date of receipt, cost, model, serial number, manufacturer, supplier information, component type, and physical location.

NIST SP 800-171 states the same expectation more briefly, and where it states it moved between revisions. In Revision 2 the inventory is folded into 3.4.1 alongside baseline configuration, covering hardware, software, firmware, and documentation across the system life cycle. Revision 3 promotes it to its own requirement, 03.04.10, which asks the organization to develop and document the inventory, review and update it at a defined frequency, and update it as part of installations, removals, and system updates. The mapping is to CM-8 and CM-8(1).

Those requirements are written for components the organization runs, which is part of why the outsourced half falls through. Externally provided services sit under SA-9 instead, where the obligation is to require providers to meet the organization's requirements, define oversight roles, and monitor compliance on an ongoing basis. An organization that reads its inventory obligation narrowly can satisfy CM-8 completely and still have no record of the service holding its most sensitive data.

Working from that base, each entry is more useful carrying:

the operating system and version, which drives nearly every configuration and patching question asked later;
the installed software, or a reliable pointer to the configuration management system that holds it, which is usually the better answer since a static list is wrong within a week;
the business owner, who is usually not in IT;
what data the system holds, and whether that puts it in scope for a specific obligation;
what happens if it is unavailable, and for how long that is tolerable. This is the field that turns an inventory into something a risk assessment and an incident responder can both use, and it varies enormously between entries that otherwise look alike. A virtualization host carrying thirty workloads takes all thirty with it. A replicated DNS server with a secondary already answering queries may not be missed until the second one fails. Recording the judgment once is far better than forming it under pressure;
how people authenticate to it, and whether the provider supports the authentication the organization's policy requires;
who has administrative access, including at the provider;
the contract, its renewal date, and what security terms it contains;
when the entry was last confirmed, and by whom.

The authentication column is where inventories earn their cost. A requirement for phishing-resistant multifactor authentication is a claim about a set of systems, and it can only be as accurate as that set. A provider that does not support it, or supports it only on a subscription tier the organization did not buy, is a gap worth finding during selection rather than during an assessment.

What an assessor does with it

An assessment usually opens with a request for the inventory: either a copy, or read-only access to wherever the organization maintains it. The second is better for everyone. An assessor who takes a copy is now holding customer data and has to protect it, retain it, and dispose of it; an assessor who is given a read-only account is looking at the live record and holding nothing. Read-only access is not always possible, and a copy is a normal outcome, but an organization that can offer the account has made the engagement simpler.

What happens next is the part organizations tend not to anticipate. The inventory is not checked off and set aside. It drives the rest of the assessment: which people get interviewed, which systems get sampled, and what evidence gets requested.

An example. If the inventory shows both Windows and Linux systems, an assessor asking about a configuration control cannot sample only Windows machines and call the control met. Meeting it on one platform says nothing about the other, so the sample has to cover both, and the evidence has to come from systems of each type. The same reasoning applies to any split the inventory reveals: laptops against servers, on-premises against cloud, employee accounts against contractor accounts.

This is the difference between evidence that is adequate and evidence that is sufficient. A single screenshot may adequately show that a setting exists somewhere. Whether it is sufficient depends on how much of the scope it covers, and the inventory is what determines the scope.

A thin inventory therefore costs more than it appears to. It does not reduce the assessment; it moves the work into the assessment, where the organization is paying for the time and where an incomplete answer looks like an incomplete program.

Keeping it true

An inventory is accurate on the day it is built and decays from there. Subscriptions renew automatically, staff leave holding the only knowledge of a tool, departments adopt something new between reviews, and providers change what they offer.

Tying confirmation to events that already happen helps more than scheduling yet another review: contract renewals, budget cycles, onboarding and departures, and the annual policy review. The measure of a good inventory is not that it was complete when written. It is how quickly the organization would notice that it no longer is.

Kenneth Ingham Consulting helps organizations establish scope and inventory as part of readiness work, which is usually where the surprises turn up.

How do you know your security policy and procedures are being followed?

2026-08-11T00:00:00-06:00

A policy is not effective merely because it has been approved, distributed, and placed in a document repository. The useful question is: how would the organization know that everybody is following it?

That question should be answered while the policy is written. If a policy requires phishing-resistant multifactor authentication, the organization should know which system inventory or configuration report demonstrates coverage. If it requires annual access reviews, it should identify who performs the review, what records it produces, and who follows up on the exceptions. If it requires incident reporting, there should be a usable reporting path and evidence that reports receive a response.

Policy and procedure divide that work: a policy says what, and a procedure says how. The question of evidence reaches both. A policy without the procedure beneath it states an intention nobody is obliged to act on, and it is the procedure that produces the artifact the policy's claim depends on.

Start with claims that can be tested

For each important requirement, identify:

the person or role responsible for carrying it out;
the systems, people, or business processes in scope;
the evidence the activity produces;
how often someone checks that evidence;
what happens when the evidence is absent or shows a failure.

The evidence does not always need to be a technical log—a record written by a system itself, showing that a particular event happened at a particular time, such as an authentication attempt, a configuration change, or an account creation. It might instead be a ticket, an approved review record, a training roster, a tabletop exercise report, or a sample showing that a procedure can be repeated. What matters is that another qualified person can reach the same conclusion from it.

Nor does all evidence carry the same weight. Auditing practice has long ranked it by how far it sits from the party with an interest in the outcome: what an assessor tests or observes directly is stronger than what the organization hands over, and a record a system produced in the course of its own operation is stronger than one assembled by hand for the occasion. This is worth knowing while the procedure is being written rather than afterward, because the strong kind of evidence is usually no harder to produce than the weak kind—it simply has to be chosen deliberately.

A worked example

Start with a fixed line of policy:

Remote access to all systems working with sensitive data requires phishing-resistant multifactor authentication. Coverage is verified regularly, and every exception is approved, compensated, and time-limited.

That is a good requirement, and on its own it cannot be verified. Nothing in it says who checks, against what, or how often. Those answers belong in the procedure the policy points to.

The procedure carries the operating detail the policy has no business holding: how the requirement is configured on each platform, how a new employee enrolls a security key and what becomes of the temporary credential used to do it, how the enrollment report is produced, and how an exception is requested. It also answers, explicitly, the five questions in the list above.

Responsibility. The identity administrator configures and maintains the authentication requirement. The IT manager approves any exception.
Scope. Every system reachable from outside the office—the identity provider and the applications behind it, the VPN, the firewall's administrative interface—and every account that can use them, including contractor, vendor support, and administrative accounts.
Evidence. A configuration export showing which authentication methods each system permits, plus an enrollment report listing every active account and the method registered to it, compared against the current roster of people who should have access.
Frequency. The enrollment report monthly; the configuration whenever it changes, and at least at each annual review.
Correction. A missing enrollment is resolved or the account is disabled within a stated period. An approved exception carries a named owner, a business reason, a compensating measure, and an expiration date.

The useful part is what that comparison finds the first time someone runs it. The applications behind single sign-on are usually clean. The firewall's administrative interface may authenticate against a local account the identity provider never sees. A vendor support account may hold an exception granted years ago for a reason no one now remembers. One-time codes sent by text message may remain enabled as a fallback for people whose security keys fail—which is to say, for anyone who says their key failed.

The harder question is which systems belonged on the list in the first place. Workstations and servers are the tractable part: an organization running centralized configuration management can show that the setting is enforced and, more usefully, which machines the tool does not manage. The outsourced systems are where scope quietly ends. The HR platform holds employee records. The learning management system holds the role-based training completions that another control depends on. The external service a few engineers use to validate their designs may hold the most sensitive technical material the company owns, and was very likely bought on a card without passing through IT. Each of those is reachable from outside the office, and each is in scope only if it appears on a current inventory.

None of that means the policy is wrong. It means the policy covers less than it claims, and the organization now knows by how much and can decide what to do about each item.

A non-technical requirement traces the same way. Managers review their staff's access annually should produce a dated record for each manager naming the accounts examined, the changes requested, and confirmation that the changes were made. The evidence is a set of records rather than a configuration export, but the test does not change.

Evidence must support the policy—not replace it

A dashboard showing that a tool is installed does not necessarily prove the control is effective. A completed checklist does not prove the underlying work occurred. Evidence has to answer the policy's actual claim and cover the full scope, including exceptions.

This is one place where maturity becomes visible. A mature process does not depend on one employee remembering how things are done. Responsibilities, evidence, review, and correction are repeatable when people change roles or leave the organization.

A practical review

Choose a small number of consequential policy statements and trace each one from the document to current evidence. Ask the responsible person to explain the process, inspect a sample, and follow one exception through correction. The gaps will usually reveal whether the policy describes reality, an aspiration, or a process that existed when the document was last edited.

A policy that cannot be verified is close to useless. A policy tied to clear, proportionate evidence can guide work, survive personnel changes, and support an assessment without turning the organization into a paperwork factory.

Kenneth Ingham Consulting writes and reviews policies and procedures against exactly this test.

A policy says what; a procedure says how

2026-08-04T00:00:00-06:00

Many organizations own a shelf of security documents and still cannot answer a plain question about any one of them: is this telling someone what to achieve, or telling them how to do it? The distinction sounds academic until an assessment, a staff departure, or a new hire makes it expensive.

The division of labor

A policy states what the organization will do and why it matters. It should make sense to someone who does not administer the systems it governs.

It also has to state its scope: which people and which systems have to meet it. People means employees, but usually also contractors, temporary staff, and vendor personnel with access. Systems means the ones the organization runs and the ones it merely pays for. A requirement with no stated scope cannot be complied with or checked, because nobody can say what falls under it.

Scope is where policies fail quietly. "All employees" leaves out the contractors who often hold the most access. "All company systems" leaves out the ones the company does not own but depends on. The scope statement also sets the size of the inventory the organization needs in order to check its own compliance.

A procedure states how the work is done: the steps, the role that performs them, the tool involved, what the work produces, who checks it, and where the result is recorded. It should be specific enough that a competent new employee can follow it without being shown.

The same requirement, split

Consider phishing-resistant multifactor authentication.

The policy is one or two sentences: remote access to all systems working with sensitive data requires phishing-resistant multifactor authentication. Coverage is verified regularly, and exceptions are approved, compensated, and time-limited.

The procedures are everything that sentence assumes: which authentication methods are permitted and which are switched off; how a new employee enrolls a security key, and what happens to the temporary credential used to do it; how the identity administrator produces the monthly enrollment report; who compares it against the current roster of people who should have access; where that comparison is filed; what happens when an account turns up without an enrolled method; how an exception is requested, what it has to contain, and who approves it.

None of that detail belongs in the policy. All of it has to exist somewhere.

Systems the organization does not run

The split matters most where the organization does not own the computer. A cloud service or an external service provider holding company sensitive data is in scope, and the organization can configure only what the provider chose to expose.

The policy does not change: the requirement follows the data, not the question of who racks the hardware. The procedure changes considerably, because the mechanisms available are contractual as much as technical. The requirement has to appear in the agreement, the provider has to be asked what it actually supports, and the answer has to be verified rather than assumed.

That check is worth doing early. Plenty of providers are absorbed in their own product and have given little thought to what their customers are obliged to meet. A provider that cannot support phishing-resistant authentication, or supports it only on a plan the organization is not buying, is a finding to discover during selection rather than during an assessment.

Providers describe the division in a responsibility matrix: which controls the provider operates, which the customer operates, and which are shared. That document is where the organization's own procedure has to start, because it determines what is left to write a procedure about. A control the matrix marks as shared is not a control that is finished.

Why separating them is practical, not tidy

Four consequences show up in ordinary operation.

Approval and change rate. A policy is approved by someone with authority to commit the organization. A procedure changes when a vendor rearranges a settings page. Merge them and one of two things happens: trivial operational changes queue up for executive approval, or the document quietly stops matching reality because nobody wants to trigger the approval cycle.

Audience. An executive, a customer, or an assessor reads the policy to learn what the organization has committed to. The person doing the work reads the procedure. A document serving both audiences usually serves neither: too much detail to approve, too little to follow.

Handover. Turnover is where the distinction stops being a documentation question and starts costing money. An administrator leaving with a written procedure hands over a document. One leaving without hands over a conversation, and whatever gets forgotten in it leaves with them.

The bill arrives twice. First on the way out, in the scramble to capture what one person turned out to be the only one who knew, conducted during a notice period against someone whose attention has already moved on. Then on the way in, because a new hire with nothing to follow has to be taught by whoever knows the work—which means taking an experienced person off their own tasks, for every hire, with a result that varies according to who did the teaching and what they happened to remember that day. Written procedures turn all of that into reading.

Evidence. A policy makes a claim about coverage. The procedure is what produces the artifact that supports the claim, and it is the procedure that names the report, the schedule, and the file it lands in. Without that, the policy asserts something nobody can demonstrate.

The line is wide and gray

Where a given statement belongs depends on the organization. A twelve-person firm may reasonably keep one short document; a twelve-hundred-person one that tried would produce something no one could approve or follow.

A workable rule of thumb: if a sentence would have to be rewritten because the organization changed vendors, reorganized a department, or upgraded a product, it is procedure. A policy that names a particular identity provider has to go back for re-approval when the organization switches to a different one, even though the underlying requirement never changed.

The same test applies to named individuals. Policies assign responsibility to roles, because people change jobs and the requirement does not.

Checking frequency sits squarely on the line, and reasonable organizations put it on either side. How often compliance is verified is a real commitment: an assessor or a customer reads it as one, and a cadence stated in the policy cannot be quietly relaxed by whoever finds the check inconvenient. Stated in the procedure instead, it can be tightened after a bad finding without waiting on a re-approval cycle.

A workable compromise states a floor in the policy—verification happens at least annually—and lets the procedure set the working cadence above it. What does not work is leaving the frequency in neither document, which is the usual arrangement and the reason so many checks turn out to have last happened whenever someone last thought of it.

Deciding where the cadence lives is the easier half. Whether the check is actually happening, and whether the answer would convince anyone, is a separate question.

Two familiar failure modes

A policy with no procedure is an intention nobody is obliged to act on. This is the usual result of buying a template document set. It reads well, covers every heading an assessor might name, and when someone asks how a requirement is actually met, the answer lives in one administrator's memory or does not exist.

A procedure with no policy is a habit that changes when its author leaves. The work may be done well, but nothing states the requirement, so there is no basis for saying that a deviation is wrong, and no way to tell whether the practice still reflects a decision anyone made.

Link them to each other

Splitting the documents creates a navigation problem, and the fix is unglamorous: each policy statement should link to the procedures that carry it out, and each procedure should name the policy it serves. Often that is more than one. A procedure for reviewing account access may answer to an access control policy and to a personnel security policy at the same time, and it should cite both.

The organization gets the first benefit. Cross-references are how anyone discovers that a requirement has no procedure at all: a policy statement with nothing to link to is the first failure mode above, found before an assessor finds it.

The second benefit arrives on assessment day. An assessor working a control follows the policy to the procedure to the evidence. Where those links exist, that path takes minutes and the conversation moves on. Where they do not, the assessor asks, someone goes looking, and the organization spends its assessment hours demonstrating that it can find things rather than demonstrating that it is secure. Assessment time is not free, and neither is the impression left by an organization that cannot navigate its own documentation.

A test worth running

Take one requirement that matters. Hand the procedure to a competent person who has not done the job and see whether they can do the work. Hand the policy to an executive who administers nothing and see whether they can tell what the organization has committed to. If both hold, the pair is doing its job.

Neither document is paperwork for its own sake. Together they are what lets the work survive a resignation, an audit, and a change of vendor.

Kenneth Ingham Consulting writes and reviews security policies and procedures, and sorting out which statement belongs in which document is usually where that work starts.