AI Penetration Testing Cannot Replace Independent Assurance

By Tom Whitfield

One illustrative Cape Endeavors model, published in a recent white paper, put AI penetration testing at about $395 per on-demand cycle, against about $4,200 a month for high-frequency continuous testing. The hypothetical environment covered 500 hosts and 20 web applications; the figures covered model-token compute only, not vendor margins, infrastructure, human review or engineering work required to fix findings.

The government’s suspension of Cybersecurity Maturity Model Certification (CMMC) Phase 2 third-party assessment requirements has reopened the procurement argument over whether automated AI testing offers a faster, cheaper substitute for independent review. Defense contractors still must demonstrate that they protect controlled unclassified information (CUI), while the government re-evaluates how to measure that protection.

Cape Endeavors CEO Terrence J. McGraw, a retired U.S. Army lieutenant colonel, set the immediate limit. “There should not be any assumption today of a universal Level 2 AI testing mandate.”

Continuous adversarial testing closes a timing gap that an assessment every three years cannot. Attackers do not wait for an assessment window; they exploit exposed devices, misconfigurations and unpatched vulnerabilities when an opening appears.

The 2026 Verizon Data Breach Investigations Report identified vulnerability exploitation as a leading initial-access path. A periodic review cannot establish whether a newly exposed service is exploitable today.

Automated tools can repeatedly check reachable assets, attempt exploitation within authorized limits and retest fixes. For technical controls, evidence that an attack actually failed may be more persuasive than a policy stating that the control should exist.

That evidence stops at the perimeter, however. Many breaches involve stolen credentials, compromised third parties or misuse of legitimate access. A perimeter tool may miss CUI copied to an unauthorized collaboration site, sent through ordinary email or left on an unmanaged device. A successful exploit also cannot show whether defenders can detect and contain a skilled attacker already inside.

Independent assessments inspect documented controls, interview personnel and examine implementation evidence. They assess one point in time and cannot guarantee security after assessors leave, but their breadth covers security governance, people practices, external service providers, incident response and actual CUI handling. None of that reduces neatly to a sequence of automated attacks.

Automation also does not create independence. A contractor may retain control over which systems are tested, how the platform is configured and how results are reported, so greater automation need not produce independent evidence.

False Claims Act enforcement has factored in the gap between self-reported cybersecurity scores and government-assigned scores. Arnold & Porter cited a contractor whose reported perfect score sharply contradicted a later Defense Industrial Base Cybersecurity Assessment Center evaluation. Self-assessment is not inherently unreliable; unverified claims create a risk separate from the technical vulnerabilities they may describe.

McGraw’s answer is layered assurance rather than replacement. Three tests define it:

• Independently confirm required controls, and verify that the stated CUI boundary matches where sensitive information actually resides.
• Test exposed technical controls continuously or after any significant change.
• Run detection and response exercises using legitimate accounts and standard administrative tools to test whether defenders can detect and contain an adversary.

These layers must feed one another. CUI discovered outside the approved boundary should change assessment scope; an exposed access path should trigger documentation, remediation and retest; undetected malicious activity should rule out treating a clean vulnerability scan as evidence of readiness.

Per-call pricing can look cheap, but a sustained campaign includes reconnaissance, exploit attempts, evidence analysis, reporting and repeated validation. Work rises with systems, applications and findings. Different architectures and testing cadences produce different totals, so the $395 and $4,200 figures remain compute illustrations rather than market quotes or all-in contractor prices.

What does this change for defense contractors?

Contractors can improve their position now. Maintain an accurate inventory of CUI and the systems that process it; keep system security plans and plans of action up to date; minimize unnecessary exposure; document changes and remediation.

Better scoping lowers security risk and the number of systems requiring tests. Fewer recurring misconfigurations also mean fewer alerts to investigate and fewer issues to retest.

For small and midsize defense industrial base businesses, a continuous-testing requirement could convert a one-off assessment into a perpetual operating cost. If policy defines scope, frequency or the class of providers allowed to test, contractors may have little flexibility in managing that obligation. Until then, McGraw’s caution stands: no universal Level 2 AI testing mandate should be assumed.

For policymakers, define the assurance outcome before choosing the tool. The method should verify implementation, find CUI outside authorized boundaries, identify exploitable weaknesses and measure detection and response. Require credible evidence and clear accountability for exclusions and outstanding findings, then assess the ongoing cost to the contractor base.

AI can accelerate technical validation and increase its frequency. It cannot by itself prove that safeguards exist, that CUI sits in the right place or that the contractor can handle compromise.

McGraw’s verdict admits less procurement convenience: “The point isn’t to pick between human assessment and machine testing. The point is to stop treating either of them as the complete answer.” Continuous testing improves assurance, but frequency is not completeness.