Quick answer: An HSM vendor’s published PQC benchmark numbers were measured on their hardware, their firmware version, and their test conditions, not yours. Before committing to a hardware refresh, run your own test plan covering eight dimensions: key generation time for the specific parameter sets you’ll deploy, signing throughput under sustained load, latency per operation, behavior under realistic concurrency, failover and clustering behavior, backup and restore of PQC key types, compatibility with your actual CA and application stack, and end-to-end certificate issuance time. This guide is that test plan.
A hardware refresh driven by PQC readiness is expensive and hard to reverse quickly once units are deployed and keys are generated on them. Vendor data sheets are a reasonable starting point for narrowing candidates, but the only numbers that should drive a purchase decision are the ones you measure against your own certificate issuance volume, application stack, and failure modes.
Key Takeaways
- ML-KEM and ML-DSA both introduce real computational and storage overhead versus classical algorithms, but the magnitude varies by HSM model, firmware version, and parameter set, which is why vendor-published numbers need independent verification.
- A complete test plan covers eight distinct dimensions, not just raw signing speed: key generation, throughput, latency, concurrency, failover, backup, provider compatibility, and certificate issuance.
- Test under your actual certificate issuance volume and concurrency pattern, not a synthetic single-operation benchmark, since HSM performance under sustained concurrent load often differs meaningfully from single-transaction latency.
- Confirm PQC key types are actually supported by your HSM’s backup, clustering, and high-availability features before assuming feature parity with classical key types.
- Run the full test plan in a lab environment before any production hardware refresh commitment, using the specific firmware version you intend to deploy, not an earlier or later one.
The Eight-Dimension Test Plan
1. Key Generation
Measure key generation time for each specific ML-KEM and ML-DSA parameter set you intend to deploy, ML-KEM-768 and ML-DSA-65 for civilian use, or the 1024/87 pairing if CNSA 2.0 applies. Run this across a meaningful sample size, not a single generation, to capture variance, since key generation for lattice-based algorithms can show more timing variability than classical RSA or ECC generation.
2. Signing Throughput
Measure signing operations per second under sustained load specifically, not a single-signature latency figure. This is the number that actually determines whether your HSM can keep pace with your real certificate issuance volume, particularly during a bulk reissuance event, which a PQC migration itself is likely to trigger for large certificate populations.
3. Latency Per Operation
Measure round-trip latency for a single signing or key-encapsulation operation under otherwise idle conditions, distinct from throughput under load. Latency matters most for interactive or synchronous workflows where an application is waiting on the HSM response directly, rather than batch issuance where throughput is the more relevant figure.
4. Concurrency Behavior
Test with multiple concurrent client connections issuing requests simultaneously, at a concurrency level matching or exceeding your actual production peak, not a single-threaded test. HSM performance under concurrent PQC operations does not always scale linearly with single-thread numbers, and this is where a vendor’s published single-operation benchmark can diverge most sharply from real deployment behavior.
5. Failover and Clustering
If your architecture depends on HSM clustering or high-availability failover, test that behavior explicitly with PQC key types, not just classical ones. Confirm that a failover event correctly maintains PQC key availability and that clustered nodes stay synchronized for PQC keys the same way they do for RSA or ECC keys, since this is exactly the kind of feature that can lag behind initial algorithm support in a vendor’s release cycle.
6. Backup and Restore
Perform an actual backup and restore cycle for PQC keys, following your vendor’s documented procedure, and verify the restored key produces identical, valid signatures to the original. Do this before production deployment, not as a first attempt during an actual disaster recovery event, since backup format compatibility for newer key types is exactly where undocumented gaps tend to surface.
7. Provider and Platform Compatibility
Confirm your CA platform, whether AD CS or another CA software, correctly recognizes and integrates with the HSM’s PQC key storage provider or PKCS#11 implementation. A CA platform and an HSM can each independently claim PQC support while still failing to interoperate correctly together, since the integration layer between them is a distinct compatibility surface from either component’s standalone capability.
8. End-to-End Certificate Issuance
Run the complete issuance path, from certificate request through HSM signing to delivered certificate, and measure total elapsed time under your actual CA and enrollment workflow, not the HSM signing operation in isolation. This is the number that reflects what your users or automated systems will actually experience, and it can differ meaningfully from the HSM’s raw signing benchmark once CA processing, template evaluation, and enrollment protocol overhead are included.
What We’d Actually Recommend
Run all eight tests against the specific firmware version you intend to deploy in production, not an earlier evaluation unit or a later, unreleased build a vendor demonstrates. Test at your actual concurrency and volume, not a synthetic single-operation number, since that is where vendor-published benchmarks most often diverge from real deployment behavior. Treat backup, restore, and failover testing as mandatory before production commitment, not optional validation, since these are the failure modes with the highest consequence if they surface for the first time during an actual incident.
How Encryption Consulting Can Help
Building an accurate picture of your actual certificate issuance volume, concurrency pattern, and CA platform dependencies, the baseline any HSM benchmark needs to test against, is exactly what CBOM Secure provides, mapping your current signing volume and infrastructure before a hardware refresh decision is made.
Our PQC Advisory Services run this eight-dimension test plan against candidate hardware as part of a structured HSM refresh evaluation, translating vendor benchmarks into verified numbers against your actual environment. Our HSM Services team brings the hands-on testing experience to execute the plan itself, rather than relying on vendor-supplied results alone.
Test Before You Commit, Not After
A PQC-driven HSM refresh is exactly the kind of infrastructure decision where the gap between a vendor’s published benchmark and your actual production experience shows up expensively, after hardware is purchased and keys are generated on it. Running the full eight-dimension test plan, key generation, throughput, latency, concurrency, failover, backup, compatibility, and end-to-end issuance, against your specific volume and firmware version before committing is what turns a hardware refresh from a leap of faith into an informed decision.
Frequently Asked Questions
Why isn’t a vendor’s published PQC benchmark sufficient for a purchase decision?
Published benchmarks reflect the vendor’s specific test hardware, firmware version, and conditions, which frequently differ from your production environment, concurrency pattern, and issuance volume. Independent testing against your own conditions is what actually validates whether the hardware meets your requirements.
What is the most commonly skipped test in an HSM PQC evaluation?
Backup and restore testing for PQC key types specifically. Teams often assume feature parity with classical key backup without verifying it, which is exactly the kind of gap that surfaces at the worst possible time, during an actual disaster recovery event.
Should I test concurrency at my current volume or expected future volume?
Test at your actual production peak concurrency, and factor in that a PQC migration itself often triggers a bulk reissuance event across large certificate populations, temporarily raising volume well above steady-state levels.
Can a CA platform and an HSM each support PQC independently but still fail to work together?
Yes. The integration layer between a CA platform and an HSM’s key storage provider or PKCS#11 implementation is a distinct compatibility surface from either component’s standalone capability, and needs its own explicit testing.
How many key generations should I sample to get a reliable performance figure?
Enough to capture variance, not a single sample. Lattice-based key generation can show more timing variability than classical RSA or ECC generation, so a meaningful sample size is needed to get a representative figure rather than a single lucky or unlucky measurement.
