10 vaults and a pager
I’ve been in SecOps long enough to remember when our “secrets manager” was a locked wiki page. Now I wrangle ten-plus secrets platforms across a company that grew by acquisition and speed. On paper, each vault solved a local problem. AWS Secrets Manager for cloud-native teams, Azure Key Vault for the data group, HashiCorp Vault for platform engineering, CyberArk for PAM, Google Secret Manager in a satellite org and a grab bag of legacy on‑prem stores. In practice, it’s a maze-like nightmare. Every incident starts with the same question. “Which vault did this service actually use?”
Organizational overhead aka my headache
My week is a carousel of access requests, exception reviews and policy “oopses”. Each vault has its own RBAC model, audit format, rotation idioms and CLI. Our onboarding runbook for engineers is 30 pages. Offboarding is longer. Compliance wants uniform rotation and attestations and I can only give them stitched-together exported “reports”. We maintain parallel playbooks for the same control across different tools and every policy change requires four to six coordination threads – platform, app teams, IAM, internal audit and the BU that “owns” that particular vault. Meanwhile, the dev teams just want their pipelines to work and will “hotfix” around “friction points” if we don’t at least meet them halfway.
The complexity SecOps has to navigate
The topology isn’t a neat hub-and-spoke. It’s a knotted up mess. One microservice calls an API whose key is in HashiCorp Vault while its job runner pulls database credentials from CyberArk and a data export uses a key in Azure. Meanwhile a legacy integration for that same app still reads from a flat file wrapper in a private datacenter. CI/CD injects some secrets via environment variables, others via sidecars and a few “one offs” via custom init containers. Our SSO groups map well in some platforms and poorly in others so the same “read-only” role means different effective privileges depending on which secrets vault we are talking about. Even our scanners and SIEM parsers need per-vault “normalization” just to answer “who accessed secret X at time Y.”
The security blind spots it creates
When a zero-day drops, the sprawl turns risk into roulette. The recent HashiCorp Vault disclosures were a gut check for the entire team. Multiple 0-days were surfaced including logic flaws that enabled username enumeration, lockout and MFA enforcement bypasses in certain configurations, privilege escalation and even remote code execution through the plugin system. This was as “worst nightmare” adjacent for me and my team as we care to get.
HashiCorp pushed fixes quickly and we patched fast. At least where we could see Vaults and had owners engaged. But the reality of ten-plus managers is that not every install is equally visible or equally maintained. A lab Vault used by a data science cluster sat a version behind because it wasn’t in our purview. Another business unit had username_as_alias enabled with LDAP and entity-level MFA, precisely the configuration called out in zero day release. They believed their gateway controls mitigated it – my team is keeping a close eye. Coordinating fixes meant mapping every instance, validating auth backends, checking plugin catalogs, reviewing policies for normalization quirks and verifying audit sinks… across teams, time zones and change windows. Pagerduty made sure I got very little sleep for over a week.
The worst part wasn’t the patching, it’s the uncertainty. If an attacker landed in any one of those vaults with a known chain can they laterally move to tokens that mint cloud roles? Could they abuse a plugin path we forgot to lock down? In a world where we don’t have 10+ vaults, our answers are crisp. In my world, they’re probabilities.
What it takes to consolidate (and why it often doesn’t happen)
We’ve tried. The board asked for consolidation after the third acquisition. We laid out a plan. Discovery and inventory of all secrets and call paths, a reference architecture with two strategic managers (one cloud-native, one enterprise-grade), a federated access model and a 12 to 18 month migration in phases. The technical steps are easy to outline and seemingly impossible for us to execute.
Discovery
You don’t just count secrets, you map their lifecycle. Which app calls which SDK? Where does CI fetch from at build vs. deploy? Which short‑lived creds are minted dynamically and which are zombie passwords in a forgotten scheduler? We used every tool we had and still somehow missed the ones buried in IaC modules and bash glue.
Refactoring
Every app, lambda, job and pipeline that reads a secret has to be changed. That means code updates for new SDKs, re-permissioning identities, reworking retries and testing failure modes. Some teams run quarterly while others deploy daily. Regulated systems need full validation. Third-party tools that only speak one vault need adapters or replacements. Multiply that by hundreds of services and the refactor effort alone can rival a platform migration in cost and risk.
Parallel vaults
We have to run old and new secrets managers side by side while we cut over. That introduces duplication and drift resulting in “two sources of truth” leading to hard to troubleshoot outages. We script rotations and backfills only to discover one consumer was still reading the “old” path because a sidecar wasn’t updated. Every wave needs comms, freeze windows and often rollbacks.
Budget
Who owns the budget? Secrets cut across org charts. No single BU wants to foot the bill to refactor another BU’s workloads. Platform wants centralization, product wants velocity, compliance wants evidence and finance wants to know why we’re paying for two enterprise vaults for a year. People move roles mid‑migration, champions change and appetite waned.
The unfortunate reality
So why doesn’t consolidation happen? Because the steady-state risk of sprawl is diffuse while the migration risk is immediate and real. It’s easier to write another exception than it is to pause a revenue‑generating team for a hardening sprint. Until a headline like the recent HashiCorp Vault 0‑day creates a jump scare. Even then the path of least resistance is “patch and proceed” not “replatform and refactor.”
It doesn’t have to be this way
With decodeRing, enterprises trapped in the risk-loop that prevents secrets manager consolidation can break out of the loop and dramatically simplify their secrets landscape while significantly improving their security posture. Developers love it, SecOps get’s their life back and finance appreciates a reduction in cost.
Resources
