The Breach That Changed Everything
We got the call on a Tuesday morning in March. Our security team had detected unusual activity on a staging server — someone was exfiltrating data through an API endpoint that shouldn't have been accessible from outside the network. By the time we contained it, the attacker had been inside for eleven days.
The entry point? A contractor's laptop, compromised through a phishing email. The contractor had VPN access, which gave them access to the staging environment, which had credentials for the production database in a config file. One phished credential, lateral movement across three systems, and eleven days of undetected access. The classic playbook.
Nobody got fired. The board asked for a plan. The plan was zero trust. I got put in charge of implementing it.
I'd read the NIST SP 800-207 standard. I'd attended two conference talks on the topic. I thought I understood what zero trust meant. I was wrong about almost everything.
What I Thought Zero Trust Meant
I thought zero trust was a product. Buy the right platform, deploy it, check the box. The vendor demos made it look that way — a dashboard, some policies, a few clicks, and suddenly every access request is verified. The marketing materials used phrases like "complete zero trust in weeks" and "turnkey architecture." I believed them.
My initial plan was straightforward: deploy a zero trust network access (ZTNA) tool to replace our VPN, add multi-factor authentication everywhere, segment the network, and call it done. Six months, maybe eight. I told the board twelve to be safe.
Eighteen months later, we're still at it. And the ZTNA tool — which we spent $40,000 on — covers maybe 30% of the problem.
The Inventory Problem Nobody Warns You About
The first thing you have to do in a zero trust migration is inventory every identity in your organization. Every human, every service account, every API key, every machine credential. You can't verify what you can't see, and you can't segment what you haven't mapped.
I thought this would take a week. We're a 200-person company. How many identities could there be?
The answer: 1,847.
Two hundred employees, sure. But also 312 service accounts — automated systems that connect to databases, APIs, and internal tools. Most of them had credentials that nobody had rotated in years. Some had admin-level access that nobody remembered granting. One service account, created by a developer who'd left the company three years earlier, still had root access to our production database. Its password was in a config file that twelve people had access to.
Then there were the API keys. Every integration — Slack, Salesforce, GitHub, Jira, three different monitoring tools — had its own set of credentials. Some were shared across environments. Some were hardcoded in scripts that nobody had touched since 2023. We found eleven API keys that were still active but whose purpose nobody could identify.
The inventory alone took six weeks. We used a combination of identity provider logs, cloud provider IAM dumps, and manual interviews with team leads. Every time we thought we were done, someone would mention another system we'd forgotten. "Oh, the data warehouse team has a separate set of service accounts." "The marketing automation tool has its own admin panel with a different user database." "The intern's project from last summer is still running on a server in AWS with root access."
This is the part that no vendor demo shows you. The zero trust architecture requires a clean identity inventory as its foundation. Building that inventory in a company that's been operating for seven years without one is forensic work. It's not a project — it's an excavation.
The Device Trust Surprise
After identity, the next layer is device trust. The idea is simple: don't just verify who the user is, verify that their device is legitimate, encrypted, patched, and managed. A valid credential from a compromised device should be treated differently from the same credential on a corporate laptop.
We enrolled our fleet in device management — about 180 company-owned laptops, easy. Then we hit the hard part: 40-something BYOD devices. Personal phones, personal tablets, personal laptops that employees used for work. Some of them had company email. Some of them had VPN clients. None of them had management profiles.
The zero trust answer is clear: unmanaged devices get restricted access. Browser-only, read-only, no downloads. Technically simple. Politically impossible.
I had two senior engineers who refused to install management profiles on their personal laptops. Not because they were hiding something — they just didn't want the company having visibility into their personal devices. Fair enough. I had a VP who worked from an iPad at home and didn't understand why she couldn't access the same systems she'd always accessed. I had a sales team that lived on their phones and treated any friction as a personal offense.
The technical implementation of device trust took a month. The organizational negotiation took four months. We ended up with a compromise: company-owned devices get full access, BYOD devices get browser-isolated access to specific applications, and a few exceptions are documented and approved by the CISO. It's not pure zero trust. It's zero trust that the organization actually uses.
The Privilege Surface Was Embarrassing
The third principle of zero trust is least-privilege access: everyone gets the minimum permissions they need, nothing more. To implement this, you need to know what everyone currently has.
We ran a permissions audit and the results were humbling. Out of 200 employees, 47 had admin-level access to systems they hadn't touched in over six months. Twelve people had standing access to the production database — not through a just-in-time elevation process, but permanent, always-on credentials. Three of those twelve had left the company in the last year. Their accounts were deactivated in the identity provider, but the database credentials were separate. Nobody had cleaned those up.
The worst finding: a shared admin account for our payment processing system. The username was admin@company.com and the password was in a shared Google Doc titled "Payment System Access — DO NOT SHARE." Twenty-three people had access to the doc. The password hadn't been changed in eighteen months.
We fixed the obvious stuff in the first month: deactivated unused accounts, removed standing admin access, rotated every credential we could find. The subtler work — scoping service accounts to single purposes, implementing just-in-time elevation, building approval workflows for sensitive access — took six more months. And it's ongoing. Every new hire, every new tool, every new integration requires a privilege review. Zero trust isn't a project with an end date. It's an operating model.
What the Vendor Won't Tell You
The ZTNA tool we bought is good. It does what it claims: verifies identity on every access request, evaluates device posture, enforces conditional access policies. It replaced our VPN for web-based applications. That part was genuinely easier than the old VPN.
But here's what the vendor didn't tell us during the sales process.
It only covers about 30% of our access patterns. Web applications, yes. But our developers SSH into servers. Our data engineers connect to databases directly. Our legacy applications use LDAP authentication that doesn't integrate with the ZTNA platform. For each of these, we needed a separate solution or a custom integration. The "single platform" we bought turned into one component of a multi-tool architecture.
The policy engine is powerful and confusing. Writing access policies that handle the complexity of a real organization — different roles, different devices, different locations, different times of day — required more engineering time than the deployment itself. Our initial policies were too broad (everyone gets access to everything during business hours) or too narrow (nobody can access anything from outside the office). Finding the right balance took months of tuning.
The logs are overwhelming. Zero trust generates an enormous amount of data. Every access request, every policy evaluation, every device attestation — it all gets logged. Without a system to analyze those logs, you're drowning in information and starving for insight. We spent another $15,000 on a SIEM tool just to make the zero trust logs useful. That wasn't in the original budget.
The user experience complaints are real. "Why do I have to verify my identity again? I just logged in." "Why can't I access this from my phone?" "Why did my access get blocked? I'm in a hotel, not a hacker." Every friction point generates help desk tickets. We saw a 40% increase in access-related support tickets in the first two months. That's a real cost that doesn't appear in any vendor ROI calculation.
What I'd Tell Someone Starting Today
If you're about to start a zero trust migration, here's what I'd actually tell you.
Start with identity, but budget three times more time than you think. The inventory will be worse than you expect. Every company has ghosts in its identity system — accounts that nobody owns, credentials that nobody rotates, permissions that nobody audits. Finding and cleaning them up is tedious, unglamorous, and absolutely essential. Don't skip it to get to the "interesting" technical work.
Get executive buy-in before you touch a single policy. Zero trust changes how people work. It adds friction. It restricts access that people are used to having. If the CEO doesn't understand why and doesn't communicate it, you'll spend your entire project fighting organizational resistance instead of building architecture. Our CEO sent a company-wide email explaining the migration after the breach. It didn't eliminate complaints, but it gave me air cover when I had to say no to a VP who wanted standing admin access.
Segment first, verify second. The network segmentation part of zero trust is the part that saves you in a breach. If one system is compromised, segmentation contains the blast radius. It's also the part that requires the most infrastructure work and the most understanding of how systems actually talk to each other. Start mapping your network dependencies now, before you need them. The map will be wrong, but having a wrong map is better than having no map.
Accept that you'll never be "done." Zero trust is not a project with a completion date. It's a security posture that requires ongoing maintenance, ongoing monitoring, and ongoing policy updates as the organization changes. New tools, new hires, new business requirements — each one affects the architecture. The organizations that succeed are the ones that build zero trust into their operational processes, not the ones that treat it as a one-time deployment.
And be honest about what it costs. Not just the tool licenses and the engineering time, but the productivity friction, the help desk volume, the organizational politics. Zero trust is the right architecture — the perimeter model is dead, and pretending otherwise is negligent. But implementing it is harder, slower, and more expensive than any vendor will tell you. Budget for that reality and you'll be fine. Pretend it's a quick deployment and you'll join the long list of zero trust projects that stalled halfway and left the organization worse off than before.
I know because I almost became one of them. The project almost died in month four, when the VP complaints reached the board and I had to justify the cost and the disruption against a theoretical threat that hadn't happened again. The breach saved me — it was recent enough that the board remembered the pain. If I'd started this project without a breach as motivation, I'm not sure it would have survived the organizational resistance. That's the uncomfortable truth about zero trust: the technology works, the architecture is sound, but the hardest part isn't technical. It's convincing a room full of people that the way they've been working for years needs to change.