Every IT environment that has been running for more than five years is full of things nobody can fully explain. A cron job that runs at 4:17 AM and nobody remembers writing. A registry key that was set by hand in 2014 and has been faithfully preserved through two migrations because removing it crashed the application the one time somebody tried. A shell script in /usr/local/bin with a name like "fix_the_thing.sh" whose content is eleven lines of sed, a curl command, and the comment "don't delete this, it keeps billing working."

I have come to think of these as IT easter eggs — artifacts left behind by previous administrators, sometimes by previous versions of yourself, each one a tiny monument to a problem that was solved the hard way at 2 AM and never documented because the person solving it was too tired to write it down. They are everywhere, they are load-bearing, and they are the reason experienced sysadmins command the salaries they do.

This is a piece about why those artifacts exist, why they matter more than we admit, and what happens when the institutional memory that produced them disappears.

The shape of institutional memory in IT

Institutional memory in IT is different from institutional memory in other disciplines. It is not written down in company handbooks or passed along in training sessions. It is stored in the heads of three or four specific people, distributed across the running configurations of systems those people have touched, and encoded in artifacts that usually have no comments explaining what they are or why they exist.

You can tell a lot about the health of an IT organization by looking at these artifacts. A shop where nothing unexplained exists is either extremely young or has been rebuilt from scratch recently. A shop where the artifacts are all labeled, dated, and annotated is run by someone unusually disciplined. The third category — a shop where the artifacts exist, they're undocumented, and the three people who know about them still work there — is the most common, and it is a stable-but-fragile equilibrium that can collapse suddenly when any one of those three people leaves.

Why the artifacts exist

There are a few specific reasons undocumented fixes accumulate, and understanding them helps with deciding what to do about them.

Pressure. Most institutional easter eggs were created during an incident. Something was on fire, the fix worked, and there was no time or energy left to write it up properly. The commit to "document this properly later" is almost always honored in the breach. Later never arrives, or when it does, the context has evaporated.

Vendor bugs worked around silently. A surprising fraction of production fixes are workarounds for bugs in commercial software. The script that resets a service at 4:17 AM exists because the vendor's memory leak has been an open ticket for six years. The registry key exists because the upgrade process doesn't migrate one specific setting correctly. These workarounds rarely get documented because documenting them feels like admitting to using broken software, and because the fix belongs to the vendor's bug rather than to the local environment.

Historical constraints that no longer apply. Sometimes the artifact existed for a reason that was valid ten years ago and is not valid now — a hardware limitation, an OS version, a security policy from a previous auditor. The fix is still running because nobody has tested whether it's still needed, and testing whether it's still needed requires potentially breaking production, which means it never happens.

Knowledge that couldn't find a home in the documentation system. Sometimes the issue is that the documentation system itself is bad — a wiki nobody reads, a SharePoint that nobody can find things in, a ticketing system where closed tickets are inaccessible after a year. The knowledge exists in someone's head because the available places to put it are worse than their head.

What happens when the artifacts disappear

Here is the part that matters most, and that every IT leader should take seriously.

When an experienced administrator leaves — retires, moves, gets poached — a specific kind of knowledge walks out the door with them. Not the formal knowledge they could put on a resume. The artifact-level knowledge. The "oh yeah, that script needs to run after the database upgrade or billing breaks" knowledge. The "don't touch that NIC, it's the one that uses the jumbo frame MTU the switches were configured for" knowledge.

The first few weeks after that person leaves, nothing noticeable happens. The systems keep running because they were designed to keep running. Then something small breaks and nobody knows why. Then something bigger breaks and the fix takes three days instead of three hours. Then an upgrade hits an unexpected interaction with an artifact that nobody knew existed, and the organization discovers it has quietly lost the ability to safely make changes to a system that is still mission-critical.

I have seen this play out in a dozen environments. The pattern is consistent. The cost is real. And it is almost always much more expensive than it would have been to spend a few thousand dollars a year on knowledge capture while the person was still around.

What to do about it

There is no silver bullet for institutional memory loss, but there are things that work. In order of how much they cost and how much they help:

Make the artifacts findable. The first step is not documenting what each one does. It is simply making a list of what exists. A directory of all cron jobs, scheduled tasks, startup scripts, and customized configuration files across your environment, with the host name and the filename. This is doable in an afternoon with some shell scripts and is more valuable than most of what passes for IT documentation.

Require a comment block on every custom script. Not a README. Just a comment at the top of the file: what it does, who wrote it, when, and what happens if you delete it. This should be enforced by code review or by a pre-commit hook. It is a small habit and it pays off for years.

Record the oral history. The best knowledge capture exercise I have ever seen was conducted by a manager who scheduled a weekly one-hour session with their senior admin for six months before the admin retired. The admin walked through every system, every quirk, every workaround, and every story. The sessions were recorded. The resulting archive was used for another decade. It cost nothing except the time, and it preserved something that would otherwise have been lost.

Run periodic archaeology days. Once or twice a year, dedicate a day to looking at what's running in your environment and asking "does this still need to be here?" Delete what is no longer needed. Document what still is. This is tedious and unglamorous and it is the cheapest form of technical debt reduction you can buy.

Reward the writing down, not just the fixing. Most organizations celebrate the person who fixed the outage. Very few celebrate the person who wrote down how they fixed it. Flip that incentive and watch the documentation quality improve. The engineers I have worked with who wrote excellent postmortems did so because they were recognized for doing it, not because they enjoyed writing.

The artifacts are not the problem

One more thing worth saying. The artifacts are not the problem. The artifacts are evidence of real work that real people did to keep real systems running under real pressure. Treating them as technical debt to be eradicated is the wrong frame. They are institutional memory in material form, and the goal is not to eliminate them — it is to make them legible so that the next generation of administrators can inherit them instead of stumbling into them.

When I walk into a new environment, I am always a little grateful for the easter eggs. They are a signal that someone before me cared enough to keep things working. My job is not to sweep them away. My job is to figure out what each one is doing, write it down, and decide whether it still belongs. That is how institutional memory gets preserved instead of lost.

And the quiet satisfaction of finding a comment from someone ten years ago explaining exactly why a particular config value is the way it is — that is one of the small pleasures of this work. Leave that gift for the next person. They will appreciate it more than you know.