rksagi.
← The Observability Series
Observability Series · Part 3

I will look for you. I will find you. I will fix you.

I will look for you. I will find you. I will fix you. — Debugging Skills, AI and Observability dashboard mockup with distributed traces, service dependencies, and AI root cause analysis.

⚠️ What follows is a work of satire. Any resemblance to actual bugs, actual node_modules, or actual 2am production incidents is entirely intentional.

* No node_modules were permanently harmed in the making of this article. The XSD file has since been updated in the actual codebase.

Cast of Characters

Bryan MillsSenior Engineer. 20+ years of production scars. Speaks fluent SOAP, Node.js, and Azure at 2am. Does not panic. Does not Google randomly. Goes into node_modules.

The DaughterThe production system. Innocent. Taken without warning. Still technically running. That’s the problem.

The KidnapperThe Bug. Confident. Silent. Leaves no stack trace. Has never once said "I don’t know."

The CIA NetworkObservability. Grafana, Loki, Tempo, Trace Ids, Span Ids, Correlation Ids. Bryan's intelligence network. Useless without Bryan.

The Incompetent Local PoliceJunior developers. Well meaning. Hardworking. Drawing a firm boundary around "my end" since 2015.

The Ransom CallThe 2am incident alert. Nobody wanted this call. It came anyway.

Scene 1 — The Ransom Call

It is 2am. Bryan Mills is asleep. His phone rings.

The production system has been taken.

Not taken down — that would be easy. Taken silently. Everything is running. Tests passed. The errors in the logs were caught and propagated. The error was “Cannot read properties of undefined” in a complex SOAP orchestration.

The local police — the junior developers — have been working on it for two days. Here is their report:

“I don’t know where this error is. I did exactly what the user guide said.”

“I upgraded the soap library version but it’s still the same.”

“No code specific changes have been done from my end.”

Bryan Mills puts on his jacket. “I’ll handle it.”

“I don’t know who you are. I don’t know what you want. But I know you are hiding somewhere between my Node module and that third party XSD file. I have a very particular set of skills. Skills I acquired over a very long career — Java, SOAP, WSDL, Node.js, Azure, CosmosDB, Service Bus. Skills that make me a nightmare for bugs like you. I will look for you. I will find you. I will fix you.”

The bug says nothing. Bugs never do. They are confident like that.

Bryan is not fazed. He has been here before. Many times. Different technologies, different systems, same silence. He knows the bug is not hiding in the obvious places — that’s where the junior developers already looked. It is hiding in the space between systems. In the contract nobody read carefully. In the library nobody thought to question.

Okay. Now, let’s be real. Enough of movie stuff. Let’s take a hypothetical situation.

You need to come up with a new web service in SOAP to access a third party legacy system. You have done everything to the T that the user guide said. You are getting an error something like “Undefined.” You cross check all WSDL files, XSD files, everything. You visit the forums and could not get any relevant information. What do you do?

Without Observability tools, you write a million log statements and still you would not get to the bottom of it. You ask AI, paste the errors. But let’s be frank. AI is not a magic hand. AI gives enough information. AI will guide you. AI will give wrong information too — with the same confidence. What’s worse, AI will even change code that is not related to the error, if you are not careful. If that does not work, it might even try to consume the error and bypass it and say “Problem solved.” 😄

With Observability tools in place — with properly configured traces, logs and alerts — you might even pinpoint the place. And in addition to this, you will get all the benefits and ill-benefits that AI gives as mentioned above too. But sometimes, even that is not enough.

What now?

They say “You can’t buy experience in the market”

It is true more often than not. Why?

There is a skill — Debugging skills. As Bryan Mills says: “I do have a very particular set of skills. Skills I have acquired over a very long career.” In movies it sounds exciting. In real life, this actually sounds reasonable.

You are surrounded by lots of tools, smart people — smarter than you — but still they can’t figure it out. Reason? They did not use SOAP in Java in 2005. Neither did AI. These are the skills you accumulate over the years. They get into your muscle memory. They will be hidden in your subconscious mind somewhere. These are the results of a million searches on StackOverflow over the years. Google searches. Before that, your own countless hours of trying to debug everything and fix it yourself.

AI won’t search node_modules that are already installed unless you specifically ask for it. The skills you acquired will give you the instincts to get into corners others don’t see. This is possible only if you had the history.

AI is writing more and more of our code now. Vibe Coding is real — I said it in Part 2 and I will say it again. User stories become functions. Functions become features. Features go to Git and then to Server. And somewhere in that chain — just like in the movie — something gets taken. Silently. Without leaving a note. AI can’t inspect your runtime.

The point I am trying to make here is simple. Never ever discount debugging. Especially now when AI can fix anything for you and you move on without knowing what is being fixed. You need to pay attention to every error. Consume that memory. It will be stored somewhere in your muscle or in your subconscious. Over time, it will take you less time than the smarter but less experienced people around you.

AI will help. Observability will help. But only when you know where to look. Only when you have developed natural instincts. Only when you don’t bypass any learnings.

How to become Bryan Mills

The skills are learnable. Bad news. There are no shortcuts.

The closing credits

By the way — what would be the fix? A simple addition of an element in your XSD file.

Days of investigation. Hours to fix. Not because Bryan is smarter. Because he knew where to look — and he was willing to go there.

I have a very particular set of skills. Acquired over a very long career. I am not letting them go extinct quietly.

What is the hardest bug you ever debugged? Did experience or tools crack it? And are you seeing debugging skills disappear in your teams? Drop your war stories in the comments. 😄

* No node_modules were permanently harmed in the making of this article. The XSD file has since been updated in the actual codebase.

Found this useful?

Let’s connect on LinkedIn →
Join the discussion.
This article is also live on LinkedIn — comments and reactions happen there.
View on LinkedIn →