Prepare well in advance
Alright,
If we want to claim we are ready – we need lot of preparation, mock runs! That’s what you should do before going oncall.
- Read your team’s runbooks and ensure you try them out, don’t simply read like a news paper. If runbook says – search logs – you should actually search logs. If runbooks says run this command – you should actually run that command on a test system. This is important, make sure you have access to a dev/test environment where you can try these without fear of destroying everything.
- Remember, once you are on-call ; You will be bombarded with issues and very little time to act. The more you prepare the more independently you can handle issues.
Shadow!
You don’t need to shadow 24/7 but if you know current on-call is good at their job try to shadow them in zoom meetings while they are actually troubleshooting, mitigating issues. I personally learned a lot in this way. Don’t ever miss a chance to participate in major incidents where a lot of folks will be pulled in and live troubleshooting will happen. That’s great learning opportunity. Make time and keep shadowing.
Be Curious
- Be curious – if you don’t know something ask around.
- Be curious – if you know a better way to do stuff say aloud.
- Be curious – Broaden your horizon and know what’s going on around. Don’t be in a state to be surprised to see a new feature built, plugged into team communication channels official and informal ones.
- Be curious – Don’t miss the most important two or three weekly team meetings. Operations reviews, Sprint Demos to name some. These are avenues to be plugged in. Be vocal and ask questions. If out of time, follow-up offline.
- Be curious – maintain proper TODO mechanism. Otherwise things will spill over and slip among cracks.
- Be curious – Be confident. If you don’t know something it is not your fault. Ask around.
- Be curious – what you don’t know is what you don’t know until you explore more.
When lost go back to happy place!
It is easy to get lost. When you are in a situation where your brain is prodding you with subtle clues “What the hell are you doing?” – It is time to go back to happy place. In other words, take a step back and write down (or type down) what is happy path for current situation. What is supposed to happen in an ideal world. What is happening now? Try to go from point A to point B and see what is supposed to happen and what is happening.
Breadcrumbs are powerful
I witnessed three kinds of devs and their troubleshooting style. (For the purpose of this document)
- The Masters, or magicians!
- These are exceptions.
- These are exceptional folks.
- They keep everything in their mind, they quickly press few buttons and viola! the issue is mitigated.
- (OK, we can be there, provided we get opportunity to fail/succeed and repeat cycle for few more years)
- The Unorganized Kind!
- These folks troubleshoot
- They have good basic concepts.
- They kind of seem to know what they are doing.
- But they get lost once every few minutes and restarts troubleshooting from scratch. Or gets frustrated. Or gives up. (depending upon which side of bed they woke up on the morning or which flavor of coffee is served in office coffee machine)
- The Organized Kind.
- These may or may not have worked on similar issue in the past,
- But they are meticulously organized.
- They keep notes and
- Theorize
- proves / disproves theory
- and moves one step at a time ahead.
- —
- These are the folks to emulate.
- They give you confidence you can also do stuff.
- They make it easier to follow along
- They demystifies everything.
- They maintain bread crumbs, so not getting lost on the way.
- At any time status is in front of you.
Expect the Unexpected
Things don’t go in the way it is easier to move forward. Expect the unexpected. Do not hesitate to get help. Send SOS to team! Remember you are not alone in this.
Be Calm and keep troubleshooting
Trust me, this is going to happen more sooner than you anticipate.
You will be on-call ; And an issue comes – which is derailing one of the important customer’s requirement who has ear to your leadership team all the way to CEO.
Now, you are sharing your screen to a group of folks, all important, eager to help. The folks will include your leads, managers, skip level managers and occasionally your CEO!
- Keep calm
- Remember you are in charge.
- Remember your training.
- Spread Breadcrumbs
- Listen to folks suggestions, make notes. But be assertive.
- Be open to feedback yet be vocal to express your opinions.
- Share your theories, and prove or disprove them one by one.
- Explain what you are doing.
- I know, it is easier said than done. But remember this too shall pass. There is always light at the end of tunnel, keep walking.