My long-standing career in project management has helped me develop some good habits over the years. I can honestly say that the most valuable lesson I have learned in the past 12 years is to assume nothing and expect the unexpected.
Assume Nothing
Assumptions are only good for half the battle. This means that we cannot rely on assumptions alone, but they are necessary to guide our efforts more effectively. Thus, it is fine to make an assumption as long as it is only a guide. Assumptions, when used, should always be verified. Have you heard that age-old phrase, “Trust, but verify”? When using assumptions, that is exactly what you must do; otherwise, you can cause major issues.
A Costly Assumption
In the world of IT, this could not be more crucial. For example, I once assumed a server was backed up before starting a risky project that nearly ended my career. Good thing God was watching out for me.
It was a warm start to a long stretch of hot days during the dog days of the summer of 2014. I had been tasked with what seemed, at the surface, to be a simple RAID expansion of a VMware ESX server. Unfortunately, the company that set up this ESX server configured the RAID in such a way that it couldn’t be expanded easily. The RAID had to be reconfigured.
A Risky Operation
I knew we had a backup server that I could fall back on, so I brought an external hard drive with me to export the VMs, which would speed up the operation.
I exported all the VMs and proceeded to wipe the array and incorporate the new drives. Once I had ESX reloaded and was ready to import the VMs, I connected the drive and proceeded to import them. All VMs imported relatively quickly, as I expected. However, when it came to the final server—their Great Plains server, which hosted their main application—the drive started to make a clicking noise.
Panic Sets In
When I heard this, I decided to go to the backup server and do a bare-metal restore. To my dismay, I could not find the server on the backup server—it had never been set up. My heart sank. The sound of a failing hard drive had literally turned my anxiety into all-out panic.
A Desperate Recovery
At that moment, I had no choice but to call my boss, who, at the very least, was an understanding man. He told me to do what I could to try to restore the server by 7 AM the next morning. In essence, he said, “You’re pulling an all-nighter.”
While praying, I heard God speak to me. He reminded me of a tool I had once used called Unstoppable Copier. So, I downloaded the tool and started copying that last VM. The tool’s log was riddled with failures and retries. All night, it ran, and all I could do was continue to pray.
By 6:53 AM, it finished. I couldn’t believe it, but it finished.
An Unexpected Miracle
I hurried up and configured the VM container with the appropriate networking and attached the disk file. At this point, I figured I had a lot of work ahead of me—I would have to repair the OS of the server and deal with a lot of corruption. To my surprise, the server started without blue-screening, and I was able to log in right away.
I quickly set the static IP of the server to what it was before and refreshed DNS.
The client arrived promptly at 7 AM. I prayed the entire walk to the CFO’s office, where I would have him verify that everything was working—or not.
A Lesson in Grace
God had worked a miracle that night. The server was fully intact, with no corruption, and was copied from a hard drive that clicked and became utterly useless after the migration.
This just doesn’t happen, folks. It shows just how much grace God has for us. However, we must never take His grace for granted.
Live and learn, they say…