This was during my time at Weirton Steel, and I worked for a guy named George. He is truly one of the best managers for whom I’ve ever worked. We were having a hell of a time with our Novell file servers. This was the big iron of the time that ran a big chunk of the company. And this thing was going up and down like a yo yo. Nobody was quite sure if it was hardware or software. George brought in engineers from our vendor that specialized in Novell and Compaq hardware. We had a ton of Novell expertise on staff as well. This should have been fixed in short order, but the problems persisted a few days.

In times of crisis, George was the one remain incredibly calm. When people started raising their voices, he would speak up and quiet people down. I recall him buying lunch for his staff during this, which allowed his people to keep working. What really impressed me through this whole crisis was not how he handled his staff, but how he handled upper management. George strategically placed himself in the corner of the data center, not only to watch from a bit of a distance, but for a much greater purpose. When a member of upper management would come into the data center, he would intercept them and usher them right back out. I recall hearing him tell the IT director that the last thing his staff needed was someone from upper management asking for a status report. He played interference for his team, making sure they could keep working.

George taught me a lot. And to this day, I won’t walk into a data center with a beverage in my hands. That was forbidden in his data centers, and that’s how it should be.

Keep in mind that I restore these databases fairly regularly, so the test should have been a formality. But this time, I had to have Derrick, our network guy, get the backup file from the long-term storage that comes back from the mine instead of my usual restores from local storage. I sent Derrick the name of the file that I needed to recover. I didn’t even care of the date on the file to satisfy this requirement.

The restore finished really quickly—almost too quickly. When I restored the database, I went to pull data from a table that would prove we had successfully restored the database. I was horrified when SQL informed me that the table I wanted didn’t exist. I scratched my brain, trying to figure out what was going on. I restored the file again with the same results. I watched my life flash before my eyes. How the ehell was I going to tell my boss that I wasn’t able to restore our ERP system from a month ago? I went back and realized that Derrick had restored the wrong backup file. After restoring the right file, my database restore ran better and the table existed again. Lesson learned: Because this system has multiple databases, the backup files can be easily mixed up. Double-check the filename before panicking.

Next came a test for our corporate controller to sign off on. I would grab a transaction from the AP invoices table so he could look it up in the system to validate that the data was entered on that date. Keep in mind that the backup was taken on Sunday, May 3. That means I should have data from Friday, May 1. As I’m looking at the data, I see transactions from Thursday, April 30. Why could I not see data from Friday? Again, I start scratching my head, trying to figure out this out. I was completely baffled by this one. I walked up to Finance to talk to Cindy in Accounts Payable. Would you believe the entire Finance team took that Friday off to go on a canoe trip?

Problem solved. After a few setbacks, my DR test passed. Now I have to document everything and get the controller to sign off. That’s usually the hard part. Not this time.

Consistency is your friend.

When I was doing Lotus Notes consulting, I had a client with virtually zero downtime. They never had the connectivity and replication problems I had seen in other environments, including my own. Their secret? It was consistency in versions. All of their Notes/Domino servers ran the same version, and the clients ran the same version. Today, it doesn’t seem like that big of a deal, but back then, when you had to upgrade everybody via a CD, that was a revolutionary idea. I had always thought that you should install the latest version. My world changed that day.

Today, in my role as a SQL DBA, I try to keep as much consistency across my database servers as possible. When I started at my current employer, we had eight different versions and service pack levels of Microsoft SQL Server. Today, we have two, SQL 2000 SP4 and SQL 2005 SP2. When I upgrade my SQL 2005 instances to SP3, they’ll all go in a very short window.

All of my database servers are backed up using the same methodology and a consistent backup script. If I get hit by a bus, a replacement could come in and quickly figure out what’s going on.

Seek out a higher power.

I was an arrogant kid when I first started. I thought I knew everything. What I didn’t know, I thought I could figure out. Working with me had to have been hell. I’ve learned that not knowing something isn’t a sign of weekness. It’s an opportunity to learn.

People are incredibly willing to help you out when you acknowledge their expertise and approach them with a “help me understand…” attitude. In today’s world of technology, one can’t know everything. I try to be good at my stuff and get help from people who know more about something than I do. Besides, saying you don’t know something makes you look human.

Write early. Write often.

In the world of technology, your work is crap if people can’t understand it. I hate writing documentation, but I have to do it. I’d like to think that should I get hit by a bus, a reasonably skilled DBA could come in and figure out my environment rather quickly. Unless something is documented, it didn’t happen.

The more I write, the better I seem to get at it. This isn’t a bad thing at all.

Know your audience.

I hate reading e-mails I don’t understand. I work for a high tech company, and our engineers talk about networking and TCP/IP using language that goes right over my head. I appreciate reading things in language I understand.

When I send out a notice that we’re upgrading a server, people in this company appreciate knowing that we’re upgrading the firmware on all of our IBM x346 servers. That makes sense to them. At my last employer, a construction management company, all our user community cared about was that a system was going to be down for maintenance. If I had sent out a message stating we were upgrading firmware, my inbox would be flooded with messages asking what the hell I was talking about.

Tonight, I was tracing some traffic to see if I could figure out how traffic gets routed, and I was a bit surprised.

Look at this trace:

  2   46 ms   69 ms   137 ms c-3-0-ubr05.boston.ma.boston.comcast.net [73.168.144.1]
  3   43 ms   39 ms   171 ms ge-2-39-ur01.boston.ma.boston.comcast.net [68.87.158.33]
  4   122 ms   67 ms   167 ms te-8-1-ur01.brookline.ma.boston.comcast.net [68.87.144.57]
  5   193 ms   83 ms   214 ms te-8-3-ar01.needham.ma.boston.comcast.net [68.87.144.53]
  6   165 ms   135 ms   156 ms po-10-ar01.springfield.ma.boston.comcast.net [68.87.146.22]
  7   177 ms   140 ms   189 ms po-11-ar01.chartford.ct.hartford.comcast.net [68.87.146.26]

Every trace is consistent with this path. It starts in Boston before being bounced to Brookline, Needham, Springfield, and then finally Hartford. Logically, it makes sense. What I find odd is that this is geographic rather than the hub/spoke type of routing I’d expect from Comcast since they’re such a large provider. It looks like all of my traffic goes to Hartford before hitting New York where it then hits the outside world. From there, it usually goes to Level3 or Global Crossing. I’m just surprised that Comcast has an internal network that big and routes it on that network rather than bouncing it to the outside in a big city like Boston.

Ah, this is what happens when a geek gets bored and doesn’t want to clean his house.

The geek in me thinks this is pretty cool. I use PayPal when I get hotel rooms on Hotwire. Why not airline tickets?

(more)

Every time I go home, the sleepy little town where I grew up seems to get smaller. For example, a young woman who grew up the street from me has a “colored boyfriend.” I’m not sure which bothers me more, the fact that they still refer to him as “colored” or the fact that they care. I’ve met said boyfriend, and he seems nice.

Here are a few other highlights from the weekend:

My dad was completely surprised by the visit.
Reggie had a great time visiting my parents.
I sat on the front porch and drank beer with my dad two nights in a row.
My sister’s boyfriend’s brother is hot. Really hot.
My mom was horrified by the fact that I didn’t go to church with her on Sunday morning. I didn’t have anything appropriate to wear.
My niece is getting so big.  She’ll be eight in a few weeks.
My dad loved my new car. He mentioned that he loved my Sirius. Little did he know that I had one in the trunk as a gift for him.
I went swimming on Sunday with the family. I got sunburned and pulled muscles that I had forgotten I had.

It was a good weekend, and I’m glad I made the trip home. Granted, I spent a little more money than I had expected, but that’s okay. My parents have spent money on me more times than I can count, so it’s okay.

The route home took ten hours each way. It certainly took longer than my usual route, but I never stopped moving. There was minimal construction and it was a smooth drive. Never did I stop moving. The one thing that sucked is that this route was a toll road from Boston to Erie, PA. I90 across two states (MA and NY) probably isn’t cheap. It’s on my EZ-pass, so I don’t know what the cost was, but I do know I paid it both directions.

Tomorrow, I’m back to work.

This item hit a little close to home.  I’ve worked with people like this.

My X41 has been acting funny the past month or so. After replacing the hard drive two weeks ago, the drive started going bad again last week.

This one is nice and big compared to the X41. It’s got a bigger screen and a bigger keyboard. It’s also much heavier. That’s not necessarily good.

We’re sending the X41 back to Lenovo while it’s still under warranty. The question is whether or not I want the X41 back when it comes back.