DevOps

Rocky Morgan: UNIX Wizard

I was a NT and Exchange administrator doing contracting work at Excel Data Corp in 1994. I was initially hired to be the internal system administrator and eventually worked my way up to contracting.

Excel had a little RedHat 1.0 server called Merlin that was setup with a Livingston Postmaster and 25 U.S. Robotics Sportster 14.4 modems. Each employee was given a remote login and there was telnet access as well (no SSH yet!).

I was in charge of setting up new accounts. This was done with a shell script I barely understood that wrote directly to the /etc/passwd and /etc/shadow file (I know I didn’t even use useradd or adduser).


To say I was a newbie isn’t enough: I got on irc and asked questions and just typed in commands people told me on #Linux on EFNET! I am so surprised the company wasn’t hacked.


One day Merlin wasn’t working quite right. Some folks couldn’t login. ls /home showed UID and GID numbers instead of names for some users.


I had no clue. The CTO at the time was a guy named Rocky Morgan. An old school UNIX guy that used to work for DEC.


Well he came over and logged into the console and ran one command. If my memory serves it was something like:


#@g%$fw#$e%@#|#$ew%#h%h@#|w$^@#hr%^h#$|^#$gdh#$^


Of course that wasn’t the command but it might as well have been.


The command spit out a couple of line lines…The users who were affected. The command showed the /etc/passwd for those users had incorrect fields. Some how the script I had used to create users broke it. I spent an hour looking at the thing and had no clue. Rocky came over and within 30 seconds showed the problem and how to fix it.


I was so impressed. At that moment I wanted to wield that kind of power. I swore that one day I would be able to write out a long command with multiple pipes and do something useful. I would do it from memory and one day I would be a UNIX master.


Fast forward to 2008. I was working at Pelago. Someone wanted to know if we were getting attacked. The web servers were slammed. I logged in and ran the following command:


awk '{ print plain" "$7 }' access.log | sort | uniq-c | sort-n

The command showed the source ip address and the URI that was getting hit. The output looked something like:


15 141.101.96.66 /careers/meet-the-team/
15 162.158.68.149 /_themes/stripe/fonts/ss-gizmo/ss-gizmo.eot?
15 162.158.76.191 /favicon.ico
15 173.245.63.186 /
17 141.101.96.83 /blog/feed/
18 108.162.245.146 /
19 108.162.238.224 /
19 173.245.63.228 /
20 162.158.76.5 /robots.txt

It was a nice distribution of ip addresses with legal URIs. We weren’t getting attacked (turns out a famous actress tweeted about our application and we were getting flash traffic).


Now this entire process took me about 30 seconds and it was run using across a fleet of web servers. I turned to the coworker who asked and explained everything looked fine. He had a surprised look on his face and he just said...


“How did you
do that?!”
I was the UNIX masterHappy It took 15 years but at that moment I thought of Rocky and that I had made my dream come true.

I posted a slightly shorter version of this story 10 months ago on
Reddit

My first UNIX Job (and second)

My first real UNIX job was at AT&T Wireless…well lets slow down a bit.

I was a contractor working at McCaw Cellular. I was helping convert their MS MAIL system that ran on DOS to this shiny new product called Exchange. My liaison was a guy name Kevin Regimbal. Next to Doug Hauger (coworker at Excel Data Corp where I worked at the time) probably the most influential person in my life. We had to interface with their SMTP gateway which ran on Solaris and used Sendmail.


I had very little experience with UNIX at this point. I played with Slackware Linux at home and helped manage a dial in server called Merlin at Excel Data Corp. Speaking of Excel Rocky Morgan was another really influential person in my life as well but I will tell that story later.


Okay so here I am doing a contract job and really focused on NT and Exchange. Kevin showed me Solaris and Sendmail and how it interfaced with the MS MAIL system and I was in love. Solaris was so cool! I mentioned to Kevin that I wish I could be a UNIX admin one day (see that was my dream job).


Kevin must have seen something in me because shortly after he was building out his team at the now AT&T Wireless and I applied and got the job.


Kevin taught me everything about being a good UNIX administrator. Kevin was kind and never yelled. He always explained his decisions and even if you disagreed with him you respected him.


Kevin taught me the importance of supporting our customers. He let me known we weren’t working on servers but helping other people do their jobs. If the mail server wasn't working people couldn’t do work and that was the real problem we were solving.


Kevin left AT&T Wireless and went to Amazon. He hired me there as well. I really don’t know where I would be in my career today if it wasn’t for him. Amazon was a lot different than AT&T Wireless. Instead of Solaris it was Linux. Instead of <50 servers for a team of five we had >1200 for a team of five.


Everything changes at scale and automation was needed. Instead of our customers being people in retail stores or business types now our customers were developers and everyone who used Amazon.com.


We had to work closely with developers. We had to understand the impact of their changes. We had to anticipate problems and proactively solve them before outages came. We had to make sure communications between operations and the different development teams was smooth. There couldn’t be a wall between us.


Work at Amazon was easily 100% more intense than at AT&T Wireless but it never seem to get to Kevin. He was always calm. Always supportive. Kevin fostered the relationships between our team and everyone else.


I remember the moment I realized the difference between Linux and Solaris. I was working on a server that threw a kernel panic. I was dutifully noting the stack trace and trying to figure out why it had crashed. Was it a bad motherboard? Maybe a disk was failing. Kevin walked over and said “Just reboot it - its Linux”. My jaw dropped. We had Solaris servers at AT&T Wireless that had been running for
years. If Solaris crashed Sun could tell us exactly why. I was shocked.

I remember thinking “This Linux fad won’t last…”

Puppet…

I am a system administrator for Pelago, makers of Whrrl. Currently I manage about 120 Linux systems (Running Ubuntu's Edgy Eft or Hardy Heron). I was looking at a better way to maintain configuration across the fleet. Currently I use a combination of ssh keys and shell scripts. So imagine my surprise when I read this on Puppet's Introduction (emphasis mine):

This, combined with the fact that organizations rather than sysadmins usually retain the copyright for these tools, means that most sysadmins start from scratch at every new company,
building ever-more-powerful systems based on ssh and a for loop or something similar.

Wow, they are talking to me personallyHappy I am going to go through the rest of the documentation and see if I can get it setup.