Goodbye Uplight...Hello Tideworks!

Posted on Sun 03 November 2019 in DevOps

I have accepted a new position at Tideworks as a Senior Developer Operations Engineer.


Continue reading

The best defnition of devops I have found

Posted on Tue 10 September 2019 in DevOps

DevOps is two things:

  1. Applying the methods of modern software development (version control, automation, DSLs...) to operations (provisioning, config, deployment, monitoring, backups...).

  2. Reducing silo barriers between devs and ops groups so that everyone is working together as a team, rather than blaming each other for poor communication and the resulting messes.

Then there are all the DevOps hijacking attempts, such as equating it to Agile or Scrum or XP, or insisting that it's a way to stop paying for expensive operations experts by making devs do it, or a way to stop paying for expensive devs by making ops do it, or a way to stop paying for expensive hardware by paying Amazon/Google/$CLOUD to do it.

No matter what your software-as-a-service company actually does, it will need to execute certain things:

  • have computers to run software

  • have computers to develop software

  • have computers to run infrastructure support

You can outsource various aspects of these things to different degrees. Anywhere you need computers, you have a choice of buying computers (and figuring out where to put them and how to run them and maintain them), or leasing computers (just a financing distinction), or renting existing computers (dedicated machines at a datacenter) or renting time on someone else's infrastructure. If you rent time, you can do so via virtual machines (which pretend to be whole servers) or containers (which pretend to be application deployments) or "serverless", which is actually a small auto-scaled container.

Docker is a management scheme for containers. VMWare provides management schemes for virtual machines. Kubernetes is an extensive management scheme for virtual machines or containers.

A continuous integration tool is, essentially, a program that notes that you have committed changes to your version control system and tries to build the resulting program. A continuous deployment system takes the CI's program and tries to put it into production (or, if you're sensible, into a QA deployment first).

Source: https://news.ycombinator.com/item?id=20894733


Continue reading

Rocky Morgan: UNIX Wizard

Posted on Tue 09 February 2016 in DevOps

I was a NT and Exchange administrator doing contracting work at Excel Data Corp in 1994. I was initially hired to be the internal system administrator and eventually worked my way up to contracting.

Excel had a little RedHat 1.0 server called Merlin that was setup with a Livingston Postmaster and 25 U.S. Robotics Sportster 14.4 modems. Each employee was given a remote login and there was telnet access as well (no SSH yet!).

I was in charge of setting up new accounts. This was done with a shell script I barely understood that wrote directly to the /etc/passwd and /etc/shadow file (I know I didn’t even use useradd or adduser).

To say I was a newbie isn’t enough: I got on irc and asked questions and just typed in commands people told me on #Linux on EFNET! I am so surprised the company wasn’t hacked.

One day Merlin wasn’t working quite right. Some folks couldn’t login. ls /home showed UID and GID numbers instead of names for some users.

I had no clue. The CTO at the time was a guy named Rocky Morgan. An old school UNIX guy that used to work for DEC.

Well he came over and logged into the console and ran one command. If my memory serves it was something like:

#@g%$fw#$e%@#|#$ew%#h%h@#|w$^@#hr%^h#$|^#$gdh#$^

Of course that wasn’t the command but it might as well have been.

The command spit out a couple of line lines…The users who were affected. The command showed the /etc/passwd for those users had incorrect fields. Some how the script I had used to create users broke it. I spent an hour looking at the thing and had no clue. Rocky came over and within 30 seconds showed the problem and how to fix it.

I was so impressed. At that moment I wanted to wield that kind of power. I swore that one day I would be able to write out a long command with multiple pipes and do something useful. I would do it from memory and one day I would be a UNIX master.

Fast forward to 2008. I was working at Pelago. Someone wanted to know if we were getting attacked. The web servers were slammed. I logged in and ran the following command:

awk '{ print $7 }' access.log | sort | uniq -c | sort -n

The command showed the source ip address and the URI that was getting hit. The output looked something like:

15 141.101.96.66 /careers/meet-the-team/
15 162.158.68.149 /_themes/stripe/fonts/ss-gizmo/ss-gizmo.eot?
15 162.158.76.191 /favicon.ico
15 173.245.63.186 /
17 141.101.96.83 /blog/feed/
18 108.162.245.146 /
19 108.162.238.224 /
19 173.245.63.228 /
20 162.158.76.5 /robots.txt

It was a nice distribution of ip addresses with legal URIs. We weren’t getting attacked (turns out a famous actress tweeted about our application and we were getting flash traffic).

Now this entire process took me about 30 seconds and it was run using across a fleet of web servers. I turned to the coworker who asked and explained everything looked fine. He had a surprised look on his face and he just said...“How did you do that?!”

I was the UNIX master It took 15 years but at that moment I thought of Rocky and that I had made my dream come true.I posted a slightly shorter version of this story 10 months ago on Reddit


Continue reading

My first UNIX Job (and second)

Posted on Mon 08 February 2016 in DevOps

My first real UNIX job was at AT&T Wireless…well lets slow down a bit.I was a contractor working at McCaw Cellular. I was helping convert their MS MAIL system that ran on DOS to this shiny new product called Exchange. My liaison was a guy name Kevin Regimbal. Next to Doug Hauger (coworker at Excel Data Corp where I worked at the time) probably the most influential person in my life. We had to interface with their SMTP gateway which ran on Solaris and used Sendmail.

I had very little experience with UNIX at this point. I played with Slackware Linux at home and helped manage a dial in server called Merlin at Excel Data Corp. Speaking of Excel Rocky Morgan was another really influential person in my life as well but I will tell that story later.

Okay so here I am doing a contract job and really focused on NT and Exchange. Kevin showed me Solaris and Sendmail and how it interfaced with the MS MAIL system and I was in love. Solaris was so cool! I mentioned to Kevin that I wish I could be a UNIX admin one day (see that was my dream job).

Kevin must have seen something in me because shortly after he was building out his team at the now AT&T Wireless and I applied and got the job.

Kevin taught me everything about being a good UNIX administrator. Kevin was kind and never yelled. He always explained his decisions and even if you disagreed with him you respected him.

Kevin taught me the importance of supporting our customers. He let me known we weren’t working on servers but helping other people do their jobs. If the mail server wasn't working people couldn’t do work and that was the real problem we were solving.

Kevin left AT&T Wireless and went to Amazon. He hired me there as well. I really don’t know where I would be in my career today if it wasn’t for him. Amazon was a lot different than AT&T Wireless. Instead of Solaris it was Linux. Instead of <50 servers for a team of five we had >1200 for a team of five.

Everything changes at scale and automation was needed. Instead of our customers being people in retail stores or business types now our customers were developers and everyone who used Amazon.com.

We had to work closely with developers. We had to understand the impact of their changes. We had to anticipate problems and proactively solve them before outages came. We had to make sure communications between operations and the different development teams was smooth. There couldn’t be a wall between us.

Work at Amazon was easily 100% more intense than at AT&T Wireless but it never seem to get to Kevin. He was always calm. Always supportive. Kevin fostered the relationships between our team and everyone else.

I remember the moment I realized the difference between Linux and Solaris. I was working on a server that threw a kernel panic. I was dutifully noting the stack trace and trying to figure out why it had crashed. Was it a bad motherboard? Maybe a disk was failing. Kevin walked over and said “Just reboot it - its Linux”. My jaw dropped. We had Solaris servers at AT&T Wireless that had been running for years. If Solaris crashed Sun could tell us exactly why. I was shocked.

I remember thinking “This Linux fad won’t last…”


Continue reading

Puppet…

Posted on Tue 24 March 2009 in DevOps

I am a system administrator for Pelago, makers of Whrrl. Currently I manage about 120 Linux systems (Running Ubuntu's Edgy Eft or Hardy Heron). I was looking at a better way to maintain configuration across the fleet. Currently I use a combination of ssh keys and shell scripts. So imagine my surprise when I read this on Puppet's Introduction (emphasis mine):

This, combined with the fact that organizations rather than sysadmins usually retain the copyright for these tools, means that most sysadmins start from scratch at every new company, building ever-more-powerful systems based on ssh and a for loop or something similar.

Wow, they are talking to me personally I am going to go through the rest of the documentation and see if I can get it setup.


Continue reading