DevOps vs NoOps

DevOps Days Austin 2016 Logo

DevOps Days Austin 2016 Logo

Speaking at DevOps Days Austin a few months back on the topic of “DevOps vs NoOps,” led to this post. Ignite presentations are great conversation starters, but can’t convey much substance. It’s a great incendiary title, however I wanted to offer a bit more context on the two “NoOps” applications I’ve been a part of the last few years.

Example #1: DevOps for a Ruby on Rails application deployed to Heroku

The facts:
This application managed the user experience for tens of thousands of customers at WP Engine. It scaled horizontally with the help of HireFire. We managed the log files with Papertrail. It had a Postgres database as the backend data store. As well as a caching layer. Everything managed directly through the Heroku Portal & CLI.

The pros:
Heroku handled much of the security. HireFire was there for horizontal scalability. Easily turn on and scale up services at the push of a button.

The cons:
Obviously, cost. You get what you pay for. Additionally, having a UI for management instead of CLI/scripts for some things. It was difficult for on-call to do more than “restart the dyno’s” due to visibility.

Example #2: DevOps for PHP & Laravel deployed on EC2 directly

The facts:
These applications supported more than a thousand customers. Laravel, Forge & Envoyer gave the team the ability to easily manage deploys and various environments. It provided us with turn-key configuration management of our application servers.

The pros:
Configuration management for initial server setup comes out of the box. It is easy to set up commit/webhook based deploys.

The cons:
Unfortunately, it did not give a true security blanket like Heroku does. Patch management was a required add-on. Additionally, much of the Envoyer/Forge configuration is not tracked by default in the version control system, so configuration drift was common. No built-in or suggested solution for monitoring.

So, do I still need DevOps?

Obviously, both of these applications still required some amount of operations support, but for the most part it was manageable by the existing team of developers as opposed to needing a specialist in monitoring, security, patching or other core DevOps structures.

I truly love DevOps for what it has done to bring development into the operations camp, and operations experience into the development side of things. Anytime we’re talking about breaking down the walls between development, operations, quality, product, or any other collaborator, I’m all for it. I’ll still be hiring DevOps engineers, and looking to build resilient platforms that make every engineer that much more effective.

Overcoming Impostor Syndrome

Speaking at SXSW is always an honor and a privilege. This year, I got the opportunity to speak at a co-branded event between SXSW & General Assembly, on “Overcoming Impostor Syndrome.” Impostor Syndrome is often encountered purely internal feelings of inadequacy in high performers. There are a number of great resources out there on Impostor Syndrome, that I won’t even try to duplicate, but this post is an anchor providing my slides and resources for those researching Impostor Syndrome.

Impostor Syndrome Takeaways

  1. Impostor Syndrome explained visuallyYou’ve got to name and claim the Impostor Syndrome feelings you’re having. By knowing what you’re dealing with, you can begin to explore the way out.
  2. Document your successes and return to them when you feel vulnerable. This list of success will help you remember the value you provide.
  3. Bodies in motion tend to stay in motion. Create a virtuous cycle of successes, instead of a downward spiral of fear and feelings of inadequacy.
  4. Use the GROW model to get moving and documenting your progress. Mapping out your goal, realities, options and the way forward will help you figure out the next first step to getting unstuck.
  5. Reference your personal board of directors to gather great advice. Collecting some mentors who’ll listen to your fears and speak truth in love to you.

My Impostor Story

Because it’s not covered well in the slides, I’m taking a moment and being vulnerable that I’ve experienced Impostor Syndrome many times.

  • Driving home in 2007 with a newborn, feeling like I couldn’t possibly be qualified to remove him from the safety of the hospital.
  • Moving into engineering management in 2010 and suddenly being thrust into managing people who were my peers the day before. I was immediately terrified that I wasn’t the right person to lead them forward.
  • Moving my family to NYC in 2011 to start the Bazaarvoice NYC office from scratch with no connections. Feeling like I couldn’t be the right person to start this team.
  • Even speaking at SXSW on Impostor Syndrome itself I felt wholly unqualified.

Resources

 

http://startupbros.com/21-ways-overcome-impostor-syndrome/

http://nymag.com/scienceofus/2015/10/new-research-on-the-impostor-cycle.htmlhttps://www.mindtools.com/pages/article/newLDR_89.htm

http://elyseholladay.github.io/tigers/

Rules of SaaS to Live By

Learning baseball is certainly about the fundamentals but it’s also about knowing the rules. You can certainly have fun in the backyard playing catch and practicing your curveball. However, when it comes time to head down to the sandlot you need to understand how the game is played. The same is true for business sectors and technology.

The rules of baseball inforgraphic

Baseball rules infographic

You have to know, or learn, the rules for the space you’re in to avoid making big mistakes. If you’re platform as a service company, you need to know how’ll you be measured. If you’re an enterprise software company, understanding what your customers expect will be crucial in designing the right team and technology to grow. A lot of my career has been in the Software as a Service (SaaS) space, discovering the rules we have to live by.

Read More

Remote Team Remotely Working?

Jedi WFH - remote team

Remote teams work from home, why shouldn’t Jedi?

Having built a remote team of software engineers in New York City a few years ago, there were several lessons learned about how to build a team that sits apart from the rest of the team and how to make them successful. A lot of companies have trepidation about starting a remote team outside of your headquarters, as they should. There are some serious pitfalls, but there are some nice benefits as well, one of which being it gives you a second base to interview and hire candidates into. There are several great articles on the pros and cons, this post focuses on how to do it well.

 

Here are four tips to make a remote team successful:

 

  1. Give them a clear project to own, soup to nuts. The less they need the other office the better in this regard. Best would be to build on top of an existing, refined and documented API. This minimizes communication delays and back and forth between headquarters and the remote team. It also helps the team at HQ get used to a remote team. (Side note: Keeping them in timezones helps communication as well.)
  2. Inject leadership from HQ. Sending a cultural & technical leader with depth of knowledge of the stack, team, and processes will help to keep them anchored to the team back home. This ties the culture and helps to make sure you hire to the same bar and that remote team members will get along with team members back at home.
  3. Get your leaders early. Give a lot of leash to hire a great group of early engineers, and sprinkle in a handful of great consultants if you need to get the project flowing quickly. Having some key wins in the first few quarters is a key to the remote team feeling like they’re a part of a bigger picture. It also gives you enough people on the remote team that you can begin to formalize your team’s culture a bit more.
  4. Hire local leadership. Backfill the cultural leader and start building a second team. Just as a team of one is not a team, an office with only one team is not an office (or at least not much of one.) People need the ability to switch teams, move between projects, and spitball on projects with people not on their immediate team. This also gives you a succession plan, flexibility for your leaders as the remote team continues to grow, and growth opportunities for leaders at your remote office.

With any successful business, you’re going to have to eventually have to learn how to have product, design, operations, development or other functions distributed into other teams. It’s only a matter of time before you’ll have to have two locations for development, so before you get to the point of needing a second center, get it spun up and firing on all cylinders.

Anti-Patterns for Quality at Scale

As I’ve been a part of several fast growing Engineering teams I’ve seen a number of approaches to building outSquare peg, round hole a Quality Assurance organization. Sometimes this has gone well, and others it has been less than effective, which will be the focus of this post. I talked about these in an Ignite style presentation today at DevOps Days Austin, but thought these would be good to share here as well.

The first anti-pattern is hiring a “pile of testers.” While manual testers have their place, and can be a useful at times, only having testers will not dramatically help you produce better quality software. The thought process goes that this is outsourcing work that otherwise your developers would be doing, which is in itself a good thing. However, simply having testers produces typically produces two things: test plans & manual tests. The bad news here is that this means that each release you want to ship the manual regression suite will grow, as will the cost of testing/number of testers required will increase. Also, my experience was that the detail oriented nature of testers produces a lot of miniscule bugs, minor UI issues, and not the more systemic release issues.

To lower these costs you can fall into the second trap, of outsourcing it to another team and/or another country. While your team overseas can be cheaper on the hourly rate, all of our validation showed that the total cost of ownership on this work amounted to an equivalent Senior Engineer based in the United States. Additionally, if you choose to outsource, they often don’t have the same level of commitment to your project. If you are just expecting a bolt-on team to add quality to your product, you’ll likely find that it doesn’t work.

The final anti-pattern is similar, throwing the process, tools, and responsibility for creating quality over to another team, whether your quality team or your DevOps team. The truth is that you need your entire company, and certainly your entire engineering team committed to quality. Engineers building software and throwing it over the wall to another team is an opportunity for confusion, frustration and lower quality.

I’ll cover some best practices in building out quality at scale in a future post, but this airbnb post provides some thoughts to get started with. I’m not so simple as to believe that you can boil quality down to only one thing, and of course your mileage may vary with these tactics above, but if I had to focus on one and only one thing it would be that every engineer owns every line of code they write.

The Origin Story

Where did this phrase “only new mistakes” come from?

It originally came from a process our development teams used of allowing people to make mistakes, but ironman1encouraging them to always learn from the mistake and to not repeat it. Each review period, quarterly when I first encountered this mentality, we would look at the three things you could have done better. Those mistakes or opportunities were a chance to place a mental speedbump on that memory so that going forward you’ll always remember the mistake and the connected ways you can handle it better going forwards.

As I moved into management, I was looking for a short, sweet version of this process so that I could say it to the members of my team and help them learn this trick and at one point I began using the phrase that it was my goal to only make “new mistakes.” Using my learning from my previous roles, companies and other life events to avoid the known mistakes I’ve seen before, or at the very least to discover new ways to make some of the same mistakes.

Where does this principle apply?

I believe it works great in the business context, but that it’s also very useful in parenting and so many other areas of life. This blog is going to focus on the types of mistakes and learnings you can use to build a great engineering team, but that doesn’t mean the learnings will end there.

This blog will primarily focus on how to apply this principle to leading software development teams, but will draw on experiences in parenting, volunteer work, business, and so much more.

Scaling on People

Building a strong software as a service (SaaS) business you’re focusing on scaling your software, but simultaneously you’re inevitably needing more supporting personnel. You’re hiring account managers, support technicians, quality assurance, sales directors, and more folks to keep the business humming and growing. Each new client often means that more people will be needed to service that account. Of course as you grow you’ll need more of these people, and many other roles as well, but the question becomes does each new account linearly add some number of additional hires to your roadmap? Can you bend that curve and change the ratio of new accounts to new hires?

The great news is that if you’re team is building software then you’ve got everything you need to change that game. Using innovation and software to eliminate, be more efficient, or to fully automate some work will drastically change the calculations on how many new employees you’ll need. But first, you’ve got to get the baseline metrics on where your team’s time is going.

exponential growth is hardMetrics will rule the world

Knowing where the time is going will give you a true data driven approach to determine how much you can change the equation. If you find that all of your team’s time is in elements that have to be done manually or by a human, then you have to look for another area for improvement. If on the other hand you find large areas of opportunity to automate you’ve got your direction. Having at least one team dedicated to internal tools and process improvements will pay huge dividends over time in at least three areas.

  1. Efficiency. Shaving off 10% of the work from a 10 person team is equivalent to giving them an extra hire. It’s not easy, but it’s a huge win that will pay off over the long term. Using the metrics on which type of issues have dominated the most of your team’s time will give you the prime targets to attack.
  2. Automation. Creating software that makes some functions of the team’s work completely push button is a huge win. This is more than just a simple efficiency gain, it’s removing a whole slice of work they previously had to do.
  3. Elimination. Once you’ve automated a piece of work it becomes much easier to move some of that work to your customers. Shadow work is a phrase commonly used to describe the work pushed to the consumer of a service which previously had been provided as a service. This is clearly the end goal for a lot of efficiency gains, giving your customer the ability to “self-service” their needs.

These things scale. Hiring more people to do these things for your customer don’t. Bend the curve by making the investment in innovation. It’ll pay off.

Moon Shot

Lessons on Failing from Dr. Astro Teller

During the final SXSW keynote with Dr. Astro Teller provided a number of great thoughts on the value of “moon shots,” and also on failure. Key quotes from the session below belong to him. I’ve sprinkled a few thoughts of my own after each quote.

Dr. Teller currently oversees the Google[x] work, swinging for the fences in a number of areas. They’ve had some big wins and some big failures too. He clearly has some great insight on the value failure when these types of quotes just flow out of him during a keynote.

It’s not about having the bumps and scrapes, it’s about getting the full value from them.

This is the value of an only new mistakes philosophy. Take each mistake and learn from it. Don’t repeat it. You should have no need to repeat it, because you’ve already learned everything you need from it. He talked about sticking with some failures even longer than you might want to in order get all the learnings from it.

If you’re not failing at least some of the time, you could be learning faster.

The startup environment inevitably forces this out of you. You’re going to be making mistakes, so you may as well be learning from those mistakes. It’s better to run fast, in the real world, with the safety off and learn from the mistakes than to build slowly and not learn the mistakes until much later.

Failing doesn’t have to mean not succeeding.

Your failures are nothing but pitstops on the highway to eventual success.

Failures are cheap if you do them first. They’re only expensive when you do them at the end.

So true in software development it’s not even funny. All the more reason for good quality engineering, quality automation & quality processes.

Having people from radically different backgrounds will give you a radical mental diversity.

According to Dr. Teller mental diversity has proved tremendously useful in bringing all the ideas to the table. I’ve loved building as diverse a team as possible to bring all the expertise and ideas into the open. I love the fact that building a racially/seniority/gender diverse team provides better solutions.

Recruit ALL the Time

Budgets don’t always allow this, but having been part of a business that wasn’t able to continuously recruit, vs one that has been able to hire for 18 months plus I can say for a fact that keeping a warm pipeline and an open role at all times is dramatically better. Let me explain a bit about why:clock

  1. Constantly filling the pipeline. They’ll be sourcing and adding new candidates regularly, so that if at any point you need to fill a bunch more req’s you’ve already got a pipeline and you can ramp up the things you’re already doing.
  2. An open role for opportunistic hiring. If you find out that a great developer is available, you’ll have a role. You may have to pivot your budget a bit to change what role you’re hiring, but at least you have a shot at great talent.
  3. Reputation as a hiring organization. If you continually hire, you’ll become known by engineers and other hiring partners (e.g. the key networkers & recruiters) as having roles, and more people will show up at your door.

Obviously, there are downsides to all of this. You’ve got to plan for your hiring to be spread across the entire year. You’ve got to have a budget that is reliable enough that you can count on it for hiring quarters in advance. Of course your budget will change, but you need some confidence. I’ll cover in a future post how to think about budgeting that will allow you to have a high level guess at your hiring all year long.

 

An introduction is in order.

lit matchOnly New Mistakes was created as a place to share what has worked, and what hasn’t for the various software development teams I’ve been a member of and lead over the last decade or so. These are my opinions and not those of my various employers past or present (National Instruments, Bazaarvoice, or WP Engine.) They are the learnings from good times, and bad times. At times they are the best answer I have discovered, and at times they are only the least bad thing I know to do in a situation.

Topics for this blog will cover everything and everything involved in running a successful software development team. Topics like hiring and recruiting, software development lifecycles, testing methodologies, individual and team development, recognition, communication, and much more will be covered. This is meant to be a firestarter to discussions, rather than some guy spouting off on the Internet, so please join in on Twitter or another social media platform.