Author Archives: RC Johnson

About RC Johnson

Engineering Director in Austin, TX. Formerly of National Instruments, Bazaarvoice, WP Engine, Lawnstarter, and currently at Indeed. Father of four. Love simple puzzle games, board games, coffee and movies.

Getting started.

A small child starting up a set of steps.
Atami, Japan by Jukan Tateisi (Unsplash)

I have 17 posts in draft form for this blog right now. This post would be number 18, however, the purpose of this post is to clear out the cobwebs. I could just post a simple, “whoops, I’m sorry,” post and say I’ll do better. That would not be starting again. That would just be apologizing for the long pause between posts. That post would not provide any positive feedback on starting again. It also would not be of any value to the people reading this.

I needed to start writing again. A post on the theme for this blog. “Getting started” is a perfect post for that!

The energy involved to start is always more than the energy involved to maintain momentum. This is one of those fun places where you can use physics to explain human behavior. In the physical world, we have to provide all of the energy to accelerate an object to a particular speed. Once we’ve attained speed we only need to provide enough energy to overcome the drag (friction, air resistance, etc.). I often default to inaction due to procrastination, feeling too busy, or a number of other excuses. Here are four reasons to get started.

Starting builds momentum.

There’s documentation in Cognitive Behavioral Therapy that dedicating five minutes to start will help with procrastination. Starting helps determine if this is a thing worth doing and gives a gentle nudge to push through to completion.

I don’t often stop at the five-minute mark. Telling myself that I can stop after five minutes gives me the freedom to start. Even when I feel I may not be able to dedicate the full time needed to complete.

Starting determines what to do.

By tackling the first phase of a task, the remaining steps are more clear and the remaining work is estimable. It will also help you size the work and determine the return on investment.

The first step I take on a major project is often to map out the phases I can see. Planning the work can often expose information I need, people I need to consult, meetings I need to schedule with busy people, or something else I can do quickly to continue to build the momentum on this project.

Starting is getting done.

Recently coming back from the Thanksgiving break I looked at my to-do list and I realized that a number of the items were simple emails, meetings, or other simple tasks. These items should never have sat on my to-do list as long as they had (embarrassingly long.) I used the “get things done” (GTD) method and completed a half-dozen items before lunch.

Starting starts getting better.

We use the phrase “get started and then get better” regularly on our engineering teams. It is useful to get something that provides value to a customer first and then we will take the time to refine the application to be of even more value to customers. If you never start, you never have anything to improve. There’s a lot of power in being able to look at something and decide that there’s a simple way to improve it.

That’s it.

Not a monumental post, but I’m back to posting again and that has value. Maybe not for you, but it has value for me. It’s good to be back.

The Innovation Toolbelt

I recently had the opportunity to speak at the Chicago CTO Summit on building a new product innovation team and I wanted to share the first few tools I’ve begun to formulate for the Innovator’s Tool Belt.

First, for those who prefer, here’s the 20 minute presentation on this topic.

 

What Toolbelt?

For those who prefer to read, first, let me explain what I mean by the Innovators Tool Belt. I’ve often used the tool belt analogy with engineers on my team. For example, when explaining some of the ways we build software I have referenced how carpenters will select their hammer and nails for some types of work. Or, alternatively, their screwdriver and screws for other types of fastening. We have talked about how there are established industry patterns for how walls, floors, stairs and more are built.

Similarly, software teams have similar concepts. Hash Maps, Linked Lists, Tries are the hammer, saw, and screwdriver of Software Engineering. Design Patterns and Microservices Architecture for instance, are established industry best practices for solving real world software problems.

But the truth is that most people aren’t carpenters, and I have no idea if wood shop is still class offered in schools. I know most people have not raised a barn, so maybe these analogies aren’t useful. So, I’ve decided it’s time to innovate on the tool belt analogy as well as the tool belt itself.

Thus the Innovator’s Tool Belt, the Swiss Army Knife of gadgets, lifting heavily from the Batman canon in the presentation above. What is the batarang of Innovation? The shark-repellent of rapid software development?

Innovation on the tool belt. Batman's utility belt - courtesy of Unspash and @serge_k
Batman’s utility belt – courtesy of Unsplash & @serge_k

The Innovation System Works If You Work The Innovation System.

A bit about the innovation program we built. While we launched about six months before “The Startup Way” by Eric Ries was released, our program has a lot of similarities. His book was a great validation that we were onto something. A few components of the program we built which helped shape innovation:

Incremental Investment – Our projects only received increased investment when they had proven they had “legs.” Constraint breeds innovation. Constraining on time and budget helps us stay lean and sets a clear deadline to show progress on business alignment or other key metrics.

Radically cross-functional teams – UX, Marketing, Product, Engineering, Program and more all acting as owners. Sitting together. Brainstorming. Everyone having a say in the product direction and when a feature could be simplified.

Innovation Economics – Being ruthlessly honest about what the costs of innovation have been and investing in innovation with a portfolio mindset. Thinking of the Three Horizons of Innovation: we can’t have 100% of projects in horizon one (incremental innovation of current products,) and horizon two (disruptive innovation of current products). That’s just playing it safe. Even though the big bets are out in horizon three (disruptive innovation of the company/industry) and we should be swinging for the fences, putting 100% of our efforts here would be overly risky.

Scaffolding – Having a tried and true framework for how we build applications has given us flexibility in staffing and a rapid innovation stack for every application. The quick start of a new Django application coupled with some custom built components makes learning how a new product would be adopted in less than three months possible.

Mistakes Were Made.

First, the bad news. We made mistakes building the new product innovation team we refer to as the Incubator and had to learn and move on But, then again, that is the theme of this blog. Here’s a few key mistakes and what we learned from them:

Hiring – We had to learn how to hire for Innovation. Flexibility. Passion. Ownership. Most importantly, a strong desire to deliver business value. These are some of the keys to an innovation team that works. Finding the leaders who possess these qualities is hard and takes time.

Globalization – We also expanded to too many sites and too many teams too quickly with too little leadership. In retrospect, we should have worked to constrain the locations and time-zones we took on in the first year (or two) in order to really pressure test the process before trying to replicate it.

Over-constraint – Finally, we used our new found process of innovation to restrict successful teams. We had great successes on our hands and ran them with teams of two engineers. Learning when and how to grow successful teams was a key differentiator in our success.

Tools That Work.

Now for the good news. There are some tools that we tried that worked and made the team even more effective at innovating.

Exits – Expect the project to fail. Work to learn as much as possible until it does. While it can lead to a nihilistic outlook at times, reminding ourselves that we’re searching for new business value often counters that attitude. Thinking with the end in mind helps keep perspective on technical debt.

Culture – An innovation team isn’t a counter-cultural team, it’s a sub-cultural team. Influence the rest of the organization. Inspire them. Take the company culture and evolve it. We emphasized parts of the broader team’s culture where it helped us, and redefined other parts to suit us better. This way our team was a partner with the broader organization, not a cancer.

Double-down – When something is working, don’t just keep funding at the same level. Invest more. More time. More money. If reasonable amounts of money can make a problem go away, do it.

The Innovation Tool Belt.

This is just the first set of tools we’ve discovered and the first attempt to publish them. There are more tools and it’s my goal this year to share even more of the tips and tricks of building a great innovation team.

DevOps vs NoOps

DevOps Days Austin 2016 Logo

DevOps Days Austin 2016 Logo

Speaking at DevOps Days Austin a few months back on the topic of “DevOps vs NoOps,” led to this post. Ignite presentations are great conversation starters, but can’t convey much substance. It’s a great incendiary title, however I wanted to offer a bit more context on the two “NoOps” applications I’ve been a part of the last few years.

Example #1: DevOps for a Ruby on Rails application deployed to Heroku

The facts:
This application managed the user experience for tens of thousands of customers at WP Engine. It scaled horizontally with the help of HireFire. We managed the log files with Papertrail. It had a Postgres database as the backend data store. As well as a caching layer. Everything managed directly through the Heroku Portal & CLI.

The pros:
Heroku handled much of the security. HireFire was there for horizontal scalability. Easily turn on and scale up services at the push of a button.

The cons:
Obviously, cost. You get what you pay for. Additionally, having a UI for management instead of CLI/scripts for some things. It was difficult for on-call to do more than “restart the dyno’s” due to visibility.

Example #2: DevOps for PHP & Laravel deployed on EC2 directly

The facts:
These applications supported more than a thousand customers. Laravel, Forge & Envoyer gave the team the ability to easily manage deploys and various environments. It provided us with turn-key configuration management of our application servers.

The pros:
Configuration management for initial server setup comes out of the box. It is easy to set up commit/webhook based deploys.

The cons:
Unfortunately, it did not give a true security blanket like Heroku does. Patch management was a required add-on. Additionally, much of the Envoyer/Forge configuration is not tracked by default in the version control system, so configuration drift was common. No built-in or suggested solution for monitoring.

So, do I still need DevOps?

Obviously, both of these applications still required some amount of operations support, but for the most part it was manageable by the existing team of developers as opposed to needing a specialist in monitoring, security, patching or other core DevOps structures.

I truly love DevOps for what it has done to bring development into the operations camp, and operations experience into the development side of things. Anytime we’re talking about breaking down the walls between development, operations, quality, product, or any other collaborator, I’m all for it. I’ll still be hiring DevOps engineers, and looking to build resilient platforms that make every engineer that much more effective.

Overcoming Impostor Syndrome

Speaking at SXSW is always an honor and a privilege. This year, I got the opportunity to speak at a co-branded event between SXSW & General Assembly, on “Overcoming Impostor Syndrome.” Impostor Syndrome is often encountered purely internal feelings of inadequacy in high performers. There are a number of great resources out there on Impostor Syndrome, that I won’t even try to duplicate, but this post is an anchor providing my slides and resources for those researching Impostor Syndrome.

Impostor Syndrome Takeaways

  1. Impostor Syndrome explained visuallyYou’ve got to name and claim the Impostor Syndrome feelings you’re having. By knowing what you’re dealing with, you can begin to explore the way out.
  2. Document your successes and return to them when you feel vulnerable. This list of success will help you remember the value you provide.
  3. Bodies in motion tend to stay in motion. Create a virtuous cycle of successes, instead of a downward spiral of fear and feelings of inadequacy.
  4. Use the GROW model to get moving and documenting your progress. Mapping out your goal, realities, options and the way forward will help you figure out the next first step to getting unstuck.
  5. Reference your personal board of directors to gather great advice. Collecting some mentors who’ll listen to your fears and speak truth in love to you.

My Impostor Story

Because it’s not covered well in the slides, I’m taking a moment and being vulnerable that I’ve experienced Impostor Syndrome many times.

  • Driving home in 2007 with a newborn, feeling like I couldn’t possibly be qualified to remove him from the safety of the hospital.
  • Moving into engineering management in 2010 and suddenly being thrust into managing people who were my peers the day before. I was immediately terrified that I wasn’t the right person to lead them forward.
  • Moving my family to NYC in 2011 to start the Bazaarvoice NYC office from scratch with no connections. Feeling like I couldn’t be the right person to start this team.
  • Even speaking at SXSW on Impostor Syndrome itself I felt wholly unqualified.

Resources

 

http://startupbros.com/21-ways-overcome-impostor-syndrome/

http://nymag.com/scienceofus/2015/10/new-research-on-the-impostor-cycle.htmlhttps://www.mindtools.com/pages/article/newLDR_89.htm

http://elyseholladay.github.io/tigers/

Rules of SaaS to Live By

Learning baseball is certainly about the fundamentals but it’s also about knowing the rules. You can certainly have fun in the backyard playing catch and practicing your curveball. However, when it comes time to head down to the sandlot you need to understand how the game is played. The same is true for business sectors and technology.

The rules of baseball inforgraphic

Baseball rules infographic

You have to know, or learn, the rules for the space you’re in to avoid making big mistakes. If you’re platform as a service company, you need to know how’ll you be measured. If you’re an enterprise software company, understanding what your customers expect will be crucial in designing the right team and technology to grow. A lot of my career has been in the Software as a Service (SaaS) space, discovering the rules we have to live by.

Read More

Remote Team Remotely Working?

Jedi WFH - remote team

Remote teams work from home, why shouldn’t Jedi?

Having built a remote team of software engineers in New York City a few years ago, there were several lessons learned about how to build a team that sits apart from the rest of the team and how to make them successful. A lot of companies have trepidation about starting a remote team outside of your headquarters, as they should. There are some serious pitfalls, but there are some nice benefits as well, one of which being it gives you a second base to interview and hire candidates into. There are several great articles on the pros and cons, this post focuses on how to do it well.

 

Here are four tips to make a remote team successful:

 

  1. Give them a clear project to own, soup to nuts. The less they need the other office the better in this regard. Best would be to build on top of an existing, refined and documented API. This minimizes communication delays and back and forth between headquarters and the remote team. It also helps the team at HQ get used to a remote team. (Side note: Keeping them in timezones helps communication as well.)
  2. Inject leadership from HQ. Sending a cultural & technical leader with depth of knowledge of the stack, team, and processes will help to keep them anchored to the team back home. This ties the culture and helps to make sure you hire to the same bar and that remote team members will get along with team members back at home.
  3. Get your leaders early. Give a lot of leash to hire a great group of early engineers, and sprinkle in a handful of great consultants if you need to get the project flowing quickly. Having some key wins in the first few quarters is a key to the remote team feeling like they’re a part of a bigger picture. It also gives you enough people on the remote team that you can begin to formalize your team’s culture a bit more.
  4. Hire local leadership. Backfill the cultural leader and start building a second team. Just as a team of one is not a team, an office with only one team is not an office (or at least not much of one.) People need the ability to switch teams, move between projects, and spitball on projects with people not on their immediate team. This also gives you a succession plan, flexibility for your leaders as the remote team continues to grow, and growth opportunities for leaders at your remote office.

With any successful business, you’re going to have to eventually have to learn how to have product, design, operations, development or other functions distributed into other teams. It’s only a matter of time before you’ll have to have two locations for development, so before you get to the point of needing a second center, get it spun up and firing on all cylinders.

Anti-Patterns for Quality at Scale

As I’ve been a part of several fast growing Engineering teams I’ve seen a number of approaches to building outSquare peg, round hole a Quality Assurance organization. Sometimes this has gone well, and others it has been less than effective, which will be the focus of this post. I talked about these in an Ignite style presentation today at DevOps Days Austin, but thought these would be good to share here as well.

The first anti-pattern is hiring a “pile of testers.” While manual testers have their place, and can be a useful at times, only having testers will not dramatically help you produce better quality software. The thought process goes that this is outsourcing work that otherwise your developers would be doing, which is in itself a good thing. However, simply having testers produces typically produces two things: test plans & manual tests. The bad news here is that this means that each release you want to ship the manual regression suite will grow, as will the cost of testing/number of testers required will increase. Also, my experience was that the detail oriented nature of testers produces a lot of miniscule bugs, minor UI issues, and not the more systemic release issues.

To lower these costs you can fall into the second trap, of outsourcing it to another team and/or another country. While your team overseas can be cheaper on the hourly rate, all of our validation showed that the total cost of ownership on this work amounted to an equivalent Senior Engineer based in the United States. Additionally, if you choose to outsource, they often don’t have the same level of commitment to your project. If you are just expecting a bolt-on team to add quality to your product, you’ll likely find that it doesn’t work.

The final anti-pattern is similar, throwing the process, tools, and responsibility for creating quality over to another team, whether your quality team or your DevOps team. The truth is that you need your entire company, and certainly your entire engineering team committed to quality. Engineers building software and throwing it over the wall to another team is an opportunity for confusion, frustration and lower quality.

I’ll cover some best practices in building out quality at scale in a future post, but this airbnb post provides some thoughts to get started with. I’m not so simple as to believe that you can boil quality down to only one thing, and of course your mileage may vary with these tactics above, but if I had to focus on one and only one thing it would be that every engineer owns every line of code they write.

The Origin Story

Where did this phrase “only new mistakes” come from?

It originally came from a process our development teams used of allowing people to make mistakes, but ironman1encouraging them to always learn from the mistake and to not repeat it. Each review period, quarterly when I first encountered this mentality, we would look at the three things you could have done better. Those mistakes or opportunities were a chance to place a mental speedbump on that memory so that going forward you’ll always remember the mistake and the connected ways you can handle it better going forwards.

As I moved into management, I was looking for a short, sweet version of this process so that I could say it to the members of my team and help them learn this trick and at one point I began using the phrase that it was my goal to only make “new mistakes.” Using my learning from my previous roles, companies and other life events to avoid the known mistakes I’ve seen before, or at the very least to discover new ways to make some of the same mistakes.

Where does this principle apply?

I believe it works great in the business context, but that it’s also very useful in parenting and so many other areas of life. This blog is going to focus on the types of mistakes and learnings you can use to build a great engineering team, but that doesn’t mean the learnings will end there.

This blog will primarily focus on how to apply this principle to leading software development teams, but will draw on experiences in parenting, volunteer work, business, and so much more.

Scaling on People

Building a strong software as a service (SaaS) business you’re focusing on scaling your software, but simultaneously you’re inevitably needing more supporting personnel. You’re hiring account managers, support technicians, quality assurance, sales directors, and more folks to keep the business humming and growing. Each new client often means that more people will be needed to service that account. Of course as you grow you’ll need more of these people, and many other roles as well, but the question becomes does each new account linearly add some number of additional hires to your roadmap? Can you bend that curve and change the ratio of new accounts to new hires?

The great news is that if you’re team is building software then you’ve got everything you need to change that game. Using innovation and software to eliminate, be more efficient, or to fully automate some work will drastically change the calculations on how many new employees you’ll need. But first, you’ve got to get the baseline metrics on where your team’s time is going.

exponential growth is hardMetrics will rule the world

Knowing where the time is going will give you a true data driven approach to determine how much you can change the equation. If you find that all of your team’s time is in elements that have to be done manually or by a human, then you have to look for another area for improvement. If on the other hand you find large areas of opportunity to automate you’ve got your direction. Having at least one team dedicated to internal tools and process improvements will pay huge dividends over time in at least three areas.

  1. Efficiency. Shaving off 10% of the work from a 10 person team is equivalent to giving them an extra hire. It’s not easy, but it’s a huge win that will pay off over the long term. Using the metrics on which type of issues have dominated the most of your team’s time will give you the prime targets to attack.
  2. Automation. Creating software that makes some functions of the team’s work completely push button is a huge win. This is more than just a simple efficiency gain, it’s removing a whole slice of work they previously had to do.
  3. Elimination. Once you’ve automated a piece of work it becomes much easier to move some of that work to your customers. Shadow work is a phrase commonly used to describe the work pushed to the consumer of a service which previously had been provided as a service. This is clearly the end goal for a lot of efficiency gains, giving your customer the ability to “self-service” their needs.

These things scale. Hiring more people to do these things for your customer don’t. Bend the curve by making the investment in innovation. It’ll pay off.

Moon Shot

Lessons on Failing from Dr. Astro Teller

During the final SXSW keynote with Dr. Astro Teller provided a number of great thoughts on the value of “moon shots,” and also on failure. Key quotes from the session below belong to him. I’ve sprinkled a few thoughts of my own after each quote.

Dr. Teller currently oversees the Google[x] work, swinging for the fences in a number of areas. They’ve had some big wins and some big failures too. He clearly has some great insight on the value failure when these types of quotes just flow out of him during a keynote.

It’s not about having the bumps and scrapes, it’s about getting the full value from them.

This is the value of an only new mistakes philosophy. Take each mistake and learn from it. Don’t repeat it. You should have no need to repeat it, because you’ve already learned everything you need from it. He talked about sticking with some failures even longer than you might want to in order get all the learnings from it.

If you’re not failing at least some of the time, you could be learning faster.

The startup environment inevitably forces this out of you. You’re going to be making mistakes, so you may as well be learning from those mistakes. It’s better to run fast, in the real world, with the safety off and learn from the mistakes than to build slowly and not learn the mistakes until much later.

Failing doesn’t have to mean not succeeding.

Your failures are nothing but pitstops on the highway to eventual success.

Failures are cheap if you do them first. They’re only expensive when you do them at the end.

So true in software development it’s not even funny. All the more reason for good quality engineering, quality automation & quality processes.

Having people from radically different backgrounds will give you a radical mental diversity.

According to Dr. Teller mental diversity has proved tremendously useful in bringing all the ideas to the table. I’ve loved building as diverse a team as possible to bring all the expertise and ideas into the open. I love the fact that building a racially/seniority/gender diverse team provides better solutions.