Self-hosted runners

CircleCI self-hosted runners are custom execution environments for jobs. They enable users to execute jobs on their own hardware.

Self-hosted runners

Fit self-hosted runners in the CircleCI

Before the introduction of self-hosted runners, CircleCI already supported the execution of jobs on Linux, Mac, Windows, and Docker images.

Orchestration layer

Workflows are considered orchestration layer. They allow people to set up when jobs execute and if they execute sequentially or in parallel.

Execution layer

Jobs are execution layers, they run commands sequentially. Depending on the type of the command (windows, linux, or mac) they need to have an appropriate executor assigned.

Self-hosted runners

Goals

  • Enable people to run their jobs on esoteric hardware (eg. custom GPU)
  • Increase security - Narrow down the gap between cloud and server products by allowing people to run security-sensitive jobs on their own hardware
  • Do not create self-hosted runners as a replacement for cloud runners
    • This feature is not intended for the majority of users, originally it was available only to organizations on a scale pricing plan
    • For some users, self-hosted runners are make it or break it feature

Team

  • Product manager
  • UX researcher
  • Product designer
  • Several developers

Design work scope

  • Enable users to create self-hosted runners
  • Enable users to manage self-hosted runners

Research

CircleCI did not have experience and customers who used self-hosted runners, but we knew that there were organizations who built their businesses on top of them.

UX research lead at the time did generative research and talked with non CircleCI customers who used self-hosted runners to get a better understanding of how they use them and learn more about their needs.

I tested and went through the process of setting up self-hosted runners using products that were considered the best in that field. My goals were:

  • To understand the process of setting up a self-hosted runner
  • To understand how this process fits into CircleCI as a product
  • To identify opportunities to simplify the process

I documented the entire process in a way that is digestible for non-technical stakeholders.

Ideation

User flow

This is high level view of how we intend users to discover, locate, set up and manage runners. Listed is information that need to be available and actions that can be performed. Each step in the flow can be broken down into smaller chunks.

Self-hosted runners

Adding self-hosted runner

We made a decision to use organization settings as a home for self-hosted runners. Landing page acts as empty state when there are no runners in the organization and as an inventory view when there are runners to show.

Self-hosted runner Self-hosted runner Self-hosted runner Self-hosted runner

Managing runners

Self-hosted runners management should provide visibility into runner fleet, insights into availability and options to debug issues with runners.

Self-hosted runner Self-hosted runner

Execution

Direction change

Product and design assumed that users would be able to install self-hosted runner software on their hardware as a package, by copying and pasting a single unit of code. Engineering changed that decision and decided to go with a fully transparent process where users need to install runner software manually. While the manual way provides full visibility about what is being installed on a runner machine, it takes much more time for users to install runners.

From my perspective, this meant that if we want to create UI for onboarding runners we would need to completely duplicate installation documentation into product UI. Having the same information in two places is not adding value to customers. It is creating multiple sources of truth where both of them require maintenance work.

In this context, my proposal was not to execute UI to onboard runners at all until we enable users to install runners as a package. We can redirect users to the docs to set up runners and enable them to manage runners in the UI.

We did a UX research project to test how much time would people need to install self-hosted runners on CircleCI and one of the competitor's products. Research results confirmed that taking a manual approach for installation resulted in increased onboarding time.

Notes

  • People can onboard runners via UI or CLI using the docs as a guide. Using UI is not a prerequisite to using the self-hosted runner feature.
  • I did not participate in executing designs as part of the delivery team. I onboarded and guided the technical design lead person.
  • This design exposed technical issues of CircleCI as a product during runner onboarding process. At two points in time we are losing control of what customers are doing because they need to go outside of our product (docs, ssh into a runner), perform specific actions, and come back, which is not ideal because we don't know if or when they get blocked.
  • The designs presented above are two years old at the time of making this case study. Visual language is outdated. There are hierarchy issues. There are too many lines and boxes.
  • Working on self-hosted runners was a growth opportunity on personal level. The feature is very technical, the challenge for me was to understand runners and make them approachable to people who are not technical.
Self-hosted runners