Insights

Leveraging data to help customers make better engineering decisions.

CircleCI insights

Problem

Large teams invest a lot in CI. They need ways to optimize CI process and monitor team performance and spending.

My role

  • Design CircleCI insights feature
  • User research
  • Future design exploraions
  • Onboarding new designer

Target audience

  • Engineering leaders (Managers)
  • DevOps
  • Developers

Team

Initially we were team of three, senior product designer, senior developer and shared engineering manager. After team expanded we had:

  • Product manager
  • Engineering manager
  • Data analyst
  • UX researcher
  • several developers

Research

In CircleCI's first design iteration there was simplified version of Insights UI which was job centric and became obsolete when we introduced workflows.

People could already get information about insights by using CircleCI insights API. That was expensive because people needed to create custom reporting tools.

My goal was to create initial design prototype based on customer needs and what is currently available in CircleCI API, test that prototype with customers, collect feedback, learn and iterate.

CircleCI insights

Goals

Short term goal was to provide information about granular usage trends like pipeline duration, credit usage and success rate so that engineering managers can make better decisions.

Long term goal is to provide developer experience people with tools to optimize pipelines and tests for speed, success rate and cost effectiveness.

Ideation

User flow

CircleCI insights

Organization level insights

High-level metrics by default show totals across all projects in the entire organization.

For engineering leaders aggregates make sense. They help them to keep track of their teams performance and to estimate and justify CI budgets.

For developer experience people this is the starting point for optimization work. This view enables them to indicate if problems exist. If yes they can dig deeper to learn more.

CircleCI insights

Project level insights

Project level insights provide a way to see what's going on with users' workflows and branches.

To optimize workflow duration developer experience people are more interested in the P95 duration of a specific workflow than total duration across all workflows.

For the main branch it is important to get to a 100% success rate which is not the case for feature branches.

Sometimes people conditionally include or exclude additional jobs based on the branch where the job executes.

CircleCI insights

Workflow level insights

Overview

Overview view enables people to discover anomalies.

CircleCI insights

Jobs view

Jobs view enables people to find couses of anomailes.

CircleCI insights

Job level insights

Jobs are composed out of steps. Steps are commands (linux, windows, mac). They execute sequentially.

CircleCI insights
CircleCI insights

Step failures

Being able to see what failed and how many times is pure awesomeness.

Getting this information before this UI was possible but tedious. It required people to manually navigate to jobs and compare steps.

Validation research

Idea behind creating design prototype as fast as possible was to get on the same page with customers literally. Instead of just leading conversations about problems and needs I wanted to see if proposed solution is actually meeting the needs and solving problems.

Rapid iterative prototype testing enabled us to update design prototype based on customer feedback while research was in progress and test updates with new participants.

CircleCI insights

Research goals

  • Validate assumption that insights UI provides actionable information about project health
  • Test user flows
    • Getting information about project health
    • Optimizing pipelines (workflow and job level)

Takeaways

  • You need P95, median, min and max to fully contextualize a value. Averages have too many outliers.
  • Trends are extremely useful, but confusing. There was not enough context about what is compared with what.
  • People wanted to see credit usage by passed and failed workflows
  • Getting deeper into step and test level insights is awesomeness

Design opportunities

  • Tests insights (flakyness)
  • Step level insights
  • Custom date ranges
  • Failure buckets
  • People

Customer advisory board meeting

Once per year CircleCI organizes meeting with engineering leaders who are also customers to discuss their needs, problems and present future product features. I got opportunity to present insights design prototype, feedback that we got was, give it to us now, and when this is going to be available. This was great signal for CircleCI leadership to increase number of insights team members and we started shipping insights to customers.

CircleCI insights

Collecting customer feedback after launch

Once we launched first iteration of a product feature and customers started using in in their own projects, feedback started to be more valuable. We used hotjar and canny to collect feedback and got in touch with customers who reported problems.

Initially I reported hothar feedback about repeated user problems and product issues to the team daily, latter weekly. We used that feedback to prioritize what to do next.