CI/CD Pipeline Optimisation That Developers Actually Thank You For

# CI/CD Pipeline Optimisation That Developers Actually Thank You For

The average developer spends 47 minutes a day waiting for CI. That's not an estimate — it's a conservative number from a 2024 DX survey, and it hasn't gotten better. What's worse is that most of that time is wasted on builds and tests that should have been optimised months ago but weren't, because pipeline improvements are invisible work.

When was the last time your team celebrated a CI speedup? Probably never. Yet the same team will spend hours debating lint rules or commit message formats.

I've built and rebuilt pipelines across a dozen codebases — from monorepos with 500+ packages to tiny polyglot microservices to sprawling enterprise monoliths. The patterns that actually move the needle on developer happiness aren't about choosing GitHub Actions over GitLab CI. They're about caching strategy, dependency graph awareness, test parallelisation wisdom, and the deployment model that doesn't make people dread Friday afternoon.

This post covers each one with real config and code.

The Optimisation Hierarchy

Most teams optimise in the wrong order. They jump into "which runner image is fastest?" before answering "are we even running the right jobs?"

Here's the priority order that actually matters:

Work top to bottom. If you skip #1 and #2, #3 and #4 are wasting money on stuff you shouldn't be running at all.

Step 1: Diff-Aware Pipelines Are Non-Negotiable

If your CI/CD pipeline runs the full test suite on a PR that only changes a README, you're burning money and developer goodwill. This sounds obvious, yet I regularly audit pipelines where a typo fix in docs/ triggers a 25-minute build, 15-minute test run, and E2E suite that opens a browser.

Mono-repo (Nx, Turborepo, Bazel)

If you're using a build system with native dependency graph awareness, the hard work is done:

# .github/workflows/ci.yml — Turborepo example
name: CI
on:
  pull_request:
    branches: [main]

env:
  TURBO_TOKEN: ${{ secrets.TURBO_TOKEN }}
  TURBO_TEAM: ${{ secrets.TURBO_TEAM }}

jobs:
  build-and-test:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0  # Required for Turborepo's filter

      - uses: actions/setup-node@v4
        with:
          node-version: 22

      - uses: pnpm/action-setup@v4

      - name: Cache Turborepo
        uses: actions/cache@v4
        with:
          path: .turbo
          key: turbo-${{ runner.os }}-${{ github.sha }}
          restore-keys: |
            turbo-${{ runner.os }}-

      - name: Run affected tests
        run: pnpm turbo run test --filter="[HEAD^1]"
        # ^ Only runs tests for packages changed in the last commit

The --filter="[HEAD^1]" trick is where the magic lives. Turborepo compares the current workspace state against the specified ref and only runs tasks for packages that changed (plus their dependents). On a monorepo with 200 packages, this reduces a 40-minute pipeline to 4 minutes for most PRs.

But what if your commit history is messy and HEAD^1 doesn't capture the full PR diff? The robust version:

- name: Find PR base branch
  id: pr-base
  run: |
    BASE=$(gh pr view ${{ github.event.pull_request.number }} --json baseRefName -q '.baseRefName')
    echo "base=$BASE" >> $GITHUB_OUTPUT

- name: Run affected tests against base
  run: pnpm turbo run test --filter="[...${{ steps.pr-base.outputs.base }}]"
``$

### Polyglot microservices

Without a monorepo tool, implement a simple diff-skipping logic:

# Only run this service's pipeline if its source changed

jobs:

changes:

runs-on: ubuntu-latest

outputs:

service-a: ${{ steps.filter.outputs.service-a }}

service-b: ${{ steps.filter.outputs.service-b }}

steps:

- uses: actions/checkout@v4

- uses: dorny/paths-filter@v3

id: filter

with:

filters: |

service-a:

- 'services/service-a/**'

- 'libs/shared/**'

service-b:

- 'services/service-b/**'

- 'libs/shared/**'

deploy-service-a:

needs: changes

if: ${{ needs.changes.outputs.service-a == 'true' }}

runs-on: ubuntu-latest

steps:

- run: echo "Deploying service-a..."

**My opinion:** If your CI pipeline runs every test on every commit and you have more than 5 packages or services, you're bleeding thousands of developer-hours per year. Install a diff-aware build system or add path filtering. This week.

---

## Step 2: Cache Strategically, Not Greedily

The most common caching mistake: caching `node_modules` (or equivalent) as a single blob. It works until it doesn't — one package changes, the cache key misses, and you're back to a cold install. The second most common mistake: caching everything forever, consuming gigabytes of cache storage and slowing restore times.

**The correct approach:** cache at the lockfile level, not at the file-tree level. And for compiled artifacts, use tiered caching.

### Dependency caching (the right way)

• name: Cache npm dependencies

with:

path: |

~/.npm

node_modules/.cache

key: npm-${{ runner.os }}-${{ hashFiles('pnpm-lock.yaml') }}

restore-keys: |

npm-${{ runner.os }}-

`$

• hashFiles('pnpm-lock.yaml') — Changes only when the actual dependency tree changes. A code change doesn't invalidate the cache.
• Cache ~/.npm (global npm cache) instead of node_modules. Restoration is faster because npm/pnpm can rehydrate from the global cache rather than downloading from the registry.
• Keep cache size under 1GB. GitHub Actions cache has a 10GB limit per repo, and oversized caches take longer to restore than they save.

# Rust — cache cargo registry + target directory
- uses: actions/cache@v4
  with:
    path: |
      ~/.cargo/registry
      ~/.cargo/git
      target
    key: cargo-${{ runner.os }}-${{ hashFiles('**/Cargo.lock') }}

# Go — build cache is built in, just mount it
- uses: actions/cache@v4
  with:
    path: |
      ~/.cache/go-build
      ~/go/pkg/mod
    key: go-${{ runner.os }}-${{ hashFiles('**/go.sum') }}

- name: Compute dependency hash
  id: dep-hash
  run: |
    HASH=$(node -e "
      const pkg = require('./package.json');
      const deps = JSON.stringify({...pkg.dependencies, ...pkg.devDependencies});
      const crypto = require('crypto');
      console.log(crypto.createHash('sha256').update(deps).digest('hex').slice(0, 12));
    ")
    echo "hash=$HASH" >> $GITHUB_OUTPUT

- uses: actions/cache@v4
  with:
    path: node_modules/.cache
    key: build-${{ runner.os }}-${{ steps.dep-hash.outputs.hash }}

# Step 1: Collect test timing data
npx jest --json --outputFile=jest-times.json
# This produces timing per test file

# Step 2: Use the timing data for subsequent runs
npx jest --shard=1/4 --testPathPattern='src/**/*.test.ts'

test:
  strategy:
    matrix:
      shard: [1, 2, 3, 4]
  steps:
    - name: Restore timing data
      uses: actions/cache@v4
      with:
        path: jest-times.json
        key: jest-times-${{ hashFiles('src/**/*.test.ts') }}

    - name: Run tests (shard ${{ matrix.shard }})
      run: npx jest --shard=${{ matrix.shard }}/4

# BUILD.bazel
py_test(
    name = "my_tests",
    srcs = glob(["test_*.py"]),
    shard_count = 10,  # Explicit shard count
    deps = [...],
)

Bazel will run each shard on a separate worker when you use --jobs=10. The sharding splits the test resources (the *_test.py files) by count, which isn't perfect, but Bazel's historic timing data (--experimental_remote_cache_compression + remote execution) improves this over time.

jobs:
  # Fast feedback — runs immediately, fails fast
  fast-feedback:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run lint
      - run: npm run typecheck
      - run: npm run test:unit -- --changedSince=HEAD^1

  # Slow feedback — runs in parallel after fast feedback passes
  build-and-integration:
    needs: [fast-feedback]
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npm run build
      - run: npm run test:integration

  # E2E — runs only after everything else passes
  e2e:
    needs: [build-and-integration]
    runs-on: ubuntu-latest
    steps:
      - run: npm run test:e2e

# .github/workflows/deploy.yml
name: Deploy

on:
  push:
    branches: [main]

concurrency:
  group: production
  cancel-in-progress: false  # Don't cancel — queue instead

jobs:
  build:
    runs-on: ubuntu-latest
    outputs:
      image-tag: ${{ steps.build.outputs.image-tag }}
    steps:
      - uses: actions/checkout@v4
      - name: Build Docker image
        id: build
        run: |
          TAG=${{ github.sha }}
          docker build -t myapp:$TAG .
          echo "image-tag=$TAG" >> $GITHUB_OUTPUT

  deploy-canary:
    needs: [build]
    runs-on: ubuntu-latest
    environment: production
    steps:
      - name: Deploy to canary (2% traffic)
        run: |
          kubectl set image deployment/myapp-canary             myapp=myapp:${{ needs.build.outputs.image-tag }}
          kubectl rollout status deployment/myapp-canary --timeout=5m

  observe:
    needs: [deploy-canary]
    runs-on: ubuntu-latest
    steps:
      - name: Observe for 10 minutes
        run: |
          echo "Watching error rates and p99 latency for 10 minutes..."
          sleep 600  # In reality, you'd check metrics
          # If all good → promote
          # If errors → abort

  promote:
    needs: [observe]
    runs-on: ubuntu-latest
    steps:
      - name: Promote to full rollout
        run: |
          kubectl set image deployment/myapp             myapp=${{ needs.build.outputs.image-tag }}
          kubectl rollout status deployment/myapp --timeout=10m
``$

**Key design decisions here:**

- **`concurrency.cancel-in-progress: false`** — This queues deployments rather than cancelling. If two commits land close together, both deploy. Cancelling means the first commit might be half-deployed when it gets killed.
- **Canary then promote** — The canary runs for a fixed observation window. This catches issues that unit tests never will (memory leaks under real traffic, integration failures, dependency version mismatches).
- **Environment protection** — The `environment: production` block ensures that deployment requires approval if you set it up that way, but for fast-moving teams, you can make it auto-approve.

---

## Step 6: The CI/CD Dashboard (The One That Matters)

Teams love building elaborate CI/CD dashboards with DORA metrics — deployment frequency, lead time, mean time to recovery, change failure rate. These are useful for management. They're not useful for developers.

**The one dashboard that actually improves developer experience:** a simple "how long until CI finishes" live tracker.

``

Or, better yet, integrate with Slack so developers get:

• 🔵 CI started (estimated 3 min)
• 🟢 CI passed (1 min feedback)
• 🔴 CI failed (lint at line 3 — "expected ';' at column 12")

The tool doesn't matter. The pattern does: surface the feedback where the developer is already looking, not in a separate dashboard they have to open.

What a Well-Optimised CI/CD Pipeline Feels Like

A developer pushes code. Within 30 seconds, they see:

1. Lint passed
2. Type checker passed
3. Unit tests for the changed packages passed

They can already merge if they want. But there's more running in the background:

1. Integration tests (3 minutes)
2. Build + containerise (2 minutes)
3. Deploy to staging (1 minute)
4. E2E tests against staging (5 minutes)
5. Deploy to production canary (1 minute)
6. Observability check (5 minutes)

The developer doesn't wait for steps 4-9. They open a PR, write the description, ask for review. By the time the reviewer looks, all 9 steps are green and the deploy is live.

This is the inversion most teams get wrong: they optimise CI/CD for machine throughput (how fast can we process all this work?) instead of human throughput (how fast can the developer get actionable feedback?).

Key Takeaways

1. Diff-awareness is the highest-ROI optimisation. Don't run what didn't change. Use Nx, Turborepo, Bazel, or path filters.
2. Cache at the lockfile level, not the filetree level. Include dependency graph hashes for compiled-language builds.
3. Parallelise by test time, not test count. Use historic timing data. Don't parallelise suites under 10 minutes — the cost isn't worth it.
4. Fail fast. Run lint and type checks first. Gate slower jobs behind faster ones.
5. Deploy every commit, but canary first. Queue deployments rather than cancelling. Never deploy straight to 100%.
6. Push feedback to where developers already are. Slack, terminal, or PR checks — not a separate dashboard.
7. Optimise for developer attention span, not machine utilization. The human waiting on the pipeline is the scarce resource. Everything else follows.

The best CI/CD pipeline is the one developers don't notice. When it's fast enough to be invisible, reliable enough to be trusted, and feedback-rich enough to not need a separate investigation — that's when you've actually optimised. Not when the dashboard shows green numbers.