Buildkite test selection

Graphify Cloud × Buildkite

Run the tests affected by a pull request. Graphify uses your repository graph to map changed code to tests, then returns a dynamic pipeline that Buildkite executes.

This integration is available in enabled Graphify Cloud workspaces. Open Integrations → BuildKite (Test Impact) in the console. If the tile says Coming soon, contact your workspace owner about availability.

Before you start

Connect and index a GitHub repository in Graphify Cloud. The commit-range endpoint used here requires a GitHub connection.
Have a workspace owner or admin create the CI token.
Configure a Buildkite pipeline with repository checkout, Git history sufficient to find the PR merge base, and an agent with Bash, Git, curl, jq, and buildkite-agent.
Ensure uploaded test steps run on agents with your test dependencies installed. The example uses pytest; configure the default queue and environment for your project.

1. Create the CI token

In Graphify Cloud, open Integrations → BuildKite (Test Impact).
Under Generate a token, enter a name such as buildkite-main and select Generate token.
Copy the token immediately. It is shown only once.
Store it as a Buildkite secret named GRAPHIFY_CI_TOKEN, with access granted to this pipeline. The script below retrieves it at job runtime. If you use another secret manager, inject the same environment variable through your agent’s secret hook instead.
Set GRAPHIFY_REPO_ID to the repository’s Graphify ID in the same workspace. This is not the GitHub repository name or numeric GitHub ID.

There is one active token for this integration per workspace. Replace token revokes the previous token; update every pipeline using it. Members can read the setup instructions but need an owner or admin to manage tokens.

2. Add the pipeline step

Add this step to .buildkite/pipeline.yml, replacing the placeholder repository ID:

.buildkite/pipeline.yml

steps:
  - label: ":buildkite: Graphify test selection"
    key: "graphify-test-selection"
    env:
      GRAPHIFY_REPO_ID: "YOUR_GRAPHIFY_REPOSITORY_ID"
    command: "bash .buildkite/graphify-select-tests.sh"

Then commit the following script as .buildkite/graphify-select-tests.sh:

.buildkite/graphify-select-tests.sh

#!/usr/bin/env bash
set +x
set -euo pipefail

full_suite() {
  printf '%s\n' '{"steps":[{"label":":test_tube: full test suite","command":"pytest"}]}' \
    | buildkite-agent pipeline upload
}

# Branch builds keep running the full suite.
if [[ "${BUILDKITE_PULL_REQUEST:-false}" == "false" ]]; then
  full_suite
  exit 0
fi

: "${GRAPHIFY_REPO_ID:?Set your Graphify repository ID}"
: "${BUILDKITE_PULL_REQUEST_BASE_BRANCH:?Missing pull request base branch}"
GRAPHIFY_CI_TOKEN="${GRAPHIFY_CI_TOKEN:-$(buildkite-agent secret get GRAPHIFY_CI_TOKEN)}"
: "${GRAPHIFY_CI_TOKEN:?Provide a Graphify CI token}"

head_sha="$(git rev-parse "${BUILDKITE_COMMIT:-HEAD}^{commit}")"
git fetch --no-tags origin "$BUILDKITE_PULL_REQUEST_BASE_BRANCH"
base_sha="$(git merge-base "$head_sha" FETCH_HEAD)"

payload="$(jq -n --arg base "$base_sha" --arg head "$head_sha" \
  '{baseSha: $base, headSha: $head, format: "buildkite", command: "pytest"}')"
response="$(mktemp)"
trap 'rm -f "$response"' EXIT

if ! status="$(curl --silent --show-error --connect-timeout 10 --max-time 180 \
  --output "$response" --write-out '%{http_code}' \
  --request POST \
  --header "Authorization: Bearer $GRAPHIFY_CI_TOKEN" \
  --header "Content-Type: application/json" \
  --data "$payload" \
  "https://api.graphify.com/v1/repos/$GRAPHIFY_REPO_ID/test-impact")"; then
  echo "Graphify could not be reached; running the full suite."
  full_suite
  exit 0
fi

case "$status" in
  200)
    jq -e '.steps | type == "array" and length > 0' "$response" > /dev/null
    buildkite-agent pipeline upload --no-interpolation "$response"
    ;;
  429|5??)
    echo "Graphify returned HTTP $status; running the full suite."
    full_suite
    ;;
  *)
    echo "Graphify returned HTTP $status. Check the CI token, repository ID, and commit range." >&2
    exit 1
    ;;
esac

Keep the token reference in the script. Buildkite expands variables in uploaded YAML before jobs run; a checked-in script reads them at runtime without embedding the token in pipeline configuration. See Buildkite’s secret handling guidance. The script resolves the PR’s base commit with git merge-base and uses BUILDKITE_COMMIT for its head. Buildkite provides BUILDKITE_PULL_REQUEST_BASE_BRANCH; BUILDKITE_PULL_REQUEST_BASE_SHA is not a built-in variable. See Buildkite environment variables.

Adapt it to your test runner

Change pytest in both the request’s command and full_suite() if your project uses another runner. The command must run the full suite with no appended paths and support selected test-file paths appended as arguments. Confirm that behavior before replacing your existing test step. The generated steps need the same dependencies, queue, and environment as your normal tests. Job-specific plugins or container configuration from your old test step are not added by this example. Preserve those requirements in your agent setup or adapt the generated steps before upload. When trialing the integration, keep your existing full-suite job and compare its results with the selected run. If later jobs must wait for dynamically uploaded tests, configure Buildkite dependencies or a wait step accordingly; see dynamic pipelines.

What the request does

POST https://api.graphify.com/v1/repos/{GRAPHIFY_REPO_ID}/test-impact

Field	Purpose
`Authorization: Bearer …`	Workspace CI token with the `test-impact` scope.
`baseSha`, `headSha`	Immutable commit SHAs for the comparison. Graphify derives the changed files through GitHub.
`format: "buildkite"`	Return a pipeline object containing `steps`, ready for `buildkite-agent pipeline upload`.
`command`	Test command used for selected paths and full-suite runs. Defaults to `pytest`.

The response contains the pipeline directly, without a data or success wrapper. Buildkite accepts the returned JSON as well as YAML. The script uses --no-interpolation when uploading that response so Buildkite does not expand variables inside the generated commands at upload time. See pipeline upload.

Full-suite fallback

Graphify requests a full run when it cannot confidently map a change to tests, including deleted tests, uncertain rename mappings, and non-code changes that require escalation. Explicitly configured non-code ignore rules can produce a no-op for an entirely ignored change. The sample adds a separate full-suite fallback for network failures, rate limits, and server errors. Authentication or invalid-request errors fail the selection step so you can fix the configuration. A malformed successful response or failed pipeline upload also fails the step. A failed request never counts as a successful test run.

Show test selection on pull requests

The modal’s Show test selection on pull requests switch adds an advisory section to Graphify reviews. An owner or admin manages it, and repository reviews must be enabled for the advisory workflow. The switch does not run, skip, or gate tests. Test execution is controlled by the Buildkite pipeline you configure above. Read pull request reviews for the review workflow.

Verify the connection

Open a pull request against the connected GitHub repository. In Buildkite, confirm that:

Graphify test selection completes and uploads a test step.
The uploaded step runs selected tests or identifies a full-suite run.
The test command executes successfully with your project’s environment and dependencies.

Troubleshooting

Symptom	Check
Token controls are unavailable	Ask a workspace owner or admin. If the integration says Coming soon, it is not enabled for that workspace.
Secret retrieval fails	Grant the pipeline access to `GRAPHIFY_CI_TOKEN`, or inject it through your configured secret manager.
Authentication or scope error	Use the active Graphify CI token for this workspace, with `test-impact` scope. Replacing a token revokes the old one.
Repository not found	Use the Graphify repository ID belonging to the token’s workspace.
HTTP 422	Confirm both SHAs are real commits and the repository has a GitHub connection. Do not combine a commit range with a manually supplied changed-file list.
Git cannot resolve the base	Fetch the PR’s target branch and enough history to compute its merge base. The script stops if it cannot establish the range.
Every run uses the full suite	Inspect the uploaded step’s label and selection-step logs. Check graph indexing, unmapped changes, non-code changes, and service errors before changing your test configuration.

Integrations

CI integrations

MCP

Buildkite test selection

Before you start

1. Create the CI token

2. Add the pipeline step

Adapt it to your test runner

What the request does

Full-suite fallback

Show test selection on pull requests

Verify the connection

Troubleshooting

​Before you start

​1. Create the CI token

​2. Add the pipeline step

​Adapt it to your test runner

​What the request does

​Full-suite fallback

​Show test selection on pull requests

​Verify the connection

​Troubleshooting

Before you start

1. Create the CI token

2. Add the pipeline step

Adapt it to your test runner

What the request does

Full-suite fallback

Show test selection on pull requests

Verify the connection

Troubleshooting