Prow

admission

Mon, 01 Jan 0001 00:00:00 +0000

This is a placeholder page. Some contents needs to be filled.

Announcements

Mon, 01 Jan 0001 00:00:00 +0000

New features

New features added to each component:

April 20, 2024 The ghcache_cache_parititions Prometheus metric has been deprecated in favor of ghcache_cache_partitions. Besides spelling both metrics are identical.
April 20, 2024 The validate-supplemental-prow-config-hirarchy check in checkconfig has been renamed to validate-supplemental-prow-config-hierarchy; the old name is now deprecated.
October 20, 2023 The update to Inrepoconfig handling will break users if they have symlinks inside .prow/ that point to targets elsewhere in the codebase. See this comment for details.

Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Prow in a Nutshell

Prow creates jobs based on various types of events, such as:

GitHub events (e.g., a new PR is created, or is merged, or a person comments “/retest” on a PR),
Pub/Sub messages,
time (these are created by Horologium and are called periodic jobs), and
retesting (triggered by Tide).

Jobs are created inside the Service Cluster as Kubernetes Custom Resources. The Prow Controller Manager takes triggered jobs and schedules them into a build cluster, where they run as Kubernetes pods. Crier then reports the results back to GitHub.

branchcleaner

Mon, 01 Jan 0001 00:00:00 +0000

The branchcleaner plugin automatically deletes source branches for merged PRs between two branches on the same repository. This is helpful to keep repos that don’t allow forking clean.

Usage

Enable the branchcleaner in the desired repos via the plugins.yaml:

plugins:
  org/repo:
  - branchcleaner

Branchprotector

Mon, 01 Jan 0001 00:00:00 +0000

branchprotector configures github branch protection according to a specified policy.

Policy configuration

Extend the primary prow config.yaml document to include a top-level branch-protection key that looks like the following:


branch-protection:
  orgs:
    kubernetes:
      repos:
        test-infra:
          # Protect all branches in kubernetes/test-infra
          protect: true
          # Always allow the org's oncall-team to push
          restrictions:
            teams: ["oncall-team"]
          # Ensure that the extra-process-followed github status context passes.
          # In addition, adds any required prow jobs (aka always_run: true)
          required_status_checks:
            contexts: ["extra-process-followed"]

presubmits:
  kubernetes/test-infra:
  - name: fancy-job-name
    context: fancy-job-name
    always_run: true
    spec:  # podspec that runs job

This config will:

checkconfig

Mon, 01 Jan 0001 00:00:00 +0000

checkconfig loads the Prow configuration given with --config-path, --job-config-path and --plugin-config in order to validate it. Use checkconfig as a pre-submit for any repository holding Prow configuration to ensure that check-ins do not break anything.

cherrypicker

Mon, 01 Jan 0001 00:00:00 +0000

Cherrypicker is an external prow plugin that can also run as a standalone bot. It automates cherry-picking merged PRs into different branches. Cherrypicks are triggered from either comments or labels in GitHub PRs that need to be cherrypicked.

For comments:

/cherrypick release-1.10

The above comment will result in opening a new PR against the release-1.10 branch once the PR where the comment was made gets merged or is already merged.

To use label, you need to apply labels that contain the name of the branch in the form:

clonerefs

Mon, 01 Jan 0001 00:00:00 +0000

clonerefs clones code under test at the specified locations. Regardless of the success or failure of clone operations, this utility will have an exit code of 0 and will record the clone operation status to the specified log file. Clone records have the form:

[
    {
        "failed": false,
        "refs": {
            "org": "kubernetes",
            "repo": "kubernetes",
            "base_ref": "master",
            "base_sha": "a36820b10cde020818b8dd437e285d0e2e7d5e98",
            "pulls": [
                {
                    "number": 123,
                    "author": "smarterclayton",
                    "sha": "2b58234a8aee0d55918b158a3b38c292d6a95ef7"
                }
            ]
        },
        "commands": [
            {
                "command": "git init",
                "output": "Reinitialized existing Git repository in /go/src/k8s.io/kubernetes/.git/",
                "error": ""
            }
        ]
    }
]

Note: the utility will exit with a non-zero status if a fatal error is detected and no clone operations can even begin to run.

cm2kc (clustermap to kubeconfig)

Mon, 01 Jan 0001 00:00:00 +0000

Description

cm2kc is a CLI tool used to convert a clustermap file to a kubeconfig file.

Usage

go run ./cmd/cm2kc <options>

The following is a list of supported options for cm2kc:

  -i, --input string    Input clustermap file. (default "/dev/stdin")
  -o, --output string   Output kubeconfig file. (default "/dev/stdout")

Examples

Add a kubeconfig file in a secret: `kubeconfig` from a clustermap file in another secret: `build-cluster` for context: `my-context`

The following command will:

config-bootstrapper

Mon, 01 Jan 0001 00:00:00 +0000

config-bootstrapper is used to bootstrap a configuration that would be incrementally updated by the config-updater Prow plugin.

When a set of configurations do not exist (for example, on a clean redeployment or in a disaster recovery situation), the config-updater plugin is not useful as it can only upload incremental updates. This tool is meant to be used in those situations to set up the config to the correct base state and hand off ownership to the plugin for updates.

Configuring Tide

Mon, 01 Jan 0001 00:00:00 +0000

Configuration of Tide is located under the config/prow/config.yaml file. All configuration for merge behavior and criteria belongs in the tide yaml struct, but it may be necessary to also configure presubmits for Tide to run against PRs (see ‘Configuring Presubmit Jobs’ below).

This document will describe the fields of the tide configuration and how to populate them, but you can also check out the GoDocs for the most up to date configuration specification.

Crier

Mon, 01 Jan 0001 00:00:00 +0000

Crier reports your prowjobs on their status changes.

Usage / How to enable existing available reporters

For any reporter you want to use, you need to mount your prow configs and specify --config-path and job-config-path flag as most of other prow controllers do.

Gerrit reporter

You can enable gerrit reporter in crier by specifying --gerrit-workers=n flag.

Similar to the gerrit adapter, you’ll need to specify --gerrit-projects for your gerrit projects, and also --cookiefile for the gerrit auth token (leave it unset for anonymous).

entrypoint

Mon, 01 Jan 0001 00:00:00 +0000

entrypoint wraps a process and records its output to stdout and stderr as well as its exit code, recording both to disk. The utility will exit with a non-zero exit code if the wrapped process fails or if the utility has a fatal error.

This utility is intended to be used with sidecar, which will watch the files written by this utility and report on the status of the wrapped process.

Exporter

Mon, 01 Jan 0001 00:00:00 +0000

The prow-exporter exposes metrics about prow jobs while the metrics are not directly related to a specific prow-component.

Metrics

Metric name	Metric type	Labels/tags
prow_job_labels	Gauge	`job_name`=<prow_job-name> `job_namespace`=<prow_job-namespace> `job_agent`=<prow_job-agent> `label_PROW_JOB_LABEL_KEY`=<PROW_JOB_LABEL_VALUE>
prow_job_annotations	Gauge	`job_name`=<prow_job-name> `job_namespace`=<prow_job-namespace> `job_agent`=<prow_job-agent> `annotation_PROW_JOB_ANNOTATION_KEY`=<PROW_JOB_ANNOTATION_VALUE>
prow_job_runtime_seconds	Histogram	`job_name`=<prow_job-name> `job_namespace`=<prow_job-namespace> `type`=<prow_job-type> `last_state`=<last-state> `state`=<state> `org`=<org> `repo`=<repo> `base_ref`=<base_ref>

For example, the metric prow_job_labels is similar to kube_pod_labels defined in kubernetes/kube-state-metrics. A typical usage of prow_job_labels is to join it with other metrics using a Prometheus matching operator.

fallbackcheck

Mon, 01 Jan 0001 00:00:00 +0000

Ensure your GCS bucket layout is what tot expects to use. Useful when you want to transition from versioning your GCS buckets away from Jenkins build numbers to build numbers vended by prow.

fallbackcheck checks the existence of latest-build.txt files as per the documented GCS layout. It ignores jobs that have no GCS buckets.

Install

go get sigs.k8s.io/prow/cmd/tot/fallbackcheck

Run

fallbackcheck -bucket GCS_BUCKET -prow-url LIVE_DECK_DEPLOYMENT

For example:

fallbackcheck -bucket https://gcsweb-ci.svc.ci.openshift.org/gcs/origin-ci-test/ -prow-url https://deck-ci.svc.ci.openshift.org/

gcsupload

Mon, 01 Jan 0001 00:00:00 +0000

gcsupload uploads artifacts to cloud storage at a path resolved from the job configuration.

gcsupload can be configured by either passing in flags or by specifying a full set of options as JSON in the $GCSUPLOAD_OPTIONS environment variable, which has the following form:

{
    "bucket": "kubernetes-jenkins",
    "sub_dir": "",
    "items": [
        "/logs/artifacts/"
    ],
    "path_strategy": "legacy",
    "default_org": "kubernetes",
    "default_repo": "kubernetes",
    "gcs_credentials_file": "/secrets/gcs/service-account.json",
    "dry_run": "false"
}

In addition to this configuration for the tool, the $JOB_SPEC environment variable should be present to provide the contents of the Prow downward API for jobs. This data is used to resolve the exact location in GCS to which artifacts and logs will be pushed.

generic-autobumper

Mon, 01 Jan 0001 00:00:00 +0000

This tool automates the version upgrading of images such as the prow.k8s.io Prow deployment. Its workflow is:

Given a local git repo containing the manifests of Prow component deployment, e.g., /config/prow/cluster folder in this repo.
Find out the most recent tags of given prefixes in gcr.io registry and modify the yaml files with them.
git-commit the change, push it to the remote repo, and create/update a PR, e.g., test-infra/pull/14249, for the change.

The cluster admins can upgrade the version of images by approving the PR.

Gerrit

Mon, 01 Jan 0001 00:00:00 +0000

Gerrit is a Prow-gerrit adapter for handling CI on gerrit workflows. It can poll gerrit changes from multiple gerrit instances, and trigger presubmits on Prow upon new patchsets on Gerrit changes, and postsubmits when Gerrit changes are merged.

Deployment Usage

When deploy the gerrit component, you need to specify --config-path to your prow config, and optionally --job-config-path to your prowjob config if you have split them up.

Set --gerrit-projects to the gerrit projects you want to poll against.

ghCache

Mon, 01 Jan 0001 00:00:00 +0000

What?

ghCache is an HTTP cache optimized for caching responses from the GitHub API (https://api.github.com). Specifically, it has the following non-standard caching behavior:

Every cache hit is revalidated with a conditional HTTP request to GitHub regardless of cache entry freshness (TTL). The ‘Cache-Control’ header is ignored and overwritten to achieve this.
Concurrent requests for the same resource are coalesced and share a single request/response from GitHub instead of each request resulting in a corresponding upstream request and response.

ghCache also provides prometheus instrumentation to expose cache activity, request duration, and API token usage/savings.

HMAC

Mon, 01 Jan 0001 00:00:00 +0000

hmac is a tool to update the HMAC token, GitHub webhooks and HMAC secret for the orgs/repos as per the managed_webhooks configuration changes in the Prow config file.

Prerequisites

To run this tool, you’ll need:

A github account that has admin permission to the orgs/repos.
A personal access token for the github account. Note the token must be granted admin:repo_hook and admin:org_hook scopes.
Permissions to read&write the hmac secret in the Prow cluster.

How to setup GitHub Oauth

Mon, 01 Jan 0001 00:00:00 +0000

This document helps configure GitHub Oauth, which is required for PR Status and for the rerun button on Prow Status. If OAuth is configured, Prow will perform GitHub actions on behalf of the authenticated users. This is necessary to fetch information about pull requests for the PR Status page and to authenticate users when checking if they have permission to rerun jobs via the rerun button on Prow Status.

Set up secrets

The following steps will show you how to set up an OAuth app.

initupload

Mon, 01 Jan 0001 00:00:00 +0000

initupload reads clone records placed by clonerefs in order to determine job status. The status and logs from the clone operations are uploaded to cloud storage at a path resolved from the job configuration. This utility will exit with a non-zero exit code if the clone records indicate that any clone operations failed, as well as if any fatal errors are encountered in this utility itself.

initupload can be configured by either passing in flags or by specifying a full set of options as JSON in the $INITUPLOAD_OPTIONS environment variable, which has the same form as that for gcsupload, plus the "log" field. See that documentation for an explanation.

invitations-accepter

Mon, 01 Jan 0001 00:00:00 +0000

The invitations-accepter tool approves all pending repository invitations.

Usage

example:

invitations-accepter --dry-run=false --github-token-path=/etc/github/oauth

using with GitHub Apps

invitations-accepter --dry-run=false --github-app-id=12345 --github-app-private-key-path=/etc/github/cert

jenkins-operator

Mon, 01 Jan 0001 00:00:00 +0000

jenkins-operator is a controller that enables Prow to use Jenkins as a backend for running jobs.

Jenkins configuration

A Jenkins master needs to be provided via --jenkins-url in order for the operator to make requests to. By default, --dry-run is set to true so the operator will not make any mutating requests to Jenkins, GitHub, and Kubernetes, but you most probably want to set it to false. The Jenkins operator expects to read the Prow configuration by default in /etc/config/config.yaml which can be configured with --config-path.

lgtm

Mon, 01 Jan 0001 00:00:00 +0000

See the documentation in the approve plugin for details on the LGTM flow.

Maintainer's Guide to Tide

Mon, 01 Jan 0001 00:00:00 +0000

Best practices

Don’t let humans (or other bots) merge especially if tests have a long duration. Every merge invalidates currently running tests for that pool.
Try to limit the total number of queries that you configure. Individual queries can cover many repos and include many criteria without using additional API tokens, but separate queries each require additional API tokens.
Ensure that merge requirements configured in GitHub match the merge requirements configured for Tide. If the requirements differ, Tide may try to merge a PR that GitHub considers unmergeable.
If you are using the lgtm plugin and requiring the lgtm label for merge, don’t make queries exclude the needs-ok-to-test label. The lgtm plugin triggers one round of testing when applied to an untrusted PR and removes the lgtm label if the PR changes so it indicates to Tide that the current version of the PR is considered trusted and can be retested safely.
Do not enable the “Require branches to be up to date before merging” GitHub setting for repos managed by Tide. This requires all PRs to be rebased before merge so that PRs are always simple fast-forwards. This is a simplistic way to ensure that PRs are tested against the most recent base branch commit, but Tide already provides this guarantee through a more sophisticated mechanism that does not force PR authors to rebase their PR whenever another PR merges first. Enabling this GH setting may cause unexpected Tide behavior, provides absolutely no benefit over Tide’s natural behavior, and forces PR author’s to needlessly rebase their PRs. Don’t use it on Tide managed repos.

Expected behavior that might seem strange

Any merge to a pool kicks all other PRs in the pool back into Queued for retest. This is because Tide requires PRs to be tested against the most recent base branch commit in order to be merged. When a merge occurs, the base branch updates so any existing or in-progress tests can no longer be used to qualify PRs for merge. All remaining PRs in the pool must be retested.
Waiting to merge a successful PR because a batch is pending. This is because Tide prioritizes batches over individual PRs and the previous point tells us that merging the individual PR would invalidate the pending batch. In this case Tide will wait for the batch to complete and will merge the individual PR only if the batch fails. If the batch succeeds, the batch is merged.
If the merge requirements for a pool change it may be necessary to “poke” or “bump” PRs to trigger an update on the PRs so that Tide will resync the status context. Alternatively, Tide can be restarted to resync all statuses.
Tide may merge a PR without retesting if the existing test results are already against the latest base branch commit.
It is possible for tide status contexts on PRs to temporarily differ from the Tide dashboard or Tide’s behavior. This is because status contexts are updated asynchronously from the main Tide sync loop and have a separate rate limit and loop period.

Troubleshooting

If Prow’s PR dashboard indicates that a PR is ready to merge and it appears to meet all merge requirements, but the PR is being ignored by Tide, you may have encountered a rare bug with GitHub’s search indexing. TLDR: If this is the problem, then any update to the PR (e.g. adding a comment) will make the PR visible to Tide again after a short delay. The longer explanation is that when GitHub’s background jobs for search indexing PRs fail, the search index becomes corrupted and the search API will have some incorrect belief about the affected PR, e.g. that it is missing a required label or still has a forbidden one. This causes the search query Tide uses to identify the mergeable PRs to incorrectly omit the PR. Since the same search engine is used by both the API and GitHub’s front end, you can confirm that the affected PR is not included in the query for mergeable PRs by using the appropriate “GitHub search link” from the expandable “Merge Requirements” section on the Tide status page. You can actually determine which particular index is corrupted by incrementally tweaking the query to remove requirements until the PR is included. Any update to the PR causes GitHub to kick off a new search indexing job in the background. Once it completes, the corrupted index should be fixed and Tide will be able to see the PR again in query results, allowing Tide to resume processing the PR. It appears any update to the PR is sufficient to trigger reindexing so we typically just leave a comment. Slack thread about an example of this.

Other resources

Configuring Tide

mkpj

Mon, 01 Jan 0001 00:00:00 +0000

This is a placeholder page. Some contents needs to be filled.

mkpod

Mon, 01 Jan 0001 00:00:00 +0000

This is a placeholder page. Some contents needs to be filled.

Peribolos

Mon, 01 Jan 0001 00:00:00 +0000

Peribolos allows the org settings, teams and memberships to be declared in a yaml file. GitHub is then updated to match the declared configuration.

See the kubernetes/org repo, in particular the merge and update.sh parts of that repo for this tool in action.

Peribolos was the subject of a KubeCon talk: How Kubernetes Uses GitOps to Manage GitHub Communities at Scale

Etymology

A peribolos is a wall that encloses a court in Greek/Roman architecture.

Org configuration

Extend the primary prow config.yaml document to include a top-level orgs key that looks like the following:

Phaino

Mon, 01 Jan 0001 00:00:00 +0000

Run prowjobs on your local workstation with phaino.

Plato believed that ideas and forms are the ultimate truth, whereas we only see the imperfect physical appearances of those idea.

He linkens this in his Allegory of the Cave to someone living in a cave who can only see the shadows projected on the wall from objects passing in front of a fire.

Phaino is act of making those imperfect shadows appear.

Phaino shares a prefix with Pharos, meaning lighthouse and in particular the ancient one in Alexandria.

Phony

Mon, 01 Jan 0001 00:00:00 +0000

phony sends fake GitHub webhooks.

Running a GitHub event manager

phony is most commonly used for testing hook and its plugins, but can be used for testing any externally exposed service configured to receive GitHub events (external plugins).

To get an idea of phony’s behavior, start a local instance of hook with this:

go run cmd/hook/main.go \
 --config-path=config/prow/config.yaml \
 --plugin-config=config/prow/plugins.yaml \
 --hmac-secret-file=path/to/hmac \
 --github-token-path=path/to/github-token

# Note:
# --hmac-secret-file is required for running locally, use the same hmac token for phony below

Usage

Once you have a running server that manages github webhook events, generate an hmac token (same process as in prow), and point a phony pull request event at it with the following:

pipeline

Mon, 01 Jan 0001 00:00:00 +0000

This is a placeholder page. Some contents needs to be filled.

Plank

Mon, 01 Jan 0001 00:00:00 +0000

Plank is the controller that manages the job execution and lifecycle for jobs running in k8s.

Usage

go run ./cmd/prow-controller-manager --help

Configuration

GCS and S3 are supported as the job log storage.

# config.yaml

plank:
  # used to link to job results for decorated jobs (with pod utilities)
  job_url_prefix_config:
    '*': https://<domain>/view
  # used to link to job results for non decorated jobs (without pod utilities)
  job_url_template: 'https://<domain>/view/<bucket-name>/pr-logs/pull/{{.Spec.Refs.Repo}}/{{with index .Spec.Refs.Pulls 0}}{{.Number}}{{end}}/{{.Spec.Job}}/{{.Status.BuildID}}'
  report_template: '[Full PR test history](https://<domain>/pr-history?org={{.Spec.Refs.Org}}&repo={{.Spec.Refs.Repo}}&pr={{with index .Spec.Refs.Pulls 0}}{{.Number}}{{end}})'
  default_decoration_config_entries:
  # All entries that match a job are used, later entries override previous values.
  # Omission of 'repo' and 'cluster' fields makes this entry match all jobs.
  - config:
      timeout: 4h
      grace_period: 15s
      utility_images: # pull specs for container images used to construct job pods
        clonerefs: gcr.io/k8s-prow/clonerefs:v20190221-d14461a
        initupload: gcr.io/k8s-prow/initupload:v20190221-d14461a
        entrypoint: gcr.io/k8s-prow/entrypoint:v20190221-d14461a
        sidecar: gcr.io/k8s-prow/sidecar:v20190221-d14461a
      gcs_configuration: # configuration for uploading job results to GCS
        bucket: <bucket-name> or s3://<bucket-name>
        path_strategy: explicit # or `legacy`, `single`
        default_org: <github-org> # should not need this if `strategy` is set to explicit
        default_repo: <github-repo> # should not need this if `strategy` is set to explicit
      gcs_credentials_secret: <secret-name> # the name of the secret that stores cloud provider credentials
      ssh_key_secrets:
        - ssh-secret # name of the secret that stores the bot's ssh keys for GitHub, doesn't matter what the key of the map is and it will just uses the values
  - repo: "^org/" # some regexp to match against <org/repo>
    config:
      timeout:2h
  - cluster: "-trusted$" #some regexp to match against the cluster name
    config:
      # example override to use k8s SA with GCP workload identity rather than
      # a GCP service account key file.
      gcs_credentials_secret: ""

PR Author's Guide to Tide

Mon, 01 Jan 0001 00:00:00 +0000

If you just want to figure out how to get your PR to merge this is the document for you!

Sources of Information

The tide status context at the bottom of your PR. The status either indicates that your PR is in the merge pool or explains why it is not in the merge pool. The ‘Details’ link will take you to either the Tide or PR dashboard.
The PR dashboard at “<deck-url>/pr” where <deck-url> is something like “https://prow.k8s.io”. This dashboard shows a card for each of your PRs. Each card shows the current test results for the PR and the difference between the PR state and the merge criteria. K8s PR dashboard
The Tide dashboard at “<deck-url>/tide”. This dashboard shows the state of every merge pool so that you can see what Tide is currently doing and what position your PR has in the retest queue. K8s Tide dashboard

Get your PR merged by asking these questions

“Is my PR in the merge pool?”

If the tide status at the bottom of your PR is successful (green) it is in the merge pool. If it is pending (yellow) it is not in the merge pool.

sidecar

Mon, 01 Jan 0001 00:00:00 +0000

sidecar watches disk for files containing a the std{out,err} output from a process as well as its exit code; when the exit code has been written, this utility uploads a status object, the logs from the process and any other specified artifacts to cloud storage. The utility will exit with the exit code of the wrapped process or otherwise non-zero if the utility has a fatal error.

This utility is intended to be used with entrypoint, which will write the files watched by this utility.

Spyglass Architecture

Mon, 01 Jan 0001 00:00:00 +0000

Spyglass is split into two major parts: the Spyglass core, and a set of independent lenses. Lenses are designed to run statelessly and without any knowledge of the world outside being provided with a list of artifacts. The core is responsible for selecting lenses and providing them with artifacts.

Spyglass Core

The Spyglass Core is split across pkg/spyglass and cmd/deck. It has the following responsibilities:

Looking up artifacts for a given job and mapping those to lenses
Generating a page that loads the required lenses
Framing lenses with their boilerplate
Facilitating communication between the lens frontends and backends

Spyglass Lenses

Spyglass Lenses currently all live in pkg/spyglass/lenses, though hopefully in the future they can live elsewhere. Spyglass lenses have the following responsibilities:

status-reconciler

Mon, 01 Jan 0001 00:00:00 +0000

status-reconciler ensures that changes to blocking presubmits in Prow configuration while PRs are in flight do not cause those PRs to get stuck.

When the set of blocking presubmits changes for a repository, one of three cases occurs:

a new blocking presubmit exists and should be triggered for every trusted pull request in flight
an existing blocking presubmit is removed and should have its’ status retired
an existing blocking presubmit is renamed and should have its’ status migrated

The status-reconciler watches the job configuration for Prow and ensures that the above actions are taken as necessary.

tackle

Mon, 01 Jan 0001 00:00:00 +0000

Prow’s tackle utility walks you through deploying a new instance of prow in a couple of minutes, try it out!

Installing tackle

Tackle at this point in time needs to be built from source. The following steps will walk you through the process:

Clone the test-infra repository:

git clone git@github.com:kubernetes/test-infra.git

Build tackle (This requires a working go installation on your system)

cd test-infra/prow/cmd/tackle && go build -o tackle

Optionally move tackle to your $PATH

sudo mv tackle /usr/sbin/tackle

Deploying prow

Note: Creating a cluster using the tackle utility assumes you have the gcloud application in your $PATH and are logged in. If you are doing this on another cloud skip to the Manual deployment below.

updateconfig

Mon, 01 Jan 0001 00:00:00 +0000

updateconfig allows prow to update configmaps when files in a repo change.

updateconfig also supports glob match, or multi-key updates.

Usage

Update your plugins.yaml file to something along the following lines:

plugins:
  my-github/repo:
    plugins:
    - config-updater

config_updater:
  maps:
    # Update the thing-config configmap whenever thing changes
    path/to/some/other/thing:
      name: thing-config
    # If cluster and namespace configuration are unset, it will be put into the default cluster in the prowjob namespace
    path/to/some/other/thing2:
      name: thing2-config
      # Specify the clusters and namespaces that the configmap targets
      # which requires that the --kubeconfig arg is enabled for Hook
      # https://docs.prow.k8s.io/docs/getting-started-deploy/#run-test-pods-in-different-clusters
      # if not set or empty, it uses the cluster where prow components are running
      # and the specified namespace(s)
      clusters: 
        others:
        - namespace1
    # Update the config configmap whenever config.yaml changes
    config/prow/config.yaml:
      name: config
    # Update the plugin configmap whenever plugins.yaml changes
    config/prow/plugins.yaml:
      name: plugin
    # Update the `this` or/and `that` key in the `data` configmap whenever `data.yaml` or/and `other-data.yaml` changes
    some/data.yaml:
      name: data
      key: this
    some/other-data.yaml:
      name: data
      key: that
    # Update the fejtaverse configmap whenever any `.yaml` file under `fejtaverse` changes
    fejtaverse/**/*.yaml:
      name: fejtaverse

Additional throttling algorithm

Mon, 01 Jan 0001 00:00:00 +0000

Motivation

An additional throttling algorithm was introduced to ghproxy to prevent secondary rate limiting issues (code 403) in large Prow installations, consisting of several organizations. Its purpose is to schedule incoming requests to adjust to the GitHub general rate-limiting guidelines.

Implementation

An incoming request is analyzed whether it is targeting GitHub API v3 or API v4. Separate queues are formed not only per API but also per organization if Prow installation is using GitHub Apps. If a user account in a form of the bot is used, every request coming from that user account is categorized as coming from the same organization. This is due to the fact, that such a request identifies not using AppID and organization name, but sha256 token hash.

Build a Spyglass Lens

Mon, 01 Jan 0001 00:00:00 +0000

Spyglass lenses consist of two components: a frontend (which may be trivial) and a backend.

Lens backend

Today, a lens backend must be linked in to the deck binary. As such, lenses must live under pkg/spyglass/lenses. Additionally lenses must be in a folder that matches the name of the lens. The content of this folder will be served by deck, enabling you to reference static content such as images, stylesheets, or scripts.

CSRF attacks

Mon, 01 Jan 0001 00:00:00 +0000

In Deck, we make a number of POST requests that require user authentication. These requests are susceptible to cross site request forgery (CSRF) attacks, in which a malicious actor tricks an already authenticated user into submitting a form to one of these endpoints and performing one of these protected actions on their behalf.

Protection

If --cookie-secret is 32 or more bytes long, CSRF protection is automatically enabled. If --rerun-creates-job is specified, CSRF protection is required, and accordingly, --cookie-secret must be 32 bytes long.

Hook

Mon, 01 Jan 0001 00:00:00 +0000

This is a placeholder page. Some contents needs to be filled.

REST API coverage lens

Mon, 01 Jan 0001 00:00:00 +0000

Presents REST endpoints statistics

Configuration

threshold_warning set threshold for warning highlight
threshold_error set threshold for error highlight

Expected input

uniqueHits total number of unique params calls (first hit of any leaf should increase this value)
expectedUniqueHits total number of params (leaves)
percent is uniqueHits * 100 / expectedUniqueHits
methodCalled whether the method was called
body body params
query query params
root root of the tree
hits number of all params hits
items collection of nodes, if not present then the node is a leaf
height height of the tree
size size of the tree

{
    "uniqueHits": 2,
    "expectedUniqueHits": 4,
    "percent": 50.00,
    "endpoints": {
        "/pets": {
            "post": {
                "uniqueHits": 2,
                "expectedUniqueHits": 4,
                "percent": 50.00,
                "methodCalled": true,
                "params": {
                    "body": {
                        "uniqueHits": 2,
                        "expectedUniqueHits": 4,
                        "percent": 50.00,
                        "root": {
                            "hits": 15,
                            "items": {
                                "origin": {
                                   "hits": 8,
                                   "items": {
                                       "country": {
                                           "hits": 8,
                                           "items": {
                                               "name": {
                                                   "hits": 8
                                               },
                                               "region": {
                                                   "hits": 0
                                               }
                                           }
                                       }
                                   }
                                },
                                "color": {
                                    "hits": 0
                                },
                                "type": {
                                    "hits": 7
                                }
                            }
                        },
                        "height": 4,
                        "size": 7
                    }
                }
            }
        }
    }
}

Fake Git Server (FGS)

Mon, 01 Jan 0001 00:00:00 +0000

FGS is actually not a fake at all. It is a real web server that serves real Git repositories them over HTTP. FGS wraps around the vanilla git http-backend subcommand that comes with Git, calling it as a CGI executable. It supports both read (e.g., git clone, git fetch) and write (e.g., git push) operations against it.

FGS is used for integration tests. See TestClonerefs for an example.

Usage in Integration Testing

The fakegitserver.go file is built automatically by hack/prowimagebuilder, and we deploy it to the KIND cluster. Inside the cluster, it accepts web traffic at the endpoint http://fakegitserver.default (http://localhost/fakegitserver from outside of the KIND cluster).

Gangway (Prow API)

Mon, 01 Jan 0001 00:00:00 +0000

Architecture

See the design doc.

Gangway uses gRPC to serve several endpoints. These can be seen in the gangway.proto file, which describes the gRPC endpoints. The proto describes the interface at a high level, and is converted into low-level Golang types into gangway.pb.go and gangway_grpc.pb.go. These low-level Golang types are then used in the gangway.go file to implement the high-level intent of the proto file.

As Gangway only understands gRPC natively, if you want to use a REST client against it you must deploy Gangway. For example, on GKE you can use Cloud Endpoints and deploy Gangway behind a reverse proxy called “ESPv2”. This ESPv2 container will forward HTTP requests made to it to the equivalent gRPC endpoint in Gangway and back again.

GKE Build Clusters

Mon, 01 Jan 0001 00:00:00 +0000

Note: This page discusses build clusters that use GKE. Technically speaking, a build cluster could be any Kubernetes cluster (not just GKE) because the only thing Prow needs is the ability to authenticate as a Kubernetes Service Account with cluster-admin role permissions to the build cluster.

Overview

By default Prow will schedule jobs in the cluster that maps to a kubeconfig alias named “default” (imagine running kubectl config set-context "default" --cluster=<CLUSTER_CONTEXT>, where the <CLUSTER_CONTEXT> could be any cluster). So the jobs can be scheduled either in the same cluster that is hosting Prow itself, or a different one. For example the “default” cluster in https://prow.k8s.io is the build cluster located in the k8s-prow-builds GCP project and not the k8s-prow GCP project where the Prow services actually run.

Horologium

Mon, 01 Jan 0001 00:00:00 +0000

This is a placeholder page. Some contents needs to be filled.

Prow-Controller-Manager

Mon, 01 Jan 0001 00:00:00 +0000

prow-controller-manager manages the job execution and lifecycle for jobs running in k8s.

It currently acts as a replacement for Plank.

It is intended to eventually replace other components, such as Sinker and Crier. See the tracking issue #17024 for details.

Advantages

Eventbased rather than cronbased, hence reacting much faster to changes in prowjobs or pods
Per-Prowjob retrying, meaning genuinely broken prowjobs will not be retried forever and transient errors will be retried much quicker
Uses a cache for the build cluster rather than doing a LIST every 30 seconds, reducing the load on the build clusters api server

Exclusion with other components

This is mutually exclusive with only Plank. Only one of them may have more than zero replicas at the same time.

Sinker

Mon, 01 Jan 0001 00:00:00 +0000

This is a placeholder page. Some contents needs to be filled.

Building, Testing, and Updating Prow

Mon, 01 Jan 0001 00:00:00 +0000

This guide is directed at Prow developers and maintainers who want to build/test individual components or deploy changes to an existing Prow cluster. “Deploying Prow” is a better reference for deploying a new Prow cluster.

How to build and test Prow

You can build, test, and deploy Prow’s binaries, container images, and cluster resources.

Build locally with:

make build-images

Push to remote with

make push-images REGISTRY=<YOUR_REGISTRY>

Unit test with:

make test

Integration test with(more details):

Deploying Prow

Mon, 01 Jan 0001 00:00:00 +0000

This document will walk you through deploying your own Prow instance to a new Kubernetes cluster. If you encounter difficulties, please open an issue so that we can make this process easier.

Prow runs in any kubernetes cluster. The guide below is focused on Google Kubernetes Engine but should work on any kubernetes distro with no/minimal changes.

GitHub App

First, you need to create a GitHub app. GitHub itself documents this. Initially, it is sufficient to set a dummy url for the Webhook. The exact set of permissions needed varies based on what functionality you use. Below is a minimum set of permissions needed. Please keep in mind that any changes to the permissions your app requests (both added and removed) require everyone to re-install it.

Developing and Contributing to Prow

Mon, 01 Jan 0001 00:00:00 +0000

Contributing

Please consider upstreaming any changes or additions you make! Contributions in any form (issues, pull requests, even constructive comments in discussions) are more than welcome! You can develop in-tree for more help and review, or out-of-tree if you need to for whatever reason. If you upstream a new feature or a change that impacts the default behavior of Prow, consider adding an announcement about it and dropping an email at the sig-testing mailing list.

Getting more out of Prow

Mon, 01 Jan 0001 00:00:00 +0000

If you want more functionality from your Prow instance this guide is for you. It primarily links to other resources that catalogue existing components and features.

Use more Prow components and plugins

Prow has a number of optional cluster components and a suite of plugins for hook that provide all sorts of automation. Check out the Components for a list of cluster components and the Plugins for information about available plugins.

Consume Prometheus metrics

Some Prow components expose prometheus metrics that can be used for monitoring, alerting, and pretty graphs. You can find details in the Metrics document.

GitHub API Library

Mon, 01 Jan 0001 00:00:00 +0000

This GitHub API library is used by multiple parts of Prow. It uses both v3 and v4 of GitHub’s API. It is subject to change as needed without notice, but you can reuse and extend it within this repository.

Its primary component is client.go, a GitHub client that sends and receives API calls.

Recommended Usage

Instantiation

An application that takes flags may want to set GitHub flags, such as a proxy endpoint. To do that, GitHubOptions has a method that returns a GitHub client.

Inrepoconfig

Mon, 01 Jan 0001 00:00:00 +0000

Inrepoconfig is a Prow feature that allows versioning Presubmit and Postsubmit jobs in the same repository that also holds the code (with a .prow directory or .prow.yaml file, akin to a .travis.yaml file). So instead of having all your jobs defined centrally, you could instead define the jobs in a distributed manner, coupled closely with the source code repos that they work on.

If enabled, Prow will use both the centrally-defined jobs and the ones defined in the code repositories. The latter ones are dynamically loaded on-demand.

Life of a Prow Job

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: This document uses 5df7636b83cab54e248e550a31dbf1e4731197a6 (July 21, 2021) as a reference point for all code links.

Let’s pretend a user comments /test all on a Pull Request (PR). In response, GitHub posts this comment to Prow via a webhook. See examples for webhook payloads.

Prow’s Kubernetes cluster uses an ingress resource for terminating TLS, and routes traffic to the hook service resource, finally sending the traffic to the hook application, which is defined as a deployment:

Prow Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Core Prow component configuration is managed by the config package and stored in the Config struct. If a configuration guide is available for a component it can be found in the “Components” directory. See jobs.md for a guide to configuring ProwJobs. Configuration for plugins is handled and stored separately. See the plugins package for details.

You can find a sample config with all possible options and a documentation of them here.

Prow Secrets Management

Mon, 01 Jan 0001 00:00:00 +0000

Secrets in prow service/build clusters are managed with Kubernetes External Secrets, which is responsible for one-way syncing secret values from major secret manager providers such as GCP, Azure, and AWS secret managers into kubernetes clusters, based on ExternalSecret custom resource defined in cluster (As shown in example below).

Note: the instructions below are only for GCP secret manager, for authenticating with other providers please refer to https://github.com/external-secrets/kubernetes-external-secrets#backends

Set Up (Prow maintainers)

This is performed by prow service/build clusters maintainer.

Gerrit

Mon, 01 Jan 0001 00:00:00 +0000

Gerrit is a free, web-based team code collaboration tool.

Prow-gerrit adapter (doc, code)
Crier (the reporter) (doc, code)

Client

We have a gerrit-client package that provides a thin wrapper around andygrunwald/go-gerrit, which is a go client library for accessing the Gerrit Code Review REST API

You can create a client instance by pass in a map of instance-name:project-ids, and pass in an oauth token path to start the client, like:

projects := map[string][]string{
 "foo.googlesource.com": {
  "project-bar",
  "project-baz",
 },
}

c, err := gerrit.NewClient(projects)
if err != nil {
 // handle error
}
c.Start(cookiefilePath)

The client will try to refetch token from the path every 10 minutes.

Sub

Mon, 01 Jan 0001 00:00:00 +0000

Sub is a Prow component that can trigger new Prow jobs (PJs) using Pub/Sub messages. The message does not need to have the full PJ defined; instead you just need to have the job name and some other key pieces of information (more on this below). The rest of the data needed to create a full-blown PJ is derived from the main Prow configuration (or inrepoconfig).

Deployment Usage

Sub can listen to Pub/Sub subscriptions (known as “pull subscriptions”).

ProwJobs

Mon, 01 Jan 0001 00:00:00 +0000

For a brief overview of how Prow runs jobs take a look at “Life of a Prow Job”.

For a brief cookbook for jobs intended for prow.k8s.io, please refer to config/jobs/README.md

Make sure Prow has been deployed correctly:

The horologium component schedules periodic jobs.
The hook component schedules presubmit and postsubmit jobs, ensuring the repo:
- enabled trigger in plugins.yaml
- sends GitHub webhooks to prow.
The plank component schedules the pod requested by a prowjob.
The crier component reports status back to github.

How to configure new jobs

To configure a new job you’ll need to add an entry into config.yaml. If you have update-config plugin deployed then the config will be automatically updated once the PR is merged, else you will need to run make update-config. This does not require redeploying any binaries, and will take effect within a few minutes.

Setting up Private Deck

Mon, 01 Jan 0001 00:00:00 +0000

1) [User] Create a PR to Set up TenantIDs for prowjobs and Repos

Prow users should create a PR creating tenantID defaults for their org/repos and clusters. Once you set up a tenantID, all prowjobs labelled with that tenantID will only be visible on Deck instances created with the same tenantID. If you already have prowjobs that you don’t want to lose access to on Deck, do this step last. If not, do it first to make sure prowjobs you want to keep sequestered do not appear on other instances of Deck.

Using Prow at Scale

Mon, 01 Jan 0001 00:00:00 +0000

If you are maintaining a Prow instance that will need to scale to handle a large load, consider using the following best practices, features, and additional tools. You may also be interested in “Getting more out of Prow”.

Features and Tools

Separate Build Cluster(s)

It is frequently not secure to run all ProwJobs in the same cluster that runs Prow’s service components (hook, plank, etc.). In particular, ProwJobs that execute presubmit tests for OSS projects should typically be isolated from Prow’s microservices. This isolation prevents a malicious PR author from modifying the presubmit test to do something evil like breaking out of the container and stealing secrets that live in the cluster or DOSing a cluster-internal Prow component service.

Understanding Started.json and Finished.json

Mon, 01 Jan 0001 00:00:00 +0000

Context

Prow uploads a host of artifacts throughout the life cycle of a job. Two of these artifacts that are present in each run are started.json and finished.json which contain a host of information pertaining to the job/run. These files have existed through the evolution of Kubernetes CI: from Jenkins -> Containerized Jenkins -> Bootstrap Containerized Jenkins -> Bootstrap Prow -> PodUtils. As of 2021, all jobs exist within either Bootstrap Prow or PodUtils. As the CI has evolved, so has started/finished.json and it’s function.

Prow

admission

Announcements

New features

Architecture

Prow in a Nutshell

branchcleaner

Usage

Branchprotector

Policy configuration

checkconfig

cherrypicker

clonerefs

cm2kc (clustermap to kubeconfig)

Description

Usage

Examples

Add a kubeconfig file in a secret: kubeconfig from a clustermap file in another secret: build-cluster for context: my-context

config-bootstrapper

Configuring Tide

Crier

Usage / How to enable existing available reporters

Gerrit reporter

entrypoint

Exporter

Metrics

fallbackcheck

Install

Run

gcsupload

generic-autobumper

Gerrit

Deployment Usage

ghCache

What?

HMAC

Prerequisites

How to setup GitHub Oauth

Set up secrets

initupload

invitations-accepter

Usage

jenkins-operator

Jenkins configuration

lgtm

Maintainer's Guide to Tide

Best practices

Expected behavior that might seem strange

Troubleshooting

Other resources

mkpj

mkpod

Peribolos

Etymology

Org configuration

Phaino

Phony

Running a GitHub event manager

Usage

pipeline

Plank

Usage

Configuration

PR Author's Guide to Tide

Sources of Information

Get your PR merged by asking these questions

“Is my PR in the merge pool?”

sidecar

Spyglass Architecture

Spyglass Core

Spyglass Lenses

status-reconciler

tackle

Installing tackle

Deploying prow

updateconfig

Usage

Additional throttling algorithm

Motivation

Implementation

Add a kubeconfig file in a secret: `kubeconfig` from a clustermap file in another secret: `build-cluster` for context: `my-context`