Documentation on Prow

Announcements

Mon, 01 Jan 0001 00:00:00 +0000

New features

New features added to each component:

April 20, 2024 The ghcache_cache_parititions Prometheus metric has been deprecated in favor of ghcache_cache_partitions. Besides spelling both metrics are identical.
April 20, 2024 The validate-supplemental-prow-config-hirarchy check in checkconfig has been renamed to validate-supplemental-prow-config-hierarchy; the old name is now deprecated.
October 20, 2023 The update to Inrepoconfig handling will break users if they have symlinks inside .prow/ that point to targets elsewhere in the codebase. See this comment for details.

GKE Build Clusters

Mon, 01 Jan 0001 00:00:00 +0000

Note: This page discusses build clusters that use GKE. Technically speaking, a build cluster could be any Kubernetes cluster (not just GKE) because the only thing Prow needs is the ability to authenticate as a Kubernetes Service Account with cluster-admin role permissions to the build cluster.

Overview

By default Prow will schedule jobs in the cluster that maps to a kubeconfig alias named “default” (imagine running kubectl config set-context "default" --cluster=<CLUSTER_CONTEXT>, where the <CLUSTER_CONTEXT> could be any cluster). So the jobs can be scheduled either in the same cluster that is hosting Prow itself, or a different one. For example the “default” cluster in https://prow.k8s.io is the build cluster located in the k8s-prow-builds GCP project and not the k8s-prow GCP project where the Prow services actually run.

Building, Testing, and Updating Prow

Mon, 01 Jan 0001 00:00:00 +0000

This guide is directed at Prow developers and maintainers who want to build/test individual components or deploy changes to an existing Prow cluster. “Deploying Prow” is a better reference for deploying a new Prow cluster.

How to build and test Prow

You can build, test, and deploy Prow’s binaries, container images, and cluster resources.

Build locally with:

make build-images

Push to remote with

make push-images REGISTRY=<YOUR_REGISTRY>

Unit test with:

make test

Integration test with(more details):

Deploying Prow

Mon, 01 Jan 0001 00:00:00 +0000

This document will walk you through deploying your own Prow instance to a new Kubernetes cluster. If you encounter difficulties, please open an issue so that we can make this process easier.

Prow runs in any kubernetes cluster. The guide below is focused on Google Kubernetes Engine but should work on any kubernetes distro with no/minimal changes.

GitHub App

First, you need to create a GitHub app. GitHub itself documents this. Initially, it is sufficient to set a dummy url for the Webhook. The exact set of permissions needed varies based on what functionality you use. Below is a minimum set of permissions needed. Please keep in mind that any changes to the permissions your app requests (both added and removed) require everyone to re-install it.

Developing and Contributing to Prow

Mon, 01 Jan 0001 00:00:00 +0000

Contributing

Please consider upstreaming any changes or additions you make! Contributions in any form (issues, pull requests, even constructive comments in discussions) are more than welcome! You can develop in-tree for more help and review, or out-of-tree if you need to for whatever reason. If you upstream a new feature or a change that impacts the default behavior of Prow, consider adding an announcement about it and dropping an email at the sig-testing mailing list.

Getting more out of Prow

Mon, 01 Jan 0001 00:00:00 +0000

If you want more functionality from your Prow instance this guide is for you. It primarily links to other resources that catalogue existing components and features.

Use more Prow components and plugins

Prow has a number of optional cluster components and a suite of plugins for hook that provide all sorts of automation. Check out the Components for a list of cluster components and the Plugins for information about available plugins.

Consume Prometheus metrics

Some Prow components expose prometheus metrics that can be used for monitoring, alerting, and pretty graphs. You can find details in the Metrics document.

GitHub API Library

Mon, 01 Jan 0001 00:00:00 +0000

This GitHub API library is used by multiple parts of Prow. It uses both v3 and v4 of GitHub’s API. It is subject to change as needed without notice, but you can reuse and extend it within this repository.

Its primary component is client.go, a GitHub client that sends and receives API calls.

Recommended Usage

Instantiation

An application that takes flags may want to set GitHub flags, such as a proxy endpoint. To do that, GitHubOptions has a method that returns a GitHub client.

Inrepoconfig

Mon, 01 Jan 0001 00:00:00 +0000

Inrepoconfig is a Prow feature that allows versioning Presubmit and Postsubmit jobs in the same repository that also holds the code (with a .prow directory or .prow.yaml file, akin to a .travis.yaml file). So instead of having all your jobs defined centrally, you could instead define the jobs in a distributed manner, coupled closely with the source code repos that they work on.

If enabled, Prow will use both the centrally-defined jobs and the ones defined in the code repositories. The latter ones are dynamically loaded on-demand.

Life of a Prow Job

Mon, 01 Jan 0001 00:00:00 +0000

NOTE: This document uses 5df7636b83cab54e248e550a31dbf1e4731197a6 (July 21, 2021) as a reference point for all code links.

Let’s pretend a user comments /test all on a Pull Request (PR). In response, GitHub posts this comment to Prow via a webhook. See examples for webhook payloads.

Prow’s Kubernetes cluster uses an ingress resource for terminating TLS, and routes traffic to the hook service resource, finally sending the traffic to the hook application, which is defined as a deployment:

Prow Configuration

Mon, 01 Jan 0001 00:00:00 +0000

Core Prow component configuration is managed by the config package and stored in the Config struct. If a configuration guide is available for a component it can be found in the “Components” directory. See jobs.md for a guide to configuring ProwJobs. Configuration for plugins is handled and stored separately. See the plugins package for details.

You can find a sample config with all possible options and a documentation of them here.

Prow Secrets Management

Mon, 01 Jan 0001 00:00:00 +0000

Secrets in prow service/build clusters are managed with Kubernetes External Secrets, which is responsible for one-way syncing secret values from major secret manager providers such as GCP, Azure, and AWS secret managers into kubernetes clusters, based on ExternalSecret custom resource defined in cluster (As shown in example below).

Note: the instructions below are only for GCP secret manager, for authenticating with other providers please refer to https://github.com/external-secrets/kubernetes-external-secrets#backends

Set Up (Prow maintainers)

This is performed by prow service/build clusters maintainer.

Gerrit

Mon, 01 Jan 0001 00:00:00 +0000

Gerrit is a free, web-based team code collaboration tool.

Prow-gerrit adapter (doc, code)
Crier (the reporter) (doc, code)

Client

We have a gerrit-client package that provides a thin wrapper around andygrunwald/go-gerrit, which is a go client library for accessing the Gerrit Code Review REST API

You can create a client instance by pass in a map of instance-name:project-ids, and pass in an oauth token path to start the client, like:

projects := map[string][]string{
 "foo.googlesource.com": {
  "project-bar",
  "project-baz",
 },
}

c, err := gerrit.NewClient(projects)
if err != nil {
 // handle error
}
c.Start(cookiefilePath)

The client will try to refetch token from the path every 10 minutes.

ProwJobs

Mon, 01 Jan 0001 00:00:00 +0000

For a brief overview of how Prow runs jobs take a look at “Life of a Prow Job”.

For a brief cookbook for jobs intended for prow.k8s.io, please refer to config/jobs/README.md

Make sure Prow has been deployed correctly:

The horologium component schedules periodic jobs.
The hook component schedules presubmit and postsubmit jobs, ensuring the repo:
- enabled trigger in plugins.yaml
- sends GitHub webhooks to prow.
The plank component schedules the pod requested by a prowjob.
The crier component reports status back to github.

How to configure new jobs

To configure a new job you’ll need to add an entry into config.yaml. If you have update-config plugin deployed then the config will be automatically updated once the PR is merged, else you will need to run make update-config. This does not require redeploying any binaries, and will take effect within a few minutes.

Setting up Private Deck

Mon, 01 Jan 0001 00:00:00 +0000

1) [User] Create a PR to Set up TenantIDs for prowjobs and Repos

Prow users should create a PR creating tenantID defaults for their org/repos and clusters. Once you set up a tenantID, all prowjobs labelled with that tenantID will only be visible on Deck instances created with the same tenantID. If you already have prowjobs that you don’t want to lose access to on Deck, do this step last. If not, do it first to make sure prowjobs you want to keep sequestered do not appear on other instances of Deck.

Using Prow at Scale

Mon, 01 Jan 0001 00:00:00 +0000

If you are maintaining a Prow instance that will need to scale to handle a large load, consider using the following best practices, features, and additional tools. You may also be interested in “Getting more out of Prow”.

Features and Tools

Separate Build Cluster(s)

It is frequently not secure to run all ProwJobs in the same cluster that runs Prow’s service components (hook, plank, etc.). In particular, ProwJobs that execute presubmit tests for OSS projects should typically be isolated from Prow’s microservices. This isolation prevents a malicious PR author from modifying the presubmit test to do something evil like breaking out of the container and stealing secrets that live in the cluster or DOSing a cluster-internal Prow component service.

Understanding Started.json and Finished.json

Mon, 01 Jan 0001 00:00:00 +0000

Context

Prow uploads a host of artifacts throughout the life cycle of a job. Two of these artifacts that are present in each run are started.json and finished.json which contain a host of information pertaining to the job/run. These files have existed through the evolution of Kubernetes CI: from Jenkins -> Containerized Jenkins -> Bootstrap Containerized Jenkins -> Bootstrap Prow -> PodUtils. As of 2021, all jobs exist within either Bootstrap Prow or PodUtils. As the CI has evolved, so has started/finished.json and it’s function.

Documentation on Prow

Announcements

New features

GKE Build Clusters

Overview

Building, Testing, and Updating Prow

How to build and test Prow

Deploying Prow

GitHub App

Developing and Contributing to Prow

Contributing

Getting more out of Prow

Use more Prow components and plugins

Consume Prometheus metrics

GitHub API Library

Recommended Usage

Instantiation

Inrepoconfig

Life of a Prow Job

Prow Configuration

Prow Secrets Management

Set Up (Prow maintainers)

Gerrit

Related Deployments

Related packages

Client

ProwJobs

How to configure new jobs

Setting up Private Deck

1) [User] Create a PR to Set up TenantIDs for prowjobs and Repos

Using Prow at Scale

Features and Tools

Separate Build Cluster(s)

Understanding Started.json and Finished.json

Context