kubernetes-csi/csi-driver-nfs
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Update csi-release-tools

Browse Source 
$ git subtree pull --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master
This commit is contained in:
Jan Wozniak 2020-02-13 13:04:05 +01:00
commit 8e4ba5d637
14 changed files with 2146 additions and 9 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
release-tools/.prow.sh Executable file
View File

@ -0,0 +1,7 @@
#! /bin/bash -e
#
# This is for testing csi-release-tools itself in Prow. All other
# repos use prow.sh for that, but as csi-release-tools isn't a normal
# repo with some Go code in it, it has a custom Prow test script.
./verify-shellcheck.sh "$(pwd)"

115
release-tools/README.md
View File

@ -49,3 +49,118 @@ Cheat sheet:
- `git subtree add --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - add release tools to a repo which does not have them yet (only once)
- `git subtree pull --prefix=release-tools https://github.com/kubernetes-csi/csi-release-tools.git master` - update local copy to latest upstream (whenever upstream changes)
- edit, `git commit`, `git subtree push --prefix=release-tools git@github.com:<user>/csi-release-tools.git <my-new-or-existing-branch>` - push to a new branch before submitting a PR
verify-shellcheck.sh
--------------------
The [verify-shellcheck.sh](./verify-shellcheck.sh) script in this repo
is a stripped down copy of the [corresponding
script](https://github.com/kubernetes/kubernetes/blob/release-1.14/hack/verify-shellcheck.sh)
in the Kubernetes repository. It can be used to check for certain
errors shell scripts, like missing quotation marks. The default
`test-shellcheck` target in [build.make](./build.make) only checks the
scripts in this directory. Components can add more directories to
`TEST_SHELLCHECK_DIRS` to check also other scripts.
End-to-end testing
------------------
A repo that wants to opt into testing via Prow must set up a top-level
`.prow.sh`. Typically that will source `prow.sh` and then transfer
control to it:
``` bash
#! /bin/bash -e
. release-tools/prow.sh
main
```
All Kubernetes-CSI repos are expected to switch to Prow. For details
on what is enabled in Prow, see
https://github.com/kubernetes/test-infra/tree/master/config/jobs/kubernetes-csi
Test results for periodic jobs are visible in
https://testgrid.k8s.io/sig-storage-csi-ci
It is possible to reproduce the Prow testing locally on a suitable machine:
- Linux host
- Docker installed
- code to be tested checkout out in `$GOPATH/src/<import path>`
- `cd $GOPATH/src/<import path> && ./.prow.sh`
Beware that the script intentionally doesn't clean up after itself and
modifies the content of `$GOPATH`, in particular the `kubernetes` and
`kind` repositories there. Better run it in an empty, disposable
`$GOPATH`.
When it terminates, the following command can be used to get access to
the Kubernetes cluster that was brought up for testing (assuming that
this step succeeded):
export KUBECONFIG="$(kind get kubeconfig-path --name="csi-prow")"
It is possible to control the execution via environment variables. See
`prow.sh` for details. Particularly useful is testing against different
Kubernetes releases:
CSI_PROW_KUBERNETES_VERSION=1.13.3 ./.prow.sh
CSI_PROW_KUBERNETES_VERSION=latest ./.prow.sh
Dependencies and vendoring
--------------------------
Most projects will (eventually) use `go mod` to manage
dependencies. `dep` is also still supported by `csi-release-tools`,
but not documented here because it's not recommended anymore.
The usual instructions for using [go
modules](https://github.com/golang/go/wiki/Modules) apply. Here's a cheat sheet
for some of the relevant commands:
- list available updates: `GO111MODULE=on go list -u -m all`
- update or add a single dependency: `GO111MODULE=on go get <package>`
- update all dependencies to their next minor or patch release:
`GO111MODULE=on go get ./...` (add `-u=patch` to limit to patch
releases)
- lock onto a specific version: `GO111MODULE=on go get <package>@<version>`
- clean up `go.mod`: `GO111MODULE=on go mod tidy`
- update vendor directory: `GO111MODULE=on go mod vendor`
`GO111MODULE=on` can be left out when using Go >= 1.13 or when the
source is checked out outside of `$GOPATH`.
`go mod tidy` must be used to ensure that the listed dependencies are
really still needed. Changing import statements or a tentative `go
get` can result in stale dependencies.
The `test-vendor` verifies that it was used when run locally or in a
pre-merge CI job. If a `vendor` directory is present, it will also
verify that it's content is up-to-date.
The `vendor` directory is optional. It is still present in projects
because it avoids downloading sources during CI builds. If this is no
longer deemed necessary, then a project can also remove the directory.
Conversion of a repository that uses `dep` to `go mod` can be done with:
GO111MODULE=on go mod init
release-tools/go-get-kubernetes.sh <current Kubernetes version from Gopkg.toml>
GO111MODULE=on go mod tidy
GO111MODULE=on go mod vendor
git rm -f Gopkg.toml Gopkg.lock
git add go.mod go.sum vendor
### Updating Kubernetes dependencies
When using packages that are part of the Kubernetes source code, the
commands above are not enough because the [lack of semantic
versioning](https://github.com/kubernetes/kubernetes/issues/72638)
prevents `go mod` from finding newer releases. Importing directly from
`kubernetes/kubernetes` also needs `replace` statements to override
the fake `v0.0.0` versions
(https://github.com/kubernetes/kubernetes/issues/79384). The
`go-get-kubernetes.sh` script can be used to update all packages in
lockstep to a different Kubernetes version. Example usage:
```
$ ./release-tools/go-get-kubernetes.sh 1.16.4
```

90
release-tools/SIDECAR_RELEASE_PROCESS.md Normal file
View File

@ -0,0 +1,90 @@
# Sidecar Release Process
This page describes the process for releasing a kubernetes-csi sidecar.
## Prerequisites
The release manager must:
* Be a member of the kubernetes-csi organization. Open an
[issue](https://github.com/kubernetes/org/issues/new?assignees=&labels=area%2Fgithub-membership&template=membership.md&title=REQUEST%3A+New+membership+for+%3Cyour-GH-handle%3E) in
kubernetes/org to request membership
* Be a top level approver for the repository. To become a top level approver,
the candidate must demonstrate ownership and deep knowledge of the repository
through active maintainence, responding to and fixing issues, reviewing PRs,
test triage.
* Be part of the maintainers or admin group for the repository. admin is a
superset of maintainers, only maintainers level is required for cutting a
release. Membership can be requested by submitting a PR to kubernetes/org.
[Example](https://github.com/kubernetes/org/pull/1467)
## Updating CI Jobs
Whenever a new Kubernetes minor version is released, our kubernetes-csi CI jobs
must be updated.
[Our CI jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) have the
naming convention `<hostpath-deployment-version>-on-<kubernetes-version>`.
1. Jobs should be actively monitored to find and fix failures in sidecars and
infrastructure changes early in the development cycle. Test failures are sent
to kubernetes-sig-storage-test-failures@googlegroups.com.
1. "-on-master" jobs are the closest reflection to the new Kubernetes version.
1. Fixes to our prow.sh CI script can be tested in the [CSI hostpath
repo](https://github.com/kubernetes-csi/csi-driver-host-path) by modifying
[prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/release-tools/prow.sh)
along with any overrides in
[.prow.sh](https://github.com/kubernetes-csi/csi-driver-host-path/blob/master/.prow.sh)
to mirror the failing environment. Once e2e tests are passing (verify-unit tests
will fail), then the prow.sh changes can be submitted to [csi-release-tools](https://github.com/kubernetes-csi/csi-release-tools).
1. Changes can then be updated in all the sidecar repos and hostpath driver repo
by following the [update
instructions](https://github.com/kubernetes-csi/csi-release-tools/blob/master/README.md#sharing-and-updating).
1. New pull and CI jobs are configured by
[gen-jobs.sh](https://github.com/kubernetes/test-infra/blob/master/config/jobs/kubernetes-csi/gen-jobs.sh).
New pull jobs that have been unverified should be initially made optional.
[Example](https://github.com/kubernetes/test-infra/pull/15055)
1. Once new pull and CI jobs have been verified, and the new Kubernetes version
is released, we can make the optional jobs required, and also remove the
Kubernetes versions that are no longer supported.
## Release Process
1. Identify all issues and ongoing PRs that should go into the release, and
drive them to resolution.
1. Download [K8s release notes
generator](https://github.com/kubernetes/release/tree/master/cmd/release-notes)
1. Generate release notes for the release. Replace arguments with the relevant
information.
```
GITHUB_TOKEN=<token> ./release-notes --start-sha=0ed6978fd199e3ca10326b82b4b8b8e916211c9b --end-sha=3cb3d2f18ed8cb40371c6d8886edcabd1f27e7b9 \
--github-org=kubernetes-csi --github-repo=external-attacher -branch=master -output out.md
```
* `--start-sha` should point to the last release from the same branch. For
example:
* `1.X-1.0` tag when releasing `1.X.0`
* `1.X.Y-1` tag when releasing `1.X.Y`
1. Compare the generated output to the new commits for the release to check if
any notable change missed a release note.
1. Reword release notes as needed. Make sure to check notes for breaking
changes and deprecations.
1. If release is a new major/minor version, create a new `CHANGELOG-<major>.<minor>.md`
file. Otherwise, add the release notes to the top of the existing CHANGELOG
file for that minor version.
1. Submit a PR for the CHANGELOG changes.
1. Submit a PR for README changes, in particular, Compatibility, Feature status,
and any other sections that may need updating.
1. Check that all [canary CI
jobs](https://k8s-testgrid.appspot.com/sig-storage-csi-ci) are passing,
and that test coverage is adequate for the changes that are going into the release.
1. Make sure that no new PRs have merged in the meantime, and no PRs are in
flight and soon to be merged.
1. Create a new release following a previous release as a template. Be sure to select the correct
branch. This requires Github release permissions as required by the prerequisites.
[external-provisioner example](https://github.com/kubernetes-csi/external-provisioner/releases/new)
1. If release was a new major/minor version, create a new `release-<minor>`
branch at that commit.
1. Update [kubernetes-csi/docs](https://github.com/kubernetes-csi/docs) sidecar
and feature pages with the new released version.
1. After all the sidecars have been released, update
CSI hostpath driver with the new sidecars in the [CSI repo](https://github.com/kubernetes-csi/csi-driver-host-path/tree/master/deploy)
and [k/k
in-tree](https://github.com/kubernetes/kubernetes/tree/master/test/e2e/testing-manifests/storage-csi/hostpath/hostpath)

80
release-tools/build.make
View File

@ -22,6 +22,9 @@
# including build.make.
REGISTRY_NAME=quay.io/k8scsi
# Can be set to -mod=vendor to ensure that the "vendor" directory is used.
GOFLAGS_VENDOR=
# Revision that gets built into each binary via the main.version
# string. Uses the `git describe` output based on the most recent
# version tag with a short revision suffix or, if nothing has been
@ -57,12 +60,18 @@ else
TESTARGS =
endif
ARCH := $(if $(GOARCH),$(GOARCH),$(shell go env GOARCH))
# Specific packages can be excluded from each of the tests below by setting the *_FILTER_CMD variables
# to something like "| grep -v 'github.com/kubernetes-csi/project/pkg/foobar'". See usage below.
build-%:
build-%: check-go-version-go
mkdir -p bin
CGO_ENABLED=0 GOOS=linux go build -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$*
CGO_ENABLED=0 GOOS=linux go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$* ./cmd/$*
if [ "$$ARCH" = "amd64" ]; then \
CGO_ENABLED=0 GOOS=windows go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*.exe ./cmd/$* ; \
CGO_ENABLED=0 GOOS=linux GOARCH=ppc64le go build $(GOFLAGS_VENDOR) -a -ldflags '-X main.version=$(REV) -extldflags "-static"' -o ./bin/$*-ppc64le ./cmd/$* ; \
fi
container-%: build-%
docker build -t $*:latest -f $(shell if [ -e ./cmd/$*/Dockerfile ]; then echo ./cmd/$*/Dockerfile; else echo Dockerfile; fi) --label revision=$(REV) .
@ -92,19 +101,19 @@ push: $(CMDS:%=push-%)
clean:
-rm -rf bin
test:
test: check-go-version-go
.PHONY: test-go
test: test-go
test-go:
@ echo; echo "### $@:"
go test `go list ./... | grep -v 'vendor' $(TEST_GO_FILTER_CMD)` $(TESTARGS)
go test $(GOFLAGS_VENDOR) `go list $(GOFLAGS_VENDOR) ./... | grep -v -e 'vendor' -e '/test/e2e$$' $(TEST_GO_FILTER_CMD)` $(TESTARGS)
.PHONY: test-vet
test: test-vet
test-vet:
@ echo; echo "### $@:"
go vet `go list ./... | grep -v vendor $(TEST_VET_FILTER_CMD)`
go vet $(GOFLAGS_VENDOR) `go list $(GOFLAGS_VENDOR) ./... | grep -v vendor $(TEST_VET_FILTER_CMD)`
.PHONY: test-fmt
test: test-fmt
@ -117,8 +126,69 @@ test-fmt:
false; \
fi
# This test only runs when dep >= 0.5 is installed, which is the case for the CI setup.
# When using 'go mod', we allow the test to be skipped in the Prow CI under some special
# circumstances, because it depends on accessing all remote repos and thus
# running it all the time would defeat the purpose of vendoring:
# - not handling a PR or
# - the fabricated merge commit leaves go.mod, go.sum and vendor dir unchanged
# - release-tools also didn't change (changing rules or Go version might lead to
# a different result and thus must be tested)
# - import statements not changed (because if they change, go.mod might have to be updated)
#
# "git diff" is intelligent enough to annotate changes inside the "import" block in
# the start of the diff hunk:
#
# diff --git a/rpc/common.go b/rpc/common.go
# index bb4a5c4..5fa4271 100644
# --- a/rpc/common.go
# +++ b/rpc/common.go
# @@ -21,7 +21,6 @@ import (
# "fmt"
# "time"
#
# - "google.golang.org/grpc"
# "google.golang.org/grpc/codes"
# "google.golang.org/grpc/status"
#
# We rely on that to find such changes.
#
# Vendoring is optional when using go.mod.
.PHONY: test-vendor
test: test-vendor
test-vendor:
@ echo; echo "### $@:"
@ ./release-tools/verify-vendor.sh
.PHONY: test-subtree
test: test-subtree
test-subtree:
@ echo; echo "### $@:"
./release-tools/verify-subtree.sh release-tools
# Components can extend the set of directories which must pass shellcheck.
# The default is to check only the release-tools directory itself.
TEST_SHELLCHECK_DIRS=release-tools
.PHONY: test-shellcheck
test: test-shellcheck
test-shellcheck:
@ echo; echo "### $@:"
@ ret=0; \
if ! command -v docker; then \
echo "skipped, no Docker"; \
exit 0; \
fi; \
for dir in $(abspath $(TEST_SHELLCHECK_DIRS)); do \
echo; \
echo "$$dir:"; \
./release-tools/verify-shellcheck.sh "$$dir" || ret=1; \
done; \
exit $$ret
# Targets in the makefile can depend on check-go-version-<path to go binary>
# to trigger a warning if the x.y version of that binary does not match
# what the project uses. Make ensures that this is only checked once per
# invocation.
.PHONY: check-go-version-%
check-go-version-%:
./release-tools/verify-go-version.sh "$*"

133
release-tools/filter-junit.go Normal file
View File

@ -0,0 +1,133 @@
/*
Copyright 2019 The Kubernetes Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
/*
* This command filters a JUnit file such that only tests with a name
* matching a regular expression are passed through. By concatenating
* multiple input files it is possible to merge them into a single file.
*/
package main
import (
"encoding/xml"
"flag"
"io/ioutil"
"os"
"regexp"
)
var (
output = flag.String("o", "-", "junit file to write, - for stdout")
tests = flag.String("t", "", "regular expression matching the test names that are to be included in the output")
)
/*
* TestSuite represents a JUnit file. Due to how encoding/xml works, we have
* represent all fields that we want to be passed through. It's therefore
* not a complete solution, but good enough for Ginkgo + Spyglass.
*/
type TestSuite struct {
XMLName string `xml:"testsuite"`
TestCases []TestCase `xml:"testcase"`
}
type TestCase struct {
Name string `xml:"name,attr"`
Time string `xml:"time,attr"`
SystemOut string `xml:"system-out,omitempty"`
Failure string `xml:"failure,omitempty"`
Skipped SkipReason `xml:"skipped,omitempty"`
}
// SkipReason deals with the special <skipped></skipped>:
// if present, we must re-encode it, even if empty.
type SkipReason string
func (s *SkipReason) UnmarshalText(text []byte) error {
*s = SkipReason(text)
if *s == "" {
*s = " "
}
return nil
}
func (s SkipReason) MarshalText() ([]byte, error) {
if s == " " {
return []byte{}, nil
}
return []byte(s), nil
}
func main() {
var junit TestSuite
var data []byte
flag.Parse()
re := regexp.MustCompile(*tests)
// Read all input files.
for _, input := range flag.Args() {
if input == "-" {
if _, err := os.Stdin.Read(data); err != nil {
panic(err)
}
} else {
var err error
data, err = ioutil.ReadFile(input)
if err != nil {
panic(err)
}
}
if err := xml.Unmarshal(data, &junit); err != nil {
panic(err)
}
}
// Keep only matching testcases. Testcases skipped in all test runs are only stored once.
filtered := map[string]TestCase{}
for _, testcase := range junit.TestCases {
if !re.MatchString(testcase.Name) {
continue
}
entry, ok := filtered[testcase.Name]
if !ok || // not present yet
entry.Skipped != "" && testcase.Skipped == "" { // replaced skipped test with real test run
filtered[testcase.Name] = testcase
}
}
junit.TestCases = nil
for _, testcase := range filtered {
junit.TestCases = append(junit.TestCases, testcase)
}
// Re-encode.
data, err := xml.MarshalIndent(junit, "", " ")
if err != nil {
panic(err)
}
// Write to output.
if *output == "-" {
if _, err := os.Stdout.Write(data); err != nil {
panic(err)
}
} else {
if err := ioutil.WriteFile(*output, data, 0644); err != nil {
panic(err)
}
}
}

104
release-tools/go-get-kubernetes.sh Executable file
View File

@ -0,0 +1,104 @@
#!/usr/bin/env bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#
# This script can be used while converting a repo from "dep" to "go mod"
# by calling it after "go mod init" or to update the Kubernetes packages
# in a repo that has already been converted. Only packages that are
# part of kubernetes/kubernetes and thus part of a Kubernetes release
# are modified. Other k8.io packages (like k8s.io/klog, k8s.io/utils)
# need to be updated separately.
set -o pipefail
cmd=$0
function help () {
echo "$cmd <kubernetes version = x.y.z> - update all components from kubernetes/kubernetes to that version"
}
if [ $# -ne 1 ]; then
help
exit 1
fi
case "$1" in -h|--help|help) help; exit 0;; esac
die () {
echo >&2 "$@"
exit 1
}
k8s="$1"
# If the repo imports k8s.io/kubernetes (directly or indirectly), then
# "go mod" will try to find "v0.0.0" versions because
# k8s.io/kubernetes has those in it's go.mod file
# (https://github.com/kubernetes/kubernetes/blob/2bd9643cee5b3b3a5ecbd3af49d09018f0773c77/go.mod#L146-L157).
# (https://github.com/kubernetes/kubernetes/issues/79384).
#
# We need to replicate the replace statements to override those fake
# versions also in our go.mod file (idea and some code from
# https://github.com/kubernetes/kubernetes/issues/79384#issuecomment-521493597).
mods=$( (set -x; curl --silent --show-error --fail "https://raw.githubusercontent.com/kubernetes/kubernetes/v${k8s}/go.mod") |
sed -n 's|.*k8s.io/\(.*\) => ./staging/src/k8s.io/.*|k8s.io/\1|p'
) || die "failed to determine Kubernetes staging modules"
for mod in $mods; do
# The presence of a potentially incomplete go.mod file affects this command,
# so move elsewhere.
modinfo=$(set -x; cd /; env GO111MODULE=on go mod download -json "$mod@kubernetes-${k8s}") ||
die "failed to determine version of $mod: $modinfo"
v=$(echo "$modinfo" | sed -n 's|.*"Version": "\(.*\)".*|\1|p')
(set -x; env GO111MODULE=on go mod edit "-replace=$mod=$mod@$v") || die "'go mod edit' failed"
done
packages=
# Beware that we have to work with packages, not modules (i.e. no -m
# flag), because some modules trigger a "no Go code except tests"
# error. Getting their packages works.
if ! packages=$( (set -x; env GO111MODULE=on go list all) | grep ^k8s.io/ | sed -e 's; *;;'); then
cat >&2 <<EOF
Warning: "GO111MODULE=on go list all" failed, trying individual packages instead.
EOF
if ! packages=$( (set -x; env GO111MODULE=on go list -f '{{ join .Deps "\n" }}' ./...) | grep ^k8s.io/); then
cat >&2 <<EOF
ERROR: could not obtain package list, both of these commands failed:
GO111MODULE=on go list all
GO111MODULE=on go list -f '{{ join .Deps "\n" }}' ./pkg/...
EOF
exit 1
fi
fi
deps=
for package in $packages; do
# Some k8s.io packages do not come from Kubernetes staging and
# thus have different versioning (or none at all...). We need to
# skip those. We know what packages are from staging because we
# now have "replace" statements for them in go.mod.
#
# shellcheck disable=SC2001
module=$(echo "$package" | sed -e 's;k8s.io/\([^/]*\)/.*;k8s.io/\1;')
if grep -q -w "$module *=>" go.mod; then
deps="$deps $(echo "$package" | sed -e "s;\$;@kubernetes-$k8s;" -e 's;^k8s.io/kubernetes\(/.*\)@kubernetes-;k8s.io/kubernetes\1@v;')"
fi
done
# shellcheck disable=SC2086
(set -x; env GO111MODULE=on go get $deps 2>&1) || die "go get failed"
echo "SUCCESS"

1183
release-tools/prow.sh Executable file
View File

File diff suppressed because it is too large Load Diff

13
release-tools/travis.yml
View File

@ -2,13 +2,20 @@ language: go
sudo: required
services:
- docker
git:
depth: false
matrix:
include:
- go: 1.11.1
- go: 1.13.3
before_script:
- mkdir -p bin
- wget https://github.com/golang/dep/releases/download/v0.5.1/dep-linux-amd64 -O bin/dep
- chmod u+x bin/dep
- export PATH=$PWD/bin:$PATH
script:
- make -k all test
- make -k all test GOFLAGS_VENDOR=$( [ -d vendor ] && echo '-mod=vendor' )
after_success:
- if [ "${TRAVIS_PULL_REQUEST}" == "false" ]; then
docker login -u "${DOCKER_USERNAME}" -p "${DOCKER_PASSWORD}" quay.io;
make push;
make push GOFLAGS_VENDOR=$( [ -d vendor ] && echo '-mod=vendor' );
fi

23
release-tools/update-vendor.sh Executable file
View File

@ -0,0 +1,23 @@
#!/usr/bin/env bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ -f Gopkg.toml ]; then
echo "Repo uses 'dep' for vendoring."
(set -x; dep ensure)
elif [ -f go.mod ]; then
release-tools/verify-go-version.sh "go"
(set -x; env GO111MODULE=on go mod tidy && env GO111MODULE=on go mod vendor)
fi

148
release-tools/util.sh Executable file
View File

@ -0,0 +1,148 @@
#!/usr/bin/env bash
# Copyright 2014 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
function kube::util::sourced_variable {
# Call this function to tell shellcheck that a variable is supposed to
# be used from other calling context. This helps quiet an "unused
# variable" warning from shellcheck and also document your code.
true
}
kube::util::sortable_date() {
date "+%Y%m%d-%H%M%S"
}
# arguments: target, item1, item2, item3, ...
# returns 0 if target is in the given items, 1 otherwise.
kube::util::array_contains() {
local search="$1"
local element
shift
for element; do
if [[ "${element}" == "${search}" ]]; then
return 0
fi
done
return 1
}
# Example: kube::util::trap_add 'echo "in trap DEBUG"' DEBUG
# See: http://stackoverflow.com/questions/3338030/multiple-bash-traps-for-the-same-signal
kube::util::trap_add() {
local trap_add_cmd
trap_add_cmd=$1
shift
for trap_add_name in "$@"; do
local existing_cmd
local new_cmd
# Grab the currently defined trap commands for this trap
existing_cmd=$(trap -p "${trap_add_name}" | awk -F"'" '{print $2}')
if [[ -z "${existing_cmd}" ]]; then
new_cmd="${trap_add_cmd}"
else
new_cmd="${trap_add_cmd};${existing_cmd}"
fi
# Assign the test. Disable the shellcheck warning telling that trap
# commands should be single quoted to avoid evaluating them at this
# point instead evaluating them at run time. The logic of adding new
# commands to a single trap requires them to be evaluated right away.
# shellcheck disable=SC2064
trap "${new_cmd}" "${trap_add_name}"
done
}
kube::util::download_file() {
local -r url=$1
local -r destination_file=$2
rm "${destination_file}" 2&> /dev/null || true
for i in $(seq 5)
do
if ! curl -fsSL --retry 3 --keepalive-time 2 "${url}" -o "${destination_file}"; then
echo "Downloading ${url} failed. $((5-i)) retries left."
sleep 1
else
echo "Downloading ${url} succeed"
return 0
fi
done
return 1
}
# Wait for background jobs to finish. Return with
# an error status if any of the jobs failed.
kube::util::wait-for-jobs() {
local fail=0
local job
for job in $(jobs -p); do
wait "${job}" || fail=$((fail + 1))
done
return ${fail}
}
# kube::util::join <delim> <list...>
# Concatenates the list elements with the delimiter passed as first parameter
#
# Ex: kube::util::join , a b c
# -> a,b,c
function kube::util::join {
local IFS="$1"
shift
echo "$*"
}
# kube::util::check-file-in-alphabetical-order <file>
# Check that the file is in alphabetical order
#
function kube::util::check-file-in-alphabetical-order {
local failure_file="$1"
if ! diff -u "${failure_file}" <(LC_ALL=C sort "${failure_file}"); then
{
echo
echo "${failure_file} is not in alphabetical order. Please sort it:"
echo
echo " LC_ALL=C sort -o ${failure_file} ${failure_file}"
echo
} >&2
false
fi
}
# Some useful colors.
if [[ -z "${color_start-}" ]]; then
declare -r color_start="\033["
declare -r color_red="${color_start}0;31m"
declare -r color_yellow="${color_start}0;33m"
declare -r color_green="${color_start}0;32m"
declare -r color_blue="${color_start}1;34m"
declare -r color_cyan="${color_start}1;36m"
declare -r color_norm="${color_start}0m"
kube::util::sourced_variable "${color_start}"
kube::util::sourced_variable "${color_red}"
kube::util::sourced_variable "${color_yellow}"
kube::util::sourced_variable "${color_green}"
kube::util::sourced_variable "${color_blue}"
kube::util::sourced_variable "${color_cyan}"
kube::util::sourced_variable "${color_norm}"
fi
# ex: ts=2 sw=2 et filetype=sh

51
release-tools/verify-go-version.sh Executable file
View File

@ -0,0 +1,51 @@
#!/usr/bin/env bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
GO="$1"
if [ ! "$GO" ]; then
echo >&2 "usage: $0 <path to go binary>"
exit 1
fi
die () {
echo "ERROR: $*"
exit 1
}
version=$("$GO" version) || die "determining version of $GO failed"
# shellcheck disable=SC2001
majorminor=$(echo "$version" | sed -e 's/.*go\([0-9]*\)\.\([0-9]*\).*/\1.\2/')
# shellcheck disable=SC2001
expected=$(grep "^ *- go:" "release-tools/travis.yml" | sed -e 's/.*go: *\([0-9]*\)\.\([0-9]*\).*/\1.\2/')
if [ "$majorminor" != "$expected" ]; then
cat >&2 <<EOF
======================================================
WARNING
This projects is tested with Go v$expected.
Your current Go version is v$majorminor.
This may or may not be close enough.
In particular test-gofmt and test-vendor
are known to be sensitive to the version of
Go.
======================================================
EOF
fi

146
release-tools/verify-shellcheck.sh Executable file
View File

@ -0,0 +1,146 @@
#!/usr/bin/env bash
# Copyright 2018 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
set -o errexit
set -o nounset
set -o pipefail
# The csi-release-tools directory.
TOOLS="$(dirname "${BASH_SOURCE[0]}")"
. "${TOOLS}/util.sh"
# Directory to check. Default is the parent of the tools themselves.
ROOT="${1:-${TOOLS}/..}"
# required version for this script, if not installed on the host we will
# use the official docker image instead. keep this in sync with SHELLCHECK_IMAGE
SHELLCHECK_VERSION="0.6.0"
# upstream shellcheck latest stable image as of January 10th, 2019
SHELLCHECK_IMAGE="koalaman/shellcheck-alpine:v0.6.0@sha256:7d4d712a2686da99d37580b4e2f45eb658b74e4b01caf67c1099adc294b96b52"
# fixed name for the shellcheck docker container so we can reliably clean it up
SHELLCHECK_CONTAINER="k8s-shellcheck"
# disabled lints
disabled=(
# this lint disallows non-constant source, which we use extensively without
# any known bugs
1090
# this lint prefers command -v to which, they are not the same
2230
)
# comma separate for passing to shellcheck
join_by() {
local IFS="$1";
shift;
echo "$*";
}
SHELLCHECK_DISABLED="$(join_by , "${disabled[@]}")"
readonly SHELLCHECK_DISABLED
# creates the shellcheck container for later use
create_container () {
# TODO(bentheelder): this is a performance hack, we create the container with
# a sleep MAX_INT32 so that it is effectively paused.
# We then repeatedly exec to it to run each shellcheck, and later rm it when
# we're done.
# This is incredibly much faster than creating a container for each shellcheck
# call ...
docker run --name "${SHELLCHECK_CONTAINER}" -d --rm -v "${ROOT}:${ROOT}" -w "${ROOT}" --entrypoint="sleep" "${SHELLCHECK_IMAGE}" 2147483647
}
# removes the shellcheck container
remove_container () {
docker rm -f "${SHELLCHECK_CONTAINER}" &> /dev/null || true
}
# ensure we're linting the source tree
cd "${ROOT}"
# find all shell scripts excluding ./_*, ./.git/*, ./vendor*,
# and anything git-ignored
all_shell_scripts=()
while IFS=$'\n' read -r script;
do git check-ignore -q "$script" || all_shell_scripts+=("$script");
done < <(find . -name "*.sh" \
-not \( \
-path ./_\* -o \
-path ./.git\* -o \
-path ./vendor\* \
\))
# detect if the host machine has the required shellcheck version installed
# if so, we will use that instead.
HAVE_SHELLCHECK=false
if which shellcheck &>/dev/null; then
detected_version="$(shellcheck --version | grep 'version: .*')"
if [[ "${detected_version}" = "version: ${SHELLCHECK_VERSION}" ]]; then
HAVE_SHELLCHECK=true
fi
fi
# tell the user which we've selected and possibly set up the container
if ${HAVE_SHELLCHECK}; then
echo "Using host shellcheck ${SHELLCHECK_VERSION} binary."
else
echo "Using shellcheck ${SHELLCHECK_VERSION} docker image."
# remove any previous container, ensure we will attempt to cleanup on exit,
# and create the container
remove_container
kube::util::trap_add 'remove_container' EXIT
if ! output="$(create_container 2>&1)"; then
{
echo "Failed to create shellcheck container with output: "
echo ""
echo "${output}"
} >&2
exit 1
fi
fi
# lint each script, tracking failures
errors=()
for f in "${all_shell_scripts[@]}"; do
set +o errexit
if ${HAVE_SHELLCHECK}; then
failedLint=$(shellcheck --exclude="${SHELLCHECK_DISABLED}" "${f}")
else
failedLint=$(docker exec -t ${SHELLCHECK_CONTAINER} \
shellcheck --exclude="${SHELLCHECK_DISABLED}" "${f}")
fi
set -o errexit
if [[ -n "${failedLint}" ]]; then
errors+=( "${failedLint}" )
fi
done
# Check to be sure all the packages that should pass lint are.
if [ ${#errors[@]} -eq 0 ]; then
echo 'Congratulations! All shell files are passing lint.'
else
{
echo "Errors from shellcheck:"
for err in "${errors[@]}"; do
echo "$err"
done
echo
echo 'Please review the above warnings. You can test via "./hack/verify-shellcheck"'
echo 'If the above warnings do not make sense, you can exempt them from shellcheck'
echo 'checking by adding the "shellcheck disable" directive'
echo '(https://github.com/koalaman/shellcheck/wiki/Directive#disable).'
echo
} >&2
false
fi

2
release-tools/verify-subtree.sh
View File

@ -30,7 +30,7 @@ if [ ! "$DIR" ]; then
exit 1
fi
REV=$(git log -n1 --format=format:%H --no-merges -- "$DIR")
REV=$(git log -n1 --remove-empty --format=format:%H --no-merges -- "$DIR")
if [ "$REV" ]; then
echo "Directory '$DIR' contains non-upstream changes:"
echo

60
release-tools/verify-vendor.sh Executable file
View File

@ -0,0 +1,60 @@
#!/usr/bin/env bash
# Copyright 2019 The Kubernetes Authors.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
if [ -f Gopkg.toml ]; then
echo "Repo uses 'dep' for vendoring."
case "$(dep version 2>/dev/null | grep 'version *:')" in
*v0.[56789]*)
if dep check; then
echo "vendor up-to-date"
else
exit 1
fi
;;
*) echo "skipping check, dep >= 0.5 required";;
esac
elif [ -f go.mod ]; then
echo "Repo uses 'go mod'."
# shellcheck disable=SC2235
if [ "${JOB_NAME}" ] &&
( [ "${JOB_TYPE}" != "presubmit" ] ||
[ "$( (git diff "${PULL_BASE_SHA}..HEAD" -- go.mod go.sum vendor release-tools;
git diff "${PULL_BASE_SHA}..HEAD" | grep -e '^@@.*@@ import (' -e '^[+-]import') |
wc -l)" -eq 0 ] ); then
echo "Skipping vendor check because the Prow pre-submit job does not affect dependencies."
elif ! (set -x; env GO111MODULE=on go mod tidy); then
echo "ERROR: vendor check failed."
exit 1
elif [ "$(git status --porcelain -- go.mod go.sum | wc -l)" -gt 0 ]; then
echo "ERROR: go module files *not* up-to-date, they did get modified by 'GO111MODULE=on go mod tidy':";
git diff -- go.mod go.sum
exit 1
elif [ -d vendor ]; then
if ! (set -x; env GO111MODULE=on go mod vendor); then
echo "ERROR: vendor check failed."
exit 1
elif [ "$(git status --porcelain -- vendor | wc -l)" -gt 0 ]; then
echo "ERROR: vendor directory *not* up-to-date, it did get modified by 'GO111MODULE=on go mod vendor':"
git status -- vendor
git diff -- vendor
exit 1
else
echo "Go dependencies and vendor directory up-to-date."
fi
else
echo "Go dependencies up-to-date."
fi
fi