### Motivation
Enables support for using the Pulsar bookies as persistent state storage for functions.
### Modifications
- Added an option to enable/disable using bookies as state storage
- Adds extra server components options to the bookkeeper to enable necessary features for bookies to be used as state storage
- Adds stateStorageServiceUrl to the broker configmap
* To address the function role vs clusterrole issue
* making backwards compatable
* updated value.yaml to include limit functions to namespace
* Added documentation to clarify the new attribute
* moved limit_to_namespace under functions.rbac
* Refactor GitHub Actions CI to a single workflow
* Handle case where "ct lint" fails because of no chart changes
* Re-order scenarios
* Remove excessive default GC logging
* Bump cert-manager version to v1.12.2
* Use compatible cert-manager version
* Install debugging tools (k9s) for ssh access
* Only apply for interactive shells
* Fix JWT symmetric test
* Fix part that was missing from #356
* Install k9s on the fly when k9s is used
- set KUBECONFIG on the fly for kubectl too
* Upgrade upgrade kind, chart releaser and helm versions
* Disable podMonitory for values-broker-tls.yaml file
- was missing from #317
* Use k8s 1.18.20
* Use ubuntu-20.04 runtime
- k8s < 1.19 doesn't support cgroup v2
* Upgrade to k8s 1.19 as baseline
* Baseline to k8s 1.20
* Set ip family to ipv4
* Add more logging to kind cluster creation
* Simplify duplicate job deletion
* use verbosity flag
* Upgrade to k8s 1.24
* Replace removed tolerate-unready-endpoints annotation with publishNotReadyAddresses
(cherry picked from commit e90926053a2b01bb95529fbaddc8d2ce2cdeec63)
* Use k8s 1.21 as baseline
* Run on ubuntu-22.04
* Use Pulsar 2.10.4
* Add RELEASE_NOTES.txt
* Add section for upgrade considerations
* Minor fixes
* Add license header
# Motivation
As part of the 3.0.0 release, I've prepared release notes. Please review them and suggest improvements.
* Copy release process doc from Apache Airflow
Source: fb741fd872/dev/README_RELEASE_HELM_CHART.md
* Adapt to Apache Pulsar
* Remove old release process notes
* Fix typo
* Apply suggestions from code review
Co-authored-by: tison <wander4096@gmail.com>
* Add sign.sh script for release artifacts
Script is copied from 395ad7110e/dev/sign.sh
* Add some updates (more to might follow)
* Add some more updates to the rest of the release plan
* Fix rat check command
Co-authored-by: tison <wander4096@gmail.com>
Relates to #290
### Motivation
Make the Apache Pulsar Helm Chart release follow ASF rules for voting, and make the helm binary available via dist.apache.org. By following the information in https://issues.apache.org/jira/browse/LEGAL-573 and in the Apache Airflow project https://github.com/apache/airflow/blob/main/dev/README_RELEASE_HELM_CHART.md, I built this new release process. It will likely need some iterative improvement.
### Modifications
* Add a release process that is based on the Apache Airflow release process
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
Fixes#287
### Motivation
The current steps to install the Apache Pulsar Helm Chart include an unnecessary script `scripts/pulsar/prepare_helm_release.sh`. It relies on tooling that has not been maintained and is not a part of the Apache Pulsar project. As such, I propose we remove these references.
Note that one of the reasons we used these scripts historically is to simplify deployment. Without these scripts, we should document what is necessary. I am tracking that work here https://github.com/apache/pulsar-helm-chart/issues/323.
* Fix PodMonitor name conflicts for multiple releases in same namespace
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
* Use pulsar.fullname for PodMonitor name prefix
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
Co-authored-by: Michael Marshall <mmarshall@apache.org>
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
Fixes#257
### Motivation
Fix PodMonitor name conflicts for multiple releases in same namespace
### Modifications
Use release name instead of hardcode `pulsar.name` for pod monitor name.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
* Add missing license headers and .rat-excludes
* Fix .rat-excludes files
### Motivation
As part of our updated release process, we need to make sure that all relevant files have license headers.
### Modifications
* Add license headers formatted appropriately for each file type
### Verifying this change
The follow script shows that the solution is complete:
```shell
$ java -jar ../apache-rat-0.15/apache-rat-0.15.jar . -E .rat-excludes
Ignored 18 lines in your exclusion files as comments or empty lines.
*****************************************************
Summary
-------
Generated at: 2022-10-20T17:54:42-05:00
Notes: 4
Binaries: 1
Archives: 0
Standards: 92
Apache Licensed: 92
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
0 Unknown Licenses
*****************************************************
Files with Apache License headers will be marked AL
Binary files (which do not require any license headers) will be marked B
Compressed archives will be marked A
Notices, licenses etc. will be marked N
AL ./.asf.yaml
AL ./.rat-excludes
N ./LICENSE
N ./NOTICE
AL ./README.md
AL ./Vagrantfile
AL ./license_test.go
AL ./charts/pulsar/.helmignore
AL ./charts/pulsar/Chart.yaml
N ./charts/pulsar/LICENSE
N ./charts/pulsar/NOTICE
AL ./charts/pulsar/values.yaml
B ./charts/pulsar/charts/kube-prometheus-stack-41.5.1.tgz
AL ./charts/pulsar/templates/_autorecovery.tpl
AL ./charts/pulsar/templates/_bookkeeper.tpl
AL ./charts/pulsar/templates/_broker.tpl
AL ./charts/pulsar/templates/_configurationstore.tpl
AL ./charts/pulsar/templates/_helpers.tpl
AL ./charts/pulsar/templates/_toolset.tpl
AL ./charts/pulsar/templates/_zookeeper.tpl
AL ./charts/pulsar/templates/autorecovery-configmap.yaml
AL ./charts/pulsar/templates/autorecovery-podmonitor.yaml
AL ./charts/pulsar/templates/autorecovery-rbac.yaml
AL ./charts/pulsar/templates/autorecovery-service.yaml
AL ./charts/pulsar/templates/autorecovery-statefulset.yaml
AL ./charts/pulsar/templates/bookkeeper-cluster-initialize.yaml
AL ./charts/pulsar/templates/bookkeeper-configmap.yaml
AL ./charts/pulsar/templates/bookkeeper-pdb.yaml
AL ./charts/pulsar/templates/bookkeeper-podmonitor.yaml
AL ./charts/pulsar/templates/bookkeeper-rbac.yaml
AL ./charts/pulsar/templates/bookkeeper-service.yaml
AL ./charts/pulsar/templates/bookkeeper-statefulset.yaml
AL ./charts/pulsar/templates/bookkeeper-storageclass.yaml
AL ./charts/pulsar/templates/broker-cluster-role-binding.yaml
AL ./charts/pulsar/templates/broker-configmap.yaml
AL ./charts/pulsar/templates/broker-hpa.yaml
AL ./charts/pulsar/templates/broker-pdb.yaml
AL ./charts/pulsar/templates/broker-podmonitor.yaml
AL ./charts/pulsar/templates/broker-rbac.yaml
AL ./charts/pulsar/templates/broker-service-account.yaml
AL ./charts/pulsar/templates/broker-service.yaml
AL ./charts/pulsar/templates/broker-statefulset.yaml
AL ./charts/pulsar/templates/dashboard-deployment.yaml
AL ./charts/pulsar/templates/dashboard-ingress.yaml
AL ./charts/pulsar/templates/dashboard-service.yaml
AL ./charts/pulsar/templates/function-worker-configmap.yaml
AL ./charts/pulsar/templates/keytool.yaml
AL ./charts/pulsar/templates/namespace.yaml
AL ./charts/pulsar/templates/proxy-configmap.yaml
AL ./charts/pulsar/templates/proxy-hpa.yaml
AL ./charts/pulsar/templates/proxy-ingress.yaml
AL ./charts/pulsar/templates/proxy-pdb.yaml
AL ./charts/pulsar/templates/proxy-podmonitor.yaml
AL ./charts/pulsar/templates/proxy-rbac.yaml
AL ./charts/pulsar/templates/proxy-service.yaml
AL ./charts/pulsar/templates/proxy-statefulset.yaml
AL ./charts/pulsar/templates/pulsar-cluster-initialize.yaml
AL ./charts/pulsar/templates/pulsar-manager-admin-secret.yaml
AL ./charts/pulsar/templates/pulsar-manager-configmap.yaml
AL ./charts/pulsar/templates/pulsar-manager-deployment.yaml
AL ./charts/pulsar/templates/pulsar-manager-ingress.yaml
AL ./charts/pulsar/templates/pulsar-manager-service.yaml
AL ./charts/pulsar/templates/tls-cert-internal-issuer.yaml
AL ./charts/pulsar/templates/tls-certs-internal.yaml
AL ./charts/pulsar/templates/toolset-configmap.yaml
AL ./charts/pulsar/templates/toolset-rbac.yaml
AL ./charts/pulsar/templates/toolset-service.yaml
AL ./charts/pulsar/templates/toolset-statefulset.yaml
AL ./charts/pulsar/templates/zookeeper-configmap.yaml
AL ./charts/pulsar/templates/zookeeper-pdb.yaml
AL ./charts/pulsar/templates/zookeeper-podmonitor.yaml
AL ./charts/pulsar/templates/zookeeper-rbac.yaml
AL ./charts/pulsar/templates/zookeeper-service.yaml
AL ./charts/pulsar/templates/zookeeper-statefulset.yaml
AL ./charts/pulsar/templates/zookeeper-storageclass.yaml
AL ./examples/values-bookkeeper-aws.yaml
AL ./examples/values-cs.yaml
AL ./examples/values-jwt-asymmetric.yaml
AL ./examples/values-jwt-symmetric.yaml
AL ./examples/values-local-cluster.yaml
AL ./examples/values-local-pv.yaml
AL ./examples/values-minikube.yaml
AL ./examples/values-no-persistence.yaml
AL ./examples/values-one-node.yaml
AL ./examples/values-tls.yaml
AL ./examples/values-zookeeper-aws.yaml
AL ./hack/common.sh
AL ./hack/kind-cluster-build.sh
AL ./scripts/set-pulsar-version.sh
AL ./scripts/cert-manager/install-cert-manager.sh
AL ./scripts/pulsar/cleanup_helm_release.sh
AL ./scripts/pulsar/common.sh
AL ./scripts/pulsar/common_auth.sh
AL ./scripts/pulsar/generate_token.sh
AL ./scripts/pulsar/generate_token_secret_key.sh
AL ./scripts/pulsar/get_token.sh
AL ./scripts/pulsar/prepare_helm_release.sh
*****************************************************
```
* Imrpove documentation and testing for PodMonitors
* Fix missed references and a typo
### Motivation
Before upgrading to 3.0.0, we want to make sure the kube-prometheus-stack is well documented.
### Modifications
* Update tests and examples to fully disable `PodMonitors` and the installation of the kube-prometheus-stack CRDs.
### Verifying this change
The current tests will cover these changes.
Fixes#309
### Motivation
Fix the metadataPrefix initialization.
### Modifications
* Fix the script by adding `&& echo`
### Verifying this change
I manually verified that this change works and correctly puts the metadata in the prefixed location.
* Allow to use selectors with volumeClaimTemplates
* Fixed naming inconsistency, added null value
Co-authored-by: Claudio Vellage <claudio.vellage@pm.me>
Co-authored-by: Michael Marshall <mmarshall@apache.org>
### Motivation
Currently it's not possible to use selectors with volumeClaimTemplates which makes it hard/impossible to bind statically provisioned PVs.
### Modifications
Added (optional) selectors to `volumeClaimTemplates` and documented in values file.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
* [CI] Do not require version bump when linting
* Fix formatting
### Motivation
With #292, we made the lint CI step require chart version bumps. That is an unnecessary requirement since we have a manual release process. Also, we didn't require it previously.
### Modifications
* Disable chart version bump
### Verifying this change
This is a trivial change.
* allow specifying the nodeSelector for the init jobs
* Use pulsar_metadata.nodeSelector
Co-authored-by: samuel <samuel.verstraete@aprimo.com>
### Motivation
When deploying pulsar to an AKS cluster with windows nodepools i was unable to specify that the Jobs of the initalize release had to run on linux nodes. With the change i can now specify a node selector for the init jobs.
### Modifications
add nodeSelector on pulsar_init and bookie_init
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
### Motivation
The current tests only produce a message. This test adds a consumer for the produced message.
### Modifications
* Add new section to the test where we consume the produced message
Related to #311
### Motivation
In #299, I updated the values without also updating the test values. As a result, I unintentionally enabled the monitoring stack in the tests and broke some examples. Because we are deploying all resources to a single node. It is possible that we are resource constrained, so I am going to re-disable the monitoring stack.
### Modifications
* Update test cluster configurations to re-disable deploying the monitoring stack
* Update examples with the new configuration
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
* Upgrade to Cert Manager 1.10.0
* Fail fast when installing cert manager
* Upgrade to 1.7.3
Here is the relevant documentation for k8s compatibility:
https://cert-manager.io/docs/installation/supported-releases/
### Motivation
The current version is out of date.
### Modifications
* Upgrade from 1.5.4 to 1.7.3
### Verifying this change
Once #306 is merged, the test suite will verify this PR.
### Motivation
In #269, we added a way to configure external zookeeper servers. However, it was added to the wrong section of the zookeeper config. The `zookeeper.configData` section is mapped directly into the zookeeper configmap.
### Modifications
Move `zookeeper.configData.ZOOKEEPER_SERVERS` to `zookeeper.externalZookeeperServerList`
### Verifying this change
This is a cosmetic change on an unreleased feature.
* Replace monitoring solution with kube-prometheus-stack dependency
* Enable pod monitors
* Download necessary chart dependencies for CI
* Actually run dependency update
* Enable missed podMonitor
* Disable alertmanager by default for feature parity
Related issues #294#65
Supersedes #296 and #297
### Motivation
Our helm chart is out of date. I propose we make a breaking change for the monitoring solution and start using the `kube-prometheus-stack` as a dependency. This should make upgrades easier and will let users leverage all of that chart's features.
This change will result in the removal of the StreamNative Grafana Dashboards. We'll need to figure out the right way to address that. The apache/pulsar project has grafana dashboards, but they have not been maintained. With this added dependency, we'll have the benefit of being able to use k8s `ConfigMap`s to configure grafana dashboards.
### Modifications
* Remove old prometheus and grafana configuration
* Add kube-prometheus-stack chart as a dependency
* Enable several components by default. I am not opinionated on these, but it is based on the other values in the chart.
### Verifying this change
This is a large change that will require manual validation, and may break deployments. I propose this triggers a helm chart 3.0.0 release.
* Use cert-manager to generate certs for tests
* Install Cert-Manager in test env
### Motivation
Currently, we use hard coded certificates for the tests. Instead, we can use Cert Manager to generate the certificates. The primary benefit of this change is that it ensure we're testing the cert manager integration.
### Modifications
* Remove `.ci/tls` directory since we no longer need these certs.
* Remove `scripts/pulsar/clean_tls.sh` (it wasn't used)
* Remove `scripts/pulsar/upload_tls.sh` since we are not uploading any certs
* Update the `helm.sh` test script
* Update the `.ci/clusters` configurations to generate the relevant cert manager manifests
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
* added pdb version detection
* refresh
* Update bookkeeper-pdb.yaml
update the capabilities syntax
* Update broker-pdb.yaml
update capability syntax
* Update proxy-pdb.yaml
update capability version syntax
* Update zookeeper-pdb.yaml
update capability version syntax
* Update zookeeper-pdb.yaml
fix typo
* Update bookkeeper-pdb.yaml
Co-authored-by: Marvin Cai <cai19930303@gmail.com>
Fixes pod disruption budget version warning
### Motivation
PDB policy api version, v1beta1 is deprecated in k8s1.21+ (not available in 1.25+).
### Modifications
zookeeper-pdb, proxy-pdb, broker-pdb and bookkeepr-pdb templates are modified. If k8s api-resources container policy/v1, the *-pdb.yaml will generate respective apiVersion.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
Co-authored-by: Stepan Mazurov <smazurov@quantummetric.com>
### Motivation
In #204, api version of the cert resources was updated to v1. This was insufficient because `v1` has different spec from `v1alpha1`
This MR finishes the work that #204 and @lhotari started.
### Modifications
Changed the spec of certs to match v1 cert manager spec.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
* Bump Apache Pulsar 2.10.1
* Do not bump .Chart.version
* Remove unnecessary jq download that was failing with Permission Denied
Co-authored-by: Michael Marshall <mmarshall@apache.org>
