* Added support for JWT secretref and key volume mount. Added admin user auto-creation.
* Removed variables accidentally re-added and comments no longer relevant
* Enabling pulsar manager test w/ manager-admin superuser for symmetric and asymmetric jwt tests
* Added verification of communication with broker to ci test-pulsar-manager
* Fixing error on line 115 of helm.sh
* More fixes
* Adding echo of envs and tenants
* Fixing LOGIN_JSESSIONID variable name
* add missing section in values.yaml for pulsar_metadata resources
* add resources to all init containers and an additional section to specify them in values.yaml
* increase memory defaults for init containers
* remove empty lines
* Add newline to end of file
* Proposal: service accounts creation should be decoupled from PodSecurityPolicy.
* Rename *-rbac.yaml to *-psp.yaml and move service account to *-service-account.yaml
* Test with psp enabled
Co-authored-by: Lari Hotari <lhotari@apache.org>
* Lowered BOOKIE_MEM and PULSAR_MEM in init containers. Default BOOKIE_MEM and PULSAR_MEM settings from conf/pulsar_env.sh and conf/bkenv.sh (-Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g) are too high for low-memory systems.
### Motivation
Enables support for using the Pulsar bookies as persistent state storage for functions.
### Modifications
- Added an option to enable/disable using bookies as state storage
- Adds extra server components options to the bookkeeper to enable necessary features for bookies to be used as state storage
- Adds stateStorageServiceUrl to the broker configmap
* To address the function role vs clusterrole issue
* making backwards compatable
* updated value.yaml to include limit functions to namespace
* Added documentation to clarify the new attribute
* moved limit_to_namespace under functions.rbac
* Refactor GitHub Actions CI to a single workflow
* Handle case where "ct lint" fails because of no chart changes
* Re-order scenarios
* Remove excessive default GC logging
* Bump cert-manager version to v1.12.2
* Use compatible cert-manager version
* Install debugging tools (k9s) for ssh access
* Only apply for interactive shells
* Fix JWT symmetric test
* Fix part that was missing from #356
* Install k9s on the fly when k9s is used
- set KUBECONFIG on the fly for kubectl too
* Upgrade upgrade kind, chart releaser and helm versions
* Disable podMonitory for values-broker-tls.yaml file
- was missing from #317
* Use k8s 1.18.20
* Use ubuntu-20.04 runtime
- k8s < 1.19 doesn't support cgroup v2
* Upgrade to k8s 1.19 as baseline
* Baseline to k8s 1.20
* Set ip family to ipv4
* Add more logging to kind cluster creation
* Simplify duplicate job deletion
* use verbosity flag
* Upgrade to k8s 1.24
* Replace removed tolerate-unready-endpoints annotation with publishNotReadyAddresses
(cherry picked from commit e90926053a2b01bb95529fbaddc8d2ce2cdeec63)
* Use k8s 1.21 as baseline
* Run on ubuntu-22.04
* Use Pulsar 2.10.4
* Fix PodMonitor name conflicts for multiple releases in same namespace
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
* Use pulsar.fullname for PodMonitor name prefix
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
Co-authored-by: Michael Marshall <mmarshall@apache.org>
Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
Fixes#257
### Motivation
Fix PodMonitor name conflicts for multiple releases in same namespace
### Modifications
Use release name instead of hardcode `pulsar.name` for pod monitor name.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
* Add missing license headers and .rat-excludes
* Fix .rat-excludes files
### Motivation
As part of our updated release process, we need to make sure that all relevant files have license headers.
### Modifications
* Add license headers formatted appropriately for each file type
### Verifying this change
The follow script shows that the solution is complete:
```shell
$ java -jar ../apache-rat-0.15/apache-rat-0.15.jar . -E .rat-excludes
Ignored 18 lines in your exclusion files as comments or empty lines.
*****************************************************
Summary
-------
Generated at: 2022-10-20T17:54:42-05:00
Notes: 4
Binaries: 1
Archives: 0
Standards: 92
Apache Licensed: 92
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
0 Unknown Licenses
*****************************************************
Files with Apache License headers will be marked AL
Binary files (which do not require any license headers) will be marked B
Compressed archives will be marked A
Notices, licenses etc. will be marked N
AL ./.asf.yaml
AL ./.rat-excludes
N ./LICENSE
N ./NOTICE
AL ./README.md
AL ./Vagrantfile
AL ./license_test.go
AL ./charts/pulsar/.helmignore
AL ./charts/pulsar/Chart.yaml
N ./charts/pulsar/LICENSE
N ./charts/pulsar/NOTICE
AL ./charts/pulsar/values.yaml
B ./charts/pulsar/charts/kube-prometheus-stack-41.5.1.tgz
AL ./charts/pulsar/templates/_autorecovery.tpl
AL ./charts/pulsar/templates/_bookkeeper.tpl
AL ./charts/pulsar/templates/_broker.tpl
AL ./charts/pulsar/templates/_configurationstore.tpl
AL ./charts/pulsar/templates/_helpers.tpl
AL ./charts/pulsar/templates/_toolset.tpl
AL ./charts/pulsar/templates/_zookeeper.tpl
AL ./charts/pulsar/templates/autorecovery-configmap.yaml
AL ./charts/pulsar/templates/autorecovery-podmonitor.yaml
AL ./charts/pulsar/templates/autorecovery-rbac.yaml
AL ./charts/pulsar/templates/autorecovery-service.yaml
AL ./charts/pulsar/templates/autorecovery-statefulset.yaml
AL ./charts/pulsar/templates/bookkeeper-cluster-initialize.yaml
AL ./charts/pulsar/templates/bookkeeper-configmap.yaml
AL ./charts/pulsar/templates/bookkeeper-pdb.yaml
AL ./charts/pulsar/templates/bookkeeper-podmonitor.yaml
AL ./charts/pulsar/templates/bookkeeper-rbac.yaml
AL ./charts/pulsar/templates/bookkeeper-service.yaml
AL ./charts/pulsar/templates/bookkeeper-statefulset.yaml
AL ./charts/pulsar/templates/bookkeeper-storageclass.yaml
AL ./charts/pulsar/templates/broker-cluster-role-binding.yaml
AL ./charts/pulsar/templates/broker-configmap.yaml
AL ./charts/pulsar/templates/broker-hpa.yaml
AL ./charts/pulsar/templates/broker-pdb.yaml
AL ./charts/pulsar/templates/broker-podmonitor.yaml
AL ./charts/pulsar/templates/broker-rbac.yaml
AL ./charts/pulsar/templates/broker-service-account.yaml
AL ./charts/pulsar/templates/broker-service.yaml
AL ./charts/pulsar/templates/broker-statefulset.yaml
AL ./charts/pulsar/templates/dashboard-deployment.yaml
AL ./charts/pulsar/templates/dashboard-ingress.yaml
AL ./charts/pulsar/templates/dashboard-service.yaml
AL ./charts/pulsar/templates/function-worker-configmap.yaml
AL ./charts/pulsar/templates/keytool.yaml
AL ./charts/pulsar/templates/namespace.yaml
AL ./charts/pulsar/templates/proxy-configmap.yaml
AL ./charts/pulsar/templates/proxy-hpa.yaml
AL ./charts/pulsar/templates/proxy-ingress.yaml
AL ./charts/pulsar/templates/proxy-pdb.yaml
AL ./charts/pulsar/templates/proxy-podmonitor.yaml
AL ./charts/pulsar/templates/proxy-rbac.yaml
AL ./charts/pulsar/templates/proxy-service.yaml
AL ./charts/pulsar/templates/proxy-statefulset.yaml
AL ./charts/pulsar/templates/pulsar-cluster-initialize.yaml
AL ./charts/pulsar/templates/pulsar-manager-admin-secret.yaml
AL ./charts/pulsar/templates/pulsar-manager-configmap.yaml
AL ./charts/pulsar/templates/pulsar-manager-deployment.yaml
AL ./charts/pulsar/templates/pulsar-manager-ingress.yaml
AL ./charts/pulsar/templates/pulsar-manager-service.yaml
AL ./charts/pulsar/templates/tls-cert-internal-issuer.yaml
AL ./charts/pulsar/templates/tls-certs-internal.yaml
AL ./charts/pulsar/templates/toolset-configmap.yaml
AL ./charts/pulsar/templates/toolset-rbac.yaml
AL ./charts/pulsar/templates/toolset-service.yaml
AL ./charts/pulsar/templates/toolset-statefulset.yaml
AL ./charts/pulsar/templates/zookeeper-configmap.yaml
AL ./charts/pulsar/templates/zookeeper-pdb.yaml
AL ./charts/pulsar/templates/zookeeper-podmonitor.yaml
AL ./charts/pulsar/templates/zookeeper-rbac.yaml
AL ./charts/pulsar/templates/zookeeper-service.yaml
AL ./charts/pulsar/templates/zookeeper-statefulset.yaml
AL ./charts/pulsar/templates/zookeeper-storageclass.yaml
AL ./examples/values-bookkeeper-aws.yaml
AL ./examples/values-cs.yaml
AL ./examples/values-jwt-asymmetric.yaml
AL ./examples/values-jwt-symmetric.yaml
AL ./examples/values-local-cluster.yaml
AL ./examples/values-local-pv.yaml
AL ./examples/values-minikube.yaml
AL ./examples/values-no-persistence.yaml
AL ./examples/values-one-node.yaml
AL ./examples/values-tls.yaml
AL ./examples/values-zookeeper-aws.yaml
AL ./hack/common.sh
AL ./hack/kind-cluster-build.sh
AL ./scripts/set-pulsar-version.sh
AL ./scripts/cert-manager/install-cert-manager.sh
AL ./scripts/pulsar/cleanup_helm_release.sh
AL ./scripts/pulsar/common.sh
AL ./scripts/pulsar/common_auth.sh
AL ./scripts/pulsar/generate_token.sh
AL ./scripts/pulsar/generate_token_secret_key.sh
AL ./scripts/pulsar/get_token.sh
AL ./scripts/pulsar/prepare_helm_release.sh
*****************************************************
```
Fixes#309
### Motivation
Fix the metadataPrefix initialization.
### Modifications
* Fix the script by adding `&& echo`
### Verifying this change
I manually verified that this change works and correctly puts the metadata in the prefixed location.
* Allow to use selectors with volumeClaimTemplates
* Fixed naming inconsistency, added null value
Co-authored-by: Claudio Vellage <claudio.vellage@pm.me>
Co-authored-by: Michael Marshall <mmarshall@apache.org>
### Motivation
Currently it's not possible to use selectors with volumeClaimTemplates which makes it hard/impossible to bind statically provisioned PVs.
### Modifications
Added (optional) selectors to `volumeClaimTemplates` and documented in values file.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
* allow specifying the nodeSelector for the init jobs
* Use pulsar_metadata.nodeSelector
Co-authored-by: samuel <samuel.verstraete@aprimo.com>
### Motivation
When deploying pulsar to an AKS cluster with windows nodepools i was unable to specify that the Jobs of the initalize release had to run on linux nodes. With the change i can now specify a node selector for the init jobs.
### Modifications
add nodeSelector on pulsar_init and bookie_init
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
### Motivation
In #269, we added a way to configure external zookeeper servers. However, it was added to the wrong section of the zookeeper config. The `zookeeper.configData` section is mapped directly into the zookeeper configmap.
### Modifications
Move `zookeeper.configData.ZOOKEEPER_SERVERS` to `zookeeper.externalZookeeperServerList`
### Verifying this change
This is a cosmetic change on an unreleased feature.
* Replace monitoring solution with kube-prometheus-stack dependency
* Enable pod monitors
* Download necessary chart dependencies for CI
* Actually run dependency update
* Enable missed podMonitor
* Disable alertmanager by default for feature parity
Related issues #294#65
Supersedes #296 and #297
### Motivation
Our helm chart is out of date. I propose we make a breaking change for the monitoring solution and start using the `kube-prometheus-stack` as a dependency. This should make upgrades easier and will let users leverage all of that chart's features.
This change will result in the removal of the StreamNative Grafana Dashboards. We'll need to figure out the right way to address that. The apache/pulsar project has grafana dashboards, but they have not been maintained. With this added dependency, we'll have the benefit of being able to use k8s `ConfigMap`s to configure grafana dashboards.
### Modifications
* Remove old prometheus and grafana configuration
* Add kube-prometheus-stack chart as a dependency
* Enable several components by default. I am not opinionated on these, but it is based on the other values in the chart.
### Verifying this change
This is a large change that will require manual validation, and may break deployments. I propose this triggers a helm chart 3.0.0 release.
* added pdb version detection
* refresh
* Update bookkeeper-pdb.yaml
update the capabilities syntax
* Update broker-pdb.yaml
update capability syntax
* Update proxy-pdb.yaml
update capability version syntax
* Update zookeeper-pdb.yaml
update capability version syntax
* Update zookeeper-pdb.yaml
fix typo
* Update bookkeeper-pdb.yaml
Co-authored-by: Marvin Cai <cai19930303@gmail.com>
Fixes pod disruption budget version warning
### Motivation
PDB policy api version, v1beta1 is deprecated in k8s1.21+ (not available in 1.25+).
### Modifications
zookeeper-pdb, proxy-pdb, broker-pdb and bookkeepr-pdb templates are modified. If k8s api-resources container policy/v1, the *-pdb.yaml will generate respective apiVersion.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
Co-authored-by: Stepan Mazurov <smazurov@quantummetric.com>
### Motivation
In #204, api version of the cert resources was updated to v1. This was insufficient because `v1` has different spec from `v1alpha1`
This MR finishes the work that #204 and @lhotari started.
### Modifications
Changed the spec of certs to match v1 cert manager spec.
### Verifying this change
- [ ] Make sure that the change passes the CI checks.
### Motivation
This is essentially the same as https://github.com/apache/pulsar-helm-chart/pull/176. Without this change, an init pod can fail and be in `Error` state even though the second pod succeeded. This will prevent misleading errors.
### Modifications
* Replace `Never` with `OnFailure`
### Verifying this change
This is a trivial change.
Co-authored-by: Michael Marshall <mmarshall@apache.org>
### Motivation
There was a suggestion [in a dev mailing list discussion](https://lists.apache.org/thread/bgkvcyt1qq6h67p2k8xwp89xlncbqn3d) that the Helm chart's appVersion should be used as the default image tag.
### Additional context
There are some limitations in Helm. It is not possible to set "appVersion" from the command line. There's in an open feature request https://github.com/helm/helm/issues/8194 to add such a feature to Helm.
### Modifications
- change default values.yaml and set the tags for the images that use the Pulsar image to an empty value
- add "defaultPulsarImageTag" to values.yaml
- add a helper template "pulsar.imageFullName" that contains the logic to fall back to .Values.defaultPulsarImageTag and if it's not set, falling back to .Chart.AppVersion
- use the helper template in all other templates that require the logic
Fixes#288
### Motivation
When specifying multiple roles in `.Values.auth.superUsers` the values are converted to a comma-separated list by piping the dict through `values` and `join` in helm templating, `values` however doesn't guarantee that the order of elements will be the same every time. Therefor it recommends also passing it through `sortAlpha` to sort the list alphabetically.
This is a problematic when `.Values.broker.restartPodsOnConfigMapChange` is enabled because the checksum of the configmap changes every time the list's order is changed, resulting in the statefulsets rolling out a new version of the pods.
### Modifications
Pass list through `sortAlpha`.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
* Add nodeSelector to cluster initialize pod
* Add option to values file
* Update charts/pulsar/templates/pulsar-cluster-initialize.yaml
Co-authored-by: Michael Marshall <mikemarsh17@gmail.com>
* Fix typo in values
Co-authored-by: Michael Marshall <mikemarsh17@gmail.com>
### Motivation
Add an option to choose where to run pulsar-cluster-initialize pod. Sometimes there is a necessity to run only on certain nodes.
### Modifications
Added nodeSelector option to the pulsar-cluster-initialize job.
* Add imagePullSecrets for zookeeper
* Add imagePullSecrets for zookeeper
Co-authored-by: Kevin Huynh <khuynh@littlebigcode.fr>
All components have the imagePullSecrets to avoid quota limit to init correctly the pods except zookeeper
Fixes https://github.com/apache/pulsar-helm-chart/issues/250
### Motivation
`httpNumThreads` is hardcoded to 8 in `charts/pulsar/templates/proxy-configmap.yaml`
When trying to override in `values.yaml` by using `proxy.configData.httpNumThreads` we get an error because the keys get duplicated.
This happens because `{{ toYaml .Values.proxy.configData | indent 2 }}` doesn't deduplicate the keys and there is no other way to set `httpNumThreads`
### Modifications
Removing the key from charts/pulsar/templates/proxy-configmap.yaml and adding it to the values yaml solves the problem.
### Verifying this change
- [x] Make sure that the change passes the CI checks.
