217 Commits

Author SHA1 Message Date
Valeriano Manassero
25e997a425
Automate initialize (#138)
- no need to do "--set initialize=true" anymore
2022-01-05 16:08:11 +02:00
matejhasul
706c8c292b
Workaround kustomize bug in pulsar cluster init (#166)
Replace folding block with multiline string to workaround https://github.com/kubernetes-sigs/kustomize/issues/4201

There are also other places where this bug is hit, but extra generated newline is not significant.

Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com>
2022-01-04 11:08:52 -06:00
Lari Hotari
d3589b1c82
[CI] Improve logging in CI scripts and add timeouts (#195)
- show events every 15 seconds
- dump logs every 5 minutes
- dump logs when timeouting
2022-01-04 17:24:28 +02:00
Lari Hotari
9b672379dd
[CI] Improve change detection by reusing solution from apache/pulsar (#194)
* [CI] Improve change detection by reusing solution from apache/pulsar

* Fix verify release
2022-01-04 11:40:40 +02:00
Shu.Wang
83bb8bd60f
Conditionally update ingress api version based on k8s version (#183)
* Update ingress api version, extension/v1beta1 will not be supported in new k8s version, this change keep backward compatibility for lower kubernetes version

* Update deprecated util Capabilities.KubeVersion.GitVersion to Capabilities.KubeVersion.Version
2022-01-04 00:53:21 -06:00
Shu.Wang
0a82ab0f9a
Fixes #177 Fix indentation of component, as it should be under the label tag (#182) 2022-01-03 21:57:45 +02:00
Lari Hotari
d74d08a89d
Use NIOServerCnxnFactory for Zookeeper to fix NPE issues with Pulsar 2.8.x+ (#180)
- follow recommendation in https://github.com/apache/pulsar/issues/11070#issuecomment-936539979
2022-01-03 11:59:58 +01:00
Lari Hotari
567bd7c5b3
[CI] Add job 45min timeout and cancel duplicate jobs (#193) 2022-01-03 10:17:21 +01:00
Jiwei Guo
0f6dea8022
Bump to Pulsar 2.7.4 (#189)
* Bump to Pulsar 2.7.4

* update

* update
pulsar-2.7.7
2021-12-30 08:55:57 +02:00
Lari Hotari
b4b2fa7b80
[Security] Workaround for CVE-2021-44228 Log4J RCE when Log4J >= 2.10.0 (#186)
* [Security] Workaround for CVE-2021-44228 Log4J RCE when Log4J >= 2.10.0

- prevents the exploit by disabling message pattern lookups

* Bump the chart version
pulsar-2.7.6
2021-12-10 18:30:01 +02:00
Lari Hotari
a16c6bbf19
Make k8s probe timeoutSeconds configurable and set default to 5s for k8s 1.20+ compatibility (#179)
- set to 5 seconds by default

- address compatibility with Kubernetes 1.20+. This impacts "bin/pulsar-zookeeper-ruok.sh" exec probe used in ZK.
  "Before Kubernetes 1.20, the field timeoutSeconds was not respected for exec probes: probes continued running indefinitely, even past their configured deadline, until a result was returned."
   https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes
2021-11-25 08:46:42 +01:00
Frank Kelly
1956a870ff
Fixes #173 Support both Role Binding and Cluster Role Binding dependi… (#174)
* Fixes #173 Support both Role Binding and Cluster Role Binding depending on rbac.limit_to_namespace

* Rev version

* Get Role/Cluster the right way around
pulsar-2.7.5
2021-11-12 07:56:35 -08:00
Frank Kelly
617308147d
Missing fix for #152. Bookie Service also needs the prefix on the port name (#172)
Fixes #158 (This is the second PR - see also https://github.com/apache/pulsar-helm-chart/pull/162)

### Motivation

* All non-standard port-names need a proper protocol prefix to support Istio
 https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/#explicit-protocol-selection
 
### Modifications

Add the prefix value before `bookie`
pulsar-2.7.4
2021-11-09 09:18:26 -08:00
Frank Kelly
5b10f48f5b
Fix #152 Add Helm chart support for Istio port naming (attempt 2) (#162)
Fixes #152 

### Motivation

Support prefix in front of port names to abide by Istio protocol rules
https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/#explicit-protocol-selection

### Modifications

Support adding a prefix
- pulsar -> tcp-pulsar
- pulsarssl --> tls-pulsarssl etc
pulsar-2.7.3
2021-09-10 08:56:16 +08:00
Peter Tinti
f307cc32af
updates pulsar ca name generation to use suffix making cert swappable (#141)
Updates CA name generation to be configurable allowing the swapping in of a CA.

### Motivation

We recently swapped out cert issuers and found that with the current helm chart we were unable to do a hot swap without downtime (via helm) because the CA cert name is not configurable. Being able to change the name of the CA allows us to create a new CA first -> Validate -> then swap over in follow up apply/release.

### Modifications

Adds the ability to specify the suffix used to generate the CA name (not the whole name in order to preserve back compatibility regardless of the release name.)
2021-08-25 23:14:03 -07:00
Frank Kelly
65dc68654b
ZooKeeper HTTP port should be exposed by service so we can use prometheus (#143)
Fixes #142 

### Motivation

Expose HTTP Port on ZooKeeper service so we can use Prometheus

### Modifications

Bug fix to expose HTTP port on ZooKeeper service
2021-08-25 23:13:47 -07:00
Aaron Johnson
c45813ffe5
added extraVolumes and extraVolumeMounts (#149)
Fixes #147

### Motivation
This gives the helm chart user the ability to specify a secret or other type of volume to be mounted into any of the statefulset pods

### Modifications
* Added conditionals to `bookkeeper`, `broker`, `proxy`, `toolset`, and `zookeeper` statefulsets which allow the chart user to specify extraVolumes and extraVolumeMounts for deployed pods.
* Added `extraVolumes` and `extraVolumeMounts` parameters to values.yaml
2021-08-25 23:13:27 -07:00
Thomas O'Neill
19d6ce6488
Add Support for imagePullSecrets (#140)
Fixes #125

### Motivation

The default images in the values.yaml are in docker hub. This PR allows us to provide image pull secrets for the containers which will allow us to get around Docker Hub's rate limiting if the nodes are not logged into Docker Hub.

### Modifications

Added a new template to generate `imagePullSecrets`, and included them in the deployments and statefulsets. This will only add them if they are specified under `images.imagePullSecrets`

### Verifying this change

- [] Make sure that the change passes the CI checks.
2021-08-20 17:22:50 -07:00
Lari Hotari
c3e4ea272b
Fix deprecation warning about rbac.authorization.k8s.io/v1beta1 (#135) 2021-07-03 10:56:58 +03:00
Florian Knip
f00c66a14a
Feature/fix typo in readme (#129)
Fixes #128 

### Motivation

Improve flow when reading the docs

### Modifications

fixed a typo in the docs
2021-06-23 21:13:41 -07:00
TC-robV
75169707fb
add enableAdminApi for prometheus (#121)
Fixes #<xyz>

### Motivation

would be nice to have this option here so people can run admin commands against the prometheus. 

### Modifications

added a new value and modified the deployment, taken from the official prom helm.

### Verifying this change

- [ ] Make sure that the change passes the CI checks.
2021-06-23 21:12:20 -07:00
MMeent
11a1d578dd
Fix indentation issue on checksum/config (#117)
Fixes #116

### Motivation

Theres indentation issues for the `checksum/config` annotation in these templates, which would either fail linting or not apply at all in some situations.

### Modifications

I've added indentation at the specified places such that this isn't an issue anymore.

### Verifying this change

- [ ] Make sure that the change passes the CI checks.
2021-06-23 21:11:38 -07:00
Peter Tinti
d6d240a123
Updates internal issuer cert to include duration and renew configs (#131)
### Motivation
* While component certs can be configured with a custom duration the CA cert for self-signed configuration uses default values. It can be convenient to have this certificate expire more than a month out.

### Modifications
* Updates the internal issuer `{{ .Release.Name }}-ca-tls` certificate to make `duration` and `renewBefore` configurable. Does not use `common` so that the CA can be configured to last much longer than individual components certs if desired.

### Verifying this change
- [x] Make sure that the change passes the CI checks.
2021-06-23 21:00:17 -07:00
Enrico Olivelli
6d0db35216
Update to Pulsar 2.7.2 (#119)
Co-authored-by: Enrico Olivelli <eolivelli@datastax.com>
pulsar-2.7.2
2021-06-03 11:31:47 +03:00
Yong Zhang
0816ac2dfd
Reduce the TLS common name length (#115)
---

*Motivation*

Reduce the TLS command name to avoid getting a too long name
that could not generate a certificate.
2021-04-23 12:43:44 +08:00
Jean Helou
ba356e5df7
makes cert-manager apiVersion configurable (#107)
This commit let's users override the apiVersion referenced in this
chart so that the chart can be used with newer cert-manager releases.
(script/cert-manager/install-cert-manager.sh installs 0.13.0 when
current version is 1.2.0...)

Fixes #68

### Motivation

cert-manager apiVersion changed after cert-manager 1.0.0 was released, which prevents the chart from provisionning certificates with newer cert-manager installation because of an incompatible apiVersion.

I have a cluster with cert-manager >1.0.0 installed, making `apiVersion` overridable makes it easy for me to install pulsar on that cluster

### Modifications

I introduced the value `certs.internal_issuer.apiVersion`, which by default uses the apiVersion that was previously hardcoded (`cert-manager.io/v1alpha2`) 
I replaced all occurrences of that apiVersion by a reference to the value so that users can override it to `cert-manager.io/v1` if they have a newer version of cert-manager installed.

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2021-03-16 00:44:38 -07:00
Yong Zhang
e0903c633c
Bump pulsar version to 2.7.1 (#109)
### Motivation

Release with pulsar 2.7.1

### Modification

- update pulsar version from 2.7.0 to 2.7.1
- add a script for updating the pulsar version
pulsar-2.7.1
2021-03-16 00:43:30 -07:00
Miecio
c059ea25d8
Feat: Dynamic superusers configuration (#104)
Adds dynamic superusers configuration

### Motivation

Allow dynamic superusers management. Adding new superuser entry to `.Values.auth.superUsers` will results in adding concatenated list to config

### Modifications

Change static list to dynamic one
2021-02-09 00:59:54 -08:00
wuYin
67818a48cb
Support common volume for journal and ledgers (#93)
### Motivation

In some case, my k8s node only have 1 large capacity ssd, for deploying 1 bookie, I need:

- Partition the ssd into 2 disks, and make 2 pv over it.
- Just make 1 pv over it, but journal & ledgers under same mount path (this PR did)

Both can't isolate IO for journal & ledgers, so I prefer the second one for reusability.


### Modifications

values.yaml
  - add `useSingleCommonVolume` option, default false

bookkeeper-statefulset.yaml
   - mount the only PV to path `/pulsar/data/bookkeeper`
   - use configured common storageClassName

bookkeeper-storageclass.yaml
  - use configured provisioner for the common storageClass 

### Others
This may not be an issue for everyone, if it's not necessary to merge, I'll just use it locally

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2021-01-30 09:28:45 -08:00
wangyufan
d73361eb1e
fix broker configmap forbidden (#95)
Fixes #94

### Motivation

fix `io.kubernetes.client.openapi.ApiException: Forbidden`

### Modifications

fix typo

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2021-01-30 09:28:00 -08:00
Miecio
b24ba1adf5
Fix namespace handling and missing dnsNames (#99)
Fixes for wrong namespace handling in some RBAC and missing dnsNames for TLS

### Motivation

Fixes old unused handling of namespace name in RBAC for autorecovery and bookkeeper.
Fixes Helm exception of missing key when not defining TLS dnsNames

### Modifications

Use namespace template in RBAC definitions for bookkeeper and autorecovery. Add if around every `toYaml .Values.tls.bookie.dnsNames` clause in TLS certs definitions.

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2021-01-30 09:27:18 -08:00
Miecio
025b263206
Extend podmonitor and add relabels (#100)
### Motivation

As I wanted to use [streamnative/apache-pulsar-grafana-dashboard](https://github.com/streamnative/apache-pulsar-grafana-dashboard) with this helm chart and own cluster wide Prometheus stack I decided that use of PodMonitor CRD is a good way. Unfortunately prometheus config has some metrics relabelings that are required by grafana dashboard. I decied to port them directly to PodMonitor definition

### Modifications

* Added missing PodMonitor for autorecovery
* Port relabelings from `prometheus-configmap.yaml` to each PodMonitor

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2021-01-30 09:24:21 -08:00
Miecio
23ba8ac948
Fix for missing PSP for bookie initialize and other (#101)
### Motivation

When using standard bookkeeper installation on PSP cluster initialization fails because has to be started as root

### Modifications

Add same ServiceAccount and SecurityContext for bookkeeper-cluster-initialize as in bookkeeper specyfication.

UPDATE: Seems that when using in cluster TLS encryption other components also require RW access to root FS, I added PSP for proxy, zookeepe, broker and toolset

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2021-01-30 09:22:52 -08:00
Miloš Matijašević
c2f672881e
Updating pods on configmap change (#73)
Fixes #71 

### Motivation

Pods are not restarting when config maps are changed after changing values.yaml file, so they need to be restarted manually in order to pick up new values from config map. 

### Modifications

As I mentioned `restartPodsOnConfigMapChange` flag for each component is added in values.yaml file whether to restart pods on configmap change or not, default is `false`.
In statefulset templates for each component is added part which is adding annotation that contains hash of corresponding configmap if `restartPodsOnConfigMapChange` is `true`, which will cause pods to restart if corresponding configmap has been changed (https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments).

### Verifying this change

- [ ] Make sure that the change passes the CI checks.
2021-01-07 21:28:11 -08:00
Miecio
667e634af0
Add basic PSP and RBAC for core components (#87)
Add PSP and add/modify RBAC. I'm open for all discussion.

### Motivation

On clusters which use PSP and restrictive default policy pulsar cannot be installed, because it uses root user and requires writable container root directory. Additionally default RBAC for broker are too permissive (usage of ClusterRoleBinding) in my opinion.

### Modifications

Add PSP and RBAC for bookkeeper and autorecovery to add
exception to allow startup even in secure environment
where containers cannot access RW on root by default.

Add option for limiting broker ClusterRoleBinding
to single namespace by replacing to RoleBinding

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2021-01-07 21:26:44 -08:00
Jiří Pinkava
8d5339f9ff
Allow use of existing secret for pulsar manager credentials (#69)
Signed-off-by: Jiří Pinkava <jiri.pinkava@rossum.ai>

Co-authored-by: Jiri Pinkava <jiri.pinkava@rossum.ai>
pulsar-2.7.0-1
2021-01-07 21:24:52 -08:00
Peter Tinti
6fbda8052c
Allows appending dnsNames to self-signed certs (#74)
Co-authored-by: Sijie Guo <sijie@apache.org>

Fixes inability to validate self-signed certs from external clients

### Motivation

Currently self-signed certificates can only be used inside of the same cluster as they are labeled with internal dns names without the possibility of appending additional values. Some use-cases require the connection of external clients. This PR aims to allow users add additional dnsNames (IP or domain) to the self-signed certificates.

### Modifications

* Adds the ability to add `dnsNames` to self-signed certificates to any component like so:

```yaml
tls:
  enabled: true
  proxy:
    enabled: true
    dnsNames:
      - test.example.com

```

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2020-12-12 16:34:35 +08:00
lipenghui
f6705f0aec
Bump Pulsar 2.7.0 (#88)
Co-authored-by: Sijie Guo <sijie@apache.org>
pulsar-2.7.0
2020-12-03 20:14:05 -08:00
Jean Helou
6c9856a1af
Use .Release.Namespace by default to handle namespaces (#80)
It remains possible to override the current release namespace by setting
the `namespace` value though this may lead to having the helm metadata
and the pulsar components in different namespaces

Fixes #66

### Motivation

Trying to deploy the chart in a namespace using the usual helm pattern fails for example
```
kubectl create ns pulsartest
helm upgrade --install pulsar -n pulsartest apache/pulsar
Error: namespaces "pulsar" not found
```
fixing that while keeping the helm metadata and the deployed objects in the same namespace requires declaring the namespace twice 
```
kubectl create ns pulsartest
helm upgrade --install pulsar -n pulsartest apache/pulsar --set namespace=pulsartest
Error: namespaces "pulsar" not found
```
This is needlessly confusing for newcomers who follow the helm documentation and is contrary to helm best practices.

### Modifications

I changed the chart to use the context namespace `.Release.Namespace` by default while preserving the ability to override that by explicitly providing a namespace on the commande line, with the this modification both  examples behave as expected
 
### Verifying this change

- [x] Make sure that the change passes the CI checks.
pulsar-2.6.2-2
2020-12-03 19:32:05 -08:00
Jiří Pinkava
eb63a19964
Local mode for kubernetes object generators (#75)
This allows operation in environemnts where direct installation of objects into
kubernetes cluster is not desired or possible. For example when using sealedsecrets
or SOPS, where the secrets are firs encrypted and then commited into repository
and deployed latter by some other deployment system.

Co-authored-by: Jiří Pinkava <jiri.pinkava@rossum.ai>
2020-11-12 20:32:40 -07:00
xiaolong ran
ebc40c3382
Bump the image version to 2.6.2 (#81)
Signed-off-by: xiaolong.ran <rxl@apache.org>

### Motivation

Bump the image version to 2.6.2

### Verifying this change

- [x] Make sure that the change passes the CI checks.
pulsar-2.6.2-1
2020-11-12 20:31:41 -07:00
xiaolong ran
36e13e1078
Upgrade chart-testing-action to 2.0.0 (#83)
Signed-off-by: xiaolong.ran <rxl@apache.org>

### Motivation

The lint ci error as follows:

```
Linting chart 'pulsar => (version: "2.6.2-1", path: "charts/pulsar")'
Checking chart 'pulsar => (version: "2.6.2-1", path: "charts/pulsar")' for a version bump...
Old chart version: 2.6.1-2
New chart version: 2.6.2-1
Chart version ok.
Validating /workdir/charts/pulsar/Chart.yaml...
Validation success! 👍
Validating maintainers...
Error: Error linting charts: Error processing charts
------------------------------------------------------------------------------------------------------------------------
 ✖︎ pulsar => (version: "2.6.2-1", path: "charts/pulsar") > Error validating maintainer 'The Apache Pulsar Team': 404 Not Found
------------------------------------------------------------------------------------------------------------------------
Error linting charts: Error processing charts
```

### Modifications

Upgrade `chart-testing-action` to 2.0.0

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2020-11-13 10:51:44 +08:00
Sijie Guo
669af788ff
Fix "unknown apiVersion: kind.sigs.k8s.io/v1alpha3" (#76)
* Fix "unknown apiVersion: kind.sigs.k8s.io/v1alpha3"

*Motivation*

The api version `kind.sigs.k8s.io/v1alpha3` is not available anymore for kind clusters.
So all the CI actions are broken now. This PR fix the issue.

Additionally it adds a helm chart lint job to lint the chart changes.

* Trigger CI when kind cluster build script is changed
2020-11-11 11:48:42 -07:00
Naveen Ramanathan
fb4c44f449
changed publishNotReadyAddresses to (#64)
### Motivation

* ```publishNotReadyAddresses``` is a service spec and not a service annotation. This is mentioned in the K8s API docs at https://v1-17.docs.kubernetes.io/docs/reference/generated/kubernetes-api/v1.15/#servicespec-v1-core

### Modifications

* Modified ```publishNotReadyAddresses``` from annotation to service spec

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2020-10-15 18:42:13 +08:00
Naveen Ramanathan
bf5db574d1
Make forceSync by default as "yes" (#63)
### Motivation

* It's not recommended to run a production zookkeeper cluster with forceSync as "no".  This is also mentioned in the forceSync section in https://pulsar.apache.org/docs/en/next/reference-configuration/#zookeeper

### Modifications

* Removed ```-Dzookeeper.forceSync=no``` from ```values.yaml``` as default ```forceSync``` is ```yes```.
2020-09-22 09:47:41 -05:00
Thomas O'Neill
bf349a8c05
Ingress optional hostname (#54)
Fixes #50 

### Motivation
The host option is not required to setup an ingress, so I made it an optional value
### Modifications

*Describe the modifications you've done.*
Made setting the host optional.
2020-09-21 13:16:20 -05:00
Elad Dolev
5049d3564a
add support for multiple clusters (#60)
Co-authored-by: Elad Dolev <elad@firebolt.io>

### Motivation

Give the ability to deploy multi-cluster instance on K8s clusters with non-default `clusterDomain`, and connect to external configuration-store

### Modifications

- give the ability to change cluster's name
- give the ability to change `clusterDomain`
- fix external configuration store functionality
- use broker ports variables
- use label templates, and add `component` label in several places

### Verifying this change

- [x] Make sure that the change passes the CI checks.
pulsar-2.6.1-2
2020-09-08 10:06:30 +08:00
Lari Hotari
6c2edba8b1
Get OS signals passed to container process by using shell built-in "exec" (#59)
### Changes 

- using "exec" to run a command replaces the shell process with the executed process
- this is required so that the process running in the container is able to receive OS signals
  - explained in https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
    and https://docs.docker.com/engine/reference/builder/#entrypoint
- receiving SIGTERM signal is required for graceful shutdown. This is explained in https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html 

This change might fix issues such as https://github.com/apache/pulsar/issues/6603 . One expectation of this fix is that graceful shutdown would allow Pulsar components such as a bookies to deregistered from Zookeeper properly before shutdown. 

### Motivation

Dockerfile best practices mention that "exec" should be used so that the process running in a container can receive OS signals. This is explained in https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
    and https://docs.docker.com/engine/reference/builder/#entrypoint .  Kubernetes documention explains pod termination in https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination : "Typically, the container runtime sends a TERM signal to the main process in each container. Once the grace period has expired, the KILL signal is sent to any remaining processes, and the Pod is then deleted from the API Server ."
Currently some issues while running Pulsar are caused by the lack of graceful shutdown. Graceful shutdown isn't happening at all since the Pulsar processes never receive the TERM signal that would allow graceful shutdown. This PR fixes that.

This PR was inspired by https://github.com/kafkaesque-io/pulsar-helm-chart/pull/31
2020-08-30 23:05:49 -06:00
冉小龙
4178c70d90
Bump the image version to 2.6.1 (#57)
Signed-off-by: xiaolong.ran rxl@apache.org

Motivation
Follow release process and bump the image version to 2.6.1
pulsar-2.6.1-1
2020-08-21 22:50:27 +08:00
Thomas O'Neill
b44b523c8a
Allow initialization to be set (#53)
Fixes #47 

### Motivation
Only create the initialize job on install. 

### Modifications

- Added an initialize value that can be set to true on install, matching the documentation in the README.md
2020-08-13 10:20:01 -07:00