40 Commits

Author SHA1 Message Date
Bruno Domenici
a9f2ba76ae
OpenID: introducing support for OpenID configuration (#509)
* feat!(openid): introducing support for openid configuration

BREAKING CHANGE: provider configuration changed from auth.authentication.provider to auth.authentication.jwt.enabled

* add upgrading to 4.1.0

* add validation for deprecated values

* add openid CI with keycloak

* fix chart-testing lint new-line-at-end-of-file

* fix keycloak dependency repository

* fix keycloak repository

* fix yaml to json convert error

* disable keycloak to validate github actions before re-enable it

* disable openid test scenario

* disable keycloak in values

* enable keycloak without authentication and authorization

* add openid test scenario

* disable test scenario other than openid

* enable all test scenario

* disable functions component

* create openid resources

* test truncate command

* test truncate command

* change client_secret generator

* change client_secret generator

* test python

* fix script

* fix script

* print python result

* test python

* test python

* fix client_secret generation

* fix create openid resources

* fix secret name

* fix mount keycloak config

* fix keycloak service

* exclude keycloak from chart

* add license

* add license

* wait keycloak is alive

* fix keycloak chart install namespace

* add test pulsar real openid config

* fix keycloak issuer url

* fix pod name

* remove check keycloak alive

* check realm pulsar openid configuration

* change keycloak service

* remove test keyclock service

* remove selector to get all pod log

* wait keycloak is alive

* check keycloak realm pulsar urls

* wait until keycloak is ready

* add wait timeout

* fix realm pulsar name

* add log to debug

* add openid for toolset

* set authorization

* set authorization

* fix client template filename

* fix install keycloak

* disable authorization

* debug sub claim value

* fix sub claim value

* cleanup

* enable all build

---------

Co-authored-by: glecroc <guillaume.lecroc@cnp.fr>
2025-05-20 14:09:12 +03:00
Haim Kortovich
77ec4cedfb
Add appAnnotations for all statefulsets (#604) 2025-05-07 09:05:19 +03:00
Lari Hotari
302db43e91
Remove PSP support (#591) 2025-03-08 12:00:35 -08:00
Lari Hotari
75119dd6d7
Remove Prometheus scrape annotations when podmonitors are enabled (#590) 2025-03-07 09:51:06 -08:00
Lari Hotari
f98ee7d69c
Replace ">" with "|" to avoid Go Yaml issue go-yaml/yaml#789 (#582) 2025-03-04 02:21:39 -08:00
Alejandro Ramirez
0b6b03002c
Fix OOM issue on broker wait-zookeeper-ready initContainer (#568) 2025-02-18 22:26:39 -08:00
Yuwei Sung
c6ce11a9b7
Add support for using Oxia as the metadata store for Pulsar and BookKeeper (#544)
Co-authored-by: Lari Hotari <lhotari@apache.org>
2024-11-21 16:52:20 -08:00
lenglet-k
db20c2bfa6
fix: broker extraEnv variable (#540)
* fix: broker extraEnv variable

* fix: comment extraEnv for broker as default values

* fix(typo): rename extreEnvs to extraEnvs
2024-10-18 00:07:24 -07:00
Shu.Wang
a45bc4bfe1
Add topologyspreadconstraint to deploy pods in sts cross different az evenly (#526)
Signed-off-by: Wang, Shu <shu.wang@fmr.com>
2024-09-26 21:37:15 -07:00
ludmanl
54401c0b9a
feat: Support to customize broker podManagementPolicy from values.yaml (#525) 2024-09-03 03:47:52 -07:00
Starry
093fa273f8
Add initContainers to templates (#516) 2024-08-05 09:40:55 -07:00
Lari Hotari
70f36ffe43
Add timeouts for cluster metadata initialization and for init containers (#218)
- Add timeouts for waiting for zk and bk to become available.
- If the waiting gets stuck for some reason, the Pulsar deployment never
  becomes starts the broker services.
  - timeouts will help failures recover eventually
2024-06-20 10:07:48 -07:00
Lari Hotari
023f902a02
Allow specifying default pull policy and functions pull policy (#507) 2024-06-12 04:16:48 -07:00
Lari Hotari
bd8bc633df
Change default statusFilePath to /pulsar/logs/status (#489)
* Change default statusFilePath to /pulsar/logs/status

* Write OK to statusFilePath
2024-04-15 05:41:17 -07:00
Lari Hotari
fdec9c69ef
Use podManagementPolicy OrderedReady for Broker sts when Functions are enabled (#474)
* Use podManagementPolicy OrderedReady for Broker sts when Functions are enabled

* Don't change podManagementPolicy when the sts already exists

* Fix template issue

* Fix apiVersion
2024-03-26 10:49:33 -07:00
Nathan Clayton
b4241f984b
Update broker statefulset to check if AWS keys secret name is defined before adding to environment. (#466) 2024-03-03 10:38:31 +02:00
Martin
4daf6d88a2
grouped init containers (#441) 2024-01-26 03:09:57 -08:00
Lari Hotari
e49bd32378 Fix indent for Tiered storage offload environment 2024-01-17 18:11:30 +02:00
pellicano
cfa156f738
Tiered Storage config (#205)
* Add tiered storage config

* Check Tiered Storage on README

* GitHub PR #205 changes (1st round)

Remove <= 2.6.0 configs.
Add missing GCS secret volumeMount.
Update GCS example name.

* Cleanup comments

* Bump chart version

* GitHub PR #205 changes (2nd round)

Moved storageOffload under broker section.
Fixed some typos.
Added AWS S3 IRSA annotation comment.

* GitHub PR #205  changes (3rd round)

Moved AWS and Azure credentials into K8S secrets using same StreamNative Helm Chart approach.

* Trim trailing spaces

---------

Co-authored-by: Lari Hotari <lhotari@apache.org>
Co-authored-by: Marcelo Pellicano <mpellicanodeoliveira@bluecatnetworks.com>
2024-01-17 03:06:16 -08:00
Martin
04b8ccebb1
Resource specs for Init Containers (#428)
* add missing section in values.yaml for pulsar_metadata resources

* add resources to all init containers and an additional section to specify them in values.yaml

* increase memory defaults for init containers

* remove empty lines

* Add newline to end of file
2024-01-15 07:53:56 -08:00
Frank Kelly
0b2d9b4d5d
Decouple Service account creation from PodSecurityPolicy (#387)
* Proposal: service accounts creation should be decoupled from PodSecurityPolicy.

* Rename *-rbac.yaml to *-psp.yaml and move service account to *-service-account.yaml

* Test with psp enabled

Co-authored-by: Lari Hotari <lhotari@apache.org>
2023-12-21 04:40:54 -08:00
Michał Koziorowski
fba51bd87e
Lowered BOOKIE_MEM and PULSAR_MEM in init containers (#386)
* Lowered BOOKIE_MEM and PULSAR_MEM in init containers. Default BOOKIE_MEM and PULSAR_MEM settings from conf/pulsar_env.sh and conf/bkenv.sh (-Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g) are too high for low-memory systems.
2023-12-11 00:50:08 -08:00
Tomasz Jegorow
042fd5b6d4
Configure custom topologyKey for podAntiAffinity (#351) 2023-07-12 18:19:49 +03:00
Atkins
b30eb6fff8
Improve HPA (#354)
* Use `autoscaling/v2` if Kubernetes version >= 1.23

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* Disable replicas when autoscaling enabled

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

---------

Signed-off-by: Atkins Chang <atkinschang@gmail.com>
2023-07-12 18:18:19 +03:00
Lari Hotari
25f355e6e2
Use appVersion as default tag for Pulsar images (#200)
Co-authored-by: Michael Marshall <mmarshall@apache.org>

### Motivation

There was a suggestion [in a dev mailing list discussion](https://lists.apache.org/thread/bgkvcyt1qq6h67p2k8xwp89xlncbqn3d) that the Helm chart's appVersion should be used as the default image tag.

### Additional context

There are some limitations in Helm. It is not possible to set "appVersion" from the command line. There's in an open feature request https://github.com/helm/helm/issues/8194 to add such a feature to Helm.

### Modifications

- change default values.yaml and set the tags for the images that use the Pulsar image to an empty value
- add "defaultPulsarImageTag" to values.yaml
- add a helper template "pulsar.imageFullName" that contains the logic to fall back to .Values.defaultPulsarImageTag and if it's not set, falling back to .Chart.AppVersion
- use the helper template in all other templates that require the logic
2022-10-17 15:42:58 -05:00
Marvin Cai
c6ab1d18e3
Support defining extra env for broker and proxy statefulsset. (#273) 2022-06-20 07:59:43 -07:00
Aaron Johnson
cee3b5c5e6
added additionalCommand parameter (#150)
Co-authored-by: Aaron Johnson <aaron.johnson@crowdstrike.com>
2022-01-05 10:26:55 -06:00
Lari Hotari
b4b2fa7b80
[Security] Workaround for CVE-2021-44228 Log4J RCE when Log4J >= 2.10.0 (#186)
* [Security] Workaround for CVE-2021-44228 Log4J RCE when Log4J >= 2.10.0

- prevents the exploit by disabling message pattern lookups

* Bump the chart version
2021-12-10 18:30:01 +02:00
Lari Hotari
a16c6bbf19
Make k8s probe timeoutSeconds configurable and set default to 5s for k8s 1.20+ compatibility (#179)
- set to 5 seconds by default

- address compatibility with Kubernetes 1.20+. This impacts "bin/pulsar-zookeeper-ruok.sh" exec probe used in ZK.
  "Before Kubernetes 1.20, the field timeoutSeconds was not respected for exec probes: probes continued running indefinitely, even past their configured deadline, until a result was returned."
   https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/#configure-probes
2021-11-25 08:46:42 +01:00
Frank Kelly
5b10f48f5b
Fix #152 Add Helm chart support for Istio port naming (attempt 2) (#162)
Fixes #152 

### Motivation

Support prefix in front of port names to abide by Istio protocol rules
https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/#explicit-protocol-selection

### Modifications

Support adding a prefix
- pulsar -> tcp-pulsar
- pulsarssl --> tls-pulsarssl etc
2021-09-10 08:56:16 +08:00
Aaron Johnson
c45813ffe5
added extraVolumes and extraVolumeMounts (#149)
Fixes #147

### Motivation
This gives the helm chart user the ability to specify a secret or other type of volume to be mounted into any of the statefulset pods

### Modifications
* Added conditionals to `bookkeeper`, `broker`, `proxy`, `toolset`, and `zookeeper` statefulsets which allow the chart user to specify extraVolumes and extraVolumeMounts for deployed pods.
* Added `extraVolumes` and `extraVolumeMounts` parameters to values.yaml
2021-08-25 23:13:27 -07:00
Thomas O'Neill
19d6ce6488
Add Support for imagePullSecrets (#140)
Fixes #125

### Motivation

The default images in the values.yaml are in docker hub. This PR allows us to provide image pull secrets for the containers which will allow us to get around Docker Hub's rate limiting if the nodes are not logged into Docker Hub.

### Modifications

Added a new template to generate `imagePullSecrets`, and included them in the deployments and statefulsets. This will only add them if they are specified under `images.imagePullSecrets`

### Verifying this change

- [] Make sure that the change passes the CI checks.
2021-08-20 17:22:50 -07:00
Miecio
23ba8ac948
Fix for missing PSP for bookie initialize and other (#101)
### Motivation

When using standard bookkeeper installation on PSP cluster initialization fails because has to be started as root

### Modifications

Add same ServiceAccount and SecurityContext for bookkeeper-cluster-initialize as in bookkeeper specyfication.

UPDATE: Seems that when using in cluster TLS encryption other components also require RW access to root FS, I added PSP for proxy, zookeepe, broker and toolset

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2021-01-30 09:22:52 -08:00
Miloš Matijašević
c2f672881e
Updating pods on configmap change (#73)
Fixes #71 

### Motivation

Pods are not restarting when config maps are changed after changing values.yaml file, so they need to be restarted manually in order to pick up new values from config map. 

### Modifications

As I mentioned `restartPodsOnConfigMapChange` flag for each component is added in values.yaml file whether to restart pods on configmap change or not, default is `false`.
In statefulset templates for each component is added part which is adding annotation that contains hash of corresponding configmap if `restartPodsOnConfigMapChange` is `true`, which will cause pods to restart if corresponding configmap has been changed (https://helm.sh/docs/howto/charts_tips_and_tricks/#automatically-roll-deployments).

### Verifying this change

- [ ] Make sure that the change passes the CI checks.
2021-01-07 21:28:11 -08:00
Jean Helou
6c9856a1af
Use .Release.Namespace by default to handle namespaces (#80)
It remains possible to override the current release namespace by setting
the `namespace` value though this may lead to having the helm metadata
and the pulsar components in different namespaces

Fixes #66

### Motivation

Trying to deploy the chart in a namespace using the usual helm pattern fails for example
```
kubectl create ns pulsartest
helm upgrade --install pulsar -n pulsartest apache/pulsar
Error: namespaces "pulsar" not found
```
fixing that while keeping the helm metadata and the deployed objects in the same namespace requires declaring the namespace twice 
```
kubectl create ns pulsartest
helm upgrade --install pulsar -n pulsartest apache/pulsar --set namespace=pulsartest
Error: namespaces "pulsar" not found
```
This is needlessly confusing for newcomers who follow the helm documentation and is contrary to helm best practices.

### Modifications

I changed the chart to use the context namespace `.Release.Namespace` by default while preserving the ability to override that by explicitly providing a namespace on the commande line, with the this modification both  examples behave as expected
 
### Verifying this change

- [x] Make sure that the change passes the CI checks.
2020-12-03 19:32:05 -08:00
Elad Dolev
5049d3564a
add support for multiple clusters (#60)
Co-authored-by: Elad Dolev <elad@firebolt.io>

### Motivation

Give the ability to deploy multi-cluster instance on K8s clusters with non-default `clusterDomain`, and connect to external configuration-store

### Modifications

- give the ability to change cluster's name
- give the ability to change `clusterDomain`
- fix external configuration store functionality
- use broker ports variables
- use label templates, and add `component` label in several places

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2020-09-08 10:06:30 +08:00
Lari Hotari
6c2edba8b1
Get OS signals passed to container process by using shell built-in "exec" (#59)
### Changes 

- using "exec" to run a command replaces the shell process with the executed process
- this is required so that the process running in the container is able to receive OS signals
  - explained in https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
    and https://docs.docker.com/engine/reference/builder/#entrypoint
- receiving SIGTERM signal is required for graceful shutdown. This is explained in https://pracucci.com/graceful-shutdown-of-kubernetes-pods.html 

This change might fix issues such as https://github.com/apache/pulsar/issues/6603 . One expectation of this fix is that graceful shutdown would allow Pulsar components such as a bookies to deregistered from Zookeeper properly before shutdown. 

### Motivation

Dockerfile best practices mention that "exec" should be used so that the process running in a container can receive OS signals. This is explained in https://docs.docker.com/develop/develop-images/dockerfile_best-practices/
    and https://docs.docker.com/engine/reference/builder/#entrypoint .  Kubernetes documention explains pod termination in https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#pod-termination : "Typically, the container runtime sends a TERM signal to the main process in each container. Once the grace period has expired, the KILL signal is sent to any remaining processes, and the Pod is then deleted from the API Server ."
Currently some issues while running Pulsar are caused by the lack of graceful shutdown. Graceful shutdown isn't happening at all since the Pulsar processes never receive the TERM signal that would allow graceful shutdown. This PR fixes that.

This PR was inspired by https://github.com/kafkaesque-io/pulsar-helm-chart/pull/31
2020-08-30 23:05:49 -06:00
Thomas O'Neill
207d697bed
Fix zookeeper antiaffinity (#52)
Fixes #39 

### Motivation

The match expression for the "app" label was incorrect breaking the antiaffinity since they would never match. Fixing this makes the podAntiAffinity work, but now requires at least N nodes to be in the cluster where N = largest replica set with affinity. Added the option to set the affinity type to preferredDuringSchedulingIgnoredDuringExecution where it will try to follow the affinity, but will still deploy a pod if it needs to break it. 

### Modifications

- Fixed app matchExpression 
- Added option to set the affinity type 
- bumped chart version

### Verifying this change

- [X] Make sure that the change passes the CI checks.
2020-08-13 10:19:01 -07:00
Sijie Guo
1c8a434ef6
Don't substitute environment variables (#28)
*Motivation*

environment variables are already taken by bash scripts. We don't need to substitute them.
2020-06-25 20:24:03 -07:00
Sijie Guo
0338d17b89
Publish chart index to gh-pages branch (#3)
*Motivation*

Release helm chart when new tags are created
2020-04-21 02:44:58 -07:00