114 Commits

Author SHA1 Message Date
Starry
093fa273f8
Add initContainers to templates (#516) 2024-08-05 09:40:55 -07:00
Lari Hotari
70f36ffe43
Add timeouts for cluster metadata initialization and for init containers (#218)
- Add timeouts for waiting for zk and bk to become available.
- If the waiting gets stuck for some reason, the Pulsar deployment never
  becomes starts the broker services.
  - timeouts will help failures recover eventually
2024-06-20 10:07:48 -07:00
Lari Hotari
023f902a02
Allow specifying default pull policy and functions pull policy (#507) 2024-06-12 04:16:48 -07:00
Lari Hotari
47c2ac442a
Add defaultPulsarImageRepository configuration (#503)
- makes it easier to use a custom image
2024-06-05 04:20:16 -07:00
Massimiliano Mirelli
6e84409b48
Support NodePort Proxy service (#500)
* Enables nodeport support for the proxy

* Correct indentation and remove null `nodePort`

Removing null `nodePort` causes k8s to pick up a random port

* Address review comment

https://github.com/apache/pulsar-helm-chart/pull/500/files#r1605762312
2024-06-04 08:46:16 -07:00
Massimiliano Mirelli
cb5c44f8ec
Allow broker's service clusterIP customisation (#498)
* Allow broker's service clusterIP customisation

This customisation is useful to configure headless vs non-headless
broker's service. The default is headless broker service, i.e. a
service for which kubernetes  does not allocate an IP
address (https://kubernetes.io/docs/concepts/services-networking/service/#type-clusterip). A
headless service is a very simple type of service that doesn't seem to work well
when pulsar service is exposed by pulsar-proxy via a nodeport.

Addresses #497.

* Address review comments

https://github.com/apache/pulsar-helm-chart/pull/498/files#r1605762934
and https://github.com/apache/pulsar-helm-chart/pull/498/files#r1605763245

* Move doc to Values.broker.service
2024-06-04 08:45:14 -07:00
Lari Hotari
59f6f74fd7
Fix prometheus node-exporter crashloop (#488) 2024-04-12 03:10:24 -07:00
Lari Hotari
ee4b7a7988
Increase default Prometheus scrape interval to 60s (#487) 2024-04-11 07:35:57 -07:00
Martin
7c7ca4a7bc
enable message peeking (#486) 2024-04-10 23:20:37 -07:00
Martin
347326e0c3
Fix pulsar-manager persistence (#485)
- only setup environment in pulsar manager if broker is deployed
- fix indent
- enable persistence for manager and move configs around
2024-04-03 21:28:46 -07:00
Lari Hotari
cc0a1acf22
Disable functions by default in values.yaml (#483) 2024-03-26 23:17:40 +01:00
doug-ba
9929b80b3c
add ability to use separate disk for zookeeper tx log (#476)
* add ability to use separate disk for zookeeper tx log

* Use absolute path

---------

Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com>
2024-03-26 07:51:31 -07:00
doug-ba
bc5862d4b0
pulsar-manager adding support for existing secret (#478) 2024-03-26 05:26:37 -07:00
doug-ba
3dee8dfe3b
making .ReleaseIsInstall optional for init jobs (#480)
* making .ReleasIsInstall optional for init jobs

* initialize simplifying an if condition based on feedback
2024-03-25 22:26:32 -07:00
Lari Hotari
17a4239733
Remove buggy and useless function-worker-config-map (#462)
Fixes #56
2024-02-21 13:47:23 -08:00
Lari Hotari
0e3251bea8
Remove deprecated "extra" key to configure components, also remove dashboard that has been replaced (#461)
- the "extra" key has been deprecated a long time ago
- the dashboard is outdated and there's a replacement with kube-prometheus-stack and #439
2024-02-21 04:53:29 -08:00
csthomas1
cb269bbaf3
Feature/pulsar manager v0.2.0 with jwt setup admin account creation (#219)
* Added support for JWT secretref and key volume mount. Added admin user auto-creation.

* Removed variables accidentally re-added and comments no longer relevant

* Enabling pulsar manager test w/ manager-admin superuser for symmetric and asymmetric jwt tests

* Added verification of communication with broker to ci test-pulsar-manager

* Fixing error on line 115 of helm.sh

* More fixes

* Adding echo of envs and tenants

* Fixing LOGIN_JSESSIONID variable name
2024-02-21 04:25:23 -08:00
Victor Fauth
29ea17b3fc
Enable persistence for pulsar-manager (#343)
* Enable persistence for pulsar-manager

* Upgrade to v0.4.0 version of pulsar-manager to get required fix

- contains https://github.com/apache/pulsar-manager/pull/501
  in https://github.com/apache/pulsar-manager/releases/tag/v0.4.0

---------

Co-authored-by: Victor Fauth <victor.fauth@thalesgroup.com>
Co-authored-by: Lari Hotari <lhotari@apache.org>
2024-02-15 01:27:40 -08:00
Martin
d0b784a953
Feature/pulsar manager initialize (#457)
* add better pulsar manager integration and init along with tests & docs

* fix pulsar manager startup args

* update pulsar manager service to ClusterIP + remove duplicate
2024-02-14 10:13:54 -08:00
Lari Hotari
9cbe03c7ee
Improve Bookkeeper default configuration (#454)
- remove minimal memory settings
- add more optimal data compaction settings
2024-01-31 03:21:04 -08:00
Martin
4daf6d88a2
grouped init containers (#441) 2024-01-26 03:09:57 -08:00
Lari Hotari
8d2d567b30
Remove pulsar_detector dash board (#446)
- not applicable for Apache Pulsar Helm chart's Pulsar deployment
2024-01-26 03:09:11 -08:00
Martin
8cd3a04812
expose admin port of pulsar manager in service (#440) 2024-01-24 23:32:25 -08:00
Lari Hotari
65a5fc0002 Fix typo in Apache License 2.0 abbrev, should be AL 2.0 2024-01-24 21:44:24 +02:00
Lari Hotari
d486e4a42d
Add default configuration for Pulsar Grafana dashboards (#439)
* Add default configuration for dashboards
2024-01-24 11:12:57 -08:00
Lari Hotari
1cb83398c8
Don't use TLS from function instances to brokers by default (#435)
- Function instances don't currently have the TLS CA cert available
2024-01-17 21:04:43 -08:00
pellicano
cfa156f738
Tiered Storage config (#205)
* Add tiered storage config

* Check Tiered Storage on README

* GitHub PR #205 changes (1st round)

Remove <= 2.6.0 configs.
Add missing GCS secret volumeMount.
Update GCS example name.

* Cleanup comments

* Bump chart version

* GitHub PR #205 changes (2nd round)

Moved storageOffload under broker section.
Fixed some typos.
Added AWS S3 IRSA annotation comment.

* GitHub PR #205  changes (3rd round)

Moved AWS and Azure credentials into K8S secrets using same StreamNative Helm Chart approach.

* Trim trailing spaces

---------

Co-authored-by: Lari Hotari <lhotari@apache.org>
Co-authored-by: Marcelo Pellicano <mpellicanodeoliveira@bluecatnetworks.com>
2024-01-17 03:06:16 -08:00
Lari Hotari
5d1275e71d
Change default value for rbac.limit_to_namespace and functions.rbac.limit_to_namespace (#418) 2024-01-17 01:02:07 -08:00
Lari Hotari
f20756abb8
Support configuring "proxyRoles" in broker (#430)
* Add support for proxyRoles

* Test produce and consume via Pulsar proxy
2024-01-16 04:48:58 -08:00
lenglet-k
e54354df87
feat: add externalTrafficPolicy and loadBalancerSourceRanges (#405)
* feat: add externalTrafficPolicy and loadBalancerSourceRanges
2024-01-15 12:59:58 -08:00
lenglet-k
c094322904
feat: add imagePullSecrets in Job (#404)
* feat: add imagePullSecrets on Job

* feat: update values.yaml
2024-01-15 12:28:05 -08:00
Martin
04b8ccebb1
Resource specs for Init Containers (#428)
* add missing section in values.yaml for pulsar_metadata resources

* add resources to all init containers and an additional section to specify them in values.yaml

* increase memory defaults for init containers

* remove empty lines

* Add newline to end of file
2024-01-15 07:53:56 -08:00
hadican
8061a8b7aa
add loadBalancerIP support for proxy service (#421)
Co-authored-by: Abdulhadi Celenlioglu <abdulhadi@intenseye.com>
2023-12-29 12:16:37 +02:00
Lari Hotari
066b40c040 Fix linting errors 2023-12-21 22:45:46 +02:00
lenglet-k
eedc21da30
feat: add metricsRelabelings on podMonitor (#406) 2023-12-21 08:24:49 -08:00
Daniel Kovacs
a725188438
feat: added support for tolerations for the init jobs (#399)
Co-authored-by: Daniel Kovacs <daniel.kovacs@directed.global>
2023-12-21 07:48:02 -08:00
Frank Kelly
0b2d9b4d5d
Decouple Service account creation from PodSecurityPolicy (#387)
* Proposal: service accounts creation should be decoupled from PodSecurityPolicy.

* Rename *-rbac.yaml to *-psp.yaml and move service account to *-service-account.yaml

* Test with psp enabled

Co-authored-by: Lari Hotari <lhotari@apache.org>
2023-12-21 04:40:54 -08:00
Jasper Vaneessen
89da2b2890
Zookeeper replicas: adhere to own recommendations (#336)
The comments say ZK replicacount should be >=3 but defaults to 1, should adhere to own recommendations by default
2023-12-11 02:36:25 -08:00
Frank Kelly
8cb3c18377
Allow Proxy and Broker HPA to specify scaling policies on scaleUp or scaleDown. (#391) 2023-09-15 14:12:12 -05:00
Ethan-Merrill
73fe688a43
Add support for stateful functions using the bookie as state storage (#171)
### Motivation

Enables support for using the Pulsar bookies as persistent state storage for functions.

### Modifications

- Added an option to enable/disable using bookies as state storage
- Adds extra server components options to the bookkeeper to enable necessary features for bookies to be used as state storage
- Adds stateStorageServiceUrl to the broker configmap
2023-07-18 21:37:03 -05:00
Tomasz Jegorow
042fd5b6d4
Configure custom topologyKey for podAntiAffinity (#351) 2023-07-12 18:19:49 +03:00
Brad Shelton
f8ad65066e
To address the function role vs clusterrole issue (#236)
* To address the function role vs clusterrole issue

* making backwards compatable

* updated value.yaml to include limit functions to namespace

* Added documentation to clarify the new attribute

* moved limit_to_namespace under functions.rbac
2023-07-12 18:11:36 +03:00
Lari Hotari
49f4acdf5a
Refactor GitHub Actions CI to a single workflow (#371)
* Refactor GitHub Actions CI to a single workflow

* Handle case where "ct lint" fails because of no chart changes

* Re-order scenarios

* Remove excessive default GC logging

* Bump cert-manager version to v1.12.2

* Use compatible cert-manager version

* Install debugging tools (k9s) for ssh access

* Only apply for interactive shells

* Fix JWT symmetric test

* Fix part that was missing from #356

* Install k9s on the fly when k9s is used

- set KUBECONFIG on the fly for kubectl too
2023-07-11 15:55:35 +03:00
mfuxi
786e182de4
add ingressClassName (#360) 2023-07-11 10:00:45 +03:00
Chris Johnson
90a26b2dc8
fix: proxy should not use priviledged port numbers (#356)
* fix: proxy should not use priviledged port numbers

This fixes issue #335

* fix: making the change backward compatible
2023-07-11 10:00:17 +03:00
Lari Hotari
f43c6f6d9e
Fix GitHub Actions based CI checks which have been failing (#370)
* Upgrade upgrade kind, chart releaser and helm versions

* Disable podMonitory for values-broker-tls.yaml file

- was missing from #317

* Use k8s 1.18.20

* Use ubuntu-20.04 runtime

- k8s < 1.19 doesn't support cgroup v2

* Upgrade to k8s 1.19 as baseline

* Baseline to k8s 1.20

* Set ip family to ipv4

* Add more logging to kind cluster creation

* Simplify duplicate job deletion

* use verbosity flag

* Upgrade to k8s 1.24

* Replace removed tolerate-unready-endpoints annotation with publishNotReadyAddresses

(cherry picked from commit e90926053a2b01bb95529fbaddc8d2ce2cdeec63)

* Use k8s 1.21 as baseline

* Run on ubuntu-22.04

* Use Pulsar 2.10.4
2023-07-10 12:45:37 -07:00
Claudio Vellage
343ce0527d
Allow to use selectors with volumeClaimTemplates (#286)
* Allow to use selectors with volumeClaimTemplates

* Fixed naming inconsistency, added null value

Co-authored-by: Claudio Vellage <claudio.vellage@pm.me>
Co-authored-by: Michael Marshall <mmarshall@apache.org>

### Motivation

Currently it's not possible to use selectors with volumeClaimTemplates which makes it hard/impossible to bind statically provisioned PVs.

### Modifications

Added (optional) selectors to `volumeClaimTemplates` and documented in values file.

### Verifying this change

- [ ] Make sure that the change passes the CI checks.
2022-10-20 13:46:23 -05:00
Samuel Verstraete
8f033bd1a5
allow specifying the nodeSelector for the init jobs (#225)
* allow specifying the nodeSelector for the init jobs

* Use pulsar_metadata.nodeSelector

Co-authored-by: samuel <samuel.verstraete@aprimo.com>

### Motivation

When deploying pulsar to an AKS cluster with windows nodepools i was unable to specify that the Jobs of the initalize release had to run on linux nodes. With the change i can now specify a node selector for the init jobs.

### Modifications

add nodeSelector on pulsar_init and bookie_init

### Verifying this change

- [ ] Make sure that the change passes the CI checks.
2022-10-19 23:41:39 -05:00
Michael Marshall
42ce7caa55
Update how to configure external zookeeper servers (#308)
### Motivation

In #269, we added a way to configure external zookeeper servers. However, it was added to the wrong section of the zookeeper config. The `zookeeper.configData` section is mapped directly into the zookeeper configmap.

### Modifications

Move `zookeeper.configData.ZOOKEEPER_SERVERS` to `zookeeper.externalZookeeperServerList`

### Verifying this change
This is a cosmetic change on an unreleased feature.
2022-10-19 16:28:33 -05:00
Michael Marshall
7f23af26b7
Replace monitoring solution with kube-prometheus-stack dependency (#299)
* Replace monitoring solution with kube-prometheus-stack dependency

* Enable pod monitors

* Download necessary chart dependencies for CI

* Actually run dependency update

* Enable missed podMonitor

* Disable alertmanager by default for feature parity

Related issues #294 #65

Supersedes #296 and #297

### Motivation

Our helm chart is out of date. I propose we make a breaking change for the monitoring solution and start using the `kube-prometheus-stack` as a dependency. This should make upgrades easier and will let users leverage all of that chart's features.

This change will result in the removal of the StreamNative Grafana Dashboards. We'll need to figure out the right way to address that. The apache/pulsar project has grafana dashboards, but they have not been maintained. With this added dependency, we'll have the benefit of being able to use k8s `ConfigMap`s to configure grafana dashboards.

### Modifications

* Remove old prometheus and grafana configuration
* Add kube-prometheus-stack chart as a dependency
* Enable several components by default. I am not opinionated on these, but it is based on the other values in the chart.

### Verifying this change

This is a large change that will require manual validation, and may break deployments. I propose this triggers a helm chart 3.0.0 release.
2022-10-19 10:23:08 -05:00