164 Commits

Author SHA1 Message Date
Martin
89c5987b17
Bugfix/pulsar manager init (#463)
* add some more logs to the pulsar manager test

* fix admin secret "double-encoding"

* make pulsar-manager-cluster-initialize.yaml "rerunnable"
2024-02-22 17:37:25 +02:00
Lari Hotari
17a4239733
Remove buggy and useless function-worker-config-map (#462)
Fixes #56
2024-02-21 13:47:23 -08:00
Lari Hotari
0e3251bea8
Remove deprecated "extra" key to configure components, also remove dashboard that has been replaced (#461)
- the "extra" key has been deprecated a long time ago
- the dashboard is outdated and there's a replacement with kube-prometheus-stack and #439
2024-02-21 04:53:29 -08:00
csthomas1
cb269bbaf3
Feature/pulsar manager v0.2.0 with jwt setup admin account creation (#219)
* Added support for JWT secretref and key volume mount. Added admin user auto-creation.

* Removed variables accidentally re-added and comments no longer relevant

* Enabling pulsar manager test w/ manager-admin superuser for symmetric and asymmetric jwt tests

* Added verification of communication with broker to ci test-pulsar-manager

* Fixing error on line 115 of helm.sh

* More fixes

* Adding echo of envs and tenants

* Fixing LOGIN_JSESSIONID variable name
2024-02-21 04:25:23 -08:00
Victor Fauth
29ea17b3fc
Enable persistence for pulsar-manager (#343)
* Enable persistence for pulsar-manager

* Upgrade to v0.4.0 version of pulsar-manager to get required fix

- contains https://github.com/apache/pulsar-manager/pull/501
  in https://github.com/apache/pulsar-manager/releases/tag/v0.4.0

---------

Co-authored-by: Victor Fauth <victor.fauth@thalesgroup.com>
Co-authored-by: Lari Hotari <lhotari@apache.org>
2024-02-15 01:27:40 -08:00
Martin
d0b784a953
Feature/pulsar manager initialize (#457)
* add better pulsar manager integration and init along with tests & docs

* fix pulsar manager startup args

* update pulsar manager service to ClusterIP + remove duplicate
2024-02-14 10:13:54 -08:00
Lari Hotari
9cbe03c7ee
Improve Bookkeeper default configuration (#454)
- remove minimal memory settings
- add more optimal data compaction settings
2024-01-31 03:21:04 -08:00
Martin
4daf6d88a2
grouped init containers (#441) 2024-01-26 03:09:57 -08:00
Lari Hotari
8d2d567b30
Remove pulsar_detector dash board (#446)
- not applicable for Apache Pulsar Helm chart's Pulsar deployment
2024-01-26 03:09:11 -08:00
Lari Hotari
72a8fb6b3e
Upgrade kube-prometheus-stack to 56.x.x version (#445)
* Upgrade to kube-prometheus-stack 56.x.x

* Add CI test case for kube-prometheus-stack upgrade

* Add "--force-conflicts" flag
2024-01-26 03:07:10 -08:00
Martin
8cd3a04812
expose admin port of pulsar manager in service (#440) 2024-01-24 23:32:25 -08:00
Lari Hotari
de4d2e7dc8
Add kubeVersion to Chart.yaml to enforce minimum Kubernetes version (#443) 2024-01-24 11:46:59 -08:00
Lari Hotari
65a5fc0002 Fix typo in Apache License 2.0 abbrev, should be AL 2.0 2024-01-24 21:44:24 +02:00
Lari Hotari
d486e4a42d
Add default configuration for Pulsar Grafana dashboards (#439)
* Add default configuration for dashboards
2024-01-24 11:12:57 -08:00
Lari Hotari
e058aa581d
Require helm version 3.10 or newer (#436)
* Add check for required helm version

* Add test scenario for helm 3.10.0
2024-01-18 19:28:09 +02:00
Lari Hotari
1cb83398c8
Don't use TLS from function instances to brokers by default (#435)
- Function instances don't currently have the TLS CA cert available
2024-01-17 21:04:43 -08:00
Lari Hotari
6db886f078 Chart: Bump version to 3.2.0
(cherry picked from commit 03b3888df449796f815ce90d12a3c64ab661ea30)
2024-01-18 00:45:52 +02:00
Lari Hotari
e49bd32378 Fix indent for Tiered storage offload environment 2024-01-17 18:11:30 +02:00
pellicano
cfa156f738
Tiered Storage config (#205)
* Add tiered storage config

* Check Tiered Storage on README

* GitHub PR #205 changes (1st round)

Remove <= 2.6.0 configs.
Add missing GCS secret volumeMount.
Update GCS example name.

* Cleanup comments

* Bump chart version

* GitHub PR #205 changes (2nd round)

Moved storageOffload under broker section.
Fixed some typos.
Added AWS S3 IRSA annotation comment.

* GitHub PR #205  changes (3rd round)

Moved AWS and Azure credentials into K8S secrets using same StreamNative Helm Chart approach.

* Trim trailing spaces

---------

Co-authored-by: Lari Hotari <lhotari@apache.org>
Co-authored-by: Marcelo Pellicano <mpellicanodeoliveira@bluecatnetworks.com>
2024-01-17 03:06:16 -08:00
Lari Hotari
5d1275e71d
Change default value for rbac.limit_to_namespace and functions.rbac.limit_to_namespace (#418) 2024-01-17 01:02:07 -08:00
Lari Hotari
f20756abb8
Support configuring "proxyRoles" in broker (#430)
* Add support for proxyRoles

* Test produce and consume via Pulsar proxy
2024-01-16 04:48:58 -08:00
lenglet-k
e54354df87
feat: add externalTrafficPolicy and loadBalancerSourceRanges (#405)
* feat: add externalTrafficPolicy and loadBalancerSourceRanges
2024-01-15 12:59:58 -08:00
lenglet-k
c094322904
feat: add imagePullSecrets in Job (#404)
* feat: add imagePullSecrets on Job

* feat: update values.yaml
2024-01-15 12:28:05 -08:00
Martin
04b8ccebb1
Resource specs for Init Containers (#428)
* add missing section in values.yaml for pulsar_metadata resources

* add resources to all init containers and an additional section to specify them in values.yaml

* increase memory defaults for init containers

* remove empty lines

* Add newline to end of file
2024-01-15 07:53:56 -08:00
hadican
8061a8b7aa
add loadBalancerIP support for proxy service (#421)
Co-authored-by: Abdulhadi Celenlioglu <abdulhadi@intenseye.com>
2023-12-29 12:16:37 +02:00
Lari Hotari
066b40c040 Fix linting errors 2023-12-21 22:45:46 +02:00
lenglet-k
eedc21da30
feat: add metricsRelabelings on podMonitor (#406) 2023-12-21 08:24:49 -08:00
Daniel Kovacs
a725188438
feat: added support for tolerations for the init jobs (#399)
Co-authored-by: Daniel Kovacs <daniel.kovacs@directed.global>
2023-12-21 07:48:02 -08:00
Frank Kelly
0b2d9b4d5d
Decouple Service account creation from PodSecurityPolicy (#387)
* Proposal: service accounts creation should be decoupled from PodSecurityPolicy.

* Rename *-rbac.yaml to *-psp.yaml and move service account to *-service-account.yaml

* Test with psp enabled

Co-authored-by: Lari Hotari <lhotari@apache.org>
2023-12-21 04:40:54 -08:00
Lari Hotari
a7247b6edb Bump chart version to 3.1.0 2023-12-11 13:32:54 +02:00
Jasper Vaneessen
89da2b2890
Zookeeper replicas: adhere to own recommendations (#336)
The comments say ZK replicacount should be >=3 but defaults to 1, should adhere to own recommendations by default
2023-12-11 02:36:25 -08:00
Lari Hotari
608b51e872
Bump appVersion to 3.0.2 (#413) 2023-12-11 01:37:34 -08:00
Đình Tuyên
a8d667a391
Correct Keytool env generation (#375) 2023-12-11 01:23:43 -08:00
Michał Koziorowski
fba51bd87e
Lowered BOOKIE_MEM and PULSAR_MEM in init containers (#386)
* Lowered BOOKIE_MEM and PULSAR_MEM in init containers. Default BOOKIE_MEM and PULSAR_MEM settings from conf/pulsar_env.sh and conf/bkenv.sh (-Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g) are too high for low-memory systems.
2023-12-11 00:50:08 -08:00
Emre Aydın
9542c7b226
Use container ports for proxy stateful set probes (#410)
Using service ports cause probes to fail.

Co-authored-by: emre <emre.aydin@zapata.ai>
2023-12-11 00:20:07 -08:00
Frank Kelly
8cb3c18377
Allow Proxy and Broker HPA to specify scaling policies on scaleUp or scaleDown. (#391) 2023-09-15 14:12:12 -05:00
Michał Koziorowski
ea5404c421
Fixed bookkeeper volume mounts indentation (#384) 2023-08-24 09:32:58 +08:00
Ethan-Merrill
73fe688a43
Add support for stateful functions using the bookie as state storage (#171)
### Motivation

Enables support for using the Pulsar bookies as persistent state storage for functions.

### Modifications

- Added an option to enable/disable using bookies as state storage
- Adds extra server components options to the bookkeeper to enable necessary features for bookies to be used as state storage
- Adds stateStorageServiceUrl to the broker configmap
2023-07-18 21:37:03 -05:00
Tomasz Jegorow
042fd5b6d4
Configure custom topologyKey for podAntiAffinity (#351) 2023-07-12 18:19:49 +03:00
Atkins
79ec5ba333
Fix pod annotations when restartPodsOnConfigMapChange is true (#353)
Signed-off-by: Atkins Chang <atkinschang@gmail.com>
2023-07-12 18:18:48 +03:00
Atkins
b30eb6fff8
Improve HPA (#354)
* Use `autoscaling/v2` if Kubernetes version >= 1.23

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* Disable replicas when autoscaling enabled

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

---------

Signed-off-by: Atkins Chang <atkinschang@gmail.com>
2023-07-12 18:18:19 +03:00
Brad Shelton
f8ad65066e
To address the function role vs clusterrole issue (#236)
* To address the function role vs clusterrole issue

* making backwards compatable

* updated value.yaml to include limit functions to namespace

* Added documentation to clarify the new attribute

* moved limit_to_namespace under functions.rbac
2023-07-12 18:11:36 +03:00
Lari Hotari
49f4acdf5a
Refactor GitHub Actions CI to a single workflow (#371)
* Refactor GitHub Actions CI to a single workflow

* Handle case where "ct lint" fails because of no chart changes

* Re-order scenarios

* Remove excessive default GC logging

* Bump cert-manager version to v1.12.2

* Use compatible cert-manager version

* Install debugging tools (k9s) for ssh access

* Only apply for interactive shells

* Fix JWT symmetric test

* Fix part that was missing from #356

* Install k9s on the fly when k9s is used

- set KUBECONFIG on the fly for kubectl too
2023-07-11 15:55:35 +03:00
huis
2d646f4efe
Fix PVC selector scope error when bookkeeper uses multiple data volumes (#342)
Fix PVC selector scope error when bookkeeper uses multiple data volumes
2023-07-11 10:03:38 +03:00
Robert Moucha
71450334cf
Fix zookeeper annotations (#348)
Fixed the case when no ZK annotations are set and `zookeeper.restartPodsOnConfigMapChange: true`
helm can not render template
2023-07-11 10:01:54 +03:00
mfuxi
786e182de4
add ingressClassName (#360) 2023-07-11 10:00:45 +03:00
Chris Johnson
90a26b2dc8
fix: proxy should not use priviledged port numbers (#356)
* fix: proxy should not use priviledged port numbers

This fixes issue #335

* fix: making the change backward compatible
2023-07-11 10:00:17 +03:00
Lari Hotari
f43c6f6d9e
Fix GitHub Actions based CI checks which have been failing (#370)
* Upgrade upgrade kind, chart releaser and helm versions

* Disable podMonitory for values-broker-tls.yaml file

- was missing from #317

* Use k8s 1.18.20

* Use ubuntu-20.04 runtime

- k8s < 1.19 doesn't support cgroup v2

* Upgrade to k8s 1.19 as baseline

* Baseline to k8s 1.20

* Set ip family to ipv4

* Add more logging to kind cluster creation

* Simplify duplicate job deletion

* use verbosity flag

* Upgrade to k8s 1.24

* Replace removed tolerate-unready-endpoints annotation with publishNotReadyAddresses

(cherry picked from commit e90926053a2b01bb95529fbaddc8d2ce2cdeec63)

* Use k8s 1.21 as baseline

* Run on ubuntu-22.04

* Use Pulsar 2.10.4
2023-07-10 12:45:37 -07:00
Michael Marshall
687060aa27
Chart: Bump version to 3.0.0 2022-10-21 00:26:42 -05:00
edward.zeng
95c218b218
Fix PodMonitor name conflicts for multiple releases in same namespace (#258)
* Fix PodMonitor name conflicts for multiple releases in same namespace

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>

* Use pulsar.fullname for PodMonitor name prefix

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
Co-authored-by: Michael Marshall <mmarshall@apache.org>

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>

Fixes #257

### Motivation

Fix PodMonitor name conflicts for multiple releases in same namespace

### Modifications

Use release name instead of hardcode `pulsar.name` for pod monitor name.

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2022-10-20 21:15:16 -05:00