179 Commits

Author SHA1 Message Date
MonicaMagoniCom
c4941b32d1
Add namespace to hpa templates (#494) 2024-05-03 11:48:02 -07:00
Lari Hotari
bd8bc633df
Change default statusFilePath to /pulsar/logs/status (#489)
* Change default statusFilePath to /pulsar/logs/status

* Write OK to statusFilePath
2024-04-15 05:41:17 -07:00
Martin
347326e0c3
Fix pulsar-manager persistence (#485)
- only setup environment in pulsar manager if broker is deployed
- fix indent
- enable persistence for manager and move configs around
2024-04-03 21:28:46 -07:00
Lari Hotari
fdd46f9b74
Add basic NOTES.txt (#482) 2024-03-27 04:32:36 -07:00
Lari Hotari
fdec9c69ef
Use podManagementPolicy OrderedReady for Broker sts when Functions are enabled (#474)
* Use podManagementPolicy OrderedReady for Broker sts when Functions are enabled

* Don't change podManagementPolicy when the sts already exists

* Fix template issue

* Fix apiVersion
2024-03-26 10:49:33 -07:00
doug-ba
9929b80b3c
add ability to use separate disk for zookeeper tx log (#476)
* add ability to use separate disk for zookeeper tx log

* Use absolute path

---------

Co-authored-by: Lari Hotari <lhotari@users.noreply.github.com>
2024-03-26 07:51:31 -07:00
Lari Hotari
eb0a878d9c
Make job.ttl.enabled consistent and effective only when k8s >= 1.23 (#481) 2024-03-26 06:23:15 -07:00
doug-ba
bc5862d4b0
pulsar-manager adding support for existing secret (#478) 2024-03-26 05:26:37 -07:00
doug-ba
3dee8dfe3b
making .ReleaseIsInstall optional for init jobs (#480)
* making .ReleasIsInstall optional for init jobs

* initialize simplifying an if condition based on feedback
2024-03-25 22:26:32 -07:00
Nathan Clayton
b4241f984b
Update broker statefulset to check if AWS keys secret name is defined before adding to environment. (#466) 2024-03-03 10:38:31 +02:00
Martin
89c5987b17
Bugfix/pulsar manager init (#463)
* add some more logs to the pulsar manager test

* fix admin secret "double-encoding"

* make pulsar-manager-cluster-initialize.yaml "rerunnable"
2024-02-22 17:37:25 +02:00
Lari Hotari
17a4239733
Remove buggy and useless function-worker-config-map (#462)
Fixes #56
2024-02-21 13:47:23 -08:00
Lari Hotari
0e3251bea8
Remove deprecated "extra" key to configure components, also remove dashboard that has been replaced (#461)
- the "extra" key has been deprecated a long time ago
- the dashboard is outdated and there's a replacement with kube-prometheus-stack and #439
2024-02-21 04:53:29 -08:00
csthomas1
cb269bbaf3
Feature/pulsar manager v0.2.0 with jwt setup admin account creation (#219)
* Added support for JWT secretref and key volume mount. Added admin user auto-creation.

* Removed variables accidentally re-added and comments no longer relevant

* Enabling pulsar manager test w/ manager-admin superuser for symmetric and asymmetric jwt tests

* Added verification of communication with broker to ci test-pulsar-manager

* Fixing error on line 115 of helm.sh

* More fixes

* Adding echo of envs and tenants

* Fixing LOGIN_JSESSIONID variable name
2024-02-21 04:25:23 -08:00
Victor Fauth
29ea17b3fc
Enable persistence for pulsar-manager (#343)
* Enable persistence for pulsar-manager

* Upgrade to v0.4.0 version of pulsar-manager to get required fix

- contains https://github.com/apache/pulsar-manager/pull/501
  in https://github.com/apache/pulsar-manager/releases/tag/v0.4.0

---------

Co-authored-by: Victor Fauth <victor.fauth@thalesgroup.com>
Co-authored-by: Lari Hotari <lhotari@apache.org>
2024-02-15 01:27:40 -08:00
Martin
d0b784a953
Feature/pulsar manager initialize (#457)
* add better pulsar manager integration and init along with tests & docs

* fix pulsar manager startup args

* update pulsar manager service to ClusterIP + remove duplicate
2024-02-14 10:13:54 -08:00
Martin
4daf6d88a2
grouped init containers (#441) 2024-01-26 03:09:57 -08:00
Martin
8cd3a04812
expose admin port of pulsar manager in service (#440) 2024-01-24 23:32:25 -08:00
Lari Hotari
e058aa581d
Require helm version 3.10 or newer (#436)
* Add check for required helm version

* Add test scenario for helm 3.10.0
2024-01-18 19:28:09 +02:00
Lari Hotari
1cb83398c8
Don't use TLS from function instances to brokers by default (#435)
- Function instances don't currently have the TLS CA cert available
2024-01-17 21:04:43 -08:00
Lari Hotari
e49bd32378 Fix indent for Tiered storage offload environment 2024-01-17 18:11:30 +02:00
pellicano
cfa156f738
Tiered Storage config (#205)
* Add tiered storage config

* Check Tiered Storage on README

* GitHub PR #205 changes (1st round)

Remove <= 2.6.0 configs.
Add missing GCS secret volumeMount.
Update GCS example name.

* Cleanup comments

* Bump chart version

* GitHub PR #205 changes (2nd round)

Moved storageOffload under broker section.
Fixed some typos.
Added AWS S3 IRSA annotation comment.

* GitHub PR #205  changes (3rd round)

Moved AWS and Azure credentials into K8S secrets using same StreamNative Helm Chart approach.

* Trim trailing spaces

---------

Co-authored-by: Lari Hotari <lhotari@apache.org>
Co-authored-by: Marcelo Pellicano <mpellicanodeoliveira@bluecatnetworks.com>
2024-01-17 03:06:16 -08:00
Lari Hotari
f20756abb8
Support configuring "proxyRoles" in broker (#430)
* Add support for proxyRoles

* Test produce and consume via Pulsar proxy
2024-01-16 04:48:58 -08:00
lenglet-k
e54354df87
feat: add externalTrafficPolicy and loadBalancerSourceRanges (#405)
* feat: add externalTrafficPolicy and loadBalancerSourceRanges
2024-01-15 12:59:58 -08:00
lenglet-k
c094322904
feat: add imagePullSecrets in Job (#404)
* feat: add imagePullSecrets on Job

* feat: update values.yaml
2024-01-15 12:28:05 -08:00
Martin
04b8ccebb1
Resource specs for Init Containers (#428)
* add missing section in values.yaml for pulsar_metadata resources

* add resources to all init containers and an additional section to specify them in values.yaml

* increase memory defaults for init containers

* remove empty lines

* Add newline to end of file
2024-01-15 07:53:56 -08:00
hadican
8061a8b7aa
add loadBalancerIP support for proxy service (#421)
Co-authored-by: Abdulhadi Celenlioglu <abdulhadi@intenseye.com>
2023-12-29 12:16:37 +02:00
lenglet-k
eedc21da30
feat: add metricsRelabelings on podMonitor (#406) 2023-12-21 08:24:49 -08:00
Daniel Kovacs
a725188438
feat: added support for tolerations for the init jobs (#399)
Co-authored-by: Daniel Kovacs <daniel.kovacs@directed.global>
2023-12-21 07:48:02 -08:00
Frank Kelly
0b2d9b4d5d
Decouple Service account creation from PodSecurityPolicy (#387)
* Proposal: service accounts creation should be decoupled from PodSecurityPolicy.

* Rename *-rbac.yaml to *-psp.yaml and move service account to *-service-account.yaml

* Test with psp enabled

Co-authored-by: Lari Hotari <lhotari@apache.org>
2023-12-21 04:40:54 -08:00
Đình Tuyên
a8d667a391
Correct Keytool env generation (#375) 2023-12-11 01:23:43 -08:00
Michał Koziorowski
fba51bd87e
Lowered BOOKIE_MEM and PULSAR_MEM in init containers (#386)
* Lowered BOOKIE_MEM and PULSAR_MEM in init containers. Default BOOKIE_MEM and PULSAR_MEM settings from conf/pulsar_env.sh and conf/bkenv.sh (-Xms2g -Xmx2g -XX:MaxDirectMemorySize=4g) are too high for low-memory systems.
2023-12-11 00:50:08 -08:00
Emre Aydın
9542c7b226
Use container ports for proxy stateful set probes (#410)
Using service ports cause probes to fail.

Co-authored-by: emre <emre.aydin@zapata.ai>
2023-12-11 00:20:07 -08:00
Frank Kelly
8cb3c18377
Allow Proxy and Broker HPA to specify scaling policies on scaleUp or scaleDown. (#391) 2023-09-15 14:12:12 -05:00
Michał Koziorowski
ea5404c421
Fixed bookkeeper volume mounts indentation (#384) 2023-08-24 09:32:58 +08:00
Ethan-Merrill
73fe688a43
Add support for stateful functions using the bookie as state storage (#171)
### Motivation

Enables support for using the Pulsar bookies as persistent state storage for functions.

### Modifications

- Added an option to enable/disable using bookies as state storage
- Adds extra server components options to the bookkeeper to enable necessary features for bookies to be used as state storage
- Adds stateStorageServiceUrl to the broker configmap
2023-07-18 21:37:03 -05:00
Tomasz Jegorow
042fd5b6d4
Configure custom topologyKey for podAntiAffinity (#351) 2023-07-12 18:19:49 +03:00
Atkins
79ec5ba333
Fix pod annotations when restartPodsOnConfigMapChange is true (#353)
Signed-off-by: Atkins Chang <atkinschang@gmail.com>
2023-07-12 18:18:48 +03:00
Atkins
b30eb6fff8
Improve HPA (#354)
* Use `autoscaling/v2` if Kubernetes version >= 1.23

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

* Disable replicas when autoscaling enabled

Signed-off-by: Atkins Chang <atkinschang@gmail.com>

---------

Signed-off-by: Atkins Chang <atkinschang@gmail.com>
2023-07-12 18:18:19 +03:00
Brad Shelton
f8ad65066e
To address the function role vs clusterrole issue (#236)
* To address the function role vs clusterrole issue

* making backwards compatable

* updated value.yaml to include limit functions to namespace

* Added documentation to clarify the new attribute

* moved limit_to_namespace under functions.rbac
2023-07-12 18:11:36 +03:00
Lari Hotari
49f4acdf5a
Refactor GitHub Actions CI to a single workflow (#371)
* Refactor GitHub Actions CI to a single workflow

* Handle case where "ct lint" fails because of no chart changes

* Re-order scenarios

* Remove excessive default GC logging

* Bump cert-manager version to v1.12.2

* Use compatible cert-manager version

* Install debugging tools (k9s) for ssh access

* Only apply for interactive shells

* Fix JWT symmetric test

* Fix part that was missing from #356

* Install k9s on the fly when k9s is used

- set KUBECONFIG on the fly for kubectl too
2023-07-11 15:55:35 +03:00
huis
2d646f4efe
Fix PVC selector scope error when bookkeeper uses multiple data volumes (#342)
Fix PVC selector scope error when bookkeeper uses multiple data volumes
2023-07-11 10:03:38 +03:00
Robert Moucha
71450334cf
Fix zookeeper annotations (#348)
Fixed the case when no ZK annotations are set and `zookeeper.restartPodsOnConfigMapChange: true`
helm can not render template
2023-07-11 10:01:54 +03:00
mfuxi
786e182de4
add ingressClassName (#360) 2023-07-11 10:00:45 +03:00
Chris Johnson
90a26b2dc8
fix: proxy should not use priviledged port numbers (#356)
* fix: proxy should not use priviledged port numbers

This fixes issue #335

* fix: making the change backward compatible
2023-07-11 10:00:17 +03:00
Lari Hotari
f43c6f6d9e
Fix GitHub Actions based CI checks which have been failing (#370)
* Upgrade upgrade kind, chart releaser and helm versions

* Disable podMonitory for values-broker-tls.yaml file

- was missing from #317

* Use k8s 1.18.20

* Use ubuntu-20.04 runtime

- k8s < 1.19 doesn't support cgroup v2

* Upgrade to k8s 1.19 as baseline

* Baseline to k8s 1.20

* Set ip family to ipv4

* Add more logging to kind cluster creation

* Simplify duplicate job deletion

* use verbosity flag

* Upgrade to k8s 1.24

* Replace removed tolerate-unready-endpoints annotation with publishNotReadyAddresses

(cherry picked from commit e90926053a2b01bb95529fbaddc8d2ce2cdeec63)

* Use k8s 1.21 as baseline

* Run on ubuntu-22.04

* Use Pulsar 2.10.4
2023-07-10 12:45:37 -07:00
edward.zeng
95c218b218
Fix PodMonitor name conflicts for multiple releases in same namespace (#258)
* Fix PodMonitor name conflicts for multiple releases in same namespace

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>

* Use pulsar.fullname for PodMonitor name prefix

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>
Co-authored-by: Michael Marshall <mmarshall@apache.org>

Signed-off-by: Edward Zeng <jie.zeng@zilliz.com>

Fixes #257

### Motivation

Fix PodMonitor name conflicts for multiple releases in same namespace

### Modifications

Use release name instead of hardcode `pulsar.name` for pod monitor name.

### Verifying this change

- [x] Make sure that the change passes the CI checks.
2022-10-20 21:15:16 -05:00
Michael Marshall
d9769a9519
Add missing license headers and .rat-excludes (#319)
* Add missing license headers and .rat-excludes

* Fix .rat-excludes files

### Motivation

As part of our updated release process, we need to make sure that all relevant files have license headers.

### Modifications

* Add license headers formatted appropriately for each file type

### Verifying this change

The follow script shows that the solution is complete:

```shell
$ java -jar ../apache-rat-0.15/apache-rat-0.15.jar . -E .rat-excludes 
Ignored 18 lines in your exclusion files as comments or empty lines.


*****************************************************
Summary
-------
Generated at: 2022-10-20T17:54:42-05:00

Notes: 4
Binaries: 1
Archives: 0
Standards: 92

Apache Licensed: 92
Generated Documents: 0

JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.

0 Unknown Licenses

*****************************************************
  Files with Apache License headers will be marked AL
  Binary files (which do not require any license headers) will be marked B
  Compressed archives will be marked A
  Notices, licenses etc. will be marked N
  AL    ./.asf.yaml
  AL    ./.rat-excludes
  N     ./LICENSE
  N     ./NOTICE
  AL    ./README.md
  AL    ./Vagrantfile
  AL    ./license_test.go
  AL    ./charts/pulsar/.helmignore
  AL    ./charts/pulsar/Chart.yaml
  N     ./charts/pulsar/LICENSE
  N     ./charts/pulsar/NOTICE
  AL    ./charts/pulsar/values.yaml
  B     ./charts/pulsar/charts/kube-prometheus-stack-41.5.1.tgz
  AL    ./charts/pulsar/templates/_autorecovery.tpl
  AL    ./charts/pulsar/templates/_bookkeeper.tpl
  AL    ./charts/pulsar/templates/_broker.tpl
  AL    ./charts/pulsar/templates/_configurationstore.tpl
  AL    ./charts/pulsar/templates/_helpers.tpl
  AL    ./charts/pulsar/templates/_toolset.tpl
  AL    ./charts/pulsar/templates/_zookeeper.tpl
  AL    ./charts/pulsar/templates/autorecovery-configmap.yaml
  AL    ./charts/pulsar/templates/autorecovery-podmonitor.yaml
  AL    ./charts/pulsar/templates/autorecovery-rbac.yaml
  AL    ./charts/pulsar/templates/autorecovery-service.yaml
  AL    ./charts/pulsar/templates/autorecovery-statefulset.yaml
  AL    ./charts/pulsar/templates/bookkeeper-cluster-initialize.yaml
  AL    ./charts/pulsar/templates/bookkeeper-configmap.yaml
  AL    ./charts/pulsar/templates/bookkeeper-pdb.yaml
  AL    ./charts/pulsar/templates/bookkeeper-podmonitor.yaml
  AL    ./charts/pulsar/templates/bookkeeper-rbac.yaml
  AL    ./charts/pulsar/templates/bookkeeper-service.yaml
  AL    ./charts/pulsar/templates/bookkeeper-statefulset.yaml
  AL    ./charts/pulsar/templates/bookkeeper-storageclass.yaml
  AL    ./charts/pulsar/templates/broker-cluster-role-binding.yaml
  AL    ./charts/pulsar/templates/broker-configmap.yaml
  AL    ./charts/pulsar/templates/broker-hpa.yaml
  AL    ./charts/pulsar/templates/broker-pdb.yaml
  AL    ./charts/pulsar/templates/broker-podmonitor.yaml
  AL    ./charts/pulsar/templates/broker-rbac.yaml
  AL    ./charts/pulsar/templates/broker-service-account.yaml
  AL    ./charts/pulsar/templates/broker-service.yaml
  AL    ./charts/pulsar/templates/broker-statefulset.yaml
  AL    ./charts/pulsar/templates/dashboard-deployment.yaml
  AL    ./charts/pulsar/templates/dashboard-ingress.yaml
  AL    ./charts/pulsar/templates/dashboard-service.yaml
  AL    ./charts/pulsar/templates/function-worker-configmap.yaml
  AL    ./charts/pulsar/templates/keytool.yaml
  AL    ./charts/pulsar/templates/namespace.yaml
  AL    ./charts/pulsar/templates/proxy-configmap.yaml
  AL    ./charts/pulsar/templates/proxy-hpa.yaml
  AL    ./charts/pulsar/templates/proxy-ingress.yaml
  AL    ./charts/pulsar/templates/proxy-pdb.yaml
  AL    ./charts/pulsar/templates/proxy-podmonitor.yaml
  AL    ./charts/pulsar/templates/proxy-rbac.yaml
  AL    ./charts/pulsar/templates/proxy-service.yaml
  AL    ./charts/pulsar/templates/proxy-statefulset.yaml
  AL    ./charts/pulsar/templates/pulsar-cluster-initialize.yaml
  AL    ./charts/pulsar/templates/pulsar-manager-admin-secret.yaml
  AL    ./charts/pulsar/templates/pulsar-manager-configmap.yaml
  AL    ./charts/pulsar/templates/pulsar-manager-deployment.yaml
  AL    ./charts/pulsar/templates/pulsar-manager-ingress.yaml
  AL    ./charts/pulsar/templates/pulsar-manager-service.yaml
  AL    ./charts/pulsar/templates/tls-cert-internal-issuer.yaml
  AL    ./charts/pulsar/templates/tls-certs-internal.yaml
  AL    ./charts/pulsar/templates/toolset-configmap.yaml
  AL    ./charts/pulsar/templates/toolset-rbac.yaml
  AL    ./charts/pulsar/templates/toolset-service.yaml
  AL    ./charts/pulsar/templates/toolset-statefulset.yaml
  AL    ./charts/pulsar/templates/zookeeper-configmap.yaml
  AL    ./charts/pulsar/templates/zookeeper-pdb.yaml
  AL    ./charts/pulsar/templates/zookeeper-podmonitor.yaml
  AL    ./charts/pulsar/templates/zookeeper-rbac.yaml
  AL    ./charts/pulsar/templates/zookeeper-service.yaml
  AL    ./charts/pulsar/templates/zookeeper-statefulset.yaml
  AL    ./charts/pulsar/templates/zookeeper-storageclass.yaml
  AL    ./examples/values-bookkeeper-aws.yaml
  AL    ./examples/values-cs.yaml
  AL    ./examples/values-jwt-asymmetric.yaml
  AL    ./examples/values-jwt-symmetric.yaml
  AL    ./examples/values-local-cluster.yaml
  AL    ./examples/values-local-pv.yaml
  AL    ./examples/values-minikube.yaml
  AL    ./examples/values-no-persistence.yaml
  AL    ./examples/values-one-node.yaml
  AL    ./examples/values-tls.yaml
  AL    ./examples/values-zookeeper-aws.yaml
  AL    ./hack/common.sh
  AL    ./hack/kind-cluster-build.sh
  AL    ./scripts/set-pulsar-version.sh
  AL    ./scripts/cert-manager/install-cert-manager.sh
  AL    ./scripts/pulsar/cleanup_helm_release.sh
  AL    ./scripts/pulsar/common.sh
  AL    ./scripts/pulsar/common_auth.sh
  AL    ./scripts/pulsar/generate_token.sh
  AL    ./scripts/pulsar/generate_token_secret_key.sh
  AL    ./scripts/pulsar/get_token.sh
  AL    ./scripts/pulsar/prepare_helm_release.sh
 
*****************************************************

```
2022-10-20 20:29:09 -05:00
Michael Marshall
9324a9a270
Fix bookkeeper metadata init when specifying metadataPrefix (#316)
Fixes #309

### Motivation

Fix the metadataPrefix initialization.

### Modifications

* Fix the script by adding `&& echo`

### Verifying this change

I manually verified that this change works and correctly puts the metadata in the prefixed location.
2022-10-20 15:24:20 -05:00
Claudio Vellage
343ce0527d
Allow to use selectors with volumeClaimTemplates (#286)
* Allow to use selectors with volumeClaimTemplates

* Fixed naming inconsistency, added null value

Co-authored-by: Claudio Vellage <claudio.vellage@pm.me>
Co-authored-by: Michael Marshall <mmarshall@apache.org>

### Motivation

Currently it's not possible to use selectors with volumeClaimTemplates which makes it hard/impossible to bind statically provisioned PVs.

### Modifications

Added (optional) selectors to `volumeClaimTemplates` and documented in values file.

### Verifying this change

- [ ] Make sure that the change passes the CI checks.
2022-10-20 13:46:23 -05:00