4 Commits

Author SHA1 Message Date
roman-popenov
56e0d05e25 [Issue-5994]: Start proxy pods when at least one broker pod is running (#6158)
### Motivation
Fixes #5994:
If the proxy service comes up before the brokers are up and reachable there will be HTTP 403 when running `bin/pulsar-admin` commands from inside the proxy pod.
 
The proxy will also not be able to connect to the brokers when data is pushed through binary port with the following error:
```bash
Caused by: org.apache.pulsar.broker.service.BrokerServiceException$PersistenceException: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
	... 14 more
Caused by: org.apache.bookkeeper.mledger.ManagedLedgerException: Not enough non-faulty bookies available
22:11:07.633 [pulsar-web-32-6] INFO  org.eclipse.jetty.server.RequestLog - 172.17.0.6 - - [24/Jan/2020:22:11:07 +0000] "PUT /admin/v2/persistent/public/functions/assignments HTTP/1.1" 500 2528 "-" "Pulsar-Java-v2.5.0" 280
```

#### Workaround:
Restart the proxy pods once brokers pods are running

#### Proposed solution:
Hold off starting of the proxies until at least one broker is reachable in the cluster. 

### Modifications

Changes are inside `proxy-deployment.yaml` helm template file that defines a new init container before proxy is started. The init container waits until broker is reachable using the nslookup on the broker service with a sleep of 30 seconds between retries and up to number of brokers times.

Alternative solution that doesn't always work was `'until nslookup broker-service; sleep 2; done;', but 403 would still sometimes (could have been a fluke, but I saw it happening once).

### Verifying this change
1) Follow the instructions on how deploying helm and run:
`helm install pulsar --values pulsar/values-mini.yaml ./pulsar/`. 
2) Wait until all the services are up and running.  
3) Connect to proxy pod and run `bin/pulsar-admin broker-stats monitoring-metrics` - no 403 or permission errors should arise
4) Set up tenant, namespace
5) Push data into a topic - No errors in the proxy logs and client is able to push data into cluster through proxies
2020-01-31 23:52:11 -08:00
Chris Bartholomew
b5a7f0a2ac Fixing pod anti-affinity rules in Kubernetes files including the Helm chart (#5381) 2019-10-15 10:41:53 -07:00
Sijie Guo
483107e2b9 [documentation][deploy] Improve helm deployment script to deploy Pulsar to minikube (#2363)
* [documentation][deploy] Update deployment instructions for deploying to Minikube

* Enable functions workers

* [documentation][deploy] Improve helm deployment script to deploy Pulsar to minikube

 ### Changes

- update the helm scripts: bookie/autorecovery/broker pods should wait until metadata is initialized
- disable `autoRecovery` on bookies since we start `AutoRecovery` in separate pods
- enable function worker on brokers
- provide a values file for minikube
- update documentation for using helm chart to deploy a cluster to minikube

* move the service type definition to values file
2018-08-16 00:25:49 -07:00
Daniel Jorge
7cfbe4a415 Helm charts for deployment on GKE (#1993)
* Helm charts for deployment on GKE

* Repackaginh helm charts under deployment/kubernetes/helm

* Formatting licences

* Removing cloud specific values to enable more generic deployments
2018-06-25 10:47:19 -07:00