Kubernetes in Production: Helm Charts, Operators, and Monitoring Strategies
Deploy Kubernetes workloads with Helm for package management, custom operators for automation, and monitoring with Prometheus and Grafana.
Kubernetes can place a pod on a node; it cannot decide whether the rollout is safe, whether the workload has enough resources, or whether an alert deserves a page. Production practice is the layer around kubectl apply: reproducible packaging, explicit controllers, resource budgets, probes, and signals tied to user impact.
Helm: The Kubernetes Package Manager
# Chart.yaml - Helm chart metadata
apiVersion: v2
name: my-app
description: Web application
version: 1.0.0
appVersion: "2.1.0"
# values.yaml - Configuration values
replicaCount: 3
image:
repository: myregistry.azurecr.io/my-app
tag: "2.1.0"
pullPolicy: IfNotPresent
service:
type: ClusterIP
port: 80
ingress:
enabled: true
className: nginx
annotations:
cert-manager.io/cluster-issuer: "letsencrypt-prod"
hosts:
- host: app.example.com
paths:
- path: /
pathType: Prefix
tls:
- secretName: app-tls
hosts:
- app.example.com
resources:
limits:
cpu: 500m
memory: 512Mi
requests:
cpu: 250m
memory: 256Mi
autoscaling:
enabled: true
minReplicas: 3
maxReplicas: 10
targetCPUUtilizationPercentage: 70
# Install:
# helm install my-app ./my-app-chart
# helm upgrade my-app ./my-app-chart --values values-production.yaml
# helm rollback my-app 1 # Instant rollback to previous versionDeployment Strategies
# Rolling Update (default - zero downtime)
apiVersion: apps/v1
kind: Deployment
metadata:
name: app
spec:
replicas: 5
strategy:
type: RollingUpdate
rollingUpdate:
maxSurge: 1 # Extra pod during update
maxUnavailable: 0 # No downtime
template:
spec:
containers:
- name: app
image: app:v2
readinessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 5
periodSeconds: 3
livenessProbe:
httpGet:
path: /health
port: 8080
initialDelaySeconds: 15
periodSeconds: 10
---
# Blue-Green Deployment (instant rollback)
apiVersion: v1
kind: Service
metadata:
name: app-service
spec:
selector:
app: app
version: green # Switch between blue/green
ports:
- protocol: TCP
port: 80
targetPort: 8080
---
# Canary Deployment (gradual rollout)
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: app
spec:
hosts:
- app.example.com
http:
- match:
- headers:
user-agent:
regex: ".*Mobile.*"
route:
- destination:
host: app
subset: v2
- route:
- destination:
host: app
subset: v1
weight: 90
- destination:
host: app
subset: v2
weight: 10 # 10% traffic to new versionResource Management and QoS
apiVersion: v1
kind: Pod
metadata:
name: guaranteed-pod
spec:
containers:
- name: app
resources:
# Guaranteed QoS: requests == limits
requests:
memory: "64Mi"
cpu: "250m"
limits:
memory: "64Mi"
cpu: "250m"
---
apiVersion: v1
kind: ResourceQuota
metadata:
name: compute-quota
namespace: production
spec:
hard:
requests.cpu: "100"
requests.memory: 200Gi
limits.cpu: "200"
limits.memory: 400Gi
pods: "100"
---
apiVersion: v1
kind: LimitRange
metadata:
name: default-limits
namespace: production
spec:
limits:
- default:
cpu: "500m"
memory: "512Mi"
defaultRequest:
cpu: "250m"
memory: "256Mi"
type: ContainerMonitoring with Prometheus
# ServiceMonitor for automatic scraping
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
name: app-metrics
spec:
selector:
matchLabels:
app: my-app
endpoints:
- port: metrics
interval: 30s
path: /metrics
---
# PrometheusRule for alerts
apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
name: app-alerts
spec:
groups:
- name: app
interval: 30s
rules:
- alert: HighErrorRate
expr: rate(http_requests_total{status=~"5.."}[5m]) > 0.05
for: 5m
labels:
severity: critical
annotations:
summary: "High error rate detected"
description: "Error rate is {{ $value | humanizePercentage }}"
- alert: PodCrashLooping
expr: rate(kube_pod_container_status_restarts_total[15m]) > 0
labels:
severity: warning
annotations:
summary: "Pod {{ $labels.pod }} is crash looping"
- alert: HighMemoryUsage
expr: container_memory_usage_bytes / container_spec_memory_limit_bytes > 0.9
for: 5m
labels:
severity: warningSecrets Management
# 1. Create secret from literal values
kubectl create secret generic app-secrets \
--from-literal=database-password=mysecretpass \
--from-literal=api-key=abc123
# 2. Create secret from file
kubectl create secret generic app-config \
--from-file=config.json
# 3. Use in pod
apiVersion: v1
kind: Pod
metadata:
name: app
spec:
containers:
- name: app
envFrom:
- secretRef:
name: app-secrets
volumeMounts:
- name: config
mountPath: /etc/config
readOnly: true
volumes:
- name: config
secret:
secretName: app-config
# 4. External secrets with Sealed Secrets
# Install controller:
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/download/v0.18.0/controller.yaml
# Seal secret (safe to commit to Git):
kubeseal --format=yaml < secret.yaml > sealed-secret.yaml
# 5. Use AWS Secrets Manager / Azure Key Vault
# Install external-secrets operator
helm install external-secrets external-secrets/external-secrets -n external-secrets-system
# SecretStore
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
name: aws-secrets
spec:
provider:
aws:
service: SecretsManager
region: us-east-1Production Checklist
- Always set resource requests and limits (prevents node exhaustion)
- Use readiness probes (prevents routing to unhealthy pods)
- Use liveness probes (restarts deadlocked pods)
- Set PodDisruptionBudgets (maintains availability during updates)
- Configure HorizontalPodAutoscaler (auto-scales based on metrics)
- Use multiple replicas across zones (tolerates zone failures)
- Implement pod anti-affinity (spreads replicas across nodes)
- Enable network policies (zero-trust security model)
- Use RBAC for access control (principle of least privilege)
- Monitor cluster health (node conditions, disk pressure, PID pressure)
- Set up log aggregation (ELK/EFK stack or cloud-native)
- Regular backups with Velero (disaster recovery)
- Image scanning in CI/CD (catch vulnerabilities early)
- Use OPA Gatekeeper for policy enforcement
- Implement circuit breakers and retries with service mesh