Post

Kubernetes Pod Security and Best Practices

Posted
By R G
15 min read

Kubernetes Pod Security and Best Practices

Security in Kubernetes is a critical concern for any organization running containerized workloads in production. Pods are the smallest deployable units in Kubernetes, and securing them is essential to maintaining a robust and secure cluster. In this comprehensive guide, we will explore Pod Security Standards, Security Contexts, Network Policies, RBAC, Secrets Management, Container Security, Image Scanning, and Runtime Security.

Understanding Kubernetes Pod Security

Pod security encompasses multiple layers of protection that work together to ensure your workloads are secure. The shift from Pod Security Policies (PSPs) to Pod Security Standards (PSS) and Pod Security Admission (PSA) represents a significant evolution in how Kubernetes handles pod security.

Pod Security Standards

As of Kubernetes 1.25, Pod Security Policies (PSPs) have been removed and replaced with Pod Security Standards. These standards define three levels of security:

Privileged

The Privileged standard is unrestricted and allows for known privilege escalations. This level should only be used for system-level workloads that require elevated privileges.

Baseline

The Baseline standard prevents known privilege escalations while remaining minimally restrictive. It is suitable for most common workloads that do not require elevated privileges.

Restricted

The Restricted standard is heavily restricted and follows current pod hardening best practices. This level is recommended for security-critical applications.

Implementing Pod Security Standards

To implement Pod Security Standards at the namespace level, you can use Pod Security Admission labels:

1
2
3
4
5
6
7
8

apiVersion: v1
kind: Namespace
metadata:
  name: production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted

The three modes available are:

  • enforce: Policy violations will cause the pod to be rejected
  • audit: Policy violations will trigger an audit annotation but pods will be allowed
  • warn: Policy violations will trigger a user-facing warning but pods will be allowed

Security Contexts

Security Contexts define privilege and access control settings for pods and containers. They allow you to specify settings such as user and group IDs, capabilities, and SELinux options.

Pod-Level Security Context

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

apiVersion: v1
kind: Pod
metadata:
  name: secure-pod
spec:
  securityContext:
    runAsUser: 1000
    runAsGroup: 3000
    fsGroup: 2000
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: nginx:1.21
    resources:
      limits:
        memory: "128Mi"
        cpu: "500m"

In this example:

  • runAsUser: Specifies the user ID to run the container process
  • runAsGroup: Specifies the primary group ID
  • fsGroup: Specifies a special supplemental group for volume access
  • seccompProfile: Enables seccomp to restrict system calls

Container-Level Security Context

Container-level security contexts override pod-level settings for specific containers:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24

apiVersion: v1
kind: Pod
metadata:
  name: secure-container
spec:
  containers:
  - name: app
    image: myapp:1.0
    securityContext:
      runAsNonRoot: true
      runAsUser: 1000
      allowPrivilegeEscalation: false
      readOnlyRootFilesystem: true
      capabilities:
        drop:
        - ALL
        add:
        - NET_BIND_SERVICE
    volumeMounts:
    - name: cache
      mountPath: /cache
  volumes:
  - name: cache
    emptyDir: {}

Key security settings:

  • runAsNonRoot: Ensures the container runs as a non-root user
  • allowPrivilegeEscalation: Prevents gaining more privileges than the parent process
  • readOnlyRootFilesystem: Makes the root filesystem read-only
  • capabilities: Drops all capabilities and only adds what is necessary

Network Policies

Network Policies control traffic flow between pods and network endpoints. By default, pods accept traffic from any source. Network Policies allow you to implement network segmentation.

Default Deny All Ingress and Egress

1
2
3
4
5
6
7
8
9
10

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny-all
  namespace: production
spec:
  podSelector: {}
  policyTypes:
  - Ingress
  - Egress

This policy denies all ingress and egress traffic to all pods in the namespace.

Allow Specific Traffic

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-frontend-to-backend
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Ingress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - protocol: TCP
      port: 8080

This policy allows traffic from pods labeled app: frontend to pods labeled app: backend on port 8080.

Egress Policy for External Services

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-and-api
  namespace: production
spec:
  podSelector:
    matchLabels:
      app: backend
  policyTypes:
  - Egress
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432

This policy allows backend pods to access DNS services and the database.

Role-Based Access Control (RBAC)

RBAC regulates access to Kubernetes resources based on the roles of individual users or service accounts.

Creating a Role

1
2
3
4
5
6
7
8
9
10
11
12

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: pod-reader
  namespace: production
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "watch", "list"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]

Creating a RoleBinding

1
2
3
4
5
6
7
8
9
10
11
12
13

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: read-pods
  namespace: production
subjects:
- kind: ServiceAccount
  name: app-service-account
  namespace: production
roleRef:
  kind: Role
  name: pod-reader
  apiGroup: rbac.authorization.k8s.io

ClusterRole for Cluster-Wide Permissions

1
2
3
4
5
6
7
8
9
10
11

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: node-reader
rules:
- apiGroups: [""]
  resources: ["nodes"]
  verbs: ["get", "list", "watch"]
- apiGroups: ["metrics.k8s.io"]
  resources: ["nodes"]
  verbs: ["get", "list"]

Service Account for Pods

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

apiVersion: v1
kind: ServiceAccount
metadata:
  name: app-service-account
  namespace: production
automountServiceAccountToken: false
---
apiVersion: v1
kind: Pod
metadata:
  name: secure-app
  namespace: production
spec:
  serviceAccountName: app-service-account
  automountServiceAccountToken: false
  containers:
  - name: app
    image: myapp:1.0

Setting automountServiceAccountToken: false prevents automatic mounting of service account tokens unless explicitly needed.

Secrets Management

Kubernetes Secrets store sensitive information such as passwords, tokens, and keys. However, default Secrets are only base64-encoded, not encrypted at rest.

Creating Secrets

1
2
3
4
5
6
7
8
9

apiVersion: v1
kind: Secret
metadata:
  name: db-credentials
  namespace: production
type: Opaque
data:
  username: YWRtaW4=
  password: cGFzc3dvcmQxMjM=

Using Secrets in Pods

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

apiVersion: v1
kind: Pod
metadata:
  name: app-with-secret
spec:
  containers:
  - name: app
    image: myapp:1.0
    env:
    - name: DB_USERNAME
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASSWORD
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password
    volumeMounts:
    - name: secret-volume
      mountPath: /etc/secrets
      readOnly: true
  volumes:
  - name: secret-volume
    secret:
      secretName: db-credentials

Encryption at Rest

Enable encryption at rest by configuring the API server:

1
2
3
4
5
6
7
8
9
10
11

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources:
  - secrets
  providers:
  - aescbc:
      keys:
      - name: key1
        secret: <base64-encoded-32-byte-key>
  - identity: {}

Pass this configuration to the API server using the --encryption-provider-config flag.

Using External Secrets Management

For production environments, consider using external secrets management solutions:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secrets-manager
  namespace: production
spec:
  provider:
    aws:
      service: SecretsManager
      region: us-east-1
      auth:
        jwt:
          serviceAccountRef:
            name: external-secrets-sa
---
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
  namespace: production
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secrets-manager
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: prod/database
      property: username
  - secretKey: password
    remoteRef:
      key: prod/database
      property: password

Container Security

Container security involves securing the container image and runtime environment.

Image Security Best Practices

  1. Use minimal base images
  2. Scan images for vulnerabilities
  3. Sign and verify images
  4. Use specific image tags, not latest
  5. Remove unnecessary packages and tools

Dockerfile Security Best Practices

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35

# Use minimal base image
FROM alpine:3.18 AS builder

# Create non-root user
RUN addgroup -g 1000 appgroup && \
    adduser -D -u 1000 -G appgroup appuser

# Install dependencies
RUN apk add --no-cache ca-certificates

# Copy application files
WORKDIR /app
COPY --chown=appuser:appgroup . .

# Build application
RUN go build -o myapp

# Final stage
FROM alpine:3.18

# Copy user from builder
COPY --from=builder /etc/passwd /etc/passwd
COPY --from=builder /etc/group /etc/group

# Copy application
COPY --from=builder --chown=appuser:appgroup /app/myapp /app/myapp

# Switch to non-root user
USER appuser

# Expose port
EXPOSE 8080

# Run application
CMD ["/app/myapp"]

Image Scanning

Image scanning identifies vulnerabilities in container images before deployment.

Using Trivy for Image Scanning

1
2
3
4
5
6
7
8
9
10
11
12
13
14

# Install Trivy
curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin

# Scan an image
trivy image nginx:1.21

# Scan with specific severity
trivy image --severity HIGH,CRITICAL nginx:1.21

# Output as JSON
trivy image --format json --output results.json nginx:1.21

# Scan filesystem
trivy fs --security-checks vuln,config .

Trivy Operator in Kubernetes

Deploy Trivy Operator to continuously scan images in your cluster:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36

apiVersion: v1
kind: Namespace
metadata:
  name: trivy-system
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: trivy-operator
  namespace: trivy-system
spec:
  replicas: 1
  selector:
    matchLabels:
      app: trivy-operator
  template:
    metadata:
      labels:
        app: trivy-operator
    spec:
      serviceAccountName: trivy-operator
      automountServiceAccountToken: true
      containers:
      - name: operator
        image: aquasecurity/trivy-operator:0.16.0
        env:
        - name: OPERATOR_NAMESPACE
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: OPERATOR_TARGET_NAMESPACES
          value: ""
        resources:
          limits:
            memory: "256Mi"
            cpu: "500m"

Admission Controllers for Image Validation

Use admission controllers to enforce image policies:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
metadata:
  name: image-validation
webhooks:
- name: validate-images.security.io
  clientConfig:
    service:
      name: image-validator
      namespace: security
      path: /validate
    caBundle: <base64-encoded-ca-cert>
  rules:
  - operations: ["CREATE", "UPDATE"]
    apiGroups: [""]
    apiVersions: ["v1"]
    resources: ["pods"]
  admissionReviewVersions: ["v1"]
  sideEffects: None
  failurePolicy: Fail

Runtime Security

Runtime security monitors and protects containers during execution.

Falco for Runtime Security

Falco is a runtime security tool that detects anomalous behavior:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94

apiVersion: v1
kind: ConfigMap
metadata:
  name: falco-config
  namespace: falco
data:
  falco.yaml: |
    rules_file:
      - /etc/falco/falco_rules.yaml
      - /etc/falco/falco_rules.local.yaml
      - /etc/falco/k8s_audit_rules.yaml
    json_output: true
    json_include_output_property: true
    log_stderr: true
    log_syslog: true
    log_level: info
    priority: debug
    syscall_event_drops:
      threshold: 0.1
      actions:
        - log
        - alert
  falco_rules.local.yaml: |
    - rule: Unauthorized Process in Container
      desc: Detect process execution outside allowed list
      condition: >
        spawned_process and
        container and
        not proc.name in (allowed_processes)
      output: >
        Unauthorized process started
        (user=%user.name command=%proc.cmdline container=%container.name)
      priority: WARNING
    - list: allowed_processes
      items: [nginx, java, python, node]
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: falco
  namespace: falco
spec:
  selector:
    matchLabels:
      app: falco
  template:
    metadata:
      labels:
        app: falco
    spec:
      serviceAccountName: falco
      hostNetwork: true
      hostPID: true
      containers:
      - name: falco
        image: falcosecurity/falco:0.36.0
        securityContext:
          privileged: true
        volumeMounts:
        - name: dev
          mountPath: /host/dev
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: boot
          mountPath: /host/boot
          readOnly: true
        - name: modules
          mountPath: /host/lib/modules
          readOnly: true
        - name: usr
          mountPath: /host/usr
          readOnly: true
        - name: config
          mountPath: /etc/falco
      volumes:
      - name: dev
        hostPath:
          path: /dev
      - name: proc
        hostPath:
          path: /proc
      - name: boot
        hostPath:
          path: /boot
      - name: modules
        hostPath:
          path: /lib/modules
      - name: usr
        hostPath:
          path: /usr
      - name: config
        configMap:
          name: falco-config

Pod Security Best Practices Checklist

Image Security

  • Use trusted base images from official sources
  • Scan images for vulnerabilities before deployment
  • Use specific image tags instead of latest
  • Implement image signing and verification
  • Regularly update base images

Container Configuration

  • Run containers as non-root users
  • Use read-only root filesystems where possible
  • Drop all capabilities and add only what is needed
  • Disable privilege escalation
  • Set resource limits for CPU and memory

Network Security

  • Implement Network Policies for ingress and egress
  • Use default deny policies
  • Segment workloads by namespace
  • Encrypt traffic between services using service mesh
  • Limit exposed ports and services

Access Control

  • Implement least privilege RBAC policies
  • Use separate service accounts for different workloads
  • Disable automatic mounting of service account tokens
  • Regularly audit RBAC policies
  • Use namespaces for logical separation

Secrets Management

  • Never store secrets in container images or environment variables
  • Use Kubernetes Secrets with encryption at rest
  • Consider external secrets management solutions
  • Rotate secrets regularly
  • Limit secret access to only necessary pods

Monitoring and Auditing

  • Enable audit logging for API server
  • Monitor runtime behavior with tools like Falco
  • Set up alerts for security events
  • Regularly review security logs
  • Implement centralized logging

Example: Secure Production Deployment

Here is a complete example of a secure production deployment:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171

apiVersion: v1
kind: Namespace
metadata:
  name: secure-production
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: secure-app-sa
  namespace: secure-production
automountServiceAccountToken: false
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: app-role
  namespace: secure-production
rules:
- apiGroups: [""]
  resources: ["configmaps"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: app-role-binding
  namespace: secure-production
subjects:
- kind: ServiceAccount
  name: secure-app-sa
  namespace: secure-production
roleRef:
  kind: Role
  name: app-role
  apiGroup: rbac.authorization.k8s.io
---
apiVersion: v1
kind: Secret
metadata:
  name: app-secrets
  namespace: secure-production
type: Opaque
stringData:
  api-key: "your-api-key-here"
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: app-network-policy
  namespace: secure-production
spec:
  podSelector:
    matchLabels:
      app: secure-app
  policyTypes:
  - Ingress
  - Egress
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: ingress-controller
    ports:
    - protocol: TCP
      port: 8080
  egress:
  - to:
    - namespaceSelector:
        matchLabels:
          name: kube-system
    ports:
    - protocol: UDP
      port: 53
  - to:
    - podSelector:
        matchLabels:
          app: database
    ports:
    - protocol: TCP
      port: 5432
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secure-app
  namespace: secure-production
spec:
  replicas: 3
  selector:
    matchLabels:
      app: secure-app
  template:
    metadata:
      labels:
        app: secure-app
    spec:
      serviceAccountName: secure-app-sa
      automountServiceAccountToken: false
      securityContext:
        runAsNonRoot: true
        runAsUser: 1000
        runAsGroup: 3000
        fsGroup: 2000
        seccompProfile:
          type: RuntimeDefault
      containers:
      - name: app
        image: myregistry.io/secure-app:1.2.3
        imagePullPolicy: Always
        securityContext:
          allowPrivilegeEscalation: false
          readOnlyRootFilesystem: true
          capabilities:
            drop:
            - ALL
        ports:
        - containerPort: 8080
          protocol: TCP
        env:
        - name: API_KEY
          valueFrom:
            secretKeyRef:
              name: app-secrets
              key: api-key
        resources:
          requests:
            memory: "128Mi"
            cpu: "100m"
          limits:
            memory: "256Mi"
            cpu: "500m"
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          initialDelaySeconds: 30
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          initialDelaySeconds: 5
          periodSeconds: 5
        volumeMounts:
        - name: cache
          mountPath: /cache
        - name: tmp
          mountPath: /tmp
      volumes:
      - name: cache
        emptyDir: {}
      - name: tmp
        emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: secure-app
  namespace: secure-production
spec:
  selector:
    app: secure-app
  ports:
  - protocol: TCP
    port: 80
    targetPort: 8080
  type: ClusterIP

Conclusion

Securing Kubernetes pods requires a comprehensive approach that addresses multiple layers of security. By implementing Pod Security Standards, Security Contexts, Network Policies, RBAC, proper Secrets Management, Container Security, Image Scanning, and Runtime Security, you can significantly reduce the attack surface of your Kubernetes workloads.

Remember that security is not a one-time task but an ongoing process. Regularly review and update your security policies, stay informed about new vulnerabilities and best practices, and continuously monitor your cluster for anomalous behavior.

References

  • Kubernetes Pod Security Standards: https://kubernetes.io/docs/concepts/security/pod-security-standards/
  • Pod Security Admission: https://kubernetes.io/docs/concepts/security/pod-security-admission/
  • Network Policies: https://kubernetes.io/docs/concepts/services-networking/network-policies/
  • RBAC Authorization: https://kubernetes.io/docs/reference/access-authn-authz/rbac/
  • Secrets Management: https://kubernetes.io/docs/concepts/configuration/secret/
  • Trivy Documentation: https://aquasecurity.github.io/trivy/
  • Falco Documentation: https://falco.org/docs/
  • CIS Kubernetes Benchmark: https://www.cisecurity.org/benchmark/kubernetes
  • OWASP Kubernetes Security Cheat Sheet: https://cheatsheetseries.owasp.org/cheatsheets/Kubernetes_Security_Cheat_Sheet.html
  • NSA/CISA Kubernetes Hardening Guide: https://media.defense.gov/2022/Aug/29/2003066362/-1/-1/0/CTR_KUBERNETES_HARDENING_GUIDANCE_1.2_20220829.PDF
DevOps
kubernetes k8s security pod-security devops containers best-practices cloud-native
This post is licensed under CC BY 4.0 by the author.