CI/CD Pipeline Design Patterns with GitHub Actions
GitHub Actions has become one of the most popular CI/CD platforms due to its tight integration with GitHub repositories, extensive marketplace, and flexible workflow design. In this comprehensive guide, we will explore various design patterns, best practices, and advanced techniques for building robust CI/CD pipelines using GitHub Actions.
Understanding GitHub Actions Fundamentals
Before diving into design patterns, let us understand the core concepts:
- Workflows: Automated processes defined in YAML files stored in
.github/workflows/ - Events: Triggers that start workflows (push, pull request, schedule, etc.)
- Jobs: Sets of steps that execute on the same runner
- Steps: Individual tasks within a job (actions or shell commands)
- Actions: Reusable units of code that can be shared across workflows
- Runners: Virtual machines that execute jobs
Workflow Design Patterns
Pattern 1: Branch-Based Workflows
Branch-based workflows execute different pipelines based on the branch being updated.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
| name: Branch-Based Deployment
on:
push:
branches:
- main
- develop
- 'release/**'
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: '20'
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
- name: Build application
run: npm run build
deploy-dev:
needs: build
if: github.ref == 'refs/heads/develop'
runs-on: ubuntu-latest
environment:
name: development
url: https://dev.example.com
steps:
- name: Deploy to Development
run: |
echo "Deploying to development environment"
# Add deployment commands here
deploy-staging:
needs: build
if: startsWith(github.ref, 'refs/heads/release/')
runs-on: ubuntu-latest
environment:
name: staging
url: https://staging.example.com
steps:
- name: Deploy to Staging
run: |
echo "Deploying to staging environment"
# Add deployment commands here
deploy-prod:
needs: build
if: github.ref == 'refs/heads/main'
runs-on: ubuntu-latest
environment:
name: production
url: https://example.com
steps:
- name: Deploy to Production
run: |
echo "Deploying to production environment"
# Add deployment commands here
|
Pattern 2: Pull Request Validation
Comprehensive validation before merging changes:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
| name: Pull Request Validation
on:
pull_request:
types: [opened, synchronize, reopened]
branches:
- main
- develop
permissions:
contents: read
pull-requests: write
checks: write
jobs:
code-quality:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- name: Install dependencies
run: |
pip install -r requirements.txt
pip install pylint black isort mypy
- name: Run Black formatter check
run: black --check .
- name: Run isort check
run: isort --check-only .
- name: Run Pylint
run: pylint src/
- name: Run MyPy type checking
run: mypy src/
unit-tests:
runs-on: ubuntu-latest
strategy:
matrix:
python-version: ['3.9', '3.10', '3.11']
steps:
- uses: actions/checkout@v4
- name: Set up Python $
uses: actions/setup-python@v5
with:
python-version: $
cache: 'pip'
- name: Install dependencies
run: pip install -r requirements.txt
- name: Run unit tests
run: pytest tests/unit -v --cov=src --cov-report=xml
- name: Upload coverage to Codecov
uses: codecov/codecov-action@v4
with:
files: ./coverage.xml
flags: unittests-py$
integration-tests:
runs-on: ubuntu-latest
services:
postgres:
image: postgres:15
env:
POSTGRES_PASSWORD: postgres
options: >-
--health-cmd pg_isready
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 5432:5432
redis:
image: redis:7
options: >-
--health-cmd "redis-cli ping"
--health-interval 10s
--health-timeout 5s
--health-retries 5
ports:
- 6379:6379
steps:
- uses: actions/checkout@v4
- name: Set up Python
uses: actions/setup-python@v5
with:
python-version: '3.11'
cache: 'pip'
- name: Install dependencies
run: pip install -r requirements.txt
- name: Run integration tests
env:
DATABASE_URL: postgresql://postgres:postgres@localhost:5432/testdb
REDIS_URL: redis://localhost:6379
run: pytest tests/integration -v
security-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Bandit security scan
run: |
pip install bandit
bandit -r src/ -f json -o bandit-report.json
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
scan-type: 'fs'
scan-ref: '.'
format: 'sarif'
output: 'trivy-results.sarif'
- name: Upload Trivy results to GitHub Security
uses: github/codeql-action/upload-sarif@v3
if: always()
with:
sarif_file: 'trivy-results.sarif'
comment-pr:
needs: [code-quality, unit-tests, integration-tests, security-scan]
if: always()
runs-on: ubuntu-latest
steps:
- name: Comment PR
uses: actions/github-script@v7
with:
script: |
const results = {
'Code Quality': '$',
'Unit Tests': '$',
'Integration Tests': '$',
'Security Scan': '$'
};
let comment = '## PR Validation Results\n\n';
for (const [check, result] of Object.entries(results)) {
const emoji = result === 'success' ? '✅' : result === 'failure' ? '❌' : '⚠️';
comment += `${emoji} **${check}**: ${result}\n`;
}
github.rest.issues.createComment({
issue_number: context.issue.number,
owner: context.repo.owner,
repo: context.repo.repo,
body: comment
});
|
Pattern 3: Reusable Workflows
Create reusable workflows to maintain consistency across repositories:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
| # .github/workflows/reusable-build.yml
name: Reusable Build Workflow
on:
workflow_call:
inputs:
node-version:
required: true
type: string
environment:
required: true
type: string
runs-on:
required: false
type: string
default: 'ubuntu-latest'
secrets:
deploy-token:
required: true
outputs:
artifact-url:
description: "URL of the build artifact"
value: $
jobs:
build:
runs-on: $
outputs:
artifact-url: $
steps:
- uses: actions/checkout@v4
- name: Set up Node.js
uses: actions/setup-node@v4
with:
node-version: $
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Run tests
run: npm test
- name: Build for $
run: npm run build
env:
NODE_ENV: $
- name: Upload artifact
id: upload
uses: actions/upload-artifact@v4
with:
name: build-$
path: dist/
retention-days: 7
|
Using the reusable workflow:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
| # .github/workflows/main.yml
name: Main Pipeline
on:
push:
branches: [main]
jobs:
build-dev:
uses: ./.github/workflows/reusable-build.yml
with:
node-version: '20'
environment: 'development'
secrets:
deploy-token: $
build-prod:
uses: ./.github/workflows/reusable-build.yml
with:
node-version: '20'
environment: 'production'
secrets:
deploy-token: $
|
Pattern 4: Matrix Builds
Test across multiple versions, operating systems, and configurations:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
| name: Matrix Build Strategy
on: [push, pull_request]
jobs:
test:
runs-on: $
strategy:
fail-fast: false
matrix:
os: [ubuntu-latest, windows-latest, macos-latest]
python-version: ['3.9', '3.10', '3.11', '3.12']
exclude:
- os: macos-latest
python-version: '3.9'
include:
- os: ubuntu-latest
python-version: '3.13-dev'
experimental: true
continue-on-error: $
steps:
- uses: actions/checkout@v4
- name: Set up Python $
uses: actions/setup-python@v5
with:
python-version: $
- name: Install dependencies
run: |
python -m pip install --upgrade pip
pip install -r requirements.txt
- name: Run tests
run: pytest tests/ -v
- name: Upload results
if: always()
uses: actions/upload-artifact@v4
with:
name: test-results-$-$
path: test-results/
|
Pattern 5: Composite Actions
Create reusable composite actions:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
| # .github/actions/setup-project/action.yml
name: Setup Project
description: Setup Node.js project with caching and dependencies
inputs:
node-version:
description: 'Node.js version to use'
required: true
default: '20'
install-dependencies:
description: 'Install npm dependencies'
required: false
default: 'true'
runs:
using: composite
steps:
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: $
cache: 'npm'
- name: Cache node modules
uses: actions/cache@v4
with:
path: ~/.npm
key: $-node-$
restore-keys: |
$-node-
- name: Install dependencies
if: inputs.install-dependencies == 'true'
shell: bash
run: npm ci
- name: Display versions
shell: bash
run: |
node --version
npm --version
|
Using the composite action:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
| name: Use Composite Action
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Setup project
uses: ./.github/actions/setup-project
with:
node-version: '20'
- name: Build
run: npm run build
|
Caching Strategies
Effective caching significantly reduces build times:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
| name: Advanced Caching
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# Cache NPM dependencies
- name: Cache NPM dependencies
uses: actions/cache@v4
with:
path: |
~/.npm
node_modules
key: $-npm-$
restore-keys: |
$-npm-
# Cache build outputs
- name: Cache build
uses: actions/cache@v4
with:
path: |
dist
.next/cache
key: $-build-$
restore-keys: |
$-build-
# Cache Docker layers
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Cache Docker layers
uses: actions/cache@v4
with:
path: /tmp/.buildx-cache
key: $-buildx-$
restore-keys: |
$-buildx-
- name: Build Docker image
uses: docker/build-push-action@v5
with:
context: .
push: false
tags: myapp:latest
cache-from: type=local,src=/tmp/.buildx-cache
cache-to: type=local,dest=/tmp/.buildx-cache-new,mode=max
- name: Move cache
run: |
rm -rf /tmp/.buildx-cache
mv /tmp/.buildx-cache-new /tmp/.buildx-cache
|
Secrets Management
Properly manage sensitive information:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
| name: Secrets Management
on: [push]
jobs:
deploy:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
# Use repository secrets
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: $
aws-secret-access-key: $
aws-region: us-east-1
# Use environment secrets
- name: Deploy to production
env:
API_KEY: $
DATABASE_URL: $
run: |
echo "Deploying with environment-specific secrets"
# Deployment commands
# Use GitHub App authentication
- name: Generate token
id: generate-token
uses: actions/create-github-app-token@v1
with:
app-id: $
private-key: $
- name: Use token
env:
GH_TOKEN: $
run: |
gh api repos/$/releases
|
Security Scanning
Implement comprehensive security scanning:
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
| name: Security Scanning
on:
push:
branches: [main]
pull_request:
branches: [main]
schedule:
- cron: '0 0 * * 0'
jobs:
codeql:
runs-on: ubuntu-latest
permissions:
security-events: write
actions: read
contents: read
steps:
- uses: actions/checkout@v4
- name: Initialize CodeQL
uses: github/codeql-action/init@v3
with:
languages: javascript, python
queries: security-and-quality
- name: Autobuild
uses: github/codeql-action/autobuild@v3
- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v3
dependency-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Run Snyk to check for vulnerabilities
uses: snyk/actions/node@master
env:
SNYK_TOKEN: $
with:
args: --severity-threshold=high
- name: Dependency Review
uses: actions/dependency-review-action@v4
if: github.event_name == 'pull_request'
secret-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Gitleaks scan
uses: gitleaks/gitleaks-action@v2
env:
GITHUB_TOKEN: $
container-scan:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build image
run: docker build -t myapp:$ .
- name: Run Trivy vulnerability scanner
uses: aquasecurity/trivy-action@master
with:
image-ref: myapp:$
format: 'sarif'
output: 'trivy-results.sarif'
severity: 'CRITICAL,HIGH'
- name: Upload Trivy results
uses: github/codeql-action/upload-sarif@v3
with:
sarif_file: 'trivy-results.sarif'
|
Deployment Strategies
Blue-Green Deployment
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
| name: Blue-Green Deployment
on:
push:
branches: [main]
jobs:
deploy:
runs-on: ubuntu-latest
environment: production
steps:
- uses: actions/checkout@v4
- name: Configure AWS credentials
uses: aws-actions/configure-aws-credentials@v4
with:
aws-access-key-id: $
aws-secret-access-key: $
aws-region: us-east-1
- name: Deploy to Green environment
run: |
aws ecs update-service \
--cluster production-cluster \
--service myapp-green \
--force-new-deployment
- name: Wait for Green deployment
run: |
aws ecs wait services-stable \
--cluster production-cluster \
--services myapp-green
- name: Run smoke tests
run: |
curl -f https://green.example.com/health || exit 1
- name: Switch traffic to Green
run: |
aws elbv2 modify-listener \
--listener-arn $ \
--default-actions Type=forward,TargetGroupArn=$
- name: Deactivate Blue environment
run: |
aws ecs update-service \
--cluster production-cluster \
--service myapp-blue \
--desired-count 0
|
Canary Deployment
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
| name: Canary Deployment
on:
push:
branches: [main]
jobs:
deploy-canary:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Deploy canary (10% traffic)
run: |
kubectl set image deployment/myapp \
myapp=myapp:$ \
-n production
kubectl patch deployment myapp \
-n production \
-p '{"spec":{"replicas":1}}'
- name: Wait and monitor metrics
run: |
sleep 300
# Check error rates, latency, etc.
- name: Scale to 50% traffic
run: |
kubectl scale deployment myapp \
--replicas=5 \
-n production
- name: Wait and monitor
run: sleep 300
- name: Full deployment
run: |
kubectl scale deployment myapp \
--replicas=10 \
-n production
|
Monitoring and Notifications
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
| name: Build with Notifications
on: [push, pull_request]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Build application
run: npm run build
- name: Run tests
run: npm test
- name: Notify Slack on success
if: success()
uses: slackapi/slack-github-action@v1
with:
payload: |
{
"text": "Build succeeded for $",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*Build Status:* ✅ Success\n*Repository:* $\n*Branch:* $\n*Commit:* $"
}
}
]
}
env:
SLACK_WEBHOOK_URL: $
- name: Notify Slack on failure
if: failure()
uses: slackapi/slack-github-action@v1
with:
payload: |
{
"text": "Build failed for $",
"blocks": [
{
"type": "section",
"text": {
"type": "mrkdwn",
"text": "*Build Status:* ❌ Failed\n*Repository:* $\n*Branch:* $\n*Commit:* $\n*Logs:* $/$/actions/runs/$"
}
}
]
}
env:
SLACK_WEBHOOK_URL: $
|
Best Practices
Workflow Organization
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
| name: Organized Workflow
on:
push:
branches: [main, develop]
paths:
- 'src/**'
- 'tests/**'
- 'package*.json'
pull_request:
branches: [main]
paths:
- 'src/**'
- 'tests/**'
concurrency:
group: $-$
cancel-in-progress: true
env:
NODE_VERSION: '20'
CACHE_VERSION: v1
jobs:
validate:
name: Validate Code
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- name: Checkout code
uses: actions/checkout@v4
- name: Setup Node.js
uses: actions/setup-node@v4
with:
node-version: $
cache: 'npm'
- name: Install dependencies
run: npm ci
- name: Lint code
run: npm run lint
- name: Check formatting
run: npm run format:check
test:
name: Run Tests
needs: validate
runs-on: ubuntu-latest
timeout-minutes: 15
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: $
cache: 'npm'
- run: npm ci
- name: Run unit tests
run: npm test -- --coverage
- name: Upload coverage
uses: codecov/codecov-action@v4
with:
token: $
build:
name: Build Application
needs: test
runs-on: ubuntu-latest
timeout-minutes: 20
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: $
cache: 'npm'
- run: npm ci
- name: Build
run: npm run build
- name: Upload build artifact
uses: actions/upload-artifact@v4
with:
name: build-output
path: dist/
retention-days: 7
|
Error Handling
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
| name: Error Handling
on: [push]
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Deploy with retries
uses: nick-invision/retry@v3
with:
timeout_minutes: 10
max_attempts: 3
retry_wait_seconds: 30
command: |
npm run deploy
- name: Rollback on failure
if: failure()
run: |
echo "Deployment failed, initiating rollback"
npm run rollback
- name: Cleanup
if: always()
run: |
echo "Cleaning up temporary resources"
npm run cleanup
|
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
| name: Optimized Pipeline
on: [push]
jobs:
build:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# Use sparse checkout for large repos
- name: Sparse checkout
run: |
git config core.sparseCheckout true
echo "src/" >> .git/info/sparse-checkout
echo "tests/" >> .git/info/sparse-checkout
git read-tree -mu HEAD
# Parallel job execution
- name: Install and test in parallel
run: |
npm ci &
PID=$!
# Do other tasks
wait $PID
npm test
# Use buildx for faster Docker builds
- name: Set up Docker Buildx
uses: docker/setup-buildx-action@v3
- name: Build with cache
uses: docker/build-push-action@v5
with:
context: .
platforms: linux/amd64,linux/arm64
cache-from: type=gha
cache-to: type=gha,mode=max
|
Conclusion
GitHub Actions provides a powerful and flexible platform for building CI/CD pipelines. By following these design patterns and best practices, you can create robust, secure, and efficient workflows that scale with your project needs.
Key takeaways:
- Use reusable workflows and composite actions to maintain consistency
- Implement proper caching strategies to improve performance
- Secure your pipelines with proper secrets management and security scanning
- Use matrix builds for comprehensive testing across platforms
- Implement appropriate deployment strategies for your use case
- Monitor and notify your team of pipeline status
References
- GitHub Actions Documentation: https://docs.github.com/en/actions
- Workflow Syntax: https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions
- GitHub Actions Marketplace: https://github.com/marketplace?type=actions
- Security Hardening: https://docs.github.com/en/actions/security-guides/security-hardening-for-github-actions
- Reusing Workflows: https://docs.github.com/en/actions/using-workflows/reusing-workflows
- Caching Dependencies: https://docs.github.com/en/actions/using-workflows/caching-dependencies-to-speed-up-workflows
- GitHub Actions Best Practices: https://docs.github.com/en/actions/learn-github-actions/best-practices-for-using-github-actions
- Environment Protection Rules: https://docs.github.com/en/actions/deployment/targeting-different-environments/using-environments-for-deployment