CI/CD Pipelines

Challenge

Design and implement automated CI/CD pipelines that ensure code quality, enable rapid deployment, and maintain system reliability.

Explanation

CI/CD Stages

• Trigger: Source code changes
• Build: Compile and package
• Test: Unit, integration, E2E tests
• Quality: Linting, code analysis
• Deploy: Staging and production
• Monitor: Health checks and alerts

Popular Platforms

• GitHub Actions: GitHub-integrated
• GitLab CI/CD: GitLab platform
• Jenkins: Self-hosted, flexible
• CircleCI: Cloud-based
• AWS CodePipeline: AWS-native

Code Examples

1. GitHub Actions Workflow

.github/workflows/ci.yml

name: CI/CD Pipeline

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]

jobs:
  build-and-test:
    runs-on: ubuntu-latest

    services:
      postgres:
        image: postgres:15
        env:
          POSTGRES_PASSWORD: postgres
        options: >-
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5
        ports:
          - 5432:5432

    steps:
      - uses: actions/checkout@v3

      - name: Setup Node.js
        uses: actions/setup-node@v3
        with:
          node-version: '18'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Lint
        run: npm run lint

      - name: Run tests
        run: npm test
        env:
          DATABASE_URL: postgres://postgres:postgres@localhost:5432/test

      - name: Generate coverage
        run: npm run test:coverage

      - name: Upload coverage
        uses: codecov/codecov-action@v3
        with:
          files: ./coverage/coverage-final.json

  security-scan:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3

      - name: Run npm audit
        run: npm audit --production

      - name: Dependency check
        uses: dependency-check/Dependency-Check_Action@main
        with:
          project: 'MyApp'
          path: '.'
          format: 'JSON'

  build-image:
    needs: [build-and-test, security-scan]
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'

    steps:
      - uses: actions/checkout@v3

      - name: Set up Docker Buildx
        uses: docker/setup-buildx-action@v2

      - name: Login to DockerHub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKER_USERNAME }}
          password: ${{ secrets.DOCKER_PASSWORD }}

      - name: Build and push
        uses: docker/build-push-action@v4
        with:
          context: .
          push: true
          tags: ${{ secrets.DOCKER_USERNAME }}/myapp:latest
          cache-from: type=registry,ref=${{ secrets.DOCKER_USERNAME }}/myapp:buildcache
          cache-to: type=registry,ref=${{ secrets.DOCKER_USERNAME }}/myapp:buildcache,mode=max

  deploy-staging:
    needs: build-image
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'

    steps:
      - name: Deploy to staging
        run: |
          curl -X POST ${{ secrets.STAGING_WEBHOOK }} \
            -H 'Content-Type: application/json' \
            -d '{"action":"deploy","version":"latest"}'

      - name: Run smoke tests
        run: |
          bash scripts/smoke-tests.sh ${{ secrets.STAGING_URL }}

  deploy-production:
    needs: deploy-staging
    runs-on: ubuntu-latest
    if: github.ref == 'refs/heads/main'

    steps:
      - name: Request approval
        uses: trstringer/manual-approval@v1
        with:
          secret: ${{ secrets.GITHUB_TOKEN }}
          approvers: team-leads
          issue-title: 'Production deployment approval'

      - name: Deploy to production
        run: |
          curl -X POST ${{ secrets.PRODUCTION_WEBHOOK }} \
            -H 'Content-Type: application/json' \
            -d '{"action":"deploy","version":"latest"}'

      - name: Health check
        run: |
          bash scripts/health-check.sh ${{ secrets.PRODUCTION_URL }}

      - name: Notify Slack
        if: success()
        uses: slackapi/slack-github-action@v1
        with:
          payload: |
            {
              "text": "Production deployment successful",
              "blocks": [
                {
                  "type": "section",
                  "text": {
                    "type": "mrkdwn",
                    "text": "Deployment to production completed successfully"
                  }
                }
              ]
            }

2. GitLab CI/CD

.gitlab-ci.yml

stages:
  - build
  - test
  - quality
  - deploy

variables:
  DOCKER_IMAGE: registry.gitlab.com/$CI_PROJECT_PATH

build:
  stage: build
  image: node:18
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/
    expire_in: 1 hour

test:unit:
  stage: test
  image: node:18
  script:
    - npm ci
    - npm run test:unit
  coverage: '/Lines\s*:\s*(\d+\.\d+)%/'

test:integration:
  stage: test
  image: node:18
  services:
    - postgres:15
  script:
    - npm ci
    - npm run test:integration
  variables:
    DATABASE_URL: postgres://postgres:postgres@postgres:5432/test

lint:
  stage: quality
  image: node:18
  script:
    - npm ci
    - npm run lint

deploy:staging:
  stage: deploy
  image: alpine:latest
  script:
    - apk add --no-cache curl
    - curl -X POST $STAGING_WEBHOOK -d "version=$CI_COMMIT_SHA"
  only:
    - develop
  environment:
    name: staging
    url: https://staging.example.com

deploy:production:
  stage: deploy
  image: alpine:latest
  script:
    - apk add --no-cache curl
    - curl -X POST $PRODUCTION_WEBHOOK -d "version=$CI_COMMIT_SHA"
  only:
    - main
  when: manual
  environment:
    name: production
    url: https://api.example.com

3. Node.js Pipeline Script

scripts/pipeline.js

const { spawn } = require('child_process');
const fs = require('fs');

class Pipeline {
  constructor() {
    this.stages = [];
    this.results = [];
  }

  addStage(name, command, args = []) {
    this.stages.push({ name, command, args });
  }

  async run() {
    console.log('Starting CI/CD pipeline...\n');

    for (const stage of this.stages) {
      const result = await this.runStage(stage);
      this.results.push(result);

      if (!result.success) {
        console.error(`\n❌ Pipeline failed at stage: ${stage.name}`);
        process.exit(1);
      }
    }

    this.printReport();
  }

  async runStage(stage) {
    const startTime = Date.now();
    console.log(`▶ ${stage.name}...`);

    return new Promise((resolve) => {
      const process = spawn(stage.command, stage.args, {
        stdio: 'inherit',
        shell: true,
      });

      process.on('exit', (code) => {
        const duration = Date.now() - startTime;
        const success = code === 0;

        console.log(`${success ? '✓' : '✗'} ${stage.name} (${duration}ms)\n`);

        resolve({
          name: stage.name,
          success,
          duration,
        });
      });
    });
  }

  printReport() {
    console.log('\n📊 Pipeline Report:');
    console.log('==================');

    this.results.forEach(result => {
      const status = result.success ? '✓' : '✗';
      console.log(`${status} ${result.name}: ${result.duration}ms`);
    });

    const totalTime = this.results.reduce((sum, r) => sum + r.duration, 0);
    console.log(`\nTotal: ${totalTime}ms`);
  }
}

// Run pipeline
const pipeline = new Pipeline();

pipeline.addStage('Install', 'npm', ['ci']);
pipeline.addStage('Lint', 'npm', ['run', 'lint']);
pipeline.addStage('Test', 'npm', ['test']);
pipeline.addStage('Build', 'npm', ['run', 'build']);
pipeline.addStage('Security Scan', 'npm', ['audit', '--production']);

pipeline.run().catch(err => {
  console.error('Pipeline error:', err);
  process.exit(1);
});

4. Deployment Script

scripts/deploy.sh

#!/bin/bash
set -e

ENV=${1:-staging}
VERSION=${2:-latest}

echo "Deploying version $VERSION to $ENV..."

# Pull latest image
docker pull $DOCKER_REGISTRY/myapp:$VERSION

# Run container
docker run \
  -d \
  --name myapp-$ENV \
  --restart unless-stopped \
  -p 3000:3000 \
  -e NODE_ENV=$ENV \
  -e DATABASE_URL=$DATABASE_URL \
  --health-cmd="curl -f http://localhost:3000/health || exit 1" \
  --health-interval=30s \
  --health-timeout=10s \
  --health-retries=3 \
  $DOCKER_REGISTRY/myapp:$VERSION

# Wait for container to be healthy
echo "Waiting for container to be healthy..."
sleep 10

if ! docker exec myapp-$ENV curl -f http://localhost:3000/health; then
  echo "Container is not healthy!"
  docker logs myapp-$ENV
  exit 1
fi

echo "✓ Deployment successful"

Best Practices

1. Pipeline Design

   • Keep stages focused
   • Fail fast
   • Parallel stages when possible
   • Clear stage names

2. Testing

   • Run tests early
   • Include unit, integration, E2E
   • Measure coverage
   • Track test trends

3. Quality Gates

   • Enforce code standards
   • Security scanning
   • Coverage thresholds
   • Performance checks

4. Deployment

   • Staging before production
   • Health checks
   • Smoke tests
   • Rollback capability

5. Monitoring

   • Track build times
   • Monitor success rate
   • Alert on failures
   • Report trends

Real-World Applications

• Continuous Delivery: Rapid release cycles
• Automated Testing: Quality assurance
• Infrastructure as Code: Automated setup
• Multi-environment: Dev, staging, production
• Security Scanning: Vulnerability detection