cicd-gitlab-pipelines-dev

name	cicd-gitlab-pipelines-dev
description	Develop and troubleshoot GitLab CI/CD pipelines. Use when creating pipeline YAML files, debugging job failures, understanding GitLab-specific features like DAG, rules, and includes, or optimizing pipeline performance.

GitLab CI/CD Pipeline Development

Guide for developing, debugging, and optimizing GitLab CI/CD pipelines.

When to Use This Skill

Creating new GitLab CI/CD pipelines
Debugging job failures from pipeline logs
Understanding GitLab-specific pipeline features
Optimizing pipeline performance and costs
Configuring runners and environments
Working with includes, extends, and templates

Pipeline Structure

File Location

Pipelines are defined in .gitlab-ci.yml at the repository root.

Basic Structure

stages:
  - build
  - test
  - deploy

variables:
  NODE_VERSION: "20"

build:
  stage: build
  image: node:${NODE_VERSION}
  script:
    - npm ci
    - npm run build
  artifacts:
    paths:
      - dist/

test:
  stage: test
  image: node:${NODE_VERSION}
  script:
    - npm ci
    - npm test

deploy:
  stage: deploy
  script:
    - ./deploy.sh
  only:
    - main

Key Concepts

Stages

Stages define the order of job execution. Jobs in the same stage run in parallel.

stages:
  - build
  - test
  - deploy
  - cleanup

Jobs

Jobs are the basic unit of execution:

job-name:
  stage: test
  image: ruby:3.2
  before_script:
    - bundle install
  script:
    - bundle exec rspec
  after_script:
    - echo "Cleanup"
  artifacts:
    paths:
      - coverage/
    expire_in: 1 week

Variables

# Global variables
variables:
  DATABASE_URL: "postgres://localhost/test"
  RAILS_ENV: test

# Job-level variables
test:
  variables:
    COVERAGE: "true"
  script:
    - echo $COVERAGE

Predefined Variables

script:
  - echo "Project: $CI_PROJECT_NAME"
  - echo "Branch: $CI_COMMIT_REF_NAME"
  - echo "Commit: $CI_COMMIT_SHA"
  - echo "Pipeline: $CI_PIPELINE_ID"
  - echo "Job: $CI_JOB_ID"
  - echo "Runner: $CI_RUNNER_ID"
  - echo "Environment: $CI_ENVIRONMENT_NAME"

Common CI Patterns

Matrix Builds (Parallel)

test:
  stage: test
  parallel:
    matrix:
      - RUBY_VERSION: ["3.1", "3.2", "3.3"]
        DATABASE: ["postgres", "mysql"]
  image: ruby:$RUBY_VERSION
  script:
    - echo "Testing Ruby $RUBY_VERSION with $DATABASE"

Rules (Conditional Execution)

deploy:
  stage: deploy
  script:
    - ./deploy.sh
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
      when: on_success
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
      when: manual
    - when: never

DAG (Directed Acyclic Graph)

Use needs for faster pipelines by skipping stage ordering:

stages:
  - build
  - test
  - deploy

build-frontend:
  stage: build
  script: npm run build:frontend

build-backend:
  stage: build
  script: npm run build:backend

test-frontend:
  stage: test
  needs: [build-frontend]  # Runs as soon as build-frontend completes
  script: npm run test:frontend

test-backend:
  stage: test
  needs: [build-backend]
  script: npm run test:backend

deploy:
  stage: deploy
  needs: [test-frontend, test-backend]
  script: ./deploy.sh

Caching

cache:
  key: ${CI_COMMIT_REF_SLUG}
  paths:
    - node_modules/
    - .npm/

# Or per-job cache
test:
  cache:
    key:
      files:
        - package-lock.json
    paths:
      - node_modules/
  script:
    - npm ci --cache .npm
    - npm test

Artifacts

build:
  script:
    - npm run build
  artifacts:
    paths:
      - dist/
    exclude:
      - dist/**/*.map
    expire_in: 1 day
    reports:
      junit: test-results.xml
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura.xml

Includes and Templates

Include External Files

include:
  # Local file
  - local: '/templates/.gitlab-ci-template.yml'

  # From another project
  - project: 'my-group/ci-templates'
    ref: main
    file: '/templates/nodejs.yml'

  # Remote URL
  - remote: 'https://example.com/templates/ci.yml'

  # GitLab templates
  - template: Auto-DevOps.gitlab-ci.yml

Extends (Inheritance)

.test-template:
  image: node:20
  before_script:
    - npm ci
  cache:
    paths:
      - node_modules/

unit-test:
  extends: .test-template
  script:
    - npm run test:unit

integration-test:
  extends: .test-template
  script:
    - npm run test:integration

Reference (YAML Anchors)

.default-rules: &default-rules
  rules:
    - if: $CI_COMMIT_BRANCH == "main"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

test:
  <<: *default-rules
  script: npm test

lint:
  <<: *default-rules
  script: npm run lint

Debugging CI Failures

View Pipeline Logs

Navigate to CI/CD > Pipelines
Click on the pipeline
Click on the failed job
Review job log output

Debug Mode

test:
  script:
    - npm test
  variables:
    CI_DEBUG_TRACE: "true"  # Verbose logging

Common Issues

Issue	Cause	Solution
Job stuck pending	No available runners	Check runner status and tags
Cache not working	Key mismatch or expired	Verify cache key and paths
Artifacts missing	Not passed between stages	Add artifacts and dependencies
Rules not matching	Incorrect conditions	Use `CI_DEBUG_TRACE` to debug
Image pull failed	Registry auth or network	Check image name and credentials

Lint Pipeline Locally

# Using GitLab CI Lint API
curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
  --data-urlencode "content=$(cat .gitlab-ci.yml)" \
  "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/ci/lint"

# Using gitlab-ci-local
npm install -g gitlab-ci-local
gitlab-ci-local --list
gitlab-ci-local test

Runner Configuration

Runner Types

Type	Best For
Shared runners	Small projects, standard builds
Group runners	Team-specific configuration
Project runners	Custom requirements, security

Runner Tags

deploy:
  tags:
    - docker
    - linux
    - production
  script:
    - ./deploy.sh

Docker Executor

test:
  image: python:3.11
  services:
    - postgres:15
    - redis:7
  variables:
    POSTGRES_DB: test
    POSTGRES_USER: runner
    POSTGRES_PASSWORD: ""
  script:
    - pytest

Environment and Secrets

Protected Variables

Configure in Settings > CI/CD > Variables:

Mark as "Protected" for protected branches only
Mark as "Masked" to hide in logs

deploy:
  script:
    - echo $DEPLOY_TOKEN  # Masked in logs
  only:
    - main  # Protected variable only available here

Environments

deploy-staging:
  stage: deploy
  environment:
    name: staging
    url: https://staging.example.com
  script:
    - ./deploy.sh staging

deploy-production:
  stage: deploy
  environment:
    name: production
    url: https://example.com
  when: manual
  script:
    - ./deploy.sh production

Performance Optimization

Interruptible Jobs

Cancel running jobs when new commits push:

test:
  interruptible: true
  script:
    - npm test

Resource Groups

Prevent concurrent deploys:

deploy:
  resource_group: production
  script:
    - ./deploy.sh

Workflow Rules

Skip pipelines for certain conditions:

workflow:
  rules:
    - if: $CI_COMMIT_MESSAGE =~ /\[skip ci\]/
      when: never
    - if: $CI_PIPELINE_SOURCE == "push"
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

Pipeline Efficiency

# Use needs for DAG
# Use rules to skip unnecessary jobs
# Cache aggressively
# Use slim images
# Parallelize where possible

test:
  parallel: 4  # Split tests across 4 jobs
  script:
    - npm run test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

Pre-commit Hooks

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/adrienverge/yamllint
    rev: v1.35.1
    hooks:
      - id: yamllint
        files: \.gitlab-ci\.yml$
        args: [-c, .yamllint.yml]

Debugging Checklist

Check .gitlab-ci.yml syntax with CI Lint
Verify runner is available and has correct tags
Check job rules/conditions are matching
Review variable values (use debug trace)
Verify artifacts are being passed correctly
Check cache is being used effectively
Review image availability and credentials
Check for resource_group conflicts

Install Skill

SKILL.md