Skip to content

Latest commit

 

History

History
338 lines (240 loc) · 18.6 KB

ci-cd.md

File metadata and controls

338 lines (240 loc) · 18.6 KB
Raw

CI/CD - Continuous Integration / Continuous Delivery

Continually tests, builds your code packages and deploys them.

TL;DR use GitHub Actions for fully hosted and Jenkins for self-hosted, more flexibility or advanced use cases.

GitHub Actions self-hosted runners are also a reasonable option.

CI/CD pipeline configurations for the below CI/CD platforms can be found throughout my public GitHub repos with hundreds of live CI/CD builds for my many open source projects.

All my GitHub repos are also mirrored to all 4 major repo providers to use their CI/CD and management systems too - GitHub, GitLab, Azure DevOps and Bitbucket.

API code for many of the following technologies can also be found in my DevOps-Bash-tools repo.

Readme Card

Self-Hosted CI/CD

Jenkins

The gold standard open source CI/CD - powerful, extensible, simple to complex to suit any need.

Readme Card

TeamCity

Proprietary by JetBrains (makers of IntelliJ, expert coders and UX).

  • decent
  • good UI
  • free for 3 agents, requires license otherwise - this is a tough hard sell vs free Jenkins

Readme Card

Concourse

Simple, lean, open source CI/CD tool.

GoCD

Open source by ThoughtWorks.

Hosted CI/CD by Git repo providers

Already available on the major Git repo providers.

All are yaml-based configuration CI/CD with no installation or administration required.

Optional self-hosted runners are available to install on your own hardware or Kubernetes, to have more control of your execution environment or offload build minutes costs if used heavily.

GitHub Actions

The 800lb gorilla of modern CI/CD.

  • easily the best choice overall for hosted CI/CD
  • unlimited free minutes for public projects
  • 50,000 build minutes included in any GitHub Enterprise Cloud plan makes this a CircleCI killer on cost alone
  • Massive community support and extensibility via GitHub Actions Marketplace
  • 3rd party actions by many prominent tech vendors like AWS and Docker
  • 1st class support - every tech vendor targets GitHub Actions as their primary CI/CD client as they're following both the tech and the market
  • GitHub Enterprise Server for on-premise (government, banks etc.)
  • self-hosted runners can be run on Kubernetes for Cloud hosted version
  • HariSekhon/GitHub-Actions - library of parameterized reusable workflows and master templates for workflows and actions
  • HariSekhon/DevOps-Bash-tools github/ - many GitHub Actions admin scripts using the GitHub API and CLI tool
  • GitHub, like LinkedIn, is now owned by Microsoft, so it's beating its own Azure DevOps below

Readme Card

GitLab CI

The second-best repo provider and CI/CD.

  • uses a single yaml, but that yaml can include other yamls to reproduce something similar to what GitHub does implicitly
    • but lacks different badges for different checks so slower CI/CD failure root cause analysis
  • not as flexible as GitHub Actions
  • good API and documentation - similar endpoints and parity of functionality with GitHub
  • lacks GitHub Actions marketplace
  • self-hosted runners can be run on Kubernetes
  • 400 free build minutes a month
    • has relegated this to a legacy CI/CD for open source - GitHub gives unlimited free build minutes for all public projects
  • GitLab server for on-premise (government, banks etc.)
  • HariSekhon/Templates - .gitlab-ci.yml
  • HariSekhon/DevOps-Bash-tools gitlab/ - several GitLab admin scripts using the GitLab API
  • do not use GitLab over GitHub unless you're going Enterprise and are forced to use it

Azure DevOps Pipelines

Microsoft's offering to accompany their Git repo hosting.

Bitbucket Pipelines

The weakest repo provider and weakest CI/CD system.

  • not as many features
  • not a great API
  • 50 free build minutes a month
  • when they dropped the free build minutes this rendered this useless for open source projects
  • unless you like getting constant emails of build failures upon every git push due to no build minutes
  • Bitbucket Data Center is the on-premise installable version
  • this product is one of Atlassian's rare fails to compete in the software market as they have many more widely used products like Jira, Confluence, Fisheye etc.
  • HariSekhon/Templates - bitbucket-pipelines.yml
  • HariSekhon/DevOps-Bash-tools bitbucket/ - several BitBucket admin scripts using the Bitbucket API
  • Legacy. Do not use Bitbucket - pick anything else.

Hosted CI/CD - Cloud

AWS CodeBuild

  • cheap pay-as-you-go integrated with your existing AWS bill, convenient for no additional accounts, approvals or purchase orders
  • too basic in functionality - poor man's CI/CD
  • good security integration with other AWS services due to all being under AWS IAM
  • most companies use GitHub Actions instead for good reason - if you're in the cloud anyway, GitHub has better features, security and integrations
  • HariSekhon/Templates- AWS CodeBuild buildspec.yaml

GCP CloudBuild

  • formerly called Container Builder which should give you a hint as to its intended purpose
  • similar advantages and drawbacks to AWS CodeBuild
  • basic functionality
  • cheap with integrated pay-as-you-go billing inside your existing Google Cloud account, no extra purchase orders and approvals needed in your enterprise
  • good security integration with other GCP services due to all being under GCP IAM
  • Kaniko integration but in my experience your mileage may vary
  • HariSekhon/Templates - cloudbuild.yaml and cloudbuild-golang.yaml

Azure DevOps Pipelines

  • Legacy
  • see details in Hosted Repo Providers section above as it's closer to them as a legacy TFS rebranding than the above cloud-native integrated offerings

Hosted CI/CD - 3rd Party (all are legacy)

CircleCI

Mature long-standing frontrunner among hosted CI/CD systems.

Travis CI

One of the first best hosted CI/CD systems.

BuildKite

Cloud control panel with self-hosted agents.

  • easy to use
  • good API
  • the most modern freshest feeling CI/CD solution in this section
  • drawback is no fully-hosted runners, they provide the control plane and you must host yourself
  • Docker / Kubernetes agents
  • HariSekhon/DevOps-Bash-tools buildkite/ - many BuildKite admin scripts using the above API

AppVeyor

Has Windows builds if you're into that sort of legacy thing.

Cirrus CI

Nothing special, yet another yaml CI/CD.

Codefresh

DroneIO

Semaphore CI

Nothing special, has its own Python environment quirks I've had to work around.

CodeShip

  • docker support
  • pricey
  • Dead
  • Acquired by CloudBees, no longer free
    • not sure why they bothered when CloudBees are Jenkins experts. Hosted Jenkins is what people really want. That or GitHub Actions whose yaml is simpler than Jenkinsfiles

Shippable

  • 1 parallel build :-/
  • Acquired by JFrog
  • Decommissioned / Dead

NimbleCI

  • only 300 build minutes per month
  • Decommissioned / Dead

Portable Code Design

Anything you put in CI/CD should ideally be easily runnable locally from the command line.

This is important for local testing and debugging, eg. being able to test code builds using Build Systems or run Terraform / Terragrunt plans before committing code, as well as deploy urgent production hotfixes even in cases where the CI/CD system may be unavailable due to a public outage (all the major vendors have them from time to time) or a self-managed maintenance window for an upgrade or similar.

Also, if you ever want to migrate to another CI/CD system, having a portable set of scripts, Makefiles and similar makes it much easier to do so.

Advanced library usage such as in HariSekhon/GitHub-Actions and HariSekhon/Jenkins sacrifice some of this portability in exchange for advanced functionality and code reuse.

Code Quality

See Code Quality.

Diagram - CI/CD Pipeline Explained

CI/CD Pipelines Explained

Only the Athenians exist... I mean... only Jenkins and GitHub Actions exist...

... and only stout wooden ships can save them...

... I mean... only strongly opinionated Senior Engineers can deliver the best CI/CD...

Jenkins and GitHub Actions Only

If companies want to run GitHub Actions I’ll implement that for them.

If companies want to run Jenkins on Kubernetes with autoscaling and cool Groovy libraries I’ll implement that for them.

If companies want to run one of the other 20 odd legacy CI/CD systems I’ll look for a new job. 😂

ps. I have good GitHub repos for both here:

Readme Card

Readme Card

Ported from private Knowledge Base page 2014+