Skip to content
This repository has been archived by the owner on Aug 10, 2018. It is now read-only.

Latest commit

 

History

History
208 lines (159 loc) · 7.21 KB

autoscaling.md

File metadata and controls

208 lines (159 loc) · 7.21 KB

Install and configure GitLab Runner for auto-scaling

The auto scale feature was introduced in GitLab Runner 1.1.0.

For an overview of the auto-scale architecture, take a look at the comprehensive documentation on auto-scaling.

Table of Contents generated with DocToc

Prepare the environment

In order to use the auto-scale feature, Docker and GitLab Runner must be installed in the same machine:

  1. Login to a new Linux-based machine that will serve as a bastion server and where Docker will spawn new machines from
  2. Install GitLab Runner following the GitLab Runner installation documentation
  3. Install Docker Machine following the Docker Machine installation documentation

Prepare the Docker Registry and Cache Server

To speedup the builds we advise to setup a personal Docker registry server working in proxy mode. A cache server is also recommended.

Install Docker Registry

Note: Read more in Distributed registry mirroring.

  1. Login to a dedicated machine where Docker registry proxy will be running

  2. Make sure that Docker Engine is installed on this machine

  3. Create a new Docker registry:

    docker run -d -p 6000:5000 \
    -e REGISTRY_PROXY_REMOTEURL=https://registry-1.docker.io \
    --restart always \
    --name registry registry:2

    You can modify the port number (6000) to expose Docker registry on a different port.

  4. Check the IP address of the server:

    hostname --ip-address

    You should preferably choose the private networking IP address. The private networking is usually the fastest solution for internal communication between machines of a single provider (DigitalOcean, AWS, Azure, etc,) Usually the private networking is also not accounted to your monthly bandwidth limit.

  5. Docker registry will be accessible under MY_REGISTRY_IP:6000.

Install the cache server

Note: You can use any other S3-compatible server, including Amazon S3. Read more in Distributed runners caching.

  1. Login to a dedicated machine where the cache server will be running

  2. Make sure that Docker Engine is installed on that machine

  3. Start minio, a simple S3-compatible server written in Go:

    docker run -it --restart always -p 9005:9000 \
        -v /.minio:/root/.minio -v /export:/export \
        --name minio \
        minio/minio:latest /export

    You can modify the port 9005 to expose the cache server on different port.

  4. Check the IP address of the server:

    hostname --ip-address

  5. Your cache server will be available at MY_CACHE_IP:9005

  6. Read the Access and Secret Key of minio with: sudo cat /.minio/config.json

  7. Create a bucket that will be used by the Runner: sudo mkdir /export/runner. runner is the name of the bucket in that case. If you choose a different bucket then it will be different

  8. All caches will be stored in the /export directory

Configure GitLab Runner

  1. Register a GitLab Runner, selecting the docker+machine executor:

    sudo gitlab-ci-multi-runner register

    Example output:

    Please enter the gitlab-ci coordinator URL (e.g. https://gitlab.com )
https://gitlab.com
Please enter the gitlab-ci token for this runner
xxx
Please enter the gitlab-ci description for this runner
my-autoscale-runner
INFO[0034] fcf5c619 Registering runner... succeeded
Please enter the executor: shell, docker, docker-ssh, docker+machine, docker-ssh+machine, ssh?
docker+machine
Please enter the Docker image (eg. ruby:2.1):
ruby:2.1
INFO[0037] Runner registered successfully. Feel free to start it, but if it's
running already the config should be automatically reloaded!

  2. Edit config.toml. You need to fill in the options for [runners.machine] and [runners.cache] and configure the MachineDriver selecting your provider. Also configure MachineOptions, limit and IdleCount.

    For more information visit the dedicated page covering detailed information about GitLab Runner Autoscaling.

    Example configuration using DigitalOcean:

    concurrent = 20

[[runners]]
executor = "docker+machine"
limit = 20
[runners.machine]
  IdleCount = 5
  MachineDriver = "digitalocean"
  MachineName = "auto-scale-runners-%s.my.domain.com"
  MachineOptions = [
      "digitalocean-image=coreos-beta",
      "digitalocean-ssh-user=core",
      "digitalocean-access-token=MY_DIGITAL_OCEAN_TOKEN",
      "digitalocean-region=nyc2",
      "digitalocean-private-networking",
      "engine-registry-mirror=http://MY_REGISTRY_IP:6000"
    ]
[runners.cache]
  Type = "s3"
  ServerAddress = "MY_CACHE_IP:9005"
  AccessKey = "ACCESS_KEY"
  SecretKey = "SECRET_KEY"
  BucketName = "runner"
  Insecure = true # Use Insecure only when using with Minio, without the TLS certificate enabled

  3. Try to build your project. In a few seconds, if you run docker-machine ls you should see a new machine being created.

Upgrading the Runner

  1. Stop the runner:

    killall -SIGQUIT gitlab-runner

    Sending the SIGQUIT signal will make the Runner to stop gracefully. It will stop accepting new jobs, and will exit as soon as the current builds are finished.

  2. Wait until the Runner exits. You can check its status with: gitlab-runner status

  3. You can now safely upgrade the Runner without interrupting any builds

Manage the Docker Machines

  1. Stop the Runner:

    killall -SIGQUIT gitlab-runner

  2. Wait until the Runner exits. You can check its status with: gitlab-runner status

  3. You can now manage (upgrade or remove) any Docker Machines with the docker-machine command