GitLab docker on Ubuntu

Today I decided to move my self-hosted gitlab instance to Docker so that I will not need to reinstall and configure lots of things when migrating from one host to another host. So I blog this entry to note steps to install gitlab docker on Ubuntu.

Install Docker CE (or Docker Compose if you want)

  1. Simply install Docker-CE:
    apt-get update -y
    apt-get install -y apt-transport-https ca-certificates curl software-properties-common
    curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
    add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
    apt-get update -y
    apt-get install -y docker-ce
    # You can also list version to install exact version with "apt-cache madison docker-ce" and "apt-get install docker-ce=<VERSION>"
  2. OR we can download and use docker-compose:
    curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
    chmod +x /usr/local/bin/docker-compose
    docker-compose --version

     

Run GitLab with Docker-CE

  1. Init and run the gitlab docker image:
    # I add a new IP address ADDITIONAL_IP_ADDRESS to docker host to run later
    # If you do not want to bind with static IP, just remove the ADDITIONAL_IP_ADDRESS part
    ip address add ADDITIONAL_IP_ADDRESS dev eno1
    docker run --detach \
        --hostname us.vngit.com \
        --env GITLAB_OMNIBUS_CONFIG="external_url 'https://us.vngit.com/'; gitlab_rails['lfs_enabled'] = true;" \
        --publish ADDITIONAL_IP_ADDRESS:443:443 --publish ADDITIONAL_IP_ADDRESS:80:80 --publish ADDITIONAL_IP_ADDRESS:22:22 \
        --name gitlab \
        --restart always \
        --volume /srv/gitlab/config:/etc/gitlab \
        --volume /srv/gitlab/logs:/var/log/gitlab \
        --volume /srv/gitlab/data:/var/opt/gitlab \
        gitlab/gitlab-ce:latest
  2. Enter the docker container (with docker exec -ti gitlab bash) to see if configuration is correct to re-run gitlab-ctl reconfigure. You will need to check the external_url variable in /etc/gitlab/gitlab.rb.
  3. Backup your current gitlab, and then transfer to current host server. Remember that in the previous step, we set /var/opt/gitlab in the gitlab container mounted to /srv/gitlab/data in the host machine. So we copy the backup file to /srv/gitlab/data/backups/ and restore the backup:
    mv /root/1517992991_2018_02_07_10.4.2_gitlab_backup.tar /srv/gitlab/data/backups/
    docker exec -t gitlab gitlab-ctl stop unicorn
    docker exec -t gitlab gitlab-ctl stop sidekiq
    docker exec -t gitlab chown -R git:git /var/opt/gitlab/backups
    docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=1517992991_2018_02_07_10.4.2
    # Restart gitlab and check
    docker exec -it gitlab gitlab-ctl start
    docker exec -it gitlab gitlab-rake gitlab:check SANITIZE=true
  4. Do final check on your new gitlab before pointing your DNS to the new IP.
  5. Setup backup with cron on the host machine: 0 0 * * * docker exec -t gitlab gitlab-rake gitlab:backup:create
  6. Setup Let’s Encrypt: we will request certs from host machine and change the gitlab.rb in gitlab container to point to the certs.
    1. Install certbot:
      add-apt-repository ppa:certbot/certbot
      apt-get update -y
      apt install -y certbot
    2. Request for certifications from host machine (we will need to stop docker container since it binds to the port 80):
      docker stop gitlab
      certbot certonly --standalone -d vngit.com -d www.vngit.com -d us.vngit.com
      docker start gitlab
    3. With the above commands, certs will be stored at /etc/letsencrypt/live/vngit.com/ of the host machine. We will copy certs to gitlab data folder to use it inside gitlab container:
      mkdir /srv/gitlab/data/certbot
      cp /etc/letsencrypt/live/vngit.com/fullchain.pem /srv/gitlab/data/certbot/fullchain.pem
      cp /etc/letsencrypt/live/vngit.com/privkey.pem /srv/gitlab/data/certbot/privkey.pem
    4. Change /srv/gitlab/config/gitlab.rb to reflect these certs. Add the following nginx lines and change external_url inside it:
      ### Add the following lines to GitLab NGINX section
      nginx['ssl_certificate'] = "/var/opt/gitlab/certbot/fullchain.pem"
      nginx['ssl_certificate_key'] = "/var/opt/gitlab/certbot/privkey.pem"
      
      ### Change this line
      external_url 'https://vngit.com/'
    5. Reconfigure gitlab:
      docker exec -it gitlab gitlab-ctl reconfigure
    6. Setup cron to renew certs on monthly basis (create a /root/cron-gitlab-certbot-renewal):
      # Create a cron with 0 0 1 * * bash  /root/cron-gitlab-certbot-renewal.sh
      docker stop gitlab
      certbot renew
      cp /etc/letsencrypt/live/vngit.com/fullchain.pem /srv/gitlab/data/certbot/fullchain.pem
      cp /etc/letsencrypt/live/vngit.com/privkey.pem /srv/gitlab/data/certbot/privkey.pem
      docker start gitlab

Upgrade GitLab to latest version

  1. Remove container, pull the latest image and run it again:
    docker stop gitlab
    docker rm gitlab
    docker pull gitlab/gitlab-ce:latest
    docker run --detach \
    --hostname us.vngit.com \
    --publish ADDITIONAL_IP_ADDRESS:443:443 --publish ADDITIONAL_IP_ADDRESS:80:80 --publish ADDITIONAL_IP_ADDRESS:22:22 \
    --name gitlab \
    --restart always \
    --volume /srv/gitlab/config:/etc/gitlab \
    --volume /srv/gitlab/logs:/var/log/gitlab \
    --volume /srv/gitlab/data:/var/opt/gitlab \
    gitlab/gitlab-ce:latest

 Backup Docker container to store remotely

  1. First, get the container ID with docker ps command.
  2. To backup a container CONTAINER_ID, user docker commit as follows (we can use either CONTAINER_ID or CONTAINER_NAME. I use name as gitlab instead):
    docker commit -p gitlab gitlab_backup
    

    After this, the backup is saved as a docker image. We can view with docker images.

  3. Save backup as tar file to rsync to remote backup storage:
    docker save -o /srv/gitlab/gitlab_container_backup.tar gitlab_backup

Restore Docker container from saved backup

  1. Load the backup saved image:
    docker load < /srv/gitlab/gitlab_container_backup.tar
  2. Start a new container with the loaded image:
    docker run --detach \
        --hostname us.vngit.com \
        --env GITLAB_OMNIBUS_CONFIG="external_url 'https://us.vngit.com/'; gitlab_rails['lfs_enabled'] = true;" \
        --publish ADDITIONAL_IP_ADDRESS:443:443 --publish ADDITIONAL_IP_ADDRESS:80:80 --publish ADDITIONAL_IP_ADDRESS:22:22 \
        --name gitlab \
        --restart always \
        --volume /srv/gitlab/config:/etc/gitlab \
        --volume /srv/gitlab/logs:/var/log/gitlab \
        --volume /srv/gitlab/data:/var/opt/gitlab \
        gitlab_backup

    Note that in this case, the base image is gitlab_backup, not gitlab/gitlab-ce:latest. We can simply rsync the /srv/gitlab folder to the new server and then run with the gitlab-ce:latest image.

 

With Docker Stack

Basically, we will need to separate GitLab services (Redis, Postgres, Prometheus) into separated ones. So I will note configurations as well as some steps as follows:

  1. Some configuration files:
    • Init docker swarm and Create a new balancer network
      docker swarm init
      docker network create -d overlay balancer
    • docker-compose.yml“:
      version: "3.4"
      
      services:
        postgres:
          image: "postgres:9.6" # The embedded postgres of gitlab till 11.x is 9.6
          volumes:
            - "/srv/gitlab-swarm/postgres:/data"
          environment:
            POSTGRES_USER: "gitlab"
            POSTGRES_PASSWORD: "gitlab"
            PGDATA: "/data"
            POSTGRES_DB: "gitlab"
      
        gitlab:
          image: "gitlab/gitlab-ce:latest"
          volumes:
            - "/srv/gitlab-swarm/gitlab/data:/var/opt/gitlab"
            - "/srv/gitlab-swarm/gitlab/logs:/var/log/gitlab"
            - "/srv/gitlab-swarm/gitlab/config:/etc/gitlab"
      #      - "/srv/gitlab/data:/var/opt/gitlab"
      #      - "/srv/gitlab/logs:/var/log/gitlab"
      #      - "/srv/gitlab/config:/etc/gitlab"
          ports:
            - "2222:22"
            - "8888:80"
          configs:
            - source: "gitlab.rb"
              target: "/etc/gitlab/gitlab.rb"
          depends_on:
            - postgres
          networks:
            - default
            - balancer
          deploy:
            replicas: 1
            restart_policy:
              condition: on-failure
            labels:
              - "traefik.port=80"
              - "traefik.backend=gitlab"
              - "traefik.docker.network=balancer"
              - "traefik.frontend.rule=Host:www.git.tienle.com,git.tienle.com"
        redis:
          image: "redis"
        prometheus:
          image: "prom/prometheus"
          command: "--config.file=/prometheus.yaml --storage.tsdb.path /data"
          volumes:
            - "/srv/gitlab-swarm/prometheus:/data"
          configs:
            - prometheus.yaml
          networks:
            - default
            - balancer
          deploy:
            labels:
              traefik.port: 9090
              traefik.frontend.rule: "Host:prometheus.git.tienle.com"
              traefik.docker.network: "balancer"
        grafana:
          image: grafana/grafana
          environment:
            GF_PATHS_CONFIG: "/grafana.ini"
          configs:
            - grafana.ini
          volumes:
            - "/srv/gitlab-swarm/grafana:/data"
          networks:
            - default
            - balancer
          deploy:
            labels:
              traefik.port: 3000
              traefik.frontend.rule: "Host:grafana.git.tienle.com"
              traefik.docker.network: "balancer"
      
      configs:
        gitlab.rb:
          file: "./swarm-configs/gitlab.rb"
        prometheus.yaml:
          file: "./swarm-configs/prometheus.yaml"
        grafana.ini:
          file: "./swarm-configs/grafana.ini"
      
      networks:
        default:
        balancer:
          external:
            name: balancer
    • swarm-configs/gitlab.rb“:
      external_url 'http://git.tienle.com'
      #registry_external_url 'http://registry.git.tienle.com'
      
      # Disable services
      postgresql['enable'] = false
      redis['enable'] = false
      prometheus['enable'] = false
      postgres_exporter['enable'] = false
      redis_exporter['enable'] = false
      
      # SMTP 
      gitlab_rails['smtp_enable'] = true
      gitlab_rails['smtp_address'] = "smtp.mailgun.org"
      gitlab_rails['smtp_port'] = 465
      gitlab_rails['smtp_user_name'] = "XXXXXXXXXXXXX"
      gitlab_rails['smtp_password'] = "XXXXXXXXXXXXX"
      gitlab_rails['smtp_domain'] = "mail.XXXXXXXXXXXX.com"
      gitlab_rails['smtp_authentication'] = "login"
      gitlab_rails['smtp_enable_starttls_auto'] = true
      gitlab_rails['smtp_tls'] = true
      
      # Postgres settings
      gitlab_rails['db_adapter'] = "postgresql"
      gitlab_rails['db_encoding'] = "unicode"
      
      # database service will be named "postgres" in the stack
      gitlab_rails['db_host'] = "postgres" 
      gitlab_rails['db_database'] = "gitlab"
      gitlab_rails['db_username'] = "gitlab"
      gitlab_rails['db_password'] = "gitlab"
      
      # Redis settings
      # redis service will be named "redis" in the stack
      gitlab_rails['redis_host'] = "redis"
      
      # Prometheus exporters
      node_exporter['listen_address'] = '0.0.0.0:9100'
      gitlab_monitor['listen_address'] = '0.0.0.0'
      gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
      gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229"
    • swarm-configs/prometheus.yaml“:
      global:
        scrape_interval: 15s
      
      scrape_configs:
        - job_name: 'prometheus'
          static_configs:
            - targets: ['localhost:9090']
      
        # gitlab monitor
        - job_name: 'gitlab_monitor'
          static_configs:
            - targets: ['gitlab:9168']
      
        # gitlab sidekiq
        - job_name: 'gitlab_sidekiq'
          metrics_path: /sidekiq
          static_configs:
            - targets: ['gitlab:9168']
      
        # gitlab process
        - job_name: 'gitlab_process'
          metrics_path: /process
          static_configs:
            - targets: ['gitlab:9168']
      
        # gitlab pages
        - job_name: 'gitlab_pages'
          static_configs:
            - targets: ['gitlab:9235']
      
        # gitaly
        - job_name: gitaly
          static_configs:
            - targets: ['gitlab:9236']
      
        # gitlab workhorse
        - job_name: workhorse
          static_configs:
            - targets: ['gitlab:9229']
      
    • swarm-configs/grafana.ini“:
      [database]
      path = "/data/grafana.db"
      
      [session]
      provider = "redis"
      provider_config = "addr=redis:6379,prefix=grafana:"
  2. Then, I start the docker stack:
    docker stack deploy -c docker-compose.yml gitlab
  3. Then, I prepare backup data for restoration. This simply copies the Gitlab backup one to the “/srv/gitlab/data/backups/” folder.
  4. Then, I need to stop the “unicorn” and “sidekiq” in the “gitlab” container before continuing on backup restoration. Then do restoration:
    docker exec -t __CONTAINER_NAME__ gitlab-ctl stop unicorn
    docker exec -t __CONTAINER_NAME__ gitlab-ctl stop sidekiq
    docker exec -it __CONTAINER_NAME__  gitlab-rake gitlab:backup:restore BACKUP=XXXX_BACKUP_TIMESTAMP_XXXX
  5. Finally, restart the Gitlab service:
    docker exec -t __CONTAINER_NAME__ gitlab-ctl restart

    And set up a cron to backup the gitlab container daily

    0 0 * * * docker exec -it $(docker ps | grep gitlab_gitlab | cut -d " " -f 1) gitlab-rake gitlab:backup:create
  6. IF you want to run multiple instances for the Gitlab service, this is the time to change the deployment parameters.

Leave a Reply