Today I decided to move my self-hosted gitlab instance to Docker so that I will not need to reinstall and configure lots of things when migrating from one host to another host. So I blog this entry to note steps to install gitlab docker on Ubuntu.

Install Docker CE (or Docker Compose if you want)

1. Simply install Docker-CE:

   apt-get update -y
   apt-get install -y apt-transport-https ca-certificates curl software-properties-common
   curl -fsSL https://download.docker.com/linux/ubuntu/gpg | apt-key add -
   add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
   apt-get update -y
   apt-get install -y docker-ce
   # You can also list version to install exact version with "apt-cache madison docker-ce" and "apt-get install docker-ce=<VERSION>"

2. OR we can download and use docker-compose:

   curl -L https://github.com/docker/compose/releases/download/1.18.0/docker-compose-`uname -s`-`uname -m` -o /usr/local/bin/docker-compose
   chmod +x /usr/local/bin/docker-compose
   docker-compose --version

    

Run GitLab with Docker-CE

1. Init and run the gitlab docker image:

   # I add a new IP address ADDITIONAL_IP_ADDRESS to docker host to run later
   # If you do not want to bind with static IP, just remove the ADDITIONAL_IP_ADDRESS part
   ip address add ADDITIONAL_IP_ADDRESS dev eno1
   docker run --detach \
       --hostname us.vngit.com \
       --env GITLAB_OMNIBUS_CONFIG="external_url 'https://us.vngit.com/'; gitlab_rails['lfs_enabled'] = true;" \
       --publish ADDITIONAL_IP_ADDRESS:443:443 --publish ADDITIONAL_IP_ADDRESS:80:80 --publish ADDITIONAL_IP_ADDRESS:22:22 \
       --name gitlab \
       --restart always \
       --volume /srv/gitlab/config:/etc/gitlab \
       --volume /srv/gitlab/logs:/var/log/gitlab \
       --volume /srv/gitlab/data:/var/opt/gitlab \
       gitlab/gitlab-ce:latest

2. Enter the docker container (with docker exec -ti gitlab bash) to see if configuration is correct to re-run gitlab-ctl reconfigure. You will need to check the external_url variable in /etc/gitlab/gitlab.rb.
3. Backup your current gitlab, and then transfer to current host server. Remember that in the previous step, we set /var/opt/gitlab in the gitlab container mounted to /srv/gitlab/data in the host machine. So we copy the backup file to /srv/gitlab/data/backups/ and restore the backup:

   mv /root/1517992991_2018_02_07_10.4.2_gitlab_backup.tar /srv/gitlab/data/backups/
   docker exec -t gitlab gitlab-ctl stop unicorn
   docker exec -t gitlab gitlab-ctl stop sidekiq
   docker exec -t gitlab chown -R git:git /var/opt/gitlab/backups
   docker exec -it gitlab gitlab-rake gitlab:backup:restore BACKUP=1517992991_2018_02_07_10.4.2
   # Restart gitlab and check
   docker exec -it gitlab gitlab-ctl start
   docker exec -it gitlab gitlab-rake gitlab:check SANITIZE=true

4. Do final check on your new gitlab before pointing your DNS to the new IP.
5. Setup backup with cron on the host machine: 0 0 * * * docker exec -t gitlab gitlab-rake gitlab:backup:create
6. Setup Let’s Encrypt: we will request certs from host machine and change the gitlab.rb in gitlab container to point to the certs.
   1. Install certbot:

      add-apt-repository ppa:certbot/certbot
      apt-get update -y
      apt install -y certbot

   2. Request for certifications from host machine (we will need to stop docker container since it binds to the port 80):

      docker stop gitlab
      certbot certonly --standalone -d vngit.com -d www.vngit.com -d us.vngit.com
      docker start gitlab

   3. With the above commands, certs will be stored at /etc/letsencrypt/live/vngit.com/ of the host machine. We will copy certs to gitlab data folder to use it inside gitlab container:

      mkdir /srv/gitlab/data/certbot
      cp /etc/letsencrypt/live/vngit.com/fullchain.pem /srv/gitlab/data/certbot/fullchain.pem
      cp /etc/letsencrypt/live/vngit.com/privkey.pem /srv/gitlab/data/certbot/privkey.pem

   4. Change /srv/gitlab/config/gitlab.rb to reflect these certs. Add the following nginx lines and change external_url inside it:

      ### Add the following lines to GitLab NGINX section
      nginx['ssl_certificate'] = "/var/opt/gitlab/certbot/fullchain.pem"
      nginx['ssl_certificate_key'] = "/var/opt/gitlab/certbot/privkey.pem"
      
      ### Change this line
      external_url 'https://vngit.com/'

   5. Reconfigure gitlab:

      docker exec -it gitlab gitlab-ctl reconfigure

   6. Setup cron to renew certs on monthly basis (create a /root/cron-gitlab-certbot-renewal):

      # Create a cron with 0 0 1 * * bash  /root/cron-gitlab-certbot-renewal.sh
      docker stop gitlab
      certbot renew
      cp /etc/letsencrypt/live/vngit.com/fullchain.pem /srv/gitlab/data/certbot/fullchain.pem
      cp /etc/letsencrypt/live/vngit.com/privkey.pem /srv/gitlab/data/certbot/privkey.pem
      docker start gitlab

Upgrade GitLab to latest version

1. Remove container, pull the latest image and run it again:

   docker stop gitlab
   docker rm gitlab
   docker pull gitlab/gitlab-ce:latest
   docker run --detach \
   --hostname us.vngit.com \
   --publish ADDITIONAL_IP_ADDRESS:443:443 --publish ADDITIONAL_IP_ADDRESS:80:80 --publish ADDITIONAL_IP_ADDRESS:22:22 \
   --name gitlab \
   --restart always \
   --volume /srv/gitlab/config:/etc/gitlab \
   --volume /srv/gitlab/logs:/var/log/gitlab \
   --volume /srv/gitlab/data:/var/opt/gitlab \
   gitlab/gitlab-ce:latest

 Backup Docker container to store remotely

1. First, get the container ID with docker ps command.
2. To backup a container CONTAINER_ID, user docker commit as follows (we can use either CONTAINER_ID or CONTAINER_NAME. I use name as gitlab instead):

   docker commit -p gitlab gitlab_backup
   

   After this, the backup is saved as a docker image. We can view with docker images.

3. Save backup as tar file to rsync to remote backup storage:

   docker save -o /srv/gitlab/gitlab_container_backup.tar gitlab_backup

Restore Docker container from saved backup

1. Load the backup saved image:

   docker load < /srv/gitlab/gitlab_container_backup.tar

2. Start a new container with the loaded image:

   docker run --detach \
       --hostname us.vngit.com \
       --env GITLAB_OMNIBUS_CONFIG="external_url 'https://us.vngit.com/'; gitlab_rails['lfs_enabled'] = true;" \
       --publish ADDITIONAL_IP_ADDRESS:443:443 --publish ADDITIONAL_IP_ADDRESS:80:80 --publish ADDITIONAL_IP_ADDRESS:22:22 \
       --name gitlab \
       --restart always \
       --volume /srv/gitlab/config:/etc/gitlab \
       --volume /srv/gitlab/logs:/var/log/gitlab \
       --volume /srv/gitlab/data:/var/opt/gitlab \
       gitlab_backup

   Note that in this case, the base image is gitlab_backup, not gitlab/gitlab-ce:latest. We can simply rsync the /srv/gitlab folder to the new server and then run with the gitlab-ce:latest image.

With Docker Stack

Basically, we will need to separate GitLab services (Redis, Postgres, Prometheus) into separated ones. So I will note configurations as well as some steps as follows:

1. Some configuration files:
   • Init docker swarm and Create a new balancer network

     docker swarm init
     docker network create -d overlay balancer

   • “docker-compose.yml“:

     version: "3.4"
     
     services:
       postgres:
         image: "postgres:9.6" # The embedded postgres of gitlab till 11.x is 9.6
         volumes:
           - "/srv/gitlab-swarm/postgres:/data"
         environment:
           POSTGRES_USER: "gitlab"
           POSTGRES_PASSWORD: "gitlab"
           PGDATA: "/data"
           POSTGRES_DB: "gitlab"
     
       gitlab:
         image: "gitlab/gitlab-ce:latest"
         volumes:
           - "/srv/gitlab-swarm/gitlab/data:/var/opt/gitlab"
           - "/srv/gitlab-swarm/gitlab/logs:/var/log/gitlab"
           - "/srv/gitlab-swarm/gitlab/config:/etc/gitlab"
     #      - "/srv/gitlab/data:/var/opt/gitlab"
     #      - "/srv/gitlab/logs:/var/log/gitlab"
     #      - "/srv/gitlab/config:/etc/gitlab"
         ports:
           - "2222:22"
           - "8888:80"
         configs:
           - source: "gitlab.rb"
             target: "/etc/gitlab/gitlab.rb"
         depends_on:
           - postgres
         networks:
           - default
           - balancer
         deploy:
           replicas: 1
           restart_policy:
             condition: on-failure
           labels:
             - "traefik.port=80"
             - "traefik.backend=gitlab"
             - "traefik.docker.network=balancer"
             - "traefik.frontend.rule=Host:www.git.tienle.com,git.tienle.com"
       redis:
         image: "redis"
       prometheus:
         image: "prom/prometheus"
         command: "--config.file=/prometheus.yaml --storage.tsdb.path /data"
         volumes:
           - "/srv/gitlab-swarm/prometheus:/data"
         configs:
           - prometheus.yaml
         networks:
           - default
           - balancer
         deploy:
           labels:
             traefik.port: 9090
             traefik.frontend.rule: "Host:prometheus.git.tienle.com"
             traefik.docker.network: "balancer"
       grafana:
         image: grafana/grafana
         environment:
           GF_PATHS_CONFIG: "/grafana.ini"
         configs:
           - grafana.ini
         volumes:
           - "/srv/gitlab-swarm/grafana:/data"
         networks:
           - default
           - balancer
         deploy:
           labels:
             traefik.port: 3000
             traefik.frontend.rule: "Host:grafana.git.tienle.com"
             traefik.docker.network: "balancer"
     
     configs:
       gitlab.rb:
         file: "./swarm-configs/gitlab.rb"
       prometheus.yaml:
         file: "./swarm-configs/prometheus.yaml"
       grafana.ini:
         file: "./swarm-configs/grafana.ini"
     
     networks:
       default:
       balancer:
         external:
           name: balancer

   • “swarm-configs/gitlab.rb“:

     external_url 'http://git.tienle.com'
     #registry_external_url 'http://registry.git.tienle.com'
     
     # Disable services
     postgresql['enable'] = false
     redis['enable'] = false
     prometheus['enable'] = false
     postgres_exporter['enable'] = false
     redis_exporter['enable'] = false
     
     # SMTP 
     gitlab_rails['smtp_enable'] = true
     gitlab_rails['smtp_address'] = "smtp.mailgun.org"
     gitlab_rails['smtp_port'] = 465
     gitlab_rails['smtp_user_name'] = "XXXXXXXXXXXXX"
     gitlab_rails['smtp_password'] = "XXXXXXXXXXXXX"
     gitlab_rails['smtp_domain'] = "mail.XXXXXXXXXXXX.com"
     gitlab_rails['smtp_authentication'] = "login"
     gitlab_rails['smtp_enable_starttls_auto'] = true
     gitlab_rails['smtp_tls'] = true
     
     # Postgres settings
     gitlab_rails['db_adapter'] = "postgresql"
     gitlab_rails['db_encoding'] = "unicode"
     
     # database service will be named "postgres" in the stack
     gitlab_rails['db_host'] = "postgres" 
     gitlab_rails['db_database'] = "gitlab"
     gitlab_rails['db_username'] = "gitlab"
     gitlab_rails['db_password'] = "gitlab"
     
     # Redis settings
     # redis service will be named "redis" in the stack
     gitlab_rails['redis_host'] = "redis"
     
     # Prometheus exporters
     node_exporter['listen_address'] = '0.0.0.0:9100'
     gitlab_monitor['listen_address'] = '0.0.0.0'
     gitaly['prometheus_listen_addr'] = "0.0.0.0:9236"
     gitlab_workhorse['prometheus_listen_addr'] = "0.0.0.0:9229"

   • “swarm-configs/prometheus.yaml“:

     global:
       scrape_interval: 15s
     
     scrape_configs:
       - job_name: 'prometheus'
         static_configs:
           - targets: ['localhost:9090']
     
       # gitlab monitor
       - job_name: 'gitlab_monitor'
         static_configs:
           - targets: ['gitlab:9168']
     
       # gitlab sidekiq
       - job_name: 'gitlab_sidekiq'
         metrics_path: /sidekiq
         static_configs:
           - targets: ['gitlab:9168']
     
       # gitlab process
       - job_name: 'gitlab_process'
         metrics_path: /process
         static_configs:
           - targets: ['gitlab:9168']
     
       # gitlab pages
       - job_name: 'gitlab_pages'
         static_configs:
           - targets: ['gitlab:9235']
     
       # gitaly
       - job_name: gitaly
         static_configs:
           - targets: ['gitlab:9236']
     
       # gitlab workhorse
       - job_name: workhorse
         static_configs:
           - targets: ['gitlab:9229']
     

   • “swarm-configs/grafana.ini“:

     [database]
     path = "/data/grafana.db"
     
     [session]
     provider = "redis"
     provider_config = "addr=redis:6379,prefix=grafana:"

2. Then, I start the docker stack:

   docker stack deploy -c docker-compose.yml gitlab

3. Then, I prepare backup data for restoration. This simply copies the Gitlab backup one to the “/srv/gitlab/data/backups/” folder.
4. Then, I need to stop the “unicorn” and “sidekiq” in the “gitlab” container before continuing on backup restoration. Then do restoration:

   docker exec -t __CONTAINER_NAME__ gitlab-ctl stop unicorn
   docker exec -t __CONTAINER_NAME__ gitlab-ctl stop sidekiq
   docker exec -it __CONTAINER_NAME__  gitlab-rake gitlab:backup:restore BACKUP=XXXX_BACKUP_TIMESTAMP_XXXX

5. Finally, restart the Gitlab service:

   docker exec -t __CONTAINER_NAME__ gitlab-ctl restart

   And set up a cron to backup the gitlab container daily

   0 0 * * * docker exec -it $(docker ps | grep gitlab_gitlab | cut -d " " -f 1) gitlab-rake gitlab:backup:create

6. IF you want to run multiple instances for the Gitlab service, this is the time to change the deployment parameters.