gitlab | panticz.de

GitLab: Docker CI pipeline

Optinal: Create nested LXD container
http://www.panticz.de/lxd/nesting

CONTAINER_NAME=gitlab-runner1-dev
lxc launch ubuntu:18.04 ${CONTAINER_NAME} -p disk-zfs -p nic-dev-mgmt -c boot.autostart=true -c security.nesting=true -c security.privileged=true
#-c volatile.dev-mgmt.hwaddr=00:11:22:33:44:55

lxc exec ${CONTAINER_NAME} -- apt update
lxc exec ${CONTAINER_NAME} -- apt dist-upgrade
lxc exec ${CONTAINER_NAME} -- apt purge -y lxd lxd-client snapd unattended-upgrades
lxc exec ${CONTAINER_NAME} -- apt autoremove

lxc file push /root/.ssh/authorized_keys ${CONTAINER_NAME}/root/.ssh/authorized_keys
lxc exec ${CONTAINER_NAME} -- bash -c "sed -i 's/eth0:/dev-mgmt:/g' /etc/netplan/50-cloud-init.yaml"
lxc exec ${CONTAINER_NAME} -- netplan apply

printf 'lxc.apparmor.profile = unconfined\nlxc.cgroup.devices.allow = a\nlxc.mount.auto=proc:rw sys:rw\nlxc.cap.drop=' | lxc config set ${CONTAINER_NAME} raw.lxc -
lxc restart ${CONTAINER_NAME}

Install Docker inside LXD container
# http://www.panticz.de/install-docker

apt update
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -
sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"
sudo apt-get install -y docker-ce

Install GitLab-runner (as Docker container)
https://docs.gitlab.com/runner/install/docker.html

docker run -d \
  --name gitlab-runner \
  --restart always \
  -v /srv/gitlab-runner/config:/etc/gitlab-runner \
  -v /var/run/docker.sock:/var/run/docker.sock \
  gitlab/gitlab-runner:latest

Configue GitLab-runner

GitLab: Backup to S3

Configure Git

# /etc/gitlab/gitlab.rb
gitlab_rails['backup_upload_connection'] = {
  'provider' => 'AWS',
  'region' => 'foo-west-1',
  'aws_access_key_id' => 'KEY123',
  'aws_secret_access_key' => 'PASS124',
  'endpoint' => 'https://s3.example.com'
}
gitlab_rails['backup_upload_remote_directory'] = 'backups'
gitlab_rails['backup_keep_time'] = 604800

gitlab-ctl reconfigure

Test

gitlab-rake gitlab:backup:create

Configure periodic backup

# crontab -e
0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1

Links
https://docs.gitlab.com/ce/raketasks/backup_restore.html#uploading-backups-to-a-remote-cloud-storage

GitLab: LFS on S3

Configure GitLab

...
gitlab_rails['lfs_enabled'] = true
gitlab_rails['lfs_object_store_enabled'] = true
gitlab_rails['lfs_object_store_proxy_download'] = true
gitlab_rails['lfs_object_store_remote_directory'] = "lfs"
gitlab_rails['lfs_object_store_connection'] = {
   'provider' => 'AWS',
   'aws_access_key_id' => 'KEY123',
   'aws_secret_access_key' => 'PASS1234',
   'endpoint' => 'https://s3.example.com',
}

gitlab-ctl reconfigure

Install (on client)

apt-get install -y git git-lfs

# create LFS testfile
head -c 1M /dev/urandom > file2.bin

Add LFS file to repository

git lfs install
git lfs track *.bin
# git lfs track "*.bin"
# git lfs track images/
git add .gitattributes *.bin
git commit -m "Commit LFS file"
git push

Fix me
LFS file download / checkout broken

Debug

git lfs ls-files
git lfs env

Links
https://about.gitlab.com/2017/01/30/getting-started-with-git-lfs-tutorial/
http://blog.testdouble.com/posts/2017-01-25-how-and-why-to-use-git-lfs

GitLab CI/CD gitlabci

ENV variables
https://docs.gitlab.com/ee/ci/variables/predefined_variables.html

Build and push Docker Images with Gitlab CI

# .gitlab-ci.yml
build:
  stage: build
  image: docker:19.03.1
  services:
    - docker:dind
  before_script:
    - echo -n $CI_JOB_TOKEN | docker login -u gitlab-ci-token --password-stdin $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE" .
    - docker push "$CI_REGISTRY_IMAGE"
  tags:
    - docker

# Dockerfile
FROM alpine:3.10

RUN apk add --no-cache nginx

Reuse commands across Gitlab jobs
https://jsramblings.com/three-ways-to-reuse-commands-across-gitlab-jobs/

.prepare_step:
  before_script:
    - echo 'prepare'

build:
  extends:
    - .prepare_step
  script:
    - echo 'build'

test:
  extends:
    - .prepare_step
  script:
    - echo 'test'

Checkout additional repository witht SSH key
https://stackoverflow.com/questions/44363537/gitlab-ci-ssh-permission-denied-publickey-password
https://docs.gitlab.com/ee/ci/ssh_keys/

script:
    - mkdir -p ~/.ssh
    - echo "$SSH_PRIVATE_KEY" | tr -d '\r' > ~/.ssh/id_rsa
    - chmod 700 ~/.ssh/id_rsa

Checkout additional repository with token

script:
    - 'git clone --branch master git@git.i.example.com:foo/bar.git $CI_BUILDS_DIR/deploy'

Links
https://about.gitlab.com/2016/05/23/gitlab-container-registry/

GitLab registry with S3 storage

Configure CEPH

touch /tmp/placeholder
s3cmd mb s3://gitlab
s3cmd put /tmp/placeholder s3://gitlab
s3cmd ls s3://gitlab
s3cmd du s3://gitlab

Configure GitLab

# /etc/gitlab/gitlab.rb
...
registry_external_url 'https://registry.example.com'
registry_nginx['enable'] = true
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/registry.example.com.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/registry.example.com.key"
#registry['log_level'] = "debug"
#registry['storage_delete_enabled'] = true
...
registry['storage'] = {
  's3' => {
     'accesskey' => 'YOUR_ACCESS_KEY',
     'secretkey' => 'YOUR_SECRET_KEY',
     'bucket'    => 'gitlab',
     'region'    => 'us-west-1',
     'regionendpoint' => 'https://s3.example.com'
  },
  'redirect' => {
    'disable' => true
  }
}

cleanup registry (test):
sudo gitlab-ctl registry-garbage-collect

Links
https://gitlab.com/gitlab-org/gitlab-ce/issues/19356
https://docs.gitlab.com/ce/administration/container_registry.html
https://docs.docker.com/registry/configuration/#storage
https://icicimov.github.io/blog/server/GitLab-server-with-LDAP-and-S3-backend/

GitLab runner

Install
https://packages.gitlab.com/app/runner/gitlab-runner/gpg
https://packages.gitlab.com/runner/gitlab-runner/install

curl -s https://packages.gitlab.com/install/repositories/runner/gitlab-runner/script.deb.sh | sudo bash
sudo apt install gitlab-runner

List / delete gitlab-runner

# List GitLab runner configuration
gitlab-runner list

# Delete all GitLab runner
gitlab-runner unregister --all-runners

Instlal with Ansible
https://github.com/panticz/ansible/tree/master/roles/gitlab-runner
https://github.com/haroldb/ansible-gitlab-runner

Relogin as gitlab-runner user

su gitlab-runner -s /bin/bash

Get token from GitLab server
http:///admin/runners

# configure DNS for GitLab server
echo "10.0.1.12 gitlab.example.com gitlab" >> /etc/hosts

Register
https://docs.gitlab.com/runner/register/

GitLab: Web-based Git repository manager

Install
http://www.panticz.de/install-gitlab

CLI

# restart gitlab
gitlab-ctl restart

# git home directory
/var/opt/gitlab

Reset admin password

# change root password
sudo gitlab-rails console
user = User.where(id: 1).first 
user.password = user.password_confirmation ='xxx' 
user.save!

Gitlab settings API

https://docs.gitlab.com/ee/api/settings.html
curl --header "PRIVATE-TOKEN: 11112222333344445555" https://gitlab.example.com/api/v4/application/settings

Disalbe register / Singup

sudo gitlab-rails console
ApplicationSetting.last.update_attributes(signup_enabled: false)

backup
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/backup_restore.md

# full backup
gitlab-rake gitlab:backup:create

# backup without reposiories
sudo gitlab-rake gitlab:backup:create SKIP=repositories

# backup target
ls -l /var/opt/gitlab/backups

gitlab_rails['backup_keep_time'] = 604800

# backup to s3
gitlab_rails['artifacts_enabled'] = true
gitlab_rails['artifacts_object_store_enabled'] = true
gitlab_rails['artifacts_object_store_remote_directory'] = "artifacts"
gitlab_rails['artifacts_object_store_connection'] = {
  'provider' => 'AWS',
  'region' => 'foo-west-1',
  'aws_access_key_id' => '1111111111111111111111',
  'aws_secret_access_key' => '22222222222222222222222222222',
  'endpoint' => 'https://s3.example.com'
}

restore

sudo gitlab-rake gitlab:backup:restore force=yes
https://gitlab.com/gitlab-org/gitlab-ce/blob/master/doc/raketasks/backup_restore.md

Send email via SMTP
https://docs.gitlab.com/omnibus/settings/smtp.html
https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/doc/settings/smtp.md

# /etc/gitlab/gitlab.rb
gitlab_rails['smtp_enable'] = true
gitlab_rails['smtp_address'] = "smtp.example.com"

# reconfigure GitLab
gitlab-ctl reconfigure

# send testmail
gitlab-rails console
Notify.test_email('foo@example.com', 'GitLab Test', 'Test message from GitLab server').deliver_now

Create backup

gitlab