s3

s3cmd

Install s3cmd (s3 CLI client)

sudo apt install -y s3cmd

Configuration file

s3cmd --configure
${HOME}/.s3cfg

CLI

s3fs

Install

sudo apt install -y s3fs

Create credentials

echo ${BUCKET}:${ACCESS_KEY}:${SECRET_KEY} >> ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

Mount S3 bucket

s3fs bucket1 /tmp/mnt -o url=https://s3.example.com

Parameter

-o passwd_file=~/.passwd-s3fs
-o url=https://s3.example.com,allow_other,umask=0000
-o use_cache=/tmp/cache

/etc/fstab

mybucket1.mydomain.org /mnt/mybucket1 fuse.s3fs _netdev,allow_other,passwd_file=/home/ftpuser/.passwd-aws-s3fs,default_acl=public-read,uid=1001,gid=65534 0 0

Links
https://github.com/s3fs-fuse/s3fs-fuse
https://gridscale.io/community/tutorials/s3-fuse-ubuntu/

GitLab: Backup to S3

Configure Git
# /etc/gitlab/gitlab.rb
gitlab_rails['backup_upload_connection'] = {
'provider' => 'AWS',
'region' => 'ew-west-1',
'aws_access_key_id' => 'KEY123',
'aws_secret_access_key' => 'PASS124',
'endpoint' => 'https://s3.example.com'
}
gitlab_rails['backup_upload_remote_directory'] = 'backups'
gitlab_rails['backup_keep_time'] = 604800

gitlab-ctl reconfigure

Test
gitlab-rake gitlab:backup:create

Configure periodic backup
# crontab -e
0 2 * * * /opt/gitlab/bin/gitlab-rake gitlab:backup:create CRON=1

GitLab: LFS on S3

Configure GitLab
...
gitlab_rails['lfs_enabled'] = true
gitlab_rails['lfs_object_store_enabled'] = true
gitlab_rails['lfs_object_store_proxy_download'] = true
gitlab_rails['lfs_object_store_remote_directory'] = "lfs"
gitlab_rails['lfs_object_store_connection'] = {
'provider' => 'AWS',
'aws_access_key_id' => 'KEY123',
'aws_secret_access_key' => 'PASS1234',
'endpoint' => 'https://s3.example.com',
}

gitlab-ctl reconfigure

Install (on client)
apt-get install -y git git-lfs

# create LFS testfile

S3 storage under Linux / Ubuntu

Install
sudo apt install -y s3fs

Configuration
# cat ~/.passwd-s3fs
AWS Access Key ID:AWS Secret Access Key

Mount
s3fs backup /media/backup -o url=https://s3.example.com,allow_other,umask=0000
# -o passwd_file=/etc/passwd-s3fs
# -o use_cache=/tmp/cache

/etc/fstab
mybucket1.mydomain.org /mnt/mybucket1 fuse.s3fs _netdev,allow_other,passwd_file=/home/ftpuser/.passwd-aws-s3fs,default_acl=public-read,uid=1001,gid=65534 0 0

Links
https://gridscale.io/community/tutorials/s3-fuse-ubuntu/

GitLab registry with S3 storage

Configure CEPH
touch /tmp/placeholder
s3cmd mb s3://gitlab
s3cmd put /tmp/placeholder s3://gitlab
s3cmd ls s3://gitlab
s3cmd du s3://gitlab

Configure GitLab
# /etc/gitlab/gitlab.rb
...
registry_external_url 'https://registry.example.com'
registry_nginx['enable'] = true
registry_nginx['ssl_certificate'] = "/etc/gitlab/ssl/registry.example.com.crt"
registry_nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/registry.example.com.key"
#registry['log_level'] = "debug"
#registry['storage_delete_enabled'] = true
...
registry['storage'] = {

Subscribe to s3