storage

s3cmd

Install s3cmd (s3 CLI client)

apt install -y s3cmd

Configuration file

s3cmd --configure
${HOME}/.s3cfg

CLI

s3cmd mb s3://buk1
s3cmd put /tmp/1m.img s3://buk1
s3cmd ls s3://buk1
s3cmd du
s3cmd rm s3://buk1/1m.img
s3cmd rb s3://buk1
s3cmd rb s3://buk1 --recursive
s3cmd rm s3://lfs/ --recursive --force
 
s3cmd \
  --access_key xxxxxxxxxxx  \
  --secret_key xxxxxxxxxxxxxx \
  --host-bucket '%(bucket)s.s3.example.com' \
  --host s3.example.com \
  --signature-v2 \
  --no-preserve \
  ls s3://lpz-duply

ACL

s3cmd setacl s3://bucket/path/to/file --acl-public
s3cmd info s3://bucket/path/to/file
s3cmd setacl s3://bucket/path/to/file --acl-private

Links
https://lollyrock.com/articles/s3cmd-with-radosgw/
https://linuxconfig.org/getting-started-with-aws-s3cmd-command-by-examples

s3fs

Install

sudo apt install -y s3fs

Create credentials

echo ACCESS_KEY_ID:SECRET_ACCESS_KEY > ~/.passwd-s3fs
chmod 600 ~/.passwd-s3fs

Mount s3 bucket

s3fs backups /media/backup \
  -o url=https://s3.example.com \
  -o passwd_file=~/.passwd-s3fs

Parameter

-o url=https://s3.example.com,allow_other,umask=0000
-o use_cache=/tmp/cache

/etc/fstab

mybucket1.mydomain.org /mnt/mybucket1 fuse.s3fs _netdev,allow_other,passwd_file=/home/ftpuser/.passwd-aws-s3fs,default_acl=public-read,uid=1001,gid=65534   0 0

Links
https://github.com/s3fs-fuse/s3fs-fuse
https://gridscale.io/community/tutorials/s3-fuse-ubuntu/

NVMe

Install

sudo apt install -y nvme-cli

CLI

# list devices
nvme list
 
nvme smart-log /dev/nvme0n1
 
isdct show -d DeviceStatus,Index,Firmware,FirmwareUpdateAvailable -intelssd

Fix nvme1: ignoring ctrl due to duplicate subnqn
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1803692

# dmesg | grep nvme
[    2.546620] nvme nvme0: pci function 0000:5e:00.0
[    2.552447] nvme nvme1: pci function 0000:5f:00.0
[    2.768347] nvme nvme1: ignoring ctrl due to duplicate subnqn (nqn.2017-12.org.nvmexpress:uuid:11111111-2222-3333-4444-555555555555).
[    2.775422] nvme nvme1: Removing after probe failure status: -22
[    2.779813]  nvme0n1: p1 p2

Fix by upgrade NVMe firmware
http://www.panticz.de/intel/nvme

Links
https://wiki.hetzner.de/index.php/NVMe

Ceph

ceph status
ceph health detail
ceph df detail
ceph osd lspools
osd pool ls detail

Monitoring
https://github.com/ceph/ceph-nagios-plugins/releases

Build nagios-plugins-ceph
sudo apt-get install -y devscripts fakeroot build-essential dh-python
git clone https://github.com/ceph/ceph-nagios-plugins.git
cd ceph-nagios-plugins#
make deb
sudo dpkg -i nagios-plugins-ceph_*_all.deb

# create wrapper for kolla-ansible installation
cat < /usr/bin/ceph
#!/bin/bash
docker exec -it ceph_mon ceph $@
EOF
chmod +x /usr/bin/ceph

DRBD

# cat /etc/drbd.d/global_common.conf 
global {
		usage-count	yes;
}
 
common {
	startup {
		degr-wfc-timeout	0;
	}
 
	net {
		cram-hmac-alg	sha1;
		shared-secret	****************;
	}
 
	disk {
		on-io-error	detach;
	}
}
 
# cat /etc/drbd.d/r0.res 
resource r0 {
	on scld.sedo.de.intern {
		volume 0 {
			device		/dev/drbd0;
			disk		/dev/vg0/lvol0;
			flexible-meta-disk	internal;
		}
		address		192.168.255.1:7788;
	}
	on ubuntu {
		volume 0 {
			device		/dev/drbd0;
			disk		/dev/sda3;
			flexible-meta-disk	internal;
		}
		address		192.168.255.2:7788;
	}