Install from repository

sudo apt-get install -y apt-transport-https software-properties-common wget
sudo mkdir -p /etc/apt/keyrings/
wget -q -O - https://apt.grafana.com/gpg.key | gpg --dearmor | sudo tee /etc/apt/keyrings/grafana.gpg > /dev/null

echo "deb [signed-by=/etc/apt/keyrings/grafana.gpg] https://apt.grafana.com stable main" | sudo tee -a /etc/apt/sources.list.d/grafana.list

sudo apt-get update

sudo apt-get install -y grafana

Credentials
http://SERVER_IP:3000/

Docker container

docker run -i -p 3000:3000 -e "GF_SERVER_ROOT_URL=http://10.0.3.164" -e "GF_SECURITY_ADMIN_PASSWORD=admin" grafana/grafana
user: admin
pass: admin

Configure Graphite datasource

Name: Graphite
Type: Graphite
Default: Y
Url: http://10.0.3.48:2003
Access: direct

Plugins

grafana-cli plugins install raintank-worldping-app
service grafana-server restart

Links
http://docs.grafana.org/installation/debian/

Install from repository

sudo apt-get install -y prometheus
# optional
sudo apt-get install -y prometheus-node-exporter
sudo apt-get install -y prometheus-alertmanager
sudo apt-get install -y prometheus-pushgateway

Ansible installation

https://github.com/panticz/ansible/tree/master/roles/prometheus
- hosts: localhost
  roles:
    - prometheus

Login
http://SERVER_IP:9090/

Commands

avg_over_time(node_memory_MemAvailable[5m])/1024/1024

Repositroy
https://github.com/cloudalchemy/ansible-prometheus

Docker

docker run --name prometheus -d -p 127.0.0.1:9090:9090 quay.io/prometheus/prometheus

Ports
https://github.com/prometheus/prometheus/wiki/Default-port-allocations

API
http://SERVER_IP:9090/metrics

Query

node_disk_io_now{device="sda"}

Deploy node_exporter without installation

# https://prometheus.io/download/#node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz -qO /tmp/node_exporter.tar.gz
tar xzf /tmp/node_exporter.tar.gz -C /tmp/
mv /tmp/node_exporter-0.16.0.linux-amd64/node_exporter /usr/bin/prometheus-node-exporter
nohup /usr/bin/prometheus-node-exporter &
prometheus-node-exporter --version

node_exporter Ansible installation

https://github.com/panticz/ansible/tree/master/roles/prometheus-node-exporter
- hosts: localhost
  roles:
    - prometheus-node-exporter

Use SSH tunnel to access node_exporter metrics
https://mkezz.wordpress.com/2017/08/04/prometheus-in-docker-ssh-tunnel-to-target-nodes/

Alerting
http://prometheus.example.com:9093/

schedule a service check

HOST_NAME=www.example.com
SERVICE_NAME=APT
/usr/bin/printf "[%lu] SCHEDULE_FORCED_SVC_CHECK;%s;%s;%s\n" $(date +%s) "${HOST_NAME}" "${SERVICE_NAME}" $(date +%s) | tee -a "${NAGIOS_CMD_SOCKET}"

schedule a host check

NAGIOS_CMD_SOCKET=/var/lib/icinga/rw/icinga.cmd
/usr/bin/printf "[%lu] SCHEDULE_HOST_CHECK;%s;%s\n" \
                 $(date +%s) \
                 "host_name" \
                 $(date +%s) | tee -a $NAGIOS_CMD_SOCKET
curl -d "cmd_typ=7&host=www.example.com&service=APT&force_check&btnSubmit=Commit" "http://monitoring.example.com/cgi-bin/icinga/cmd.cgi" -u 'foo:bar'

Schedule downtime
https://assets.nagios.com/downloads/nagioscore/docs/externalcmds/cmdinfo.php?command_id=118

HOST=www.example.com
STARTDATE=$(date +%s)
ENDDATE=$(($(date +%s) + 7200))
printf "[%lu] SCHEDULE_HOST_DOWNTIME;${HOST};${STARTDATE};${ENDDATE};0;0;7200;rundeck;Deployment" ${STARTDATE} > /data/icinga/var/rw/icinga.cmd

Schedule downtime for specific services

IFS=$(echo -en "\n\b")
STARTDATE=$(date +%s)
ENDDATE=$(date +%s -d "next monday 12:00:00")
DURATION=$((${ENDDATE} - ${STARTDATE}))

KNOWN_ISSUES="
srv1-dev,Swap Usage
srv3-prod,Hardware
"

for ISSUE in ${KNOWN_ISSUES}; do
    HOST=$(echo ${ISSUE} | cut -d"," -f1)
    SERVICE="$(echo ${ISSUE} | cut -d"," -f2)"
    echo "HOST: ${HOST}"
    echo "SERVICE: ${SERVICE}"

    printf "[%lu] SCHEDULE_SVC_DOWNTIME;${HOST};${SERVICE};${STARTDATE};${ENDDATE};0;0;${DURATION};SysAdmin1;FixMe" ${STARTDATE} > /data/icinga/var/rw/icinga.cmd
    sleep 1
done

Links
http://www.box293.com/guides/tricks/nagios-command-pipe-and-cgis
https://icinga.com/docs/icinga1/latest/en/cgiparams.html
https://stackoverflow.com/questions/6842683/how-to-set-downtime-for-any-specific-nagios-host-for-certain-time-from-commandli