Install prometheus under Ubuntu

Install from repository
sudo apt-get install -y prometheus

sudo apt-get install -y prometheus-node-exporter
sudo apt-get install -y prometheus-alertmanager
sudo apt-get install -y prometheus-pushgateway

Login
http://SERVER_IP:9090/

Commands
avg_over_time(node_memory_MemAvailable[5m])/1024/1024

Ansible
https://github.com/cloudalchemy/ansible-prometheus

Docker
docker run --name prometheus -d -p 127.0.0.1:9090:9090 quay.io/prometheus/prometheus

Ports
https://github.com/prometheus/prometheus/wiki/Default-port-allocations

API
http://SERVER_IP:9090/metrics

Query
node_disk_io_now{device="sda"}

Deploy node_exporter without installation
# https://prometheus.io/download/#node_exporter
wget https://github.com/prometheus/node_exporter/releases/download/v0.16.0/node_exporter-0.16.0.linux-amd64.tar.gz -qO /tmp/node_exporter.tar.gz
tar xzf /tmp/node_exporter.tar.gz -C /tmp/
mv /tmp/node_exporter-0.16.0.linux-amd64/node_exporter /usr/bin/prometheus-node-exporter
nohup /usr/bin/prometheus-node-exporter &
prometheus-node-exporter --version

Use SSH tunnel to access node_exporter metrics
https://mkezz.wordpress.com/2017/08/04/prometheus-in-docker-ssh-tunnel-to-target-nodes/

Alerting
http://prometheus.example.com:9093/

# /etc/prometheus/prometheus.yml
...
rule_files:
- alert.rules.yml

# cat /etc/prometheus/alert.rules.yml
groups:
- name: node.rules
rules:
- alert: InstanceDown
expr: up{job="node"} == 0
for: 1m

Exporters
https://prometheus.io/docs/instrumenting/exporters/

https://github.com/prometheus/alertmanager
https://www.robustperception.io/alerting-on-down-instances
https://pierrevincent.github.io/2017/12/prometheus-blog-series-part-5-alerting-rules/
https://en.fabernovel.com/insights/tech-en/alerting-in-prometheus-or-how-i-can-sleep-well-at-night
https://www.digitalocean.com/community/tutorials/how-to-use-alertmanager-and-blackbox-exporter-to-monitor-your-web-server-on-ubuntu-16-04

unsee
https://github.com/cloudflare/unsee

wget https://github.com/cloudflare/unsee/releases/download/v0.9.2/unsee-linux-amd64.tar.gz
tar xzf unsee-linux-amd64.tar.gz unsee-linux-amd64
./unsee-linux-amd64 --alertmanager.uri http://localhost:9093

http://prometheus.example.com:8080/

Links
https://github.com/prometheus/pushgateway
https://prometheus.io/docs/introduction/overview/
https://www.digitalocean.com/community/tutorials/how-to-install-prometheus-on-ubuntu-16-04