MONA

// Documentation

System Documentation

Deployment guides, node exporters setup, and machine learning anomaly detection architecture.

Deployment

MONA runs in two ways. Both give you the same API, Celery workers and web UI. They differ in the monitoring stack around them.

Terraform and HelmDocker Compose
Best forFull setup with cluster monitoring and alertsA quick local run or a single host
IncludesEverything in Compose, plus Grafana, Loki, Alertmanager and the cluster PrometheusAPI, Celery, PostgreSQL, Redis, web UI and the Prometheus for devices
NeedsDocker Desktop, kind, helm, TerraformDocker Desktop
Configterraform.tfvars, mona-chart/values.yaml.env
  1. Install Docker Desktop, kind, helm and Terraform.
  2. Run node exporter on every device you want to monitor (see Node exporter).
  3. Clone the repository.
  4. Copy terraform/terraform.tfvars.example to terraform/terraform.tfvars and fill in the admin and user accounts, PostgreSQL settings, the Grafana password and, for alerts, the Telegram bot token and chat ID.
  5. Optional: list devices in mona-chart/values.yaml, or add them later in the Admin panel.
  6. Optional: tune ML and rate limiter settings in config.py.
  7. Deploy.
bash
git clone https://github.com/gwill1337/MONA.git
cd MONA
Windows
.\deploy.ps1 -all # or .\deploy.ps1 -deploy
Linux and macOS
make all

After changing code

Images are built locally as mona-*:local and loaded into the kind cluster on every apply. Run build first, then deploy.

Node exporter

MONA reads CPU and RAM of each device from Prometheus, which scrapes an exporter on port 9100. Install one on every machine you want to monitor.

Package manager (Debian, Ubuntu)

bash
sudo apt update && sudo apt install -y prometheus-node-exporter

Binary with systemd (any distro)

bash
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvf node_exporter-*.tar.gz
sudo cp node_exporter-*/node_exporter /usr/local/bin/
/etc/systemd/system/node_exporter.service
[Unit]
Description=Node Exporter
After=network.target

[Service]
User=nobody
ExecStart=/usr/local/bin/node_exporter

[Install]
WantedBy=multi-user.target
bash
sudo systemctl daemon-reload
sudo systemctl enable --now node_exporter

Check that it works

bash
curl http://localhost:9100/metrics

You should see lines starting with # HELP. If the request hangs, a firewall is in the way.

Open the port only to MONA

bash
sudo ufw allow from <MONA-HOST-IP> to any port 9100 proto tcp

No authentication

Exporters serve metrics to anyone who can reach them. Never expose port 9100 to the internet, and allow only your MONA host or LAN subnet.

Other exporters

Prometheus scrapes anything that serves metrics over HTTP. Servers, network gear and services plug in the same way: run an exporter, then register its host and port.

TargetExporterPortNotes
Linux, macOS, BSDnode_exporter (opens in a new tab)9100The default for MONA
Windowswindows_exporter (opens in a new tab)9182MSI installer, runs as a service
Routers, switches, UPS, printerssnmp_exporter (opens in a new tab)9116Anything that speaks SNMP
Server hardware sensorsipmi_exporter (opens in a new tab)9290iDRAC, iLO and other BMCs
Docker containerscAdvisor (opens in a new tab)8080CPU and memory per container
HTTP, TCP and ping checksblackbox_exporter (opens in a new tab)9115Probes a target, like SNMP
NVIDIA GPUsdcgm-exporter (opens in a new tab)9400
PostgreSQLpostgres_exporter (opens in a new tab)9187
Redisredis_exporter (opens in a new tab)9121

What feeds anomaly detection

Anomaly detection works on CPU and RAM. Data from other exporters is visible in Prometheus and, with the Terraform setup, in Grafana through the Prometheus-PC data source.

Example: a network device over SNMP

Run snmp_exporter once, then add one scrape job per device. The job passes the device address to the exporter as a parameter.

bash
docker run -d --name snmp-exporter -p 9116:9116 prom/snmp-exporter
scrape_configs
- job_name: 'snmp-switch'
  static_configs:
    - targets: ['192.168.1.254']
  metrics_path: /snmp
  params:
    module: [if_mib]
  relabel_configs:
    - source_labels: [__address__]
      target_label: __param_target
    - source_labels: [__param_target]
      target_label: instance

Add the job to prometheus/prometheus.yml for Docker Compose, or to prometheus-configmap.yaml in the Helm chart, then reload Prometheus without a restart:

bash
curl -X POST http://localhost:9090/-/reload

Connect a device

Register a device in any of three ways. Each one ends up as a Prometheus target.

  1. Open the Admin panel and choose Add device.
  2. Enter a name and the device IP.
  3. Wait for Prometheus to pick it up. The Helm setup refreshes the list every 30 seconds.

Best for adding and removing devices on the fly. The exporter must listen on port 9100.

Reachability

The address must be reachable from the Prometheus pod or container, not only from your laptop. Check routing and firewalls between MONA and the device.

Check that the device is scraped

Open Prometheus for devices, then Status and Targets. The device should be UP. You can also read the list MONA serves to Prometheus (no login needed):

bash
curl http://localhost:30080/api/prometheus/targets

How ML works

MONA looks for unusual CPU and RAM behavior with Isolation Forest from scikit-learn. The model does not know what bad looks like. It learns what is usual for your devices and flags points that do not fit.

The path of a data point

  1. 01Collect

    Celery beat schedules a task. A Celery worker reads CPU and RAM for every device from Prometheus and stores them in the metrics table. Detection runs on a schedule, every 60 seconds by default.

  2. 02Build features

    Each point carries CPU, RAM and how much both changed since the previous point (deltas). Deltas let the model catch sudden jumps, not only high values.

  3. 03Score

    Isolation Forest isolates points with random splits. Unusual points need fewer splits, so they get a lower score. The model flags about 5% of points (CONTAMINATION = 0.05), even on a healthy machine.

  4. 04Filter

    A flagged point survives only if its score is below SCORE_THRESHOLD = -0.05, and a combined CPU and RAM anomaly is dropped while both stay under 80%.

  5. 05Save

    What is left goes to the anomalies table with the device, values, reason and score, and is available on the dashboard and through the API.

Auto and Manual

Auto needs no setup and follows the current load. Manual lets you fix a reference period and keep the model.

AutoManual
Training dataThe most recent 50–500 points, a sliding windowA time range you choose
Model storageNot stored, rebuilt in memorySaved in PostgreSQL with period, point count and a note
RetrainingBefore every detection, every 60 seconds by defaultOnly when you run training again
Follows load changesYes, the window moves with the current loadNo, it keeps judging by the period it learned
Watch out forA long problem fills the window and starts to look normalNeeds retraining after workload changes
Use it whenYou are getting started or the load changes over timeThe load is regular and you can point at a healthy period

Training a model manually

Training is an admin action and runs in Celery, so the API answers right away. It needs at least 30 points in the chosen range. Without a stored model MONA works in Auto mode. The endpoints are listed in Swagger:

  • POST /api/v1/train — starts training for a time range
  • GET /api/v1/task-status/{task_id} — shows task progress
  • GET /api/v1/model-info — shows the stored model
  • DELETE /api/v1/model — removes it

Anomaly thresholds

Isolation Forest always flags a fixed share of points, even on a healthy machine. Three rules turn that raw output into something worth looking at.

CONTAMINATION = 0.05Share of points flagged
Isolation Forest is told to expect up to 5% outliers, so it flags that share of points even when nothing is wrong. On its own this setting produces false alarms. It is the raw output that the next two filters clean up.
SCORE_THRESHOLD = -0.05Minimum strength
A flagged point must score below −0.05, and lower means more unusual. Borderline points that only just crossed the line are discarded. The score is internal to the model: it is not a percentage, a probability or a severity.
CPU and RAM < 80%Quiet machines
The model compares a point with the device’s own history, not with a healthy level. On an idle host a jump from 3% to 25% CPU is statistically rare but harmless. When both CPU and RAM stay under 80%, a combined anomaly is ignored.

What counts as an anomaly

A point the model flagged, whose score is below the threshold and which was not dropped by the 80% rule.

Example: a quiet server idles at 3% CPU and 20% RAM. CPU jumps to 25% for a minute. The model finds it unusual, but both values are under 80%, so MONA drops it. If CPU reaches 95% while RAM climbs to 85% and the score is below −0.05, MONA saves it.

Unusual is not broken

An anomaly means the point differs from the learned pattern, not that something failed. The absence of anomalies does not prove a device is healthy either, so check that metrics are arriving. ML settings live in config.py; set them before you deploy.

Troubleshooting

Common problems first, then the commands you will reach for.

SymptomWhat to check
Target is DOWN in PrometheusOpen http://<device-ip>:9100/metrics from the MONA host. Check the firewall and that the exporter listens on all interfaces, not only 127.0.0.1.
Device added, no dataConfirm the target is UP, then read the Celery worker and Celery beat logs.
Windows device not foundwindows_exporter uses port 9182. Reinstall with LISTEN_PORT=9100 or register it in values.yaml with port 9182.
No anomaliesThe model needs enough history. On idle machines the 80% rule filters most spikes, which is expected.
401 or 403401 means the session expired (it lasts 12 hours). 403 means the account is a user: users are read-only, and adding devices or training needs an admin.
Code changes do not show upWith Terraform, rebuild the images, then deploy again. With Compose, start with --build.
bash
kubectl get pods -n mona
kubectl logs deployment/fastapi-deployment -n mona