System Documentation
Deployment guides, node exporters setup, and machine learning anomaly detection architecture.
Deployment
MONA runs in two ways. Both give you the same API, Celery workers and web UI. They differ in the monitoring stack around them.
| Terraform and Helm | Docker Compose | |
|---|---|---|
| Best for | Full setup with cluster monitoring and alerts | A quick local run or a single host |
| Includes | Everything in Compose, plus Grafana, Loki, Alertmanager and the cluster Prometheus | API, Celery, PostgreSQL, Redis, web UI and the Prometheus for devices |
| Needs | Docker Desktop, kind, helm, Terraform | Docker Desktop |
| Config | terraform.tfvars, mona-chart/values.yaml | .env |
- Install Docker Desktop, kind, helm and Terraform.
- Run node exporter on every device you want to monitor (see Node exporter).
- Clone the repository.
- Copy
terraform/terraform.tfvars.exampletoterraform/terraform.tfvarsand fill in the admin and user accounts, PostgreSQL settings, the Grafana password and, for alerts, the Telegram bot token and chat ID. - Optional: list devices in
mona-chart/values.yaml, or add them later in the Admin panel. - Optional: tune ML and rate limiter settings in
config.py. - Deploy.
git clone https://github.com/gwill1337/MONA.git
cd MONA.\deploy.ps1 -all # or .\deploy.ps1 -deploymake allAfter changing code
Images are built locally asmona-*:local and loaded into the kind cluster on every apply. Run build first, then deploy.Node exporter
MONA reads CPU and RAM of each device from Prometheus, which scrapes an exporter on port 9100. Install one on every machine you want to monitor.
Package manager (Debian, Ubuntu)
sudo apt update && sudo apt install -y prometheus-node-exporterBinary with systemd (any distro)
wget https://github.com/prometheus/node_exporter/releases/download/v1.7.0/node_exporter-1.7.0.linux-amd64.tar.gz
tar xvf node_exporter-*.tar.gz
sudo cp node_exporter-*/node_exporter /usr/local/bin/[Unit]
Description=Node Exporter
After=network.target
[Service]
User=nobody
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.targetsudo systemctl daemon-reload
sudo systemctl enable --now node_exporterCheck that it works
curl http://localhost:9100/metricsYou should see lines starting with # HELP. If the request hangs, a firewall is in the way.
Open the port only to MONA
sudo ufw allow from <MONA-HOST-IP> to any port 9100 proto tcpNo authentication
Exporters serve metrics to anyone who can reach them. Never expose port 9100 to the internet, and allow only your MONA host or LAN subnet.Other exporters
Prometheus scrapes anything that serves metrics over HTTP. Servers, network gear and services plug in the same way: run an exporter, then register its host and port.
| Target | Exporter | Port | Notes |
|---|---|---|---|
| Linux, macOS, BSD | node_exporter (opens in a new tab) | 9100 | The default for MONA |
| Windows | windows_exporter (opens in a new tab) | 9182 | MSI installer, runs as a service |
| Routers, switches, UPS, printers | snmp_exporter (opens in a new tab) | 9116 | Anything that speaks SNMP |
| Server hardware sensors | ipmi_exporter (opens in a new tab) | 9290 | iDRAC, iLO and other BMCs |
| Docker containers | cAdvisor (opens in a new tab) | 8080 | CPU and memory per container |
| HTTP, TCP and ping checks | blackbox_exporter (opens in a new tab) | 9115 | Probes a target, like SNMP |
| NVIDIA GPUs | dcgm-exporter (opens in a new tab) | 9400 | |
| PostgreSQL | postgres_exporter (opens in a new tab) | 9187 | |
| Redis | redis_exporter (opens in a new tab) | 9121 |
What feeds anomaly detection
Anomaly detection works on CPU and RAM. Data from other exporters is visible in Prometheus and, with the Terraform setup, in Grafana through thePrometheus-PC data source.Example: a network device over SNMP
Run snmp_exporter once, then add one scrape job per device. The job passes the device address to the exporter as a parameter.
docker run -d --name snmp-exporter -p 9116:9116 prom/snmp-exporter- job_name: 'snmp-switch'
static_configs:
- targets: ['192.168.1.254']
metrics_path: /snmp
params:
module: [if_mib]
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instanceAdd the job to prometheus/prometheus.yml for Docker Compose, or to prometheus-configmap.yaml in the Helm chart, then reload Prometheus without a restart:
curl -X POST http://localhost:9090/-/reloadConnect a device
Register a device in any of three ways. Each one ends up as a Prometheus target.
- Open the Admin panel and choose Add device.
- Enter a name and the device IP.
- Wait for Prometheus to pick it up. The Helm setup refreshes the list every 30 seconds.
Best for adding and removing devices on the fly. The exporter must listen on port 9100.
Reachability
The address must be reachable from the Prometheus pod or container, not only from your laptop. Check routing and firewalls between MONA and the device.Check that the device is scraped
Open Prometheus for devices, then Status and Targets. The device should be UP. You can also read the list MONA serves to Prometheus (no login needed):
curl http://localhost:30080/api/prometheus/targetsLinks and ports
Addresses use the host you opened this page from. Ports are the defaults from the Helm chart and the Compose file.
| Service | Address | Terraform | Compose |
|---|---|---|---|
| Web UIAdmin panel | localhost:30081 (opens in a new tab) | Available | Available |
| DashboardMetrics and anomalies | localhost:30081/dashboard (opens in a new tab) | Available | Available |
| Userboard | localhost:30081/userboard (opens in a new tab) | Available | Available |
| SwaggerInteractive API reference | localhost:30080/docs (opens in a new tab) | Available | Available |
| Prometheus, devicesTargets scraped by MONA | localhost:30391 (opens in a new tab) | Available | Available |
| MetricsFastAPI /metrics | localhost:30080/metrics (opens in a new tab) | Available | Available |
| GrafanaDashboards and Loki logs | localhost:30300 (opens in a new tab) | Available | Not available |
| Prometheus, clusterPods and nodes | localhost:30091 (opens in a new tab) | Available | Not available |
MONA has two Prometheus instances. The one for devices is part of the product: it scrapes your exporters, and MONA reads CPU and RAM from it. The cluster one watches pods and nodes and exists only in the Terraform setup, together with Grafana.
Grafana access
The Grafana admin password is thegrafana_admin_password value from terraform.tfvars.How ML works
MONA looks for unusual CPU and RAM behavior with Isolation Forest from scikit-learn. The model does not know what bad looks like. It learns what is usual for your devices and flags points that do not fit.
The path of a data point
01Collect
Celery beat schedules a task. A Celery worker reads CPU and RAM for every device from Prometheus and stores them in the
metricstable. Detection runs on a schedule, every 60 seconds by default.02Build features
Each point carries CPU, RAM and how much both changed since the previous point (deltas). Deltas let the model catch sudden jumps, not only high values.
03Score
Isolation Forest isolates points with random splits. Unusual points need fewer splits, so they get a lower score. The model flags about 5% of points (
CONTAMINATION = 0.05), even on a healthy machine.04Filter
A flagged point survives only if its score is below
SCORE_THRESHOLD = -0.05, and a combined CPU and RAM anomaly is dropped while both stay under 80%.05Save
What is left goes to the
anomaliestable with the device, values, reason and score, and is available on the dashboard and through the API.
Auto and Manual
Auto needs no setup and follows the current load. Manual lets you fix a reference period and keep the model.
| Auto | Manual | |
|---|---|---|
| Training data | The most recent 50–500 points, a sliding window | A time range you choose |
| Model storage | Not stored, rebuilt in memory | Saved in PostgreSQL with period, point count and a note |
| Retraining | Before every detection, every 60 seconds by default | Only when you run training again |
| Follows load changes | Yes, the window moves with the current load | No, it keeps judging by the period it learned |
| Watch out for | A long problem fills the window and starts to look normal | Needs retraining after workload changes |
| Use it when | You are getting started or the load changes over time | The load is regular and you can point at a healthy period |
Training a model manually
Training is an admin action and runs in Celery, so the API answers right away. It needs at least 30 points in the chosen range. Without a stored model MONA works in Auto mode. The endpoints are listed in Swagger:
- POST /api/v1/train — starts training for a time range
- GET /api/v1/task-status/{task_id} — shows task progress
- GET /api/v1/model-info — shows the stored model
- DELETE /api/v1/model — removes it
Anomaly thresholds
Isolation Forest always flags a fixed share of points, even on a healthy machine. Three rules turn that raw output into something worth looking at.
CONTAMINATION = 0.05Share of points flagged- Isolation Forest is told to expect up to 5% outliers, so it flags that share of points even when nothing is wrong. On its own this setting produces false alarms. It is the raw output that the next two filters clean up.
SCORE_THRESHOLD = -0.05Minimum strength- A flagged point must score below −0.05, and lower means more unusual. Borderline points that only just crossed the line are discarded. The score is internal to the model: it is not a percentage, a probability or a severity.
CPU and RAM < 80%Quiet machines- The model compares a point with the device’s own history, not with a healthy level. On an idle host a jump from 3% to 25% CPU is statistically rare but harmless. When both CPU and RAM stay under 80%, a combined anomaly is ignored.
What counts as an anomaly
A point the model flagged, whose score is below the threshold and which was not dropped by the 80% rule.
Example: a quiet server idles at 3% CPU and 20% RAM. CPU jumps to 25% for a minute. The model finds it unusual, but both values are under 80%, so MONA drops it. If CPU reaches 95% while RAM climbs to 85% and the score is below −0.05, MONA saves it.
Unusual is not broken
An anomaly means the point differs from the learned pattern, not that something failed. The absence of anomalies does not prove a device is healthy either, so check that metrics are arriving. ML settings live inconfig.py; set them before you deploy.Troubleshooting
Common problems first, then the commands you will reach for.
| Symptom | What to check |
|---|---|
| Target is DOWN in Prometheus | Open http://<device-ip>:9100/metrics from the MONA host. Check the firewall and that the exporter listens on all interfaces, not only 127.0.0.1. |
| Device added, no data | Confirm the target is UP, then read the Celery worker and Celery beat logs. |
| Windows device not found | windows_exporter uses port 9182. Reinstall with LISTEN_PORT=9100 or register it in values.yaml with port 9182. |
| No anomalies | The model needs enough history. On idle machines the 80% rule filters most spikes, which is expected. |
| 401 or 403 | 401 means the session expired (it lasts 12 hours). 403 means the account is a user: users are read-only, and adding devices or training needs an admin. |
| Code changes do not show up | With Terraform, rebuild the images, then deploy again. With Compose, start with --build. |
kubectl get pods -n mona
kubectl logs deployment/fastapi-deployment -n mona