mirror of
https://github.com/CyberMind-FR/secubox-deb.git
synced 2026-06-29 10:08:36 +00:00
Compare commits
17 Commits
740cbd291f
...
47076b24d3
| Author | SHA1 | Date | |
|---|---|---|---|
| 47076b24d3 | |||
| cb6eee4b0b | |||
|
|
5d505cae16 | ||
| 1c6a978748 | |||
| 9696cdad13 | |||
| fd9f535e63 | |||
| d91f6a8b67 | |||
| c1c4810f3e | |||
| d05deee7ff | |||
| b30a69316a | |||
| 851af7d172 | |||
| 55814ac3a0 | |||
| e6260215a4 | |||
| 6ae107db91 | |||
| 60f876ad08 | |||
| bdafced25a | |||
| 5b283b6bff |
|
|
@ -3,6 +3,63 @@
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
## 2026-06-27 — Netboot live PROUVÉ + première install SecuBox Debian sur c3box (second MOCHAbin) (#748 #737)
|
||||||
|
|
||||||
|
Grande session hardware : netboot gk2→c3box validé de bout en bout, premier SecuBox Debian installé
|
||||||
|
sur un vrai MOCHAbin, et le blocage U-Boot qui empêche #748 de fermer est formellement documenté.
|
||||||
|
|
||||||
|
### A. Netboot gk2 → c3box : validé en prod
|
||||||
|
|
||||||
|
- **c3box** (second MOCHAbin, Armada 7040) a booté l'installeur SecuBox Debian servi par gk2 via
|
||||||
|
TFTP : factory U-Boot 2020.10 → `tftpboot Image/dtb/initrd` → `booti` → rescue shell installeur,
|
||||||
|
kernel custom 6.12.85 #5secubox. Le FIT signé (49 Mo) était servi en HTTP sur `:8099`.
|
||||||
|
- Le long détour cabling était une impasse LAB (prouvé via gk2 bridge-FDB + test DHCP) — aucun
|
||||||
|
bug logiciel.
|
||||||
|
- **Learnings opérationnels réutilisables** (documentés dans `wiki/Netboot-Install.md`) :
|
||||||
|
- Factory U-Boot 2020.10 s'interrompt sur **Enter** (pas Ctrl-C), `bootdelay=2`.
|
||||||
|
- Son env n'est PAS dans SPI mtd2 (env étranger fossile) → `fw_setenv` depuis Linux n'a aucun
|
||||||
|
effet ; seule la config U-Boot interne compte.
|
||||||
|
- Seul le port cuivre RJ45 unique = `mvpp2-2` est bootable par le factory U-Boot (les 4 ports
|
||||||
|
switch nécessitent le driver MV88E6XXX DSA, absent au boot).
|
||||||
|
- Kernel load à `0x02080000` = adresse mémoire réservée → crash immédiat ; utiliser `0x0a000000`.
|
||||||
|
- `setenv tftpblocksize 1468` pour TFTP rapide.
|
||||||
|
|
||||||
|
### B. #748 enhanced Tow-Boot (HTTP/wget bootloader) — DIFFÉRÉ, bloquant documenté
|
||||||
|
|
||||||
|
Branche `feature/748-enhanced-tow-boot-http-netboot-serial-fl` (stackée sur #737) :
|
||||||
|
spec+plan (`docs/superpowers/`), Kconfig Tow-Boot, `build-uboot-overlay.sh --tow-boot`,
|
||||||
|
plan serial-flasher, CI `.github/workflows/build-tow-boot.yml` (push-triggered).
|
||||||
|
|
||||||
|
**Bloquant dur (ciseau)** : le board MOCHAbin n'existe que dans le fork U-Boot 2022.07 de
|
||||||
|
Tow-Boot (pas de `wget`) ; `wget` n'existe que dans U-Boot stock ≥2023.07 (pas de board
|
||||||
|
mochabin/DTS). Bump à stock 2023.07 = `wget` compile mais build sans DTS. Pour débloquer :
|
||||||
|
backporter wget/TCP dans le fork Tow-Boot 2022.07, OU porter le board mochabin vers mainline
|
||||||
|
≥2023.07. Pas un tweak de config.
|
||||||
|
|
||||||
|
### C. PREMIÈRE INSTALL — c3box → SecuBox Debian (la headline)
|
||||||
|
|
||||||
|
- **Image** : artefact CI `secubox-mochabin-bookworm` (run 27426515472, 1,8 Go gzip / 8,0 Gio
|
||||||
|
décompressé), téléchargée sur gk2 `/data`, SHA256SUMS vérifié.
|
||||||
|
- **Signature** : clé `secubox-netboot.key` de gk2. Vérifié : cette clé FIT == `netboot-image.pub`
|
||||||
|
embarquée dans l'installeur (modulus match + roundtrip sign/verify). `sbx.img.gz` + `.sig`
|
||||||
|
publiés dans le root HTTP netboot, servis sur `:8099` (symlink depuis `/data`).
|
||||||
|
- **Install automatisé depuis le rescue shell** :
|
||||||
|
`wget sbx.img.gz` (en RAM, c3box a 8 Go) →
|
||||||
|
`openssl dgst -verify` contre `netboot-image.pub` (résultat : Verified OK) →
|
||||||
|
`gunzip | dd of=/dev/mmcblk0 bs=4M conv=fsync` (8 Gio, progression 32→62→94→100%) → sync.
|
||||||
|
- **c3box démarre SecuBox Debian v1.9.0** — hostname `secubox-mochabin`, kernel Debian
|
||||||
|
6.1.0-47-arm64, stack complète : secuboxd, hub, grafana, zigbee, mqtt, authelia,
|
||||||
|
sentinel/rogue-BTS (layers WALL+MIND). Creds root/secubox, Web UI `:9443`.
|
||||||
|
- **Fix auto-boot persistant** : l'image utilise `extlinux.conf` à `0x02080000` (adresse réservée
|
||||||
|
factory U-Boot → reset immédiat) et ne livre pas de `boot.scr` compilé. Construit
|
||||||
|
`/boot/boot.scr` (kernel@`0x0a000000`, initrd@`0x10000000`, `console=ttyS0` + earlycon,
|
||||||
|
`root=LABEL=rootfs`) : le factory U-Boot charge `boot.scr` depuis mmc et démarre Debian sans
|
||||||
|
intervention. **VÉRIFIÉ** : reboot sans intervention → login Debian.
|
||||||
|
- **Layout eMMC installé** : GPT p1=boot (FAT, `/boot`) p2=ROOT (`/`) p3=DATA. c3box était
|
||||||
|
OpenWrt ; eMMC écrasé (install RAM-only, pas de risque sur l'OS tournant avant le `dd`).
|
||||||
|
- **Rig netboot temporaire gk2 encore actif** : `lan1=192.168.77.1/24`, dnsmasq test (DHCP) sur
|
||||||
|
`lan1`, `nft iif lan1 accept`, nginx boot-vhost extra listen `192.168.77.1:8099`.
|
||||||
|
|
||||||
## 2026-06-24 (cont.) — R4 analyst mode: MITM-everything + media reverse-catcher + clone (#736)
|
## 2026-06-24 (cont.) — R4 analyst mode: MITM-everything + media reverse-catcher + clone (#736)
|
||||||
|
|
||||||
New "R4" doctrine — visibility over performance. Delivered + live on gk2:
|
New "R4" doctrine — visibility over performance. Delivered + live on gk2:
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,53 @@
|
||||||
# TODO — SecuBox-DEB Backlog
|
# TODO — SecuBox-DEB Backlog
|
||||||
*Mis à jour : 2026-06-22*
|
*Mis à jour : 2026-06-27*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ⚪ T5 — Images / OS variants / Hardware (ajouts 2026-06-27)
|
||||||
|
|
||||||
|
### ⬜ MOCHAbin — bootloader propre (adresses réservées + extlinux)
|
||||||
|
|
||||||
|
> Workaround actif : `/boot/boot.scr` compilé forçant le kernel à `0x0a000000`. Fix durable requis.
|
||||||
|
|
||||||
|
- [ ] **Option A — Corriger l'image** : patcher `extlinux.conf` généré par le CI pour utiliser
|
||||||
|
`0x0a000000` (kernel) et `0x10000000` (initrd) au lieu de `0x02080000` (adresse réservée
|
||||||
|
factory U-Boot 2020.10 → reset immédiat). Boot.scr deviendrait redondant.
|
||||||
|
- [ ] **Option B — Enhanced Tow-Boot (#748)** : bloqué par le ciseau U-Boot (voir ci-dessous) ;
|
||||||
|
déverrouille wget/HTTP natif dans U-Boot, supprime le besoin de TFTP pour les futures installs.
|
||||||
|
- [ ] **Valider** que le fix d'adresse tient sur les deux MOCHAbin (gk2 + c3box).
|
||||||
|
|
||||||
|
### ⬜ #748 — wget dans U-Boot pour MOCHAbin (bloquant documenté)
|
||||||
|
|
||||||
|
> Bloquant dur (ciseau) confirmé 2026-06-27. Branche
|
||||||
|
> `feature/748-enhanced-tow-boot-http-netboot-serial-fl` : spec + plan + Kconfig +
|
||||||
|
> `build-uboot-overlay.sh --tow-boot` + CI `.github/workflows/build-tow-boot.yml` en place.
|
||||||
|
> Problème : board mochabin UNIQUEMENT dans fork Tow-Boot U-Boot 2022.07 (pas de `wget`) ;
|
||||||
|
> `wget`/TCP UNIQUEMENT dans stock U-Boot ≥2023.07 (pas de board mochabin/DTS).
|
||||||
|
|
||||||
|
- [ ] **Voie 1** : backporter le stack TCP + `wget` de U-Boot ≥2023.07 dans le fork Tow-Boot
|
||||||
|
2022.07 (mochabin board natif). Diff TCP = `net/wget.c` + dépendances `CONFIG_NET_WGET`.
|
||||||
|
- [ ] **Voie 2** : porter le board mochabin (DTS Armada 7040 + PHY + eMMC) vers U-Boot mainline
|
||||||
|
≥2023.07 (sans Tow-Boot). Plus long mais durable.
|
||||||
|
- [ ] Choisir une voie, débloquer #748.
|
||||||
|
|
||||||
|
### ⬜ Packager le flow netboot + install signé (rig temporaire → procédure reproductible)
|
||||||
|
|
||||||
|
> Actuellement rig manuel sur gk2 : `lan1=192.168.77.1/24`, dnsmasq DHCP, nft, nginx `:8099`.
|
||||||
|
|
||||||
|
- [ ] Scripter la publication de l'image signée dans le root HTTP netboot (wget + sha256 + sig).
|
||||||
|
- [ ] Documenter / packager la config dnsmasq + nft + nginx pour un segment `lan1` dédié.
|
||||||
|
- [ ] Intégrer dans `scripts/deploy-netboot.sh` ou équivalent.
|
||||||
|
|
||||||
|
### ⬜ Teardown rig netboot temporaire gk2
|
||||||
|
|
||||||
|
> Le rig (lan1 bridge, dnsmasq, nft iif lan1 accept, nginx extra listen) reste actif jusqu'à
|
||||||
|
> ce que c3box soit autonome en prod.
|
||||||
|
|
||||||
|
- [ ] Retirer la règle nft `iif lan1 accept` (risque : tout le segment lan1 est accepté sans filtrage).
|
||||||
|
- [ ] Désactiver / retirer dnsmasq test sur lan1.
|
||||||
|
- [ ] Retirer le extra listen `192.168.77.1:8099` du vhost nginx netboot (ou couper le vhost si
|
||||||
|
plus nécessaire).
|
||||||
|
- [ ] Vérifier que c3box auto-boot sans rig (boot.scr en place → OK).
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -1,5 +1,34 @@
|
||||||
# WIP — Work In Progress
|
# WIP — Work In Progress
|
||||||
*Mis à jour : 2026-06-22*
|
*Mis à jour : 2026-06-27*
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ 2026-06-27 : c3box → SecuBox Debian — première install réussie · netboot prouvé (#748 #737)
|
||||||
|
|
||||||
|
### ✅ Fait (session 2026-06-27)
|
||||||
|
|
||||||
|
- **Netboot gk2→c3box prouvé** — factory U-Boot 2020.10 → TFTP → rescue shell installeur
|
||||||
|
(kernel 6.12.85 #5secubox). Détour cabling résolu (impasse LAB, pas logiciel).
|
||||||
|
- **Première install SecuBox Debian sur un MOCHAbin physique (c3box)** — image CI artefact
|
||||||
|
`secubox-mochabin-bookworm` (run 27426515472, 8 Gio), SHA256 + signature vérifiés,
|
||||||
|
`gunzip|dd` en RAM → eMMC. c3box boot Debian v1.9.0 avec stack complète.
|
||||||
|
- **boot.scr workaround déployé** — extlinux.conf charge le kernel à `0x02080000` (réservé
|
||||||
|
factory U-Boot → reset). Construit `/boot/boot.scr` (kernel@`0x0a000000`) ; auto-boot
|
||||||
|
Debian sans intervention vérifié après reboot.
|
||||||
|
- **#748 bloquant documenté** — ciseau U-Boot : mochabin board UNIQUEMENT dans fork Tow-Boot
|
||||||
|
2022.07 (pas de `wget`) ↔ `wget` UNIQUEMENT dans stock ≥2023.07 (pas de board mochabin).
|
||||||
|
Branche `feature/748-enhanced-tow-boot-http-netboot-serial-fl` parkée (spec+CI+Kconfig en
|
||||||
|
place, dépend du backport wget OU port board mainline).
|
||||||
|
|
||||||
|
### ⬜ Rig netboot temporaire gk2 à démonter (quand c3box autonome)
|
||||||
|
|
||||||
|
- `lan1=192.168.77.1/24` avec dnsmasq DHCP + `nft iif lan1 accept` + nginx `:8099` encore actifs.
|
||||||
|
- À retirer une fois c3box en prod (voir TODO T5 — teardown rig).
|
||||||
|
|
||||||
|
### ⬜ Bootloader propre à faire (#748 ou alternative)
|
||||||
|
|
||||||
|
- boot.scr = workaround ; fix durable = enhanced Tow-Boot (#748, bloqué ciseau) OU corriger
|
||||||
|
les adresses de boot dans l'image (extlinux.conf → `0x0a000000`). Voir TODO T5.
|
||||||
|
|
||||||
---
|
---
|
||||||
|
|
||||||
|
|
|
||||||
|
|
@ -6,16 +6,21 @@ network:
|
||||||
renderer: networkd
|
renderer: networkd
|
||||||
|
|
||||||
ethernets:
|
ethernets:
|
||||||
# WAN — connecté à l'opérateur
|
# WAN candidate (SFP+, eth0) — connecté à l'opérateur via fibre/module SFP.
|
||||||
eth0:
|
eth0:
|
||||||
dhcp4: true
|
dhcp4: true
|
||||||
dhcp6: false
|
dhcp6: false
|
||||||
optional: true
|
optional: true
|
||||||
|
|
||||||
# LAN — ports GbE (DSA ou directs selon la config switch)
|
# LAN — port GbE switch (DSA 88E6341)
|
||||||
eth1:
|
eth1:
|
||||||
optional: true
|
optional: true
|
||||||
|
# WAN candidate (RJ45 cuivre, eth2 = mvpp2-2). Sur MOCHAbin le seul RJ45
|
||||||
|
# direct ; sert d'uplink quand l'opérateur arrive en cuivre. Le port WAN
|
||||||
|
# câblé (eth0 SFP+ OU eth2 cuivre) obtient le bail DHCP ; l'autre reste idle.
|
||||||
eth2:
|
eth2:
|
||||||
|
dhcp4: true
|
||||||
|
dhcp6: false
|
||||||
optional: true
|
optional: true
|
||||||
eth3:
|
eth3:
|
||||||
optional: true
|
optional: true
|
||||||
|
|
@ -31,7 +36,7 @@ network:
|
||||||
bridges:
|
bridges:
|
||||||
# Bridge LAN
|
# Bridge LAN
|
||||||
br-lan:
|
br-lan:
|
||||||
interfaces: [eth1, eth2, eth3, eth4]
|
interfaces: [eth1, eth3, eth4]
|
||||||
addresses: [192.168.1.1/24]
|
addresses: [192.168.1.1/24]
|
||||||
dhcp4: false
|
dhcp4: false
|
||||||
parameters:
|
parameters:
|
||||||
|
|
|
||||||
1638
docs/superpowers/plans/2026-06-27-nft-based-network-stats.md
Normal file
1638
docs/superpowers/plans/2026-06-27-nft-based-network-stats.md
Normal file
File diff suppressed because it is too large
Load Diff
|
|
@ -0,0 +1,211 @@
|
||||||
|
# Design — nft-based network stats into the dashboard (#758)
|
||||||
|
|
||||||
|
**Date:** 2026-06-27
|
||||||
|
**Issue:** [#758](https://github.com/CyberMind-FR/secubox-deb/issues/758)
|
||||||
|
**Status:** Approved design, pending implementation plan
|
||||||
|
**Branch:** `feature/758-nft-based-network-stats-log-counter-drop`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 1. Problem
|
||||||
|
|
||||||
|
The `#ads` dashboard breakdown (issue #755) exposes a **"Drops réseau"** KPI fed by
|
||||||
|
`network_drops`, which currently always reads **0**. Its source is an anonymous
|
||||||
|
inline counter in the `inet secubox_blacklist` `enforce` chain, read via
|
||||||
|
`nft -j list table inet secubox_blacklist` — and it only covers blacklist /
|
||||||
|
quarantine drops, which sit at 0 until the blacklist-sync daemon populates the
|
||||||
|
sets. There is no view of general firewall drops, attack/blocked-traffic
|
||||||
|
categories, or in/out interface throughput.
|
||||||
|
|
||||||
|
This work feeds **real network-layer stats from nftables** (plus interface
|
||||||
|
counters) into the dashboard: categorized drops, attack/blocked traffic,
|
||||||
|
ad-blocks (cross-referenced), and in/out throughput — with a 24h time-series and
|
||||||
|
charts.
|
||||||
|
|
||||||
|
## 2. Goals / Non-goals
|
||||||
|
|
||||||
|
### Goals
|
||||||
|
- Real, categorized **drops** and **attacks** sourced from nft **named counters**.
|
||||||
|
- **In/out** throughput per interface.
|
||||||
|
- **Time-series** retention (SQLite) powering 24h charts.
|
||||||
|
- A new **"Réseau"** tab in the toolbox dashboard.
|
||||||
|
- The existing `#ads` "Drops réseau" KPI becomes real.
|
||||||
|
|
||||||
|
### Non-goals (v1)
|
||||||
|
- No per-flow / per-connection accounting (that is nDPId / metrics territory).
|
||||||
|
- No injection of named counters into the externally-managed `inet crowdsec`
|
||||||
|
table (read-only best-effort instead).
|
||||||
|
- No new heavy charting dependency if the dashboard already ships one.
|
||||||
|
|
||||||
|
## 3. Decisions (from brainstorming)
|
||||||
|
|
||||||
|
| Decision | Choice |
|
||||||
|
|---|---|
|
||||||
|
| v1 scope | Full dashboard with time-series |
|
||||||
|
| Ownership | **Hub collects** (reuses its root nft poller + sudoers), **toolbox renders** |
|
||||||
|
| Instrumentation | **Hybrid** — named nft counters (drops/attacks) + `/proc/net/dev` (in/out) + app-layer `ad_block_stats` (ads) |
|
||||||
|
| Store | **SQLite** time-series (`/var/lib/secubox/hub/netstats.db`) |
|
||||||
|
| UI surface | New **"Réseau"** tab in the toolbox dashboard |
|
||||||
|
|
||||||
|
## 4. Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
[kernel nft named counters] ──┐
|
||||||
|
[/proc/net/dev interfaces] ──┼─► secubox-netstats-collect (root oneshot, 30s timer, in secubox-hub)
|
||||||
|
[inet crowdsec counters (RO)]─┘ │
|
||||||
|
├─► SQLite /var/lib/secubox/hub/netstats.db (cumulative samples, time-series)
|
||||||
|
└─► snapshot /var/lib/secubox/hub/netstats.json (latest + instantaneous rates, 0644)
|
||||||
|
│
|
||||||
|
hub api (user secubox, under aggregator) reads DB/json ──► GET /api/v1/hub/netstats/{summary,series}
|
||||||
|
│
|
||||||
|
toolbox "Réseau" tab JS ──► fetch /api/v1/hub/netstats/* (charts + tiles)
|
||||||
|
toolbox /admin/ad-stats ──► reads netstats.json → real network_drops (#ads KPI)
|
||||||
|
```
|
||||||
|
|
||||||
|
**Root/user split.** The collector runs as **root** (oneshot + timer, mirrors the
|
||||||
|
existing `secubox-nft-cache.timer` pattern in hub) because `nft -j list counters`
|
||||||
|
needs `CAP_NET_ADMIN`. The hub API and toolbox API run as user `secubox` inside
|
||||||
|
the aggregator process and only **read** the DB / snapshot file. A new dedicated
|
||||||
|
collector is used rather than overloading `secubox-nft-cache`, because this one
|
||||||
|
also does SQLite inserts, `/proc` reads, and crowdsec reads.
|
||||||
|
|
||||||
|
## 5. nft instrumentation — named counters
|
||||||
|
|
||||||
|
Add **declared named counter objects** to the owning tables and reference them
|
||||||
|
with `counter name "…"` on the existing drop rules. Naming convention:
|
||||||
|
`sbx_<category>[_<af>]`.
|
||||||
|
|
||||||
|
| Category | Counter(s) | File (owner package) | Change |
|
||||||
|
|---|---|---|---|
|
||||||
|
| C2 blacklist drops | `sbx_drop_blacklist_v4`, `sbx_drop_blacklist_v6` | `secubox-blacklist.nft` (secubox-toolbox) | add |
|
||||||
|
| Quarantine drops | `sbx_drop_quarantine_v4`, `sbx_drop_quarantine_v6` | `secubox-blacklist.nft` (secubox-toolbox) | add |
|
||||||
|
| DoH/DoT detect | `sbx_doh_detect_v4`, `sbx_doh_detect_v6` | `secubox-blacklist.nft` (secubox-toolbox, `doh_watch`) | add |
|
||||||
|
| WAF rate-limit (scanners/bots) | `sbx_drop_wafrl` | `secubox-waf-ratelimit.nft` (secubox-mitmproxy) | add |
|
||||||
|
| Unsolicited inbound (policy drop) | `sbx_drop_input_policy` | **new** `zz-secubox-netstats-tap.nft` (secubox-hub) | add |
|
||||||
|
| CrowdSec bouncer drops | existing counters in `inet crowdsec` | — | **read-only, best-effort** |
|
||||||
|
|
||||||
|
Notes:
|
||||||
|
- Named counter objects **must be declared in the same table** as the rules that
|
||||||
|
reference them (no cross-table counter refs). Each owning `.nft` declares its
|
||||||
|
own counters at table scope, then references them inline:
|
||||||
|
`ip daddr @blacklist_v4 limit rate 20/second log prefix "SBX-BL-DROP " counter name "sbx_drop_blacklist_v4" drop`.
|
||||||
|
- The **unsolicited-inbound** tap is a hub-owned drop-in that declares
|
||||||
|
`counter inet filter sbx_drop_input_policy` and appends a tail
|
||||||
|
`add rule inet filter input counter name "sbx_drop_input_policy"` — a bare
|
||||||
|
counter at the end of the default-drop `input` chain counts exactly what the
|
||||||
|
policy would drop. It **must load after** all accept rules, hence the `zz-`
|
||||||
|
prefix (same ordering precedent as `zz-secubox-toolbox-wg-fanout.nft`).
|
||||||
|
- `inet crowdsec` is created/regenerated by `crowdsec-firewall-bouncer`; we must
|
||||||
|
**not** rewrite it. The collector reads its counters if present and treats them
|
||||||
|
as best-effort (0 when absent).
|
||||||
|
- **Counters reset to 0 on every `nft -f` reload.** The collector and the series
|
||||||
|
math are reset-aware (§8).
|
||||||
|
|
||||||
|
## 6. Collector + store
|
||||||
|
|
||||||
|
**Script:** `/usr/lib/secubox/hub/netstats-collect.py` (Python, shipped by secubox-hub).
|
||||||
|
**Units:** `secubox-netstats.service` (`Type=oneshot`, root) + `secubox-netstats.timer`
|
||||||
|
(`OnBootSec=15s`, `OnUnitActiveSec=30s`, `AccuracySec=5s`).
|
||||||
|
|
||||||
|
Each tick:
|
||||||
|
1. `nft -j list counters` → map counter-name → cumulative `{packets, bytes}` per category.
|
||||||
|
2. `/proc/net/dev` → per-interface `rx_bytes, rx_packets, tx_bytes, tx_packets`.
|
||||||
|
Interfaces from hub config (default: all non-`lo`).
|
||||||
|
3. Best-effort read of `inet crowdsec` counters.
|
||||||
|
4. INSERT samples into SQLite (cumulative values + `ts`).
|
||||||
|
5. Write `netstats.json` snapshot: latest cumulative + **instantaneous rates**
|
||||||
|
computed vs the previous sample (reset-aware) + `updated` ts.
|
||||||
|
6. Retention: `DELETE FROM … WHERE ts < now - 7d` (keep a week; charts default 24h).
|
||||||
|
|
||||||
|
**SQLite schema** (long-and-narrow, easy to query/downsample; avoids a wide
|
||||||
|
dynamic-interface table):
|
||||||
|
|
||||||
|
```sql
|
||||||
|
CREATE TABLE IF NOT EXISTS counter_samples (
|
||||||
|
ts INTEGER NOT NULL,
|
||||||
|
name TEXT NOT NULL, -- e.g. sbx_drop_blacklist_v4
|
||||||
|
packets INTEGER NOT NULL,
|
||||||
|
bytes INTEGER NOT NULL,
|
||||||
|
PRIMARY KEY (ts, name)
|
||||||
|
);
|
||||||
|
CREATE TABLE IF NOT EXISTS iface_samples (
|
||||||
|
ts INTEGER NOT NULL,
|
||||||
|
iface TEXT NOT NULL, -- e.g. eth0, br-lan
|
||||||
|
rx_bytes INTEGER NOT NULL, rx_packets INTEGER NOT NULL,
|
||||||
|
tx_bytes INTEGER NOT NULL, tx_packets INTEGER NOT NULL,
|
||||||
|
PRIMARY KEY (ts, iface)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_counter_ts ON counter_samples(ts);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_iface_ts ON iface_samples(ts);
|
||||||
|
```
|
||||||
|
|
||||||
|
File perms: DB + json written by root, dir `0755`, files `0644` (user `secubox`
|
||||||
|
reads). DB opened read-only by the API.
|
||||||
|
|
||||||
|
## 7. API (secubox-hub, read-only)
|
||||||
|
|
||||||
|
- `GET /api/v1/hub/netstats/summary`
|
||||||
|
Latest snapshot: per-category cumulative + current rate (pkt/s, bit/s),
|
||||||
|
per-interface in/out throughput (bit/s), `updated` ts, `stale` flag.
|
||||||
|
- `GET /api/v1/hub/netstats/series?window=24h&step=5m&metric=<cat|iface>`
|
||||||
|
Downsampled buckets; deltas/rates computed server-side from cumulative
|
||||||
|
samples, **reset-aware**.
|
||||||
|
|
||||||
|
`network_drops` for the `#ads` KPI = sum of `sbx_drop_*` categories (+ crowdsec
|
||||||
|
best-effort), read by toolbox from `netstats.json`.
|
||||||
|
|
||||||
|
## 8. Correctness / error handling
|
||||||
|
|
||||||
|
- **Counter resets:** if a cumulative value is **lower** than the previous
|
||||||
|
sample, an `nft -f` reload occurred → that interval's delta = the current value
|
||||||
|
(never negative). Same logic for `/proc` interface counters (rare 64-bit wrap /
|
||||||
|
iface re-create).
|
||||||
|
- Each source is wrapped independently: missing counter/table/iface ⇒ 0, logged,
|
||||||
|
others proceed.
|
||||||
|
- Collector down ⇒ API serves last snapshot with `stale: true` + `updated` age;
|
||||||
|
frontend renders a staleness badge.
|
||||||
|
- SQLite write failure ⇒ snapshot still written; error logged to journald.
|
||||||
|
- Read/write privilege split enforced by file ownership (root writes, secubox
|
||||||
|
reads).
|
||||||
|
|
||||||
|
## 9. Frontend — new "Réseau" tab (`packages/secubox-toolbox/www/toolbox/index.html`)
|
||||||
|
|
||||||
|
- **Throughput chart:** in/out bit/s per interface, 24h (from `/series`).
|
||||||
|
- **Drops/attacks trend:** stacked-by-category, 24h (from `/series`).
|
||||||
|
- **Breakdown tiles:** total drops, by category, top sources (from `/summary`).
|
||||||
|
- **#ads KPI:** "Drops réseau" repointed to the real `network_drops`.
|
||||||
|
- **Charts:** reuse the dashboard's existing charting lib if present; otherwise a
|
||||||
|
lightweight inline SVG sparkline — **no new heavy dependency**. (Confirmed
|
||||||
|
during implementation.)
|
||||||
|
- Staleness badge when `stale: true`.
|
||||||
|
|
||||||
|
## 10. Packages touched + deployment
|
||||||
|
|
||||||
|
| Package | Change | Bump |
|
||||||
|
|---|---|---|
|
||||||
|
| **secubox-hub** | collector script, `.service`+`.timer`, `zz-secubox-netstats-tap.nft`, SQLite schema, API endpoints (sudoers already grants `nft -j list *`) | yes |
|
||||||
|
| **secubox-toolbox** | `secubox-blacklist.nft` named counters, new "Réseau" tab, repoint `network_drops` | yes |
|
||||||
|
| **secubox-mitmproxy** | `secubox-waf-ratelimit.nft` named counters | yes |
|
||||||
|
| secubox-crowdsec | none (read-only) | no |
|
||||||
|
|
||||||
|
- Each package's `postinst` reloads its own `.nft` (existing idempotent
|
||||||
|
delete+recreate pattern). `nft -c -f` syntax check added in CI for changed/new
|
||||||
|
`.nft` files.
|
||||||
|
- Postinst must preserve runtime state (try-restart for the timer; redeploy
|
||||||
|
operator drop-ins) per existing project guidance.
|
||||||
|
|
||||||
|
## 11. Testing
|
||||||
|
|
||||||
|
- **Unit:** counter-name→category map; `/proc/net/dev` parser; reset-aware
|
||||||
|
delta/rate math; SQLite insert + downsample query; snapshot staleness logic.
|
||||||
|
- **Integration:** one collector run yields rows + valid `/summary` and `/series`
|
||||||
|
shapes; graceful when DB / counters / crowdsec table are absent.
|
||||||
|
- **nft:** `nft -c -f` syntax check on each modified/new `.nft`.
|
||||||
|
|
||||||
|
## 12. Open items (resolve during implementation)
|
||||||
|
|
||||||
|
- Confirm the dashboard's existing charting approach before choosing chart impl.
|
||||||
|
- Confirm whether to expose a `forward` policy-drop tap in addition to `input`
|
||||||
|
(v1: input only).
|
||||||
|
- Interface selection: config-driven allow-list vs all-non-`lo` (v1: all-non-`lo`,
|
||||||
|
config override available).
|
||||||
|
|
@ -14,6 +14,7 @@ import asyncio
|
||||||
import os
|
import os
|
||||||
import time
|
import time
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
import netstats # #758 — shared collector/reader module
|
||||||
|
|
||||||
app = FastAPI(title="secubox-hub", version="1.7.0", root_path="/api/v1/hub")
|
app = FastAPI(title="secubox-hub", version="1.7.0", root_path="/api/v1/hub")
|
||||||
|
|
||||||
|
|
@ -756,6 +757,29 @@ async def network_summary(user=Depends(require_jwt)):
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/netstats/summary")
|
||||||
|
async def netstats_summary(user=Depends(require_jwt)) -> dict:
|
||||||
|
"""#758 — latest network-stats snapshot (categories, interfaces, drops).
|
||||||
|
Read-only; served from the collector's JSON snapshot (cheap)."""
|
||||||
|
return netstats.read_snapshot()
|
||||||
|
|
||||||
|
|
||||||
|
@router.get("/netstats/series")
|
||||||
|
async def netstats_series(window: int = 86400, step: int = 300, user=Depends(require_jwt)) -> dict:
|
||||||
|
"""#758 — reset-aware drops/throughput time-series for the dashboard charts.
|
||||||
|
Read-only over the collector's SQLite DB."""
|
||||||
|
import sqlite3 as _sql
|
||||||
|
w = max(300, min(window, 7 * 86400))
|
||||||
|
s = max(30, min(step, 3600))
|
||||||
|
if not netstats.DB_PATH.exists():
|
||||||
|
return {"window_s": w, "step_s": s, "drops": {}, "in_bps": {}, "out_bps": {}}
|
||||||
|
conn = _sql.connect(f"file:{netstats.DB_PATH}?mode=ro", uri=True)
|
||||||
|
try:
|
||||||
|
return netstats.query_series(conn, window_s=w, step_s=s)
|
||||||
|
finally:
|
||||||
|
conn.close()
|
||||||
|
|
||||||
|
|
||||||
@router.get("/quick_actions")
|
@router.get("/quick_actions")
|
||||||
async def quick_actions(user=Depends(require_jwt)):
|
async def quick_actions(user=Depends(require_jwt)):
|
||||||
"""Actions rapides disponibles."""
|
"""Actions rapides disponibles."""
|
||||||
|
|
|
||||||
336
packages/secubox-hub/api/netstats.py
Normal file
336
packages/secubox-hub/api/netstats.py
Normal file
|
|
@ -0,0 +1,336 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
"""SecuBox-Deb :: secubox-hub network-stats (#758).
|
||||||
|
|
||||||
|
Shared by the root collector (write path: collect_once/main) and the FastAPI
|
||||||
|
app (read path: read_snapshot/query_series). Pure functions are unit-tested;
|
||||||
|
the privileged collect path is integration-tested with monkeypatched sources.
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
import subprocess
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
DB_PATH = Path("/var/lib/secubox/hub/netstats.db")
|
||||||
|
SNAP_PATH = Path("/var/lib/secubox/hub/netstats.json")
|
||||||
|
DATA_DIR = DB_PATH.parent
|
||||||
|
STALE_AFTER_S = 120 # snapshot older than this is flagged stale
|
||||||
|
|
||||||
|
# counter-name → category. Named counters live in the owning packages' tables.
|
||||||
|
CATEGORY_MAP = {
|
||||||
|
"sbx_drop_blacklist_v4": "blacklist", "sbx_drop_blacklist_v6": "blacklist",
|
||||||
|
"sbx_drop_quarantine_v4": "quarantine", "sbx_drop_quarantine_v6": "quarantine",
|
||||||
|
"sbx_doh_detect_v4": "doh", "sbx_doh_detect_v6": "doh",
|
||||||
|
"sbx_drop_wafrl": "waf_ratelimit",
|
||||||
|
"sbx_drop_input_policy": "input_policy",
|
||||||
|
"sbx_drop_crowdsec": "crowdsec",
|
||||||
|
}
|
||||||
|
# Categories that count toward "network_drops" (doh is detect-only, excluded).
|
||||||
|
DROP_CATEGORIES = {"blacklist", "quarantine", "waf_ratelimit", "input_policy", "crowdsec"}
|
||||||
|
|
||||||
|
|
||||||
|
def category_for(name: str) -> str | None:
|
||||||
|
return CATEGORY_MAP.get(name)
|
||||||
|
|
||||||
|
|
||||||
|
def parse_proc_net_dev(text: str) -> dict[str, dict]:
|
||||||
|
"""Parse /proc/net/dev → {iface: {rx_bytes,rx_packets,tx_bytes,tx_packets}}.
|
||||||
|
Skips the two header lines and the loopback interface.
|
||||||
|
"""
|
||||||
|
out: dict[str, dict] = {}
|
||||||
|
for line in text.splitlines():
|
||||||
|
if ":" not in line:
|
||||||
|
continue
|
||||||
|
name, _, rest = line.partition(":")
|
||||||
|
name = name.strip()
|
||||||
|
if name == "lo" or not name:
|
||||||
|
continue
|
||||||
|
f = rest.split()
|
||||||
|
if len(f) < 16:
|
||||||
|
continue
|
||||||
|
out[name] = {
|
||||||
|
"rx_bytes": int(f[0]), "rx_packets": int(f[1]),
|
||||||
|
"tx_bytes": int(f[8]), "tx_packets": int(f[9]),
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def parse_nft_counters_json(data: dict) -> dict[str, dict]:
|
||||||
|
"""Parse `nft -j list counters` (or list table) → {name: {packets,bytes}}."""
|
||||||
|
out: dict[str, dict] = {}
|
||||||
|
for item in data.get("nftables", []):
|
||||||
|
c = item.get("counter")
|
||||||
|
if isinstance(c, dict) and "name" in c:
|
||||||
|
out[c["name"]] = {
|
||||||
|
"packets": int(c.get("packets", 0) or 0),
|
||||||
|
"bytes": int(c.get("bytes", 0) or 0),
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def reset_aware_delta(prev: int, cur: int) -> int:
|
||||||
|
"""Monotonic-counter delta that tolerates resets (nft reload → cur < prev)."""
|
||||||
|
if cur < prev:
|
||||||
|
return cur
|
||||||
|
return cur - prev
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# SQLite store (Task 5)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def init_db(conn: sqlite3.Connection) -> None:
|
||||||
|
conn.executescript(
|
||||||
|
"""
|
||||||
|
CREATE TABLE IF NOT EXISTS counter_samples (
|
||||||
|
ts INTEGER NOT NULL, name TEXT NOT NULL,
|
||||||
|
packets INTEGER NOT NULL, bytes INTEGER NOT NULL,
|
||||||
|
PRIMARY KEY (ts, name)
|
||||||
|
);
|
||||||
|
CREATE TABLE IF NOT EXISTS iface_samples (
|
||||||
|
ts INTEGER NOT NULL, iface TEXT NOT NULL,
|
||||||
|
rx_bytes INTEGER NOT NULL, rx_packets INTEGER NOT NULL,
|
||||||
|
tx_bytes INTEGER NOT NULL, tx_packets INTEGER NOT NULL,
|
||||||
|
PRIMARY KEY (ts, iface)
|
||||||
|
);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_counter_ts ON counter_samples(ts);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_iface_ts ON iface_samples(ts);
|
||||||
|
"""
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def insert_sample(conn: sqlite3.Connection, ts: int, counters: dict, ifaces: dict) -> None:
|
||||||
|
for name, v in counters.items():
|
||||||
|
conn.execute(
|
||||||
|
"INSERT OR REPLACE INTO counter_samples(ts,name,packets,bytes) VALUES(?,?,?,?)",
|
||||||
|
(ts, name, int(v.get("packets", 0)), int(v.get("bytes", 0))),
|
||||||
|
)
|
||||||
|
for iface, v in ifaces.items():
|
||||||
|
conn.execute(
|
||||||
|
"INSERT OR REPLACE INTO iface_samples(ts,iface,rx_bytes,rx_packets,tx_bytes,tx_packets) "
|
||||||
|
"VALUES(?,?,?,?,?,?)",
|
||||||
|
(ts, iface, int(v["rx_bytes"]), int(v["rx_packets"]),
|
||||||
|
int(v["tx_bytes"]), int(v["tx_packets"])),
|
||||||
|
)
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
def _bucket(ts: int, step_s: int) -> int:
|
||||||
|
return ts - (ts % step_s)
|
||||||
|
|
||||||
|
|
||||||
|
def query_series(conn: sqlite3.Connection, window_s: int, step_s: int) -> dict:
|
||||||
|
"""Reset-aware deltas/rates bucketed to step_s over the last window_s.
|
||||||
|
Rates are attributed to the bucket of the later sample in each pair.
|
||||||
|
"""
|
||||||
|
now_row = conn.execute("SELECT MAX(ts) FROM counter_samples").fetchone()
|
||||||
|
max_ts_c = now_row[0] if now_row and now_row[0] is not None else None
|
||||||
|
now_row2 = conn.execute("SELECT MAX(ts) FROM iface_samples").fetchone()
|
||||||
|
max_ts_i = now_row2[0] if now_row2 and now_row2[0] is not None else None
|
||||||
|
max_ts = max(t for t in (max_ts_c, max_ts_i) if t is not None) if any(t is not None for t in (max_ts_c, max_ts_i)) else 0
|
||||||
|
floor = max_ts - window_s
|
||||||
|
|
||||||
|
drops: dict[str, dict[int, int]] = {}
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT ts,name,packets FROM counter_samples WHERE ts>=? ORDER BY name,ts",
|
||||||
|
(floor - step_s,),
|
||||||
|
).fetchall()
|
||||||
|
prev: dict[str, tuple[int, int]] = {}
|
||||||
|
for ts, name, pk in rows:
|
||||||
|
cat = category_for(name)
|
||||||
|
if cat is None:
|
||||||
|
continue
|
||||||
|
if name in prev:
|
||||||
|
d = reset_aware_delta(prev[name][1], pk)
|
||||||
|
b = _bucket(ts, step_s)
|
||||||
|
if b >= _bucket(floor, step_s):
|
||||||
|
drops.setdefault(cat, {}).setdefault(b, 0)
|
||||||
|
drops[cat][b] += d
|
||||||
|
prev[name] = (ts, pk)
|
||||||
|
|
||||||
|
in_bps: dict[str, dict[int, int]] = {}
|
||||||
|
out_bps: dict[str, dict[int, int]] = {}
|
||||||
|
irows = conn.execute(
|
||||||
|
"SELECT ts,iface,rx_bytes,tx_bytes FROM iface_samples WHERE ts>=? ORDER BY iface,ts",
|
||||||
|
(floor - step_s,),
|
||||||
|
).fetchall()
|
||||||
|
iprev: dict[str, tuple[int, int, int]] = {}
|
||||||
|
for ts, iface, rx, tx in irows:
|
||||||
|
if iface in iprev:
|
||||||
|
pts, prx, ptx = iprev[iface]
|
||||||
|
dt = ts - pts
|
||||||
|
if dt > 0:
|
||||||
|
b = _bucket(ts, step_s)
|
||||||
|
if b >= _bucket(floor, step_s):
|
||||||
|
in_bps.setdefault(iface, {})[b] = reset_aware_delta(prx, rx) * 8 // dt
|
||||||
|
out_bps.setdefault(iface, {})[b] = reset_aware_delta(ptx, tx) * 8 // dt
|
||||||
|
iprev[iface] = (ts, rx, tx)
|
||||||
|
|
||||||
|
def _flatten(d: dict[str, dict[int, int]]) -> dict[str, list]:
|
||||||
|
return {k: [[b, v[b]] for b in sorted(v)] for k, v in d.items()}
|
||||||
|
|
||||||
|
return {
|
||||||
|
"window_s": window_s, "step_s": step_s,
|
||||||
|
"drops": _flatten(drops), "in_bps": _flatten(in_bps), "out_bps": _flatten(out_bps),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def prune(conn: sqlite3.Connection, keep_s: int) -> None:
|
||||||
|
for tbl in ("counter_samples", "iface_samples"):
|
||||||
|
row = conn.execute(f"SELECT MAX(ts) FROM {tbl}").fetchone()
|
||||||
|
if row and row[0] is not None:
|
||||||
|
conn.execute(f"DELETE FROM {tbl} WHERE ts < ?", (row[0] - keep_s,))
|
||||||
|
conn.commit()
|
||||||
|
|
||||||
|
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
# Snapshot builder + privileged collector (Task 6)
|
||||||
|
# ---------------------------------------------------------------------------
|
||||||
|
|
||||||
|
def build_snapshot(conn: sqlite3.Connection, now: int) -> dict:
|
||||||
|
"""Latest cumulative per category + instantaneous rates vs the previous
|
||||||
|
sample. network_drops = sum of DROP_CATEGORIES packet counts (doh excluded).
|
||||||
|
"""
|
||||||
|
cats: dict[str, dict] = {}
|
||||||
|
# latest cumulative per counter-name → fold into categories
|
||||||
|
rows = conn.execute(
|
||||||
|
"SELECT name, packets, bytes, ts FROM counter_samples "
|
||||||
|
"WHERE ts=(SELECT MAX(ts) FROM counter_samples)"
|
||||||
|
).fetchall()
|
||||||
|
for name, pk, by, _ts in rows:
|
||||||
|
cat = category_for(name)
|
||||||
|
if cat is None:
|
||||||
|
continue
|
||||||
|
c = cats.setdefault(cat, {"packets": 0, "bytes": 0})
|
||||||
|
c["packets"] += int(pk)
|
||||||
|
c["bytes"] += int(by)
|
||||||
|
ser = query_series(conn, window_s=120, step_s=30)
|
||||||
|
for cat, pts in ser["drops"].items():
|
||||||
|
if pts:
|
||||||
|
cats.setdefault(cat, {"packets": 0, "bytes": 0})["pps"] = pts[-1][1] / 30.0
|
||||||
|
|
||||||
|
ifaces: dict[str, dict] = {}
|
||||||
|
irow = conn.execute(
|
||||||
|
"SELECT iface, rx_bytes, tx_bytes FROM iface_samples "
|
||||||
|
"WHERE ts=(SELECT MAX(ts) FROM iface_samples)"
|
||||||
|
).fetchall()
|
||||||
|
for iface, rx, tx in irow:
|
||||||
|
ifaces[iface] = {"rx_bytes": int(rx), "tx_bytes": int(tx)}
|
||||||
|
for iface, pts in ser["in_bps"].items():
|
||||||
|
if pts:
|
||||||
|
ifaces.setdefault(iface, {})["rx_bps"] = pts[-1][1]
|
||||||
|
for iface, pts in ser["out_bps"].items():
|
||||||
|
if pts:
|
||||||
|
ifaces.setdefault(iface, {})["tx_bps"] = pts[-1][1]
|
||||||
|
|
||||||
|
network_drops = sum(
|
||||||
|
cats.get(cat, {}).get("packets", 0) for cat in DROP_CATEGORIES
|
||||||
|
)
|
||||||
|
return {
|
||||||
|
"updated": now,
|
||||||
|
"stale": False,
|
||||||
|
"categories": cats,
|
||||||
|
"interfaces": ifaces,
|
||||||
|
"network_drops": int(network_drops),
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def read_snapshot() -> dict:
|
||||||
|
"""Read the latest snapshot (API path). Flags stale by `updated` age."""
|
||||||
|
try:
|
||||||
|
d = json.loads(SNAP_PATH.read_text())
|
||||||
|
except Exception:
|
||||||
|
return {"updated": 0, "stale": True, "categories": {}, "interfaces": {}, "network_drops": 0}
|
||||||
|
age = max(0, int(time.time()) - int(d.get("updated", 0)))
|
||||||
|
d["stale"] = age > STALE_AFTER_S
|
||||||
|
return d
|
||||||
|
|
||||||
|
|
||||||
|
def _run_nft_json(args: list[str]) -> dict:
|
||||||
|
try:
|
||||||
|
r = subprocess.run(["/usr/sbin/nft", "-j", "list"] + args,
|
||||||
|
capture_output=True, text=True, timeout=5)
|
||||||
|
if r.returncode == 0:
|
||||||
|
return json.loads(r.stdout or "{}")
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _read_nft_counters() -> dict:
|
||||||
|
return parse_nft_counters_json(_run_nft_json(["counters"]))
|
||||||
|
|
||||||
|
|
||||||
|
def _read_crowdsec() -> dict:
|
||||||
|
"""Best-effort: sum the externally-managed inet crowdsec table counters
|
||||||
|
into a single synthetic counter mapped to the 'crowdsec' category.
|
||||||
|
"""
|
||||||
|
data = _run_nft_json(["table", "inet", "crowdsec"])
|
||||||
|
total = 0
|
||||||
|
for item in data.get("nftables", []):
|
||||||
|
c = item.get("counter")
|
||||||
|
if isinstance(c, dict):
|
||||||
|
total += int(c.get("packets", 0) or 0)
|
||||||
|
rule = item.get("rule")
|
||||||
|
if isinstance(rule, dict):
|
||||||
|
for ex in rule.get("expr", []):
|
||||||
|
cc = ex.get("counter")
|
||||||
|
if isinstance(cc, dict):
|
||||||
|
total += int(cc.get("packets", 0) or 0)
|
||||||
|
if total:
|
||||||
|
# synthetic name so category_for resolves to 'crowdsec'
|
||||||
|
return {"sbx_drop_crowdsec": {"packets": total, "bytes": 0}}
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _read_ifaces() -> dict:
|
||||||
|
try:
|
||||||
|
return parse_proc_net_dev(Path("/proc/net/dev").read_text())
|
||||||
|
except Exception:
|
||||||
|
return {}
|
||||||
|
|
||||||
|
|
||||||
|
def _open_db(conn: sqlite3.Connection | None):
|
||||||
|
if conn is not None:
|
||||||
|
return conn, False
|
||||||
|
DATA_DIR.mkdir(parents=True, exist_ok=True)
|
||||||
|
try:
|
||||||
|
DATA_DIR.chmod(0o755) # our subdir only — NEVER the /var/lib/secubox parent
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
c = sqlite3.connect(str(DB_PATH))
|
||||||
|
init_db(c)
|
||||||
|
return c, True
|
||||||
|
|
||||||
|
|
||||||
|
def collect_once(now: int | None = None, conn: sqlite3.Connection | None = None) -> dict:
|
||||||
|
if now is None:
|
||||||
|
now = int(time.time())
|
||||||
|
c, owns = _open_db(conn)
|
||||||
|
try:
|
||||||
|
counters = dict(_read_nft_counters())
|
||||||
|
counters.update(_read_crowdsec())
|
||||||
|
ifaces = _read_ifaces()
|
||||||
|
insert_sample(c, now, counters, ifaces)
|
||||||
|
prune(c, keep_s=7 * 86400)
|
||||||
|
snap = build_snapshot(c, now)
|
||||||
|
try:
|
||||||
|
SNAP_PATH.write_text(json.dumps(snap))
|
||||||
|
SNAP_PATH.chmod(0o644)
|
||||||
|
except Exception:
|
||||||
|
pass
|
||||||
|
return snap
|
||||||
|
finally:
|
||||||
|
if owns:
|
||||||
|
c.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> None:
|
||||||
|
collect_once()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
main()
|
||||||
|
|
@ -1,3 +1,12 @@
|
||||||
|
secubox-hub (1.5.0-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
|
* feat(#758): nft-based network-stats collector — root oneshot+timer samples
|
||||||
|
nft named counters + /proc/net/dev into /var/lib/secubox/hub/netstats.{db,json};
|
||||||
|
new read-only endpoints /api/v1/hub/netstats/{summary,series}; inet filter
|
||||||
|
input policy-drop tap drop-in (zz-secubox-netstats-tap.nft).
|
||||||
|
|
||||||
|
-- Gerald KERMA <devel@cybermind.fr> Sat, 27 Jun 2026 12:00:00 +0200
|
||||||
|
|
||||||
secubox-hub (1.4.7-1~bookworm1) bookworm; urgency=medium
|
secubox-hub (1.4.7-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
* #494: drop the ExecStartPre chown/chmod of the shared /run/secubox parent.
|
* #494: drop the ExecStartPre chown/chmod of the shared /run/secubox parent.
|
||||||
|
|
|
||||||
|
|
@ -12,6 +12,19 @@ case "$1" in
|
||||||
systemctl start secubox-hub.service || true
|
systemctl start secubox-hub.service || true
|
||||||
install -d -m 755 /etc/nginx/secubox.d
|
install -d -m 755 /etc/nginx/secubox.d
|
||||||
systemctl reload nginx 2>/dev/null || true
|
systemctl reload nginx 2>/dev/null || true
|
||||||
|
# #758 — deploy + load the network-stats input policy-drop tap
|
||||||
|
if [ -f /usr/share/secubox/hub/nftables.d/zz-secubox-netstats-tap.nft ]; then
|
||||||
|
install -d -m 0755 /etc/nftables.d
|
||||||
|
install -m 0644 /usr/share/secubox/hub/nftables.d/zz-secubox-netstats-tap.nft \
|
||||||
|
/etc/nftables.d/zz-secubox-netstats-tap.nft
|
||||||
|
if systemctl is-active --quiet nftables.service; then
|
||||||
|
systemctl reload nftables.service 2>/dev/null \
|
||||||
|
|| /usr/sbin/nft -f /etc/nftables.d/zz-secubox-netstats-tap.nft 2>/dev/null || true
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
# #758 — enable the collector timer (idempotent)
|
||||||
|
systemctl daemon-reload 2>/dev/null || true
|
||||||
|
systemctl enable --now secubox-netstats.timer 2>/dev/null || true
|
||||||
;;
|
;;
|
||||||
esac
|
esac
|
||||||
#DEBHELPER#
|
#DEBHELPER#
|
||||||
|
|
|
||||||
|
|
@ -24,14 +24,27 @@ override_dh_auto_install:
|
||||||
install -d debian/secubox-hub/etc/systemd/system/timers.target.wants
|
install -d debian/secubox-hub/etc/systemd/system/timers.target.wants
|
||||||
ln -sf /lib/systemd/system/secubox-nft-cache.timer \
|
ln -sf /lib/systemd/system/secubox-nft-cache.timer \
|
||||||
debian/secubox-hub/etc/systemd/system/timers.target.wants/secubox-nft-cache.timer
|
debian/secubox-hub/etc/systemd/system/timers.target.wants/secubox-nft-cache.timer
|
||||||
|
# #758 — network-stats collector + timer
|
||||||
|
install -m 0755 sbin/secubox-netstats-collect debian/secubox-hub/usr/sbin/secubox-netstats-collect
|
||||||
|
install -m 0644 debian/secubox-netstats.service debian/secubox-hub/lib/systemd/system/
|
||||||
|
install -m 0644 debian/secubox-netstats.timer debian/secubox-hub/lib/systemd/system/
|
||||||
|
ln -sf /lib/systemd/system/secubox-netstats.timer \
|
||||||
|
debian/secubox-hub/etc/systemd/system/timers.target.wants/secubox-netstats.timer
|
||||||
|
# #758 — ship the inet filter input policy-drop tap drop-in
|
||||||
|
install -d debian/secubox-hub/usr/share/secubox/hub/nftables.d
|
||||||
|
install -m 0644 nftables.d/zz-secubox-netstats-tap.nft \
|
||||||
|
debian/secubox-hub/usr/share/secubox/hub/nftables.d/
|
||||||
# sudoers fragment so the hub service (User=secubox) can fall back to
|
# sudoers fragment so the hub service (User=secubox) can fall back to
|
||||||
# realtime `nft list` when the cache is stale (read-only nft, no risk).
|
# realtime `nft list` when the cache is stale (read-only nft, no risk).
|
||||||
# Two patterns because `-j` is a global option for nft and must precede
|
# Two patterns because `-j` is a global option for nft and must precede
|
||||||
# the command (`nft -j list ruleset`, not `nft list ruleset -j`).
|
# the command (`nft -j list ruleset`, not `nft list ruleset -j`).
|
||||||
install -d debian/secubox-hub/etc/sudoers.d
|
install -d debian/secubox-hub/etc/sudoers.d
|
||||||
printf '%s\n%s\n%s\n' \
|
# #758 — netstats collector reads `nft -j list table inet crowdsec`
|
||||||
|
# (best-effort, covered by the existing `nft -j list *` grant above).
|
||||||
|
printf '%s\n%s\n%s\n%s\n' \
|
||||||
'secubox ALL=(root) NOPASSWD: /usr/sbin/nft list *' \
|
'secubox ALL=(root) NOPASSWD: /usr/sbin/nft list *' \
|
||||||
'secubox ALL=(root) NOPASSWD: /usr/sbin/nft -j list *' \
|
'secubox ALL=(root) NOPASSWD: /usr/sbin/nft -j list *' \
|
||||||
'secubox ALL=(root) NOPASSWD: /usr/bin/systemctl --no-block start secubox-nft-cache.service' \
|
'secubox ALL=(root) NOPASSWD: /usr/bin/systemctl --no-block start secubox-nft-cache.service' \
|
||||||
|
'secubox ALL=(root) NOPASSWD: /usr/bin/systemctl --no-block start secubox-netstats.service' \
|
||||||
> debian/secubox-hub/etc/sudoers.d/secubox-hub-nft
|
> debian/secubox-hub/etc/sudoers.d/secubox-hub-nft
|
||||||
chmod 0440 debian/secubox-hub/etc/sudoers.d/secubox-hub-nft
|
chmod 0440 debian/secubox-hub/etc/sudoers.d/secubox-hub-nft
|
||||||
|
|
|
||||||
12
packages/secubox-hub/debian/secubox-netstats.service
Normal file
12
packages/secubox-hub/debian/secubox-netstats.service
Normal file
|
|
@ -0,0 +1,12 @@
|
||||||
|
[Unit]
|
||||||
|
Description=SecuBox network-stats collector (nft named counters + /proc/net/dev → SQLite)
|
||||||
|
Documentation=https://github.com/CyberMind-FR/secubox-deb/issues/758
|
||||||
|
After=nftables.service
|
||||||
|
ConditionPathExists=/usr/sbin/nft
|
||||||
|
|
||||||
|
[Service]
|
||||||
|
Type=oneshot
|
||||||
|
ExecStart=/usr/sbin/secubox-netstats-collect
|
||||||
|
# Runs as root via systemd — nft list + /proc throughput need privilege.
|
||||||
|
StandardOutput=journal
|
||||||
|
StandardError=journal
|
||||||
12
packages/secubox-hub/debian/secubox-netstats.timer
Normal file
12
packages/secubox-hub/debian/secubox-netstats.timer
Normal file
|
|
@ -0,0 +1,12 @@
|
||||||
|
[Unit]
|
||||||
|
Description=SecuBox network-stats collector timer (every 30s)
|
||||||
|
Documentation=https://github.com/CyberMind-FR/secubox-deb/issues/758
|
||||||
|
|
||||||
|
[Timer]
|
||||||
|
OnBootSec=15s
|
||||||
|
OnUnitActiveSec=30s
|
||||||
|
AccuracySec=5s
|
||||||
|
Unit=secubox-netstats.service
|
||||||
|
|
||||||
|
[Install]
|
||||||
|
WantedBy=timers.target
|
||||||
14
packages/secubox-hub/nftables.d/zz-secubox-netstats-tap.nft
Normal file
14
packages/secubox-hub/nftables.d/zz-secubox-netstats-tap.nft
Normal file
|
|
@ -0,0 +1,14 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
#
|
||||||
|
# #758 — network-stats tap. Counts packets that fall through to the
|
||||||
|
# default-drop policy of the base inet filter input chain = "unsolicited
|
||||||
|
# inbound" volume. ADDITIVE ONLY: `table inet filter { counter … }` adds the
|
||||||
|
# counter to the pre-existing base table without deleting it; the tail
|
||||||
|
# `add rule` appends a bare counter AFTER every accept rule. The zz- filename
|
||||||
|
# guarantees this loads last among /etc/nftables.d/*.nft. Counter resets on
|
||||||
|
# reload — the collector is reset-aware. Never drops or accepts; pure observe.
|
||||||
|
table inet filter {
|
||||||
|
counter sbx_drop_input_policy {}
|
||||||
|
}
|
||||||
|
add rule inet filter input counter name "sbx_drop_input_policy" comment "sbx-netstats-input-policy-tap"
|
||||||
6
packages/secubox-hub/sbin/secubox-netstats-collect
Executable file
6
packages/secubox-hub/sbin/secubox-netstats-collect
Executable file
|
|
@ -0,0 +1,6 @@
|
||||||
|
#!/bin/sh
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
# #758 — root oneshot: sample nft named counters + /proc/net/dev into
|
||||||
|
# /var/lib/secubox/hub/netstats.{db,json}. Runs as root via systemd timer.
|
||||||
|
exec /usr/bin/python3 -c "import sys; sys.path.insert(0, '/usr/lib/secubox/hub/api'); import netstats; netstats.main()"
|
||||||
39
packages/secubox-hub/tests/test_netstats_api.py
Normal file
39
packages/secubox-hub/tests/test_netstats_api.py
Normal file
|
|
@ -0,0 +1,39 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
# Source-Disclosed License — All rights reserved except as expressly granted.
|
||||||
|
# See LICENCE-CMSD-1.0.md for terms.
|
||||||
|
"""Hub netstats endpoints (ref #758)."""
|
||||||
|
import asyncio
|
||||||
|
import importlib
|
||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "api"))
|
||||||
|
main = importlib.import_module("main")
|
||||||
|
netstats = importlib.import_module("netstats")
|
||||||
|
|
||||||
|
|
||||||
|
def test_summary_reads_snapshot(tmp_path, monkeypatch):
|
||||||
|
snap = tmp_path / "netstats.json"
|
||||||
|
snap.write_text(json.dumps({
|
||||||
|
"updated": 0, "categories": {"blacklist": {"packets": 4}},
|
||||||
|
"interfaces": {}, "network_drops": 4,
|
||||||
|
}))
|
||||||
|
monkeypatch.setattr(netstats, "SNAP_PATH", snap)
|
||||||
|
out = asyncio.run(main.netstats_summary())
|
||||||
|
assert out["network_drops"] == 4
|
||||||
|
assert out["stale"] is True
|
||||||
|
|
||||||
|
|
||||||
|
def test_series_queries_db(tmp_path, monkeypatch):
|
||||||
|
db = tmp_path / "netstats.db"
|
||||||
|
monkeypatch.setattr(netstats, "DB_PATH", db)
|
||||||
|
c = sqlite3.connect(str(db))
|
||||||
|
netstats.init_db(c)
|
||||||
|
netstats.insert_sample(c, 0, {"sbx_drop_wafrl": {"packets": 0, "bytes": 0}}, {})
|
||||||
|
netstats.insert_sample(c, 30, {"sbx_drop_wafrl": {"packets": 12, "bytes": 0}}, {})
|
||||||
|
c.close()
|
||||||
|
out = asyncio.run(main.netstats_series(window=3600, step=30))
|
||||||
|
assert out["drops"]["waf_ratelimit"][-1][1] == 12
|
||||||
29
packages/secubox-hub/tests/test_netstats_packaging.py
Normal file
29
packages/secubox-hub/tests/test_netstats_packaging.py
Normal file
|
|
@ -0,0 +1,29 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
# Source-Disclosed License — All rights reserved except as expressly granted.
|
||||||
|
# See LICENCE-CMSD-1.0.md for terms.
|
||||||
|
"""Packaging wires the collector, timer, nft tap, and sudoers (ref #758)."""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
PKG = Path(__file__).resolve().parents[1]
|
||||||
|
|
||||||
|
|
||||||
|
def test_units_exist():
|
||||||
|
assert (PKG / "debian" / "secubox-netstats.service").exists()
|
||||||
|
assert (PKG / "debian" / "secubox-netstats.timer").exists()
|
||||||
|
|
||||||
|
|
||||||
|
def test_rules_installs_collector_and_units_and_tap():
|
||||||
|
rules = (PKG / "debian" / "rules").read_text()
|
||||||
|
assert "sbin/secubox-netstats-collect" in rules
|
||||||
|
assert "secubox-netstats.service" in rules
|
||||||
|
assert "secubox-netstats.timer" in rules
|
||||||
|
assert "zz-secubox-netstats-tap.nft" in rules
|
||||||
|
# crowdsec read grant added to the sudoers fragment
|
||||||
|
assert "inet crowdsec" in rules
|
||||||
|
|
||||||
|
|
||||||
|
def test_postinst_deploys_tap_and_enables_timer():
|
||||||
|
post = (PKG / "debian" / "postinst").read_text()
|
||||||
|
assert "zz-secubox-netstats-tap.nft" in post
|
||||||
|
assert "secubox-netstats.timer" in post
|
||||||
48
packages/secubox-hub/tests/test_netstats_parse.py
Normal file
48
packages/secubox-hub/tests/test_netstats_parse.py
Normal file
|
|
@ -0,0 +1,48 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
"""Pure parsers for the network-stats collector (ref #758)."""
|
||||||
|
import importlib
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "api"))
|
||||||
|
netstats = importlib.import_module("netstats")
|
||||||
|
|
||||||
|
PROC = (
|
||||||
|
"Inter-| Receive | Transmit\n"
|
||||||
|
" face |bytes packets errs drop fifo frame compressed multicast|bytes packets errs drop fifo colls carrier compressed\n"
|
||||||
|
" lo: 123 4 0 0 0 0 0 0 123 4 0 0 0 0 0 0\n"
|
||||||
|
" eth0: 1000 10 0 0 0 0 0 0 2000 20 0 0 0 0 0 0\n"
|
||||||
|
)
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_proc_net_dev_skips_lo_header():
|
||||||
|
out = netstats.parse_proc_net_dev(PROC)
|
||||||
|
assert "lo" not in out
|
||||||
|
assert out["eth0"] == {"rx_bytes": 1000, "rx_packets": 10, "tx_bytes": 2000, "tx_packets": 20}
|
||||||
|
|
||||||
|
|
||||||
|
def test_parse_nft_counters_json():
|
||||||
|
data = {"nftables": [
|
||||||
|
{"counter": {"name": "sbx_drop_blacklist_v4", "packets": 5, "bytes": 500}},
|
||||||
|
{"counter": {"name": "irrelevant", "packets": 9, "bytes": 9}},
|
||||||
|
{"rule": {"chain": "x"}},
|
||||||
|
]}
|
||||||
|
out = netstats.parse_nft_counters_json(data)
|
||||||
|
assert out["sbx_drop_blacklist_v4"] == {"packets": 5, "bytes": 500}
|
||||||
|
assert "irrelevant" in out # parser keeps all; mapping happens via category_for
|
||||||
|
|
||||||
|
|
||||||
|
def test_category_for():
|
||||||
|
assert netstats.category_for("sbx_drop_blacklist_v6") == "blacklist"
|
||||||
|
assert netstats.category_for("sbx_drop_quarantine_v4") == "quarantine"
|
||||||
|
assert netstats.category_for("sbx_drop_wafrl") == "waf_ratelimit"
|
||||||
|
assert netstats.category_for("sbx_drop_input_policy") == "input_policy"
|
||||||
|
assert netstats.category_for("sbx_doh_detect_v4") == "doh"
|
||||||
|
assert netstats.category_for("nope") is None
|
||||||
|
|
||||||
|
|
||||||
|
def test_reset_aware_delta():
|
||||||
|
assert netstats.reset_aware_delta(100, 150) == 50 # normal
|
||||||
|
assert netstats.reset_aware_delta(150, 10) == 10 # reset → treat cur as delta
|
||||||
|
assert netstats.reset_aware_delta(0, 0) == 0
|
||||||
54
packages/secubox-hub/tests/test_netstats_snapshot.py
Normal file
54
packages/secubox-hub/tests/test_netstats_snapshot.py
Normal file
|
|
@ -0,0 +1,54 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
"""Snapshot builder + collect_once with monkeypatched privileged sources (#758)."""
|
||||||
|
import importlib
|
||||||
|
import json
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
import time
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "api"))
|
||||||
|
netstats = importlib.import_module("netstats")
|
||||||
|
|
||||||
|
|
||||||
|
def test_build_snapshot_network_drops_excludes_doh():
|
||||||
|
c = sqlite3.connect(":memory:")
|
||||||
|
netstats.init_db(c)
|
||||||
|
counters = {
|
||||||
|
"sbx_drop_blacklist_v4": {"packets": 4, "bytes": 0},
|
||||||
|
"sbx_drop_wafrl": {"packets": 6, "bytes": 0},
|
||||||
|
"sbx_doh_detect_v4": {"packets": 99, "bytes": 0},
|
||||||
|
}
|
||||||
|
netstats.insert_sample(c, 1000, counters, {})
|
||||||
|
snap = netstats.build_snapshot(c, now=1000)
|
||||||
|
assert snap["network_drops"] == 10 # 4 + 6, doh excluded
|
||||||
|
assert snap["categories"]["doh"]["packets"] == 99
|
||||||
|
assert snap["updated"] == 1000
|
||||||
|
|
||||||
|
|
||||||
|
def test_collect_once_writes_row_and_snapshot(tmp_path, monkeypatch):
|
||||||
|
db = tmp_path / "netstats.db"
|
||||||
|
snap = tmp_path / "netstats.json"
|
||||||
|
monkeypatch.setattr(netstats, "DB_PATH", db)
|
||||||
|
monkeypatch.setattr(netstats, "SNAP_PATH", snap)
|
||||||
|
monkeypatch.setattr(netstats, "DATA_DIR", tmp_path)
|
||||||
|
monkeypatch.setattr(netstats, "_read_nft_counters",
|
||||||
|
lambda: {"sbx_drop_wafrl": {"packets": 3, "bytes": 30}})
|
||||||
|
monkeypatch.setattr(netstats, "_read_crowdsec", lambda: {})
|
||||||
|
monkeypatch.setattr(netstats, "_read_ifaces",
|
||||||
|
lambda: {"eth0": {"rx_bytes": 1, "rx_packets": 1, "tx_bytes": 1, "tx_packets": 1}})
|
||||||
|
out = netstats.collect_once(now=1234)
|
||||||
|
assert out["network_drops"] == 3
|
||||||
|
assert snap.exists()
|
||||||
|
written = json.loads(snap.read_text())
|
||||||
|
assert written["updated"] == 1234
|
||||||
|
# a second tick must not raise (DB reused)
|
||||||
|
netstats.collect_once(now=1264)
|
||||||
|
|
||||||
|
|
||||||
|
def test_read_snapshot_marks_stale(tmp_path, monkeypatch):
|
||||||
|
snap = tmp_path / "netstats.json"
|
||||||
|
snap.write_text(json.dumps({"updated": 0, "categories": {}, "interfaces": {}, "network_drops": 0}))
|
||||||
|
monkeypatch.setattr(netstats, "SNAP_PATH", snap)
|
||||||
|
out = netstats.read_snapshot()
|
||||||
|
assert out["stale"] is True
|
||||||
63
packages/secubox-hub/tests/test_netstats_store.py
Normal file
63
packages/secubox-hub/tests/test_netstats_store.py
Normal file
|
|
@ -0,0 +1,63 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
"""SQLite store + reset-aware series for network-stats (ref #758)."""
|
||||||
|
import importlib
|
||||||
|
import sqlite3
|
||||||
|
import sys
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
sys.path.insert(0, str(Path(__file__).resolve().parents[1] / "api"))
|
||||||
|
netstats = importlib.import_module("netstats")
|
||||||
|
|
||||||
|
|
||||||
|
def _conn():
|
||||||
|
c = sqlite3.connect(":memory:")
|
||||||
|
netstats.init_db(c)
|
||||||
|
return c
|
||||||
|
|
||||||
|
|
||||||
|
def test_insert_and_query_drops_delta():
|
||||||
|
c = _conn()
|
||||||
|
# t=0 blacklist=100 ; t=30 blacklist=160 → delta 60 in the bucket at t=30
|
||||||
|
netstats.insert_sample(c, 0, {"sbx_drop_blacklist_v4": {"packets": 100, "bytes": 0}}, {})
|
||||||
|
netstats.insert_sample(c, 30, {"sbx_drop_blacklist_v4": {"packets": 160, "bytes": 0}}, {})
|
||||||
|
out = netstats.query_series(c, window_s=3600, step_s=30)
|
||||||
|
pts = out["drops"]["blacklist"]
|
||||||
|
assert pts[-1][1] == 60
|
||||||
|
|
||||||
|
|
||||||
|
def test_query_drops_reset_aware():
|
||||||
|
c = _conn()
|
||||||
|
netstats.insert_sample(c, 0, {"sbx_drop_blacklist_v4": {"packets": 100, "bytes": 0}}, {})
|
||||||
|
netstats.insert_sample(c, 30, {"sbx_drop_blacklist_v4": {"packets": 5, "bytes": 0}}, {}) # reload
|
||||||
|
out = netstats.query_series(c, window_s=3600, step_s=30)
|
||||||
|
assert out["drops"]["blacklist"][-1][1] == 5 # not negative
|
||||||
|
|
||||||
|
|
||||||
|
def test_query_throughput_bps():
|
||||||
|
c = _conn()
|
||||||
|
# 30s apart, +30000 rx bytes → 30000*8/30 = 8000 bps
|
||||||
|
netstats.insert_sample(c, 0, {}, {"eth0": {"rx_bytes": 0, "rx_packets": 0, "tx_bytes": 0, "tx_packets": 0}})
|
||||||
|
netstats.insert_sample(c, 30, {}, {"eth0": {"rx_bytes": 30000, "rx_packets": 0, "tx_bytes": 0, "tx_packets": 0}})
|
||||||
|
out = netstats.query_series(c, window_s=3600, step_s=30)
|
||||||
|
assert out["in_bps"]["eth0"][-1][1] == 8000
|
||||||
|
|
||||||
|
|
||||||
|
def test_query_drops_sums_multiple_names_per_category():
|
||||||
|
c = _conn()
|
||||||
|
# blacklist_v4 and blacklist_v6 both map to category "blacklist"
|
||||||
|
netstats.insert_sample(c, 0, {"sbx_drop_blacklist_v4": {"packets": 0, "bytes": 0},
|
||||||
|
"sbx_drop_blacklist_v6": {"packets": 0, "bytes": 0}}, {})
|
||||||
|
netstats.insert_sample(c, 30, {"sbx_drop_blacklist_v4": {"packets": 10, "bytes": 0},
|
||||||
|
"sbx_drop_blacklist_v6": {"packets": 5, "bytes": 0}}, {})
|
||||||
|
out = netstats.query_series(c, window_s=3600, step_s=30)
|
||||||
|
assert out["drops"]["blacklist"][-1][1] == 15 # 10 + 5 summed, not overwritten
|
||||||
|
|
||||||
|
|
||||||
|
def test_prune_drops_old_rows():
|
||||||
|
c = _conn()
|
||||||
|
netstats.insert_sample(c, 0, {"sbx_drop_wafrl": {"packets": 1, "bytes": 0}}, {})
|
||||||
|
netstats.insert_sample(c, 1_000_000, {"sbx_drop_wafrl": {"packets": 2, "bytes": 0}}, {})
|
||||||
|
netstats.prune(c, keep_s=10) # relative to max ts
|
||||||
|
rows = c.execute("SELECT COUNT(*) FROM counter_samples").fetchone()[0]
|
||||||
|
assert rows == 1
|
||||||
16
packages/secubox-hub/tests/test_netstats_tap.py
Normal file
16
packages/secubox-hub/tests/test_netstats_tap.py
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
"""The hub input policy-drop tap declares + references its counter (ref #758)."""
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
NFT = Path(__file__).resolve().parents[1] / "nftables.d" / "zz-secubox-netstats-tap.nft"
|
||||||
|
|
||||||
|
|
||||||
|
def test_tap_counter_present_and_zz_ordered():
|
||||||
|
assert NFT.name.startswith("zz-"), "tap must sort after accept rules"
|
||||||
|
text = NFT.read_text()
|
||||||
|
assert re.search(r'counter\s+sbx_drop_input_policy\s*\{', text)
|
||||||
|
assert re.search(r'add rule inet filter input .*counter name "sbx_drop_input_policy"', text)
|
||||||
|
# additive only — must NOT delete or flush the base filter table
|
||||||
|
assert "delete table inet filter" not in text
|
||||||
|
assert "flush ruleset" not in text
|
||||||
|
|
@ -1,3 +1,10 @@
|
||||||
|
secubox-mitmproxy (1.0.10-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
|
* feat(nft): named counter sbx_drop_wafrl on the WAF rate-limit drops so the
|
||||||
|
secubox-hub network-stats collector can attribute attack drops (ref #758).
|
||||||
|
|
||||||
|
-- Gerald KERMA <devel@cybermind.fr> Sat, 27 Jun 2026 12:00:00 +0200
|
||||||
|
|
||||||
secubox-mitmproxy (1.0.9-1~bookworm1) bookworm; urgency=medium
|
secubox-mitmproxy (1.0.9-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
* feat(robustness): self-healing WAF inspector watchdog (closes #624).
|
* feat(robustness): self-healing WAF inspector watchdog (closes #624).
|
||||||
|
|
|
||||||
|
|
@ -37,6 +37,9 @@ table inet secubox_waf_ratelimit {
|
||||||
elements = { 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 }
|
elements = { 127.0.0.0/8, 10.0.0.0/8, 172.16.0.0/12, 192.168.0.0/16 }
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# #758 — named counter for the network-stats collector (attacks family).
|
||||||
|
counter sbx_drop_wafrl {}
|
||||||
|
|
||||||
chain input {
|
chain input {
|
||||||
type filter hook input priority -10; policy accept;
|
type filter hook input priority -10; policy accept;
|
||||||
|
|
||||||
|
|
@ -44,14 +47,14 @@ table inet secubox_waf_ratelimit {
|
||||||
ip saddr @whitelist_v4 accept
|
ip saddr @whitelist_v4 accept
|
||||||
|
|
||||||
# Drop active offenders fast (no further matching)
|
# Drop active offenders fast (no further matching)
|
||||||
ip saddr @offenders_v4 counter drop
|
ip saddr @offenders_v4 counter name "sbx_drop_wafrl" drop
|
||||||
ip6 saddr @offenders_v6 counter drop
|
ip6 saddr @offenders_v6 counter name "sbx_drop_wafrl" drop
|
||||||
|
|
||||||
# Rate-limit only on TCP SYN to 80/443 (new public conn attempts)
|
# Rate-limit only on TCP SYN to 80/443 (new public conn attempts)
|
||||||
# nft 'limit rate over X' syntax matches packets exceeding the rate.
|
# nft 'limit rate over X' syntax matches packets exceeding the rate.
|
||||||
tcp flags syn tcp dport { 80, 443 } \
|
tcp flags syn tcp dport { 80, 443 } \
|
||||||
limit rate over 30/second burst 50 packets \
|
limit rate over 30/second burst 50 packets \
|
||||||
add @offenders_v4 { ip saddr timeout 5m } \
|
add @offenders_v4 { ip saddr timeout 5m } \
|
||||||
log prefix "[secubox-rl] " level info counter drop
|
log prefix "[secubox-rl] " level info counter name "sbx_drop_wafrl" drop
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
18
packages/secubox-mitmproxy/tests/test_waf_counter.py
Normal file
18
packages/secubox-mitmproxy/tests/test_waf_counter.py
Normal file
|
|
@ -0,0 +1,18 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
# Source-Disclosed License — All rights reserved except as expressly granted.
|
||||||
|
# See LICENCE-CMSD-1.0.md for terms.
|
||||||
|
|
||||||
|
"""WAF rate-limit drops carry a named counter (ref #758)."""
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
NFT = Path(__file__).resolve().parents[1] / "nftables" / "secubox-waf-ratelimit.nft"
|
||||||
|
|
||||||
|
|
||||||
|
def test_wafrl_counter_declared_and_referenced():
|
||||||
|
text = NFT.read_text()
|
||||||
|
decls = set(re.findall(r'counter\s+([a-z0-9_]+)\s*\{', text))
|
||||||
|
refs = set(re.findall(r'counter name "([a-z0-9_]+)"', text))
|
||||||
|
assert "sbx_drop_wafrl" in refs
|
||||||
|
assert "sbx_drop_wafrl" in decls
|
||||||
|
|
@ -1,3 +1,12 @@
|
||||||
|
secubox-toolbox (2.8.0-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
|
* feat(#758): new "Réseau" dashboard tab — interface throughput sparklines +
|
||||||
|
per-category drop/attack trends from the hub netstats collector; the #ads
|
||||||
|
"Drops réseau" KPI now reads real data. Named nft counters added to the
|
||||||
|
blacklist spine.
|
||||||
|
|
||||||
|
-- Gerald KERMA <devel@cybermind.fr> Sat, 27 Jun 2026 12:00:00 +0200
|
||||||
|
|
||||||
secubox-toolbox (2.7.24-1~bookworm1) bookworm; urgency=medium
|
secubox-toolbox (2.7.24-1~bookworm1) bookworm; urgency=medium
|
||||||
|
|
||||||
* ui: remove the redundant ♥ Liveness dashboard card (generic status/version);
|
* ui: remove the redundant ♥ Liveness dashboard card (generic status/version);
|
||||||
|
|
|
||||||
|
|
@ -53,18 +53,29 @@ table inet secubox_blacklist {
|
||||||
flags interval, timeout
|
flags interval, timeout
|
||||||
}
|
}
|
||||||
|
|
||||||
|
# #758 — named counters for the network-stats collector. Declared at
|
||||||
|
# table scope so `nft list counters` exposes them by name; referenced
|
||||||
|
# by the enforce/doh_watch rules below. Reset to 0 on reload (collector
|
||||||
|
# is reset-aware).
|
||||||
|
counter sbx_drop_quarantine_v4 {}
|
||||||
|
counter sbx_drop_quarantine_v6 {}
|
||||||
|
counter sbx_drop_blacklist_v4 {}
|
||||||
|
counter sbx_drop_blacklist_v6 {}
|
||||||
|
counter sbx_doh_detect_v4 {}
|
||||||
|
counter sbx_doh_detect_v6 {}
|
||||||
|
|
||||||
chain enforce {
|
chain enforce {
|
||||||
type filter hook forward priority -10; policy accept;
|
type filter hook forward priority -10; policy accept;
|
||||||
# Quarantine first : a quarantined device is fully cut off.
|
# Quarantine first : a quarantined device is fully cut off.
|
||||||
ip saddr @quarantine_v4 counter drop
|
ip saddr @quarantine_v4 counter name "sbx_drop_quarantine_v4" drop
|
||||||
ip6 saddr @quarantine_v6 counter drop
|
ip6 saddr @quarantine_v6 counter name "sbx_drop_quarantine_v6" drop
|
||||||
# C2 blacklist (13.A). Phase 13.C : rate-limited log so the
|
# C2 blacklist (13.A). Phase 13.C : rate-limited log so the
|
||||||
# attribution tailer can bucket drops per source device without
|
# attribution tailer can bucket drops per source device without
|
||||||
# flooding the kernel log.
|
# flooding the kernel log.
|
||||||
ip daddr @blacklist_v4 limit rate 20/second log prefix "SBX-BL-DROP " counter drop
|
ip daddr @blacklist_v4 limit rate 20/second log prefix "SBX-BL-DROP " counter name "sbx_drop_blacklist_v4" drop
|
||||||
ip saddr @blacklist_v4 counter drop
|
ip saddr @blacklist_v4 counter name "sbx_drop_blacklist_v4" drop
|
||||||
ip6 daddr @blacklist_v6 limit rate 20/second log prefix "SBX-BL-DROP " counter drop
|
ip6 daddr @blacklist_v6 limit rate 20/second log prefix "SBX-BL-DROP " counter name "sbx_drop_blacklist_v6" drop
|
||||||
ip6 saddr @blacklist_v6 counter drop
|
ip6 saddr @blacklist_v6 counter name "sbx_drop_blacklist_v6" drop
|
||||||
}
|
}
|
||||||
|
|
||||||
# Phase 13.B (#522) — DoH/DoT detection. COUNT-ONLY by default : a
|
# Phase 13.B (#522) — DoH/DoT detection. COUNT-ONLY by default : a
|
||||||
|
|
@ -86,7 +97,7 @@ table inet secubox_blacklist {
|
||||||
type filter hook forward priority -11; policy accept;
|
type filter hook forward priority -11; policy accept;
|
||||||
# Phase 13.C : rate-limited log (new connections only) so the
|
# Phase 13.C : rate-limited log (new connections only) so the
|
||||||
# attribution tailer can bucket DoH attempts per device.
|
# attribution tailer can bucket DoH attempts per device.
|
||||||
ip daddr @doh_detect_v4 tcp dport { 443, 853 } ct state new limit rate 5/second log prefix "SBX-DOH " counter
|
ip daddr @doh_detect_v4 tcp dport { 443, 853 } ct state new limit rate 5/second log prefix "SBX-DOH " counter name "sbx_doh_detect_v4"
|
||||||
ip6 daddr @doh_detect_v6 tcp dport { 443, 853 } ct state new limit rate 5/second log prefix "SBX-DOH " counter
|
ip6 daddr @doh_detect_v6 tcp dport { 443, 853 } ct state new limit rate 5/second log prefix "SBX-DOH " counter name "sbx_doh_detect_v6"
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
|
||||||
|
|
@ -45,6 +45,8 @@ try:
|
||||||
_HAS_TRANSPARENCY = True
|
_HAS_TRANSPARENCY = True
|
||||||
except ImportError:
|
except ImportError:
|
||||||
_HAS_TRANSPARENCY = False
|
_HAS_TRANSPARENCY = False
|
||||||
|
from pathlib import Path as _Path
|
||||||
|
NETSTATS_SNAPSHOT = _Path("/var/lib/secubox/hub/netstats.json")
|
||||||
from .config import load_config, resolve_secret
|
from .config import load_config, resolve_secret
|
||||||
from .models import AcceptResp, ClientRow, Config, StatusResp
|
from .models import AcceptResp, ClientRow, Config, StatusResp
|
||||||
|
|
||||||
|
|
@ -2971,17 +2973,14 @@ async def admin_blacklist() -> dict:
|
||||||
out["doh_detect_v4"] = n
|
out["doh_detect_v4"] = n
|
||||||
elif name == "doh_detect_v6":
|
elif name == "doh_detect_v6":
|
||||||
out["doh_detect_v6"] = n
|
out["doh_detect_v6"] = n
|
||||||
if "rule" in item:
|
if "counter" in item and isinstance(item["counter"], dict):
|
||||||
chain = item["rule"].get("chain", "")
|
cobj = item["counter"]
|
||||||
for ex in item["rule"].get("expr", []):
|
cname = cobj.get("name", "")
|
||||||
c = ex.get("counter")
|
pk = int(cobj.get("packets", 0) or 0)
|
||||||
if not c:
|
if cname.startswith("sbx_doh_detect"):
|
||||||
continue
|
out["doh_hits"] += pk
|
||||||
pk = int(c.get("packets", 0) or 0)
|
elif cname.startswith(("sbx_drop_blacklist", "sbx_drop_quarantine")):
|
||||||
if chain == "doh_watch":
|
out["drops"] += pk
|
||||||
out["doh_hits"] += pk
|
|
||||||
else:
|
|
||||||
out["drops"] += pk
|
|
||||||
out["active"] = True
|
out["active"] = True
|
||||||
except Exception as e: # noqa: BLE001
|
except Exception as e: # noqa: BLE001
|
||||||
log.warning("admin_blacklist nft parse failed: %s", e)
|
log.warning("admin_blacklist nft parse failed: %s", e)
|
||||||
|
|
@ -3096,13 +3095,22 @@ async def admin_ad_stats(hours: int = 24) -> dict:
|
||||||
"""Contextual ad-block metrics for the #ads tab (read-only, kbin-safe)."""
|
"""Contextual ad-block metrics for the #ads tab (read-only, kbin-safe)."""
|
||||||
h = max(1, min(int(hours if hours is not None else 24), 168))
|
h = max(1, min(int(hours if hours is not None else 24), 168))
|
||||||
out = store.ad_stats(hours=h)
|
out = store.ad_stats(hours=h)
|
||||||
# #755 — network-layer drops (blacklist nft sets). Best-effort; 0 when the
|
# #758 — real network-layer drops from the hub netstats collector snapshot.
|
||||||
# blacklist is inert or unreadable. Reuses the admin_blacklist parse.
|
# Fall back to the legacy blacklist nft parse when the snapshot is absent.
|
||||||
|
nd = None
|
||||||
try:
|
try:
|
||||||
bl = await admin_blacklist()
|
import json as _json
|
||||||
out["network_drops"] = int(bl.get("drops", 0) or 0)
|
snap = _json.loads(NETSTATS_SNAPSHOT.read_text())
|
||||||
|
nd = int(snap.get("network_drops", 0) or 0)
|
||||||
except Exception:
|
except Exception:
|
||||||
out["network_drops"] = 0
|
nd = None
|
||||||
|
if nd is None:
|
||||||
|
try:
|
||||||
|
bl = await admin_blacklist()
|
||||||
|
nd = int(bl.get("drops", 0) or 0)
|
||||||
|
except Exception:
|
||||||
|
nd = 0
|
||||||
|
out["network_drops"] = nd
|
||||||
return out
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
|
|
||||||
26
packages/secubox-toolbox/tests/test_admin_blacklist_named.py
Normal file
26
packages/secubox-toolbox/tests/test_admin_blacklist_named.py
Normal file
|
|
@ -0,0 +1,26 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
"""admin_blacklist must sum named counter OBJECTS (ref #758)."""
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
import types
|
||||||
|
from secubox_toolbox import api
|
||||||
|
|
||||||
|
|
||||||
|
def _fake_nft_json():
|
||||||
|
return json.dumps({"nftables": [
|
||||||
|
{"set": {"name": "blacklist_v4", "elem": ["1.2.3.4"]}},
|
||||||
|
{"counter": {"name": "sbx_drop_blacklist_v4", "packets": 7, "bytes": 700}},
|
||||||
|
{"counter": {"name": "sbx_drop_quarantine_v4", "packets": 3, "bytes": 300}},
|
||||||
|
{"counter": {"name": "sbx_doh_detect_v4", "packets": 5, "bytes": 500}},
|
||||||
|
]})
|
||||||
|
|
||||||
|
|
||||||
|
def test_admin_blacklist_sums_named_counters(monkeypatch):
|
||||||
|
def fake_run(cmd, **kw):
|
||||||
|
return types.SimpleNamespace(returncode=0, stdout=_fake_nft_json(), stderr="")
|
||||||
|
monkeypatch.setattr("subprocess.run", fake_run)
|
||||||
|
out = asyncio.run(api.admin_blacklist())
|
||||||
|
assert out["drops"] == 10 # blacklist 7 + quarantine 3 (NOT doh)
|
||||||
|
assert out["doh_hits"] == 5
|
||||||
|
assert out["v4_count"] == 1
|
||||||
25
packages/secubox-toolbox/tests/test_blacklist_counters.py
Normal file
25
packages/secubox-toolbox/tests/test_blacklist_counters.py
Normal file
|
|
@ -0,0 +1,25 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
"""Every named counter referenced in the blacklist spine is declared (ref #758)."""
|
||||||
|
import re
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
NFT = Path(__file__).resolve().parents[1] / "nftables.d" / "secubox-blacklist.nft"
|
||||||
|
EXPECTED = {
|
||||||
|
"sbx_drop_blacklist_v4", "sbx_drop_blacklist_v6",
|
||||||
|
"sbx_drop_quarantine_v4", "sbx_drop_quarantine_v6",
|
||||||
|
"sbx_doh_detect_v4", "sbx_doh_detect_v6",
|
||||||
|
}
|
||||||
|
|
||||||
|
|
||||||
|
def _decls_and_refs(text):
|
||||||
|
decls = set(re.findall(r'counter\s+([a-z0-9_]+)\s*\{', text))
|
||||||
|
refs = set(re.findall(r'counter name "([a-z0-9_]+)"', text))
|
||||||
|
return decls, refs
|
||||||
|
|
||||||
|
|
||||||
|
def test_named_counters_declared_and_referenced():
|
||||||
|
text = NFT.read_text()
|
||||||
|
decls, refs = _decls_and_refs(text)
|
||||||
|
assert EXPECTED <= refs, f"missing refs: {EXPECTED - refs}"
|
||||||
|
assert refs <= decls, f"undeclared counters referenced: {refs - decls}"
|
||||||
32
packages/secubox-toolbox/tests/test_network_drops_source.py
Normal file
32
packages/secubox-toolbox/tests/test_network_drops_source.py
Normal file
|
|
@ -0,0 +1,32 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
"""network_drops comes from the hub netstats snapshot (ref #758)."""
|
||||||
|
import asyncio
|
||||||
|
import json
|
||||||
|
from secubox_toolbox import api
|
||||||
|
|
||||||
|
|
||||||
|
def test_network_drops_from_snapshot(tmp_path, monkeypatch):
|
||||||
|
snap = tmp_path / "netstats.json"
|
||||||
|
snap.write_text(json.dumps({"network_drops": 42, "updated": 9_999_999_999}))
|
||||||
|
monkeypatch.setattr(api, "NETSTATS_SNAPSHOT", snap)
|
||||||
|
monkeypatch.setattr(api.store, "ad_stats", lambda hours: {"total_blocked": 0})
|
||||||
|
|
||||||
|
async def _boom():
|
||||||
|
raise AssertionError("must not fall back to nft when snapshot present")
|
||||||
|
monkeypatch.setattr(api, "admin_blacklist", _boom)
|
||||||
|
|
||||||
|
out = asyncio.run(api.admin_ad_stats(hours=24))
|
||||||
|
assert out["network_drops"] == 42
|
||||||
|
|
||||||
|
|
||||||
|
def test_network_drops_fallback_to_blacklist(tmp_path, monkeypatch):
|
||||||
|
monkeypatch.setattr(api, "NETSTATS_SNAPSHOT", tmp_path / "missing.json")
|
||||||
|
monkeypatch.setattr(api.store, "ad_stats", lambda hours: {"total_blocked": 0})
|
||||||
|
|
||||||
|
async def _bl():
|
||||||
|
return {"drops": 7}
|
||||||
|
monkeypatch.setattr(api, "admin_blacklist", _bl)
|
||||||
|
|
||||||
|
out = asyncio.run(api.admin_ad_stats(hours=24))
|
||||||
|
assert out["network_drops"] == 7
|
||||||
16
packages/secubox-toolbox/tests/test_reseau_tab_present.py
Normal file
16
packages/secubox-toolbox/tests/test_reseau_tab_present.py
Normal file
|
|
@ -0,0 +1,16 @@
|
||||||
|
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||||
|
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||||
|
"""The Réseau tab is wired into the toolbox dashboard (ref #758)."""
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
HTML = Path(__file__).resolve().parents[1] / "www" / "toolbox" / "index.html"
|
||||||
|
|
||||||
|
|
||||||
|
def test_reseau_tab_button_panel_and_loader():
|
||||||
|
t = HTML.read_text()
|
||||||
|
assert 'data-tab="reseau"' in t
|
||||||
|
assert 'id="panel-reseau"' in t
|
||||||
|
assert "loadNetstats" in t
|
||||||
|
# talks to the hub netstats endpoints
|
||||||
|
assert "/api/v1/hub/netstats/summary" in t
|
||||||
|
assert "/api/v1/hub/netstats/series" in t
|
||||||
|
|
@ -83,6 +83,7 @@
|
||||||
<button class="tab" data-tab="filtres" onclick="switchTab('filtres')">🚦 Filtres MITM</button>
|
<button class="tab" data-tab="filtres" onclick="switchTab('filtres')">🚦 Filtres MITM</button>
|
||||||
<button class="tab" data-tab="social" onclick="switchTab('social')">🕸️ Cartographie sociale</button>
|
<button class="tab" data-tab="social" onclick="switchTab('social')">🕸️ Cartographie sociale</button>
|
||||||
<button class="tab" data-tab="ads" onclick="switchTab('ads')">🛑 Pubs</button>
|
<button class="tab" data-tab="ads" onclick="switchTab('ads')">🛑 Pubs</button>
|
||||||
|
<button class="tab" data-tab="reseau" onclick="switchTab('reseau')">📡 Réseau</button>
|
||||||
<button class="tab" data-tab="tor" onclick="switchTab('tor')">🧅 Tor</button>
|
<button class="tab" data-tab="tor" onclick="switchTab('tor')">🧅 Tor</button>
|
||||||
<button class="tab" data-tab="config" onclick="switchTab('config')">⚙ Config</button>
|
<button class="tab" data-tab="config" onclick="switchTab('config')">⚙ Config</button>
|
||||||
</nav>
|
</nav>
|
||||||
|
|
@ -194,6 +195,17 @@
|
||||||
</div>
|
</div>
|
||||||
</section>
|
</section>
|
||||||
|
|
||||||
|
<!-- #758 — network-layer stats (nft named counters + interface throughput) -->
|
||||||
|
<section class="panel" id="panel-reseau">
|
||||||
|
<h2>📡 Statistiques réseau (nftables)</h2>
|
||||||
|
<div id="ns-stale" style="display:none;color:var(--cinnabar,#e63946);font-size:.85em">⚠ données périmées (collecteur arrêté ?)</div>
|
||||||
|
<div id="ns-kpi" class="kpis"></div>
|
||||||
|
<h3>Débit interfaces (24h)</h3>
|
||||||
|
<div id="ns-throughput"></div>
|
||||||
|
<h3>Drops & attaques par catégorie (24h)</h3>
|
||||||
|
<div id="ns-drops"></div>
|
||||||
|
</section>
|
||||||
|
|
||||||
<!-- Tor egress quick-switch (#683) -->
|
<!-- Tor egress quick-switch (#683) -->
|
||||||
<section class="panel" id="panel-tor">
|
<section class="panel" id="panel-tor">
|
||||||
<div class="toolbar">
|
<div class="toolbar">
|
||||||
|
|
@ -248,6 +260,7 @@ function switchTab(name) {
|
||||||
if (name === 'filtres') loadFilters();
|
if (name === 'filtres') loadFilters();
|
||||||
if (name === 'social') loadSocial();
|
if (name === 'social') loadSocial();
|
||||||
if (name === 'ads') loadAds();
|
if (name === 'ads') loadAds();
|
||||||
|
if (name === 'reseau') loadNetstats();
|
||||||
if (name === 'tor') loadTor();
|
if (name === 'tor') loadTor();
|
||||||
location.hash = name;
|
location.hash = name;
|
||||||
}
|
}
|
||||||
|
|
@ -643,13 +656,62 @@ async function loadAdsClient(mh) {
|
||||||
: '<div class="empty">aucune pub bloquée pour ce visiteur</div>');
|
: '<div class="empty">aucune pub bloquée pour ce visiteur</div>');
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// ── Réseau tab (#758) — hub netstats endpoints ──
|
||||||
|
async function Jhub(path) {
|
||||||
|
try {
|
||||||
|
const r = await fetch('/api/v1/hub' + path, { credentials: 'same-origin' });
|
||||||
|
if (!r.ok) return { __error: 'HTTP ' + r.status };
|
||||||
|
return await r.json();
|
||||||
|
} catch (e) { return { __error: String(e) }; }
|
||||||
|
}
|
||||||
|
|
||||||
|
function sparkline(points, color) {
|
||||||
|
// points: [[ts, value], …] → inline SVG polyline, auto-scaled.
|
||||||
|
if (!points || !points.length) return '<span style="opacity:.5">—</span>';
|
||||||
|
const W = 240, H = 40, vals = points.map(p => p[1]);
|
||||||
|
const max = Math.max(1, ...vals), n = points.length;
|
||||||
|
const path = points.map((p, i) =>
|
||||||
|
`${(i / Math.max(1, n - 1) * W).toFixed(1)},${(H - p[1] / max * H).toFixed(1)}`).join(' ');
|
||||||
|
return `<svg width="${W}" height="${H}" viewBox="0 0 ${W} ${H}">`
|
||||||
|
+ `<polyline fill="none" stroke="${color}" stroke-width="1.5" points="${path}"/></svg>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function fmtBps(b) {
|
||||||
|
if (b > 1e6) return (b / 1e6).toFixed(1) + ' Mb/s';
|
||||||
|
if (b > 1e3) return (b / 1e3).toFixed(1) + ' kb/s';
|
||||||
|
return (b || 0) + ' b/s';
|
||||||
|
}
|
||||||
|
|
||||||
|
// Endpoints: /api/v1/hub/netstats/summary /api/v1/hub/netstats/series
|
||||||
|
async function loadNetstats() {
|
||||||
|
const sum = await Jhub('/netstats/summary');
|
||||||
|
const kpi = document.getElementById('ns-kpi');
|
||||||
|
document.getElementById('ns-stale').style.display = (sum && sum.stale) ? 'block' : 'none';
|
||||||
|
if (!sum || sum.__error) { kpi.innerHTML = `<span class="k">err</span><span class="v">${(sum && sum.__error) || 'no data'}</span>`; return; }
|
||||||
|
const cats = sum.categories || {};
|
||||||
|
kpi.innerHTML = `<span class="k">Drops réseau (total)</span> <span class="v">${sum.network_drops || 0}</span>`
|
||||||
|
+ Object.keys(cats).map(c => ` <span class="k">${c}</span> <span class="v">${(cats[c].packets) || 0}</span>`).join('');
|
||||||
|
|
||||||
|
const ser = await Jhub('/netstats/series?window=86400&step=300');
|
||||||
|
const tp = document.getElementById('ns-throughput');
|
||||||
|
const dr = document.getElementById('ns-drops');
|
||||||
|
if (!ser || ser.__error) { tp.textContent = 'no data'; dr.textContent = ''; return; }
|
||||||
|
const inb = ser.in_bps || {}, outb = ser.out_bps || {};
|
||||||
|
tp.innerHTML = Object.keys(inb).map(ifc =>
|
||||||
|
`<div class="row"><b>${ifc}</b> ↓ ${fmtBps((inb[ifc].slice(-1)[0] || [0, 0])[1])} ${sparkline(inb[ifc], '#00d4ff')}`
|
||||||
|
+ ` ↑ ${fmtBps(((outb[ifc] || []).slice(-1)[0] || [0, 0])[1])} ${sparkline(outb[ifc] || [], '#c9a84c')}</div>`).join('') || '<span style="opacity:.5">—</span>';
|
||||||
|
const drops = ser.drops || {};
|
||||||
|
dr.innerHTML = Object.keys(drops).map(c =>
|
||||||
|
`<div class="row"><b>${c}</b> ${sparkline(drops[c], '#e63946')}</div>`).join('') || '<span style="opacity:.5">—</span>';
|
||||||
|
}
|
||||||
|
|
||||||
async function refreshAll() {
|
async function refreshAll() {
|
||||||
await Promise.all([loadCfg(), loadClients(), loadHealth(), loadMetrics()]);
|
await Promise.all([loadCfg(), loadClients(), loadHealth(), loadMetrics()]);
|
||||||
}
|
}
|
||||||
|
|
||||||
// Deep-link : open the tab named in the URL hash on load.
|
// Deep-link : open the tab named in the URL hash on load.
|
||||||
const initial = (location.hash || '').replace('#', '');
|
const initial = (location.hash || '').replace('#', '');
|
||||||
if (['overview','clients','filtres','social','ads','config'].includes(initial)) switchTab(initial);
|
if (['overview','clients','filtres','social','ads','reseau','config'].includes(initial)) switchTab(initial);
|
||||||
|
|
||||||
refreshAll();
|
refreshAll();
|
||||||
setInterval(() => {
|
setInterval(() => {
|
||||||
|
|
|
||||||
167
wiki/Netboot-Install.md
Normal file
167
wiki/Netboot-Install.md
Normal file
|
|
@ -0,0 +1,167 @@
|
||||||
|
# Netboot Install — MOCHAbin from gk2
|
||||||
|
|
||||||
|
Network-boot a second MOCHAbin (c3box) from gk2's netboot rig and perform a
|
||||||
|
fully automated, cryptographically verified SecuBox Debian install to eMMC.
|
||||||
|
|
||||||
|
Validated 2026-06-27 (factory U-Boot 2020.10, custom kernel 6.12.85 #5secubox,
|
||||||
|
image `secubox-mochabin-bookworm` CI run 27426515472).
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
| Item | Detail |
|
||||||
|
|------|--------|
|
||||||
|
| Cable | Straight copper RJ45 from gk2 **lan1** to c3box **mvpp2-2** (the single copper port, not the 4-port switch) |
|
||||||
|
| Signing key | `secubox-netboot.key` on gk2 (matches `netboot-image.pub` embedded in the installer FIT) |
|
||||||
|
| Image | `secubox-mochabin-bookworm` artifact from CI (latest successful build-packages + build-image run) |
|
||||||
|
| gk2 services | dnsmasq DHCP on `192.168.77.0/24`, TFTP root `/srv/tftp`, nginx `:8099` serving signed image |
|
||||||
|
| Serial console | USB-UART to c3box debug header (115200 8N1) to catch factory U-Boot |
|
||||||
|
|
||||||
|
> **Port note**: Only `mvpp2-2` (the single copper RJ45) is reachable by the factory U-Boot.
|
||||||
|
> The 4 switch ports require the MV88E6XXX DSA driver which is absent at early boot.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 1 — Publish the signed image on gk2
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Sign the image
|
||||||
|
openssl dgst -sha256 -sign /etc/secubox/netboot/secubox-netboot.key \
|
||||||
|
-out /srv/netboot/sbx.img.gz.sig /data/sbx.img.gz
|
||||||
|
|
||||||
|
# Publish (symlink or copy into the nginx root)
|
||||||
|
ln -sf /data/sbx.img.gz /srv/netboot/sbx.img.gz
|
||||||
|
ln -sf /data/sbx.img.gz.sig /srv/netboot/sbx.img.gz.sig
|
||||||
|
|
||||||
|
# Verify the public key matches what the installer FIT embeds
|
||||||
|
openssl rsa -in /etc/secubox/netboot/secubox-netboot.key -pubout \
|
||||||
|
| diff - /etc/secubox/netboot/netboot-image.pub
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 2 — Catch the factory U-Boot on c3box
|
||||||
|
|
||||||
|
Power-cycle c3box. On the serial console you have **2 seconds** (`bootdelay=2`).
|
||||||
|
|
||||||
|
> **Important**: press **Enter** to interrupt autoboot — NOT Ctrl-C.
|
||||||
|
|
||||||
|
```
|
||||||
|
Hit any key to stop autoboot: 2
|
||||||
|
Marvell>>
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 3 — Configure TFTP environment
|
||||||
|
|
||||||
|
```
|
||||||
|
# Set TFTP block size for fast transfer
|
||||||
|
setenv tftpblocksize 1468
|
||||||
|
|
||||||
|
# Set the TFTP server (gk2 lan1 address)
|
||||||
|
setenv serverip 192.168.77.1
|
||||||
|
setenv ipaddr 192.168.77.100
|
||||||
|
|
||||||
|
# Save (optional — env is NOT in SPI mtd2 on this board, lost on power cycle)
|
||||||
|
# saveenv
|
||||||
|
```
|
||||||
|
|
||||||
|
> **U-Boot env note**: `fw_setenv` from Linux has no effect on this board — the factory
|
||||||
|
> U-Boot does not store its env in SPI mtd2 (a stale foreign env sits there). All env
|
||||||
|
> changes must be made interactively in the U-Boot shell.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 4 — Load installer via TFTP and boot
|
||||||
|
|
||||||
|
```
|
||||||
|
# Load kernel, DTB, initrd into free RAM addresses (NOT 0x02080000 — reserved)
|
||||||
|
tftpboot 0x0a000000 Image
|
||||||
|
tftpboot 0x09000000 armada-7040-mochabin.dtb
|
||||||
|
tftpboot 0x10000000 initrd.img
|
||||||
|
|
||||||
|
# Boot
|
||||||
|
booti 0x0a000000 0x10000000 0x09000000
|
||||||
|
```
|
||||||
|
|
||||||
|
The installer rescue shell appears. The 49 MB signed FIT is fetched automatically
|
||||||
|
from `http://192.168.77.1:8099/` during installer init.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 5 — Automated verified install from the rescue shell
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Download image to RAM (c3box has 8 GB — fits comfortably)
|
||||||
|
wget http://192.168.77.1:8099/sbx.img.gz -O /tmp/sbx.img.gz
|
||||||
|
wget http://192.168.77.1:8099/sbx.img.gz.sig -O /tmp/sbx.img.gz.sig
|
||||||
|
|
||||||
|
# Verify signature against the embedded public key
|
||||||
|
openssl dgst -sha256 -verify /etc/secubox/installer/netboot-image.pub \
|
||||||
|
-signature /tmp/sbx.img.gz.sig /tmp/sbx.img.gz
|
||||||
|
# Expected output: Verified OK
|
||||||
|
|
||||||
|
# Stream to eMMC (progress tracked via status=progress)
|
||||||
|
gunzip -c /tmp/sbx.img.gz | dd of=/dev/mmcblk0 bs=4M conv=fsync status=progress
|
||||||
|
sync
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Step 6 — Fix auto-boot persistence (boot.scr)
|
||||||
|
|
||||||
|
The image ships `extlinux.conf` pointing to `0x02080000` (reserved on factory U-Boot —
|
||||||
|
causes an immediate reset). It also ships no compiled `boot.scr`. Build one before rebooting:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# From the rescue shell, after dd completes, mount the boot partition
|
||||||
|
mount /dev/mmcblk0p1 /mnt
|
||||||
|
|
||||||
|
cat > /tmp/boot.cmd << 'EOF'
|
||||||
|
setenv bootargs "console=ttyS0,115200 earlycon root=LABEL=rootfs rw"
|
||||||
|
load mmc 0:1 0x0a000000 Image
|
||||||
|
load mmc 0:1 0x10000000 initrd.img
|
||||||
|
load mmc 0:1 0x09000000 armada-7040-mochabin.dtb
|
||||||
|
booti 0x0a000000 0x10000000:${filesize} 0x09000000
|
||||||
|
EOF
|
||||||
|
|
||||||
|
mkimage -C none -A arm64 -T script -d /tmp/boot.cmd /mnt/boot.scr
|
||||||
|
sync
|
||||||
|
umount /mnt
|
||||||
|
```
|
||||||
|
|
||||||
|
Reboot. The factory U-Boot picks up `boot.scr` from mmc and boots Debian unattended.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Gotchas
|
||||||
|
|
||||||
|
| Symptom | Root cause | Fix |
|
||||||
|
|---------|-----------|-----|
|
||||||
|
| Board resets immediately after kernel load | Kernel loaded at `0x02080000` (reserved memory region) | Use `0x0a000000` for kernel, `0x10000000` for initrd |
|
||||||
|
| TFTP very slow | Default block size too small | `setenv tftpblocksize 1468` |
|
||||||
|
| `fw_setenv` from Linux has no effect | Factory U-Boot env is NOT in SPI mtd2 (stale foreign env there) | Reconfigure U-Boot env interactively each time, or use boot.scr |
|
||||||
|
| Autoboot not interrupted | Interrupt key is **Enter**, not Ctrl-C | Press Enter within 2 seconds of power-on |
|
||||||
|
| Switch ports (4-port block) unreachable at boot | MV88E6XXX DSA driver absent in factory U-Boot | Cable must go to `mvpp2-2` (single copper RJ45) |
|
||||||
|
| Image boots but SecuBox stack not starting | First boot — wait ~90 s for systemd units to settle | Check `journalctl -u secubox-aggregator` |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## eMMC layout after install
|
||||||
|
|
||||||
|
| Partition | Type | Mount | Purpose |
|
||||||
|
|-----------|------|-------|---------|
|
||||||
|
| mmcblk0p1 | FAT32 | `/boot` | Kernel, DTB, initrd, boot.scr, extlinux.conf |
|
||||||
|
| mmcblk0p2 | ext4 | `/` | Root filesystem |
|
||||||
|
| mmcblk0p3 | ext4 | `/data` | Persistent data |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Related
|
||||||
|
|
||||||
|
- [[ARM-Installation]] — general ARM installation guide
|
||||||
|
- [[Hardware-Matrix]] — board support matrix
|
||||||
|
- `feature/748-enhanced-tow-boot-http-netboot-serial-fl` — enhanced Tow-Boot with native HTTP/wget (blocked, see #748)
|
||||||
|
- GitHub: [#748](https://github.com/CyberMind-FR/secubox-deb/issues/748) · [#737](https://github.com/CyberMind-FR/secubox-deb/issues/737)
|
||||||
|
|
@ -27,6 +27,7 @@
|
||||||
* [[Live-USB]] | [FR](Live-USB-FR) | [DE](Live-USB-DE) | [中文](Live-USB-ZH)
|
* [[Live-USB]] | [FR](Live-USB-FR) | [DE](Live-USB-DE) | [中文](Live-USB-ZH)
|
||||||
* [[Installation]] | [FR](Installation-FR) | [DE](Installation-DE) | [中文](Installation-ZH)
|
* [[Installation]] | [FR](Installation-FR) | [DE](Installation-DE) | [中文](Installation-ZH)
|
||||||
* [[ARM-Installation]] | [FR](ARM-Installation-FR) | [DE](ARM-Installation-DE) | [中文](ARM-Installation-ZH)
|
* [[ARM-Installation]] | [FR](ARM-Installation-FR) | [DE](ARM-Installation-DE) | [中文](ARM-Installation-ZH)
|
||||||
|
* [[Netboot-Install]] 🔧 MOCHAbin netboot + signed install
|
||||||
* [[ESPRESSObin]] | [FR](ESPRESSObin-FR) | [DE](ESPRESSObin-DE) | [中文](ESPRESSObin-ZH)
|
* [[ESPRESSObin]] | [FR](ESPRESSObin-FR) | [DE](ESPRESSObin-DE) | [中文](ESPRESSObin-ZH)
|
||||||
* [[Eye-Remote]] 📡
|
* [[Eye-Remote]] 📡
|
||||||
* [[Android-ToolBox]] 📱 one-tap R3
|
* [[Android-ToolBox]] 📱 one-tap R3
|
||||||
|
|
|
||||||
Loading…
Reference in New Issue
Block a user