Compare commits

...

7 Commits

Author SHA1 Message Date
3064fb61fd feat(toolbox): Phase 11.C — consent probe + bilingual FR/EN evidence PDF (ref #508)
Completes Phase 11 (social mapping per device): A backend + B d3 view + C evidence/PDF.

  - social_graph.py: consent-platform probe (OneTrust/Didomi/Quantcast/
    Sourcepoint cookies + loader URLs) per (peer,site). Edges stamped
    consent_state none_seen|pre_consent|post_consent. pre_consent =
    tracker fired while site runs a CMP but no consent cookie yet
    (RGPD art. 6.1.a + 7).
  - social.py: consent_state column + GeoIP node columns + EU/EEA
    whitelist + GeoIP fold + evidence() (carried from 55626e51).
  - social_report.py: NEW bilingual FR/EN evidence PDF via fpdf2.
    Robust _mc() helper (set_x + epw) avoids fpdf 'not enough
    horizontal space'. Fact-only: cover + summary + pre-consent table
    + extra-EU table + RGPD article refs. Text fallback if no fpdf2.
  - api.py: GET /social/report/{token}.pdf (HMAC-token gated).
  - social_view.html.j2 + i18n + css: real PDF download button,
    active evidence-card summary.
  - changelog: 2.6.3-1~bookworm1.

Live on gk2: PDF 200 / 41870 bytes / valid PDF v1.3 via uvicorn + kbin.
Consent state machine verified (pre->post->none). Rebased on #513 so
the toolbox sub-tabs ship together.
2026-06-10 13:03:43 +02:00
a7183b643e wip(toolbox): Phase 11.C backend schema + GeoIP fold + evidence helper (ref #508)
Checkpoint — backend evidence layer scaffolded.  Frontend wire + PDF
generator + addon consent probe land in follow-up commits.

  - social.py schema : social_edges.consent_state column ; social_nodes
    gains country_iso / asn_org / eu_inside / pre_consent_hits.
  - Idempotent _migrate_phase11c() ALTERs the pre-existing tables on
    2.6.0 → 2.6.x upgrades (no destructive recreate).
  - _EU_EEA_ISO whitelist (27 EU + 3 EFTA + UK adequacy) + is_eu_iso().
  - _geo_for() LRU-cached (4096) wrapper around the existing geo
    module ; fold time populates the GeoIP fields on every node row.
  - record_edge() accepts consent_state (default 'none_seen') ; fold
    accumulates pre_consent_hits into the per-node aggregate.
  - evidence(mac_hash) helper : returns the two legal-grade buckets
    (pre_consent + extra_eu) consumed by the PDF in the next commit.

Pivoting to admin tab routing per user request — Phase 11.C resumes
after that lands.
2026-06-10 12:58:47 +02:00
19f9ee21a7 feat(toolbox): WebUI sub-tab nav + remove redundant kbin /admin/ inline UI (ref #513)
Consolidates the two toolbox admin surfaces onto the canonical
admin.gk2.secubox.in/toolbox/ WebUI.

  - www/toolbox/index.html: 5-tab nav (Vue d'ensemble / Clients /
    Filtres MITM / Cartographie sociale / Config). Lazy-load per tab,
    live refresh only polls the visible tab, URL-hash deep-links.
    Clients tab folds in the R0-R3 level switcher from the old kbin
    admin UI. New Cartographie sociale tab surfaces the Phase 11
    operator aggregate (KPI + top trackers + anonymized clients).
  - api.py: DELETED the inline kbin /admin/ HTML route (admin_index,
    ~230 lines). All /admin/* JSON API routes preserved.
  - conf/landing.html.j2: removed the Admin quicknav icon.
  - debian/changelog: 2.6.1 -> 2.6.2.

Live on gk2: /toolbox/ 200 (19853 bytes, 5 tabs), kbin /admin/ now 404
by design, social-aggregate shows 225 trackers / 3 clients, all tab
endpoints (clients/rich, filter-control/list, social-aggregate) 200.
2026-06-10 12:50:38 +02:00
bc0905af70 fix: converge /var/log/secubox postinsts to 0755 traversable (closes #511)
Shared-directory ownership race: secubox-toolbox + secubox-mesh created
/var/log/secubox at 0750 with their own module owner, secubox-admin at
750 root:secubox, while ~10 other packages use the correct 0755
secubox:secubox.  Whichever postinst ran last won — and when toolbox or
mesh won, the aggregator (user secubox) lost traversal and the /waf/ +
/soc/ dashboards went blank (regressed on gk2 2026-06-10).

  - secubox-toolbox/debian/postinst:47  0750 -> 0755 (+ guard comment)
  - secubox-mesh/debian/postinst:23     0750 -> 0755
  - secubox-admin/debian/postinst:11    750  -> 0755

Mode 0755 makes the shared parent world-traversable regardless of which
package's owner wins.  Per-module log files + subdirs inside keep their
own restricted perms.  Same class as the /etc/secubox traversal
constraint + the /usr/share/secubox/www chmod (#507).
2026-06-10 10:04:42 +02:00
8b4874e8ed docs: update WIP/HISTORY/TODO for Phase 11 A+B + gk2 system triage + v2.13.14
- Phase 11 social mapping A (#506/2.6.0) + B (#507/2.6.1) deployed live;
    C (#508) checkpointed at 55626e51.
  - gk2 triage: CrowdSec bouncer tables, /var/log/secubox traversal,
    WAF double-buffer cache (#509/1.2.2), PeerTube/PhotoPrism restart.
  - CI: espressobin disable (#504), WAF perf (#510) merged, v2.13.14 tagged.
  - Carried: Round Eye link, toolbox tab decision, var/log postinst patch.
2026-06-10 10:04:42 +02:00
2c1d88b942 perf(waf): double-buffered cache for /waf/stats (memory + disk) with incremental log reading (closes #509)
User reported /waf/ + /soc/ dashboards showing empty threats and tracked
attackers cards.  Live triage on gk2 found _get_threat_stats() iterating
all 332k JSONL entries in the 110 MB waf-threats.log on every request,
pinning the aggregator at 89% CPU with 8+ concurrent open handles.

  - api/main.py: _load_stats_disk_cache + _save_stats_disk_cache helpers
    persist counters + last-read byte position to
    /var/lib/secubox/waf/stats-disk-cache.json (atomic .tmp -> rename so
    a crash mid-write never corrupts).
  - api/main.py: _get_threat_stats reseats the file at the saved byte
    position, reads only the new tail since then, persists the new
    position before returning.
  - api/main.py: log rotation / truncation detected via size shrink;
    counters reset cleanly.
  - api/main.py: day rollover resets threats_today only; cumulative
    counters survive.
  - api/main.py: ip_countries dict capped at 1200 entries (most-active
    1000 retained on overflow) so the cache file stays small.

Live on gk2 after deploy: /waf/stats steady-state 30-37 ms (vs 30s+
timeout before).  /waf/alerts, /waf/bans/history, /soc/* all 200 in
under 200 ms.  Aggregator CPU drops from 89% to 52% under same load.

  - debian/changelog: 1.2.1 -> 1.2.2-1~bookworm1.
2026-06-10 10:04:42 +02:00
76eee956c9 ci: drop espressobin-v7 + espressobin-ultra from scheduled image matrix (ref #503)
The two boards fail in the cross-arm64 chroot stage of build-image.yml
and (even with fail-fast: false) block the downstream release.yml job
from publishing the OTHER boards' images.  Releases v2.13.9 through
v2.13.12 all hit this trap.

This change keeps both entries available via workflow_dispatch (operators
can still build them on-demand), but removes them from the push-tag
scheduled matrix so mochabin / vm-x64 / rpi400 actually ship on tag.

Board support files (board/espressobin-*/, image/build-image.sh
--board espressobin-*) stay in tree.

  - build-image.yml matrix excludes espressobin-v7 + espressobin-ultra
    on push and workflow_call.
  - build-image.yml workflow_dispatch choice list keeps both entries
    flagged as on-demand only.
  - Release notes template drops the two image rows, adds a note
    explaining the on-demand path.
  - Install instruction adjusted (MOCHAbin + Raspberry Pi 400).
2026-06-10 10:04:42 +02:00
20 changed files with 1234 additions and 351 deletions

View File

@ -3,6 +3,79 @@
---
## 2026-06-10 — Phase 11 social mapping (A+B) + system triage + v2.13.14 (ref #502-#509)
### Package bumps
| Package | from → to |
|---|---|
| secubox-toolbox | 2.5.2 → **2.6.0** (#505 Phase 11.A backend) |
| secubox-toolbox | 2.6.0 → **2.6.1** (#507 Phase 11.B frontend) |
| secubox-waf | 1.2.1 → **1.2.2** (#509 double-buffer cache) |
| Release tag | **v2.13.14** |
### Phase 11 — Social mapping per device (#502)
**11.A backend** (`secubox-toolbox 2.6.0`, PR #506) — `social.py`
correlation engine + 3 SQLite tables (`social_edges` / `social_nodes`
/ `social_links`), `social_graph.py` mitm addon (cookie_id_hash =
sha256, never persists raw values), `/social/graph/{token}` +
`/social/wipe/{token}` (RGPD art. 17) + `/admin/social-aggregate`
endpoints, fold + purge background tasks.
**11.B frontend** (`secubox-toolbox 2.6.1`, #507) — d3 force-directed
graph view at `/social/{token}`, FR/EN i18n, server-side favicon proxy
(7d cache), wipe modal with 3s countdown, full-viewport layout with
pan/pinch-zoom + pre-warm + autoFit. Splash menu link `/social/me`
(🕸️ Ma carto) resolving R3 peers via X-R3-Peer sentinel.
**Live result** : graph renders real cross-site tracking on gk2 — the
ad-tech relay `35.214.136.108` bridging 360yield + seedtag +
smartadserver + smilewanted publishers, surfacing exactly the
fingerprint reuse Phase 11 targets.
**Critical live-deploy fixes** : addon relative-import never resolved
(mitmproxy loads addons top-level) → inlined; **PYTHONPATH missing in
mitm-wg launcher** silently degraded every addon's `secubox_toolbox`
imports → fixed globally (also un-degraded inject_banner's host
classification + GeoIP); i18n moved to `<script>` block (FR
apostrophes broke JSON.parse); StaticFiles mount + chmod 0755 www
(kbin HAProxy path bypasses nginx).
**11.C** (#508) — WIP checkpoint `55626e51` : schema (consent_state +
GeoIP columns), EU/EEA whitelist, GeoIP fold enrichment, evidence()
helper. PDF generator + consent-probe addon + frontend wire pending.
### System triage on gk2
- **CrowdSec firewall** — bouncer ran healthy but had no nft tables
(external flush). Restart recreated `ip crowdsec` + `ip6 crowdsec6`,
100 live decisions.
- **WAF + SOC empty cards**`/var/log/secubox` was 0750
secubox-toolbox, blocking the aggregator (user `secubox`) from
traversing to read `waf-threats.log`. chmod 0755 live.
- **WAF /stats 30s+ timeout**`_get_threat_stats()` re-parsed the full
110 MB / 332k-entry JSONL on every request (89% aggregator CPU).
Fixed via #509 double-buffered cache : disk-persisted counters +
byte-position incremental tail reading. `/waf/stats` now 30-37 ms.
- **PeerTube + PhotoPrism 502** — LXCs were STOPPED; `lxc-start` → live.
### CI + release
- #503/PR #504 — drop espressobin-v7 + ultra from the scheduled
build-image matrix (cause of the v2.13.9-12 release failures).
- #509/PR #510 — double-buffer WAF cache.
- Merged both to master (`3ebb4477`, `a6f44807`), tagged **v2.13.14**.
### Carried forward
- Round Eye gadget remote-link to gk2 (shows local metrics only) —
needs Pi-side investigation.
- admin.gk2/toolbox/ tab surfacing decision (proxy/iframe/sub-tab).
- `/var/log/secubox` 0755 source-side postinst patch (live-only for now).
---
## 2026-06-09 — Phase 10 banner injection perf quick wins + postinst regression fix (ref #501)
### Package bumps

View File

@ -1,10 +1,40 @@
# TODO — SecuBox-DEB Backlog
*Mis à jour : 2026-06-09*
*Mis à jour : 2026-06-10*
---
## 🔥 P0 — Immediate (in flight)
### Phase 11 — Social mapping per device (#502)
- [x] **11.A backend** (#505 / PR #506, `secubox-toolbox 2.6.0`) — correlation
engine + SQLite + API. Déployé live gk2.
- [x] **11.B frontend** (#507, `2.6.1`) — d3 graph + i18n FR/EN + favicon
proxy + wipe modal + full-viewport pan/zoom. Live `/social/me`.
- [ ] **11.C evidence + PDF** (#508) — reprendre depuis checkpoint
`55626e51` : consent-probe addon (OneTrust/Didomi/Quantcast/Sourcepoint)
+ extra-EU flag + PDF bilingue FR/EN + wire frontend (remplacer le
placeholder "Rapport PDF arrive en Phase 11.C").
- [ ] **Merger PR #506** (11.A backend) → master quand prêt.
- [ ] **11.D opérateur** (futur) — dashboard agrégat `/admin/social/` HTML
(l'endpoint JSON `/admin/social-aggregate` existe déjà depuis 2.6.0).
### Système — bugs gk2 (2026-06-10)
- [x] **CrowdSec firewall** — restart bouncer → tables nft recréées.
- [x] **WAF /var/log/secubox traversal** — chmod 0755 live.
- [x] **WAF /stats perf** (#509 / PR #510, `secubox-waf 1.2.2`) — double-buffer
cache. Mergé + `v2.13.14`.
- [x] **PeerTube + PhotoPrism** — LXC redémarrés.
- [ ] **Round Eye gadget** — ne voit plus gk2, métriques locales only.
Investigation Pi Zero (dashboard `localhost:8000` proxie vers gk2 via OTG).
- [ ] **admin.gk2/toolbox/ tab** — toolbox déjà wiré (`/toolbox/` alias +
sidebar). User veut surfacer l'UI kbin/admin dedans — décision en
attente : proxy_pass `10.99.0.1:8088/admin/` (cleanest) / iframe (CSP) /
sous-tab frontend-only.
- [ ] **Postinst `/var/log/secubox` 0755** — porter le fix live en source
(même pattern que `/etc/secubox` parent + `/usr/share/secubox/www`).
### Phase 10 — Banner injection perf (#501) — ✅ shipped 2026-06-09
- [x] **Banner perf quick wins** (`secubox-toolbox` 2.5.1, commit `ce059d0f`)

View File

@ -1,5 +1,73 @@
# WIP — Work In Progress
*Mis à jour : 2026-06-09*
*Mis à jour : 2026-06-10*
---
## 🔄 2026-06-10 : Phase 11 social mapping (A+B) + system triage round (ref #502-#509)
Grosse journée : Phase 11 social mapping shippé jusqu'au frontend live,
puis une cascade de fixes système découverts par l'utilisateur en
production sur gk2.
### ✅ Done — Phase 11 social mapping (#502 parent)
| Issue | Phase | État |
|---|---|---|
| #505 / PR #506 | **11.A backend** : correlation engine + SQLite + API | ✅ mergeable, déployé live `secubox-toolbox 2.6.0` |
| #507 | **11.B frontend** : d3 graph + i18n FR/EN + favicon proxy + wipe modal | ✅ déployé live `2.6.1`, branche poussée |
| #508 | **11.C evidence + PDF** | 🔄 WIP checkpoint `55626e51` (schema + GeoIP fold + evidence helper) |
**Design** : 2 rounds de design lock sur #502 (Gemini + GPT mockups),
edge-thickness + animated-pulse + tracker bottom-sheet + 3s wipe
countdown verrouillés.
**Live URL** : `https://kbin.gk2.secubox.in/social/me` (splash → 🕸️ Ma carto).
Le graphe montre les trackers cross-site réels (relais ad-tech
`35.214.136.108` reliant 360yield + seedtag + smartadserver + smilewanted).
**Fixes live-deploy critiques découverts** :
- `social_graph.py` : `from . import local_store` ne résolvait jamais
(mitmproxy charge les addons en top-level) → inliné le WG peer hash.
- **PYTHONPATH manquant dans le launcher mitm-wg** : TOUS les addons
(`inject_banner` dpi/geo/store, `social_graph`) avaient leurs
`from secubox_toolbox import …` silencieusement dégradés. Fix global.
- i18n déplacé de `data-*` attr vers `<script>` (apostrophes FR
cassaient `JSON.parse`).
- StaticFiles mount + chmod 0755 `/usr/share/secubox/www` (kbin passe
par HAProxy direct uvicorn, bypass nginx).
- d3 : full-viewport + pan/pinch-zoom + pre-warm 300 ticks + autoFit
data-based (146 nodes spread off-screen avant).
### ✅ Done — triage système gk2 (2026-06-10)
| Bug | Cause racine | Fix |
|---|---|---|
| CrowdSec firewall status faux | bouncer tournait mais sans tables nft (flush externe) | restart bouncer → `ip crowdsec` + `ip6 crowdsec6` recréées, 100 décisions live |
| WAF /threats + tracked attackers vides | `/var/log/secubox` 0750 secubox-toolbox bloquait traversal aggregator (user `secubox`) | chmod 0755 live |
| WAF /stats timeout 30s+ | `_get_threat_stats()` re-parsait 110 MB / 332k JSONL à CHAQUE requête (CPU 89%) | **#509 double-buffer cache** (disk + byte-position incrémental) `secubox-waf 1.2.2` |
| SOC /soc/ status WAF+firewall faux | consommait les mêmes endpoints WAF cassés | résolu en cascade par le fix WAF |
| PeerTube + PhotoPrism 502 | LXC STOPPED | `lxc-start` → 200 / 307 |
### ✅ Done — CI + release
- **#503 / PR #504** : drop espressobin-v7 + ultra du matrix build-image
scheduled (faisaient échouer le pipeline release v2.13.9-12).
- **#509 / PR #510** : double-buffer WAF cache.
- **Merge #504 + #510 → master** (`3ebb4477`, `a6f44807`).
- **Tag `v2.13.14`** poussé.
### ⬜ Next up
- **Round Eye gadget** : ne voit plus le lien gk2, montre ses métriques
locales. iface `eye-remote` UP côté gk2, route `/api/v1/eye-remote/*`
renvoie page erreur. Investigation côté Pi Zero nécessaire.
- **admin.gk2/toolbox/ tab** : le toolbox est DÉJÀ wiré (`/toolbox/`
alias + sidebar). User veut surfacer l'UI kbin/admin dedans —
décision en attente : proxy_pass / iframe / sous-tab.
- **Phase 11.C** : reprendre depuis `55626e51` (consent probe addon +
extra-EU flag + PDF bilingue + wire frontend).
- **Postinst patch** : `/var/log/secubox` 0755 en source (pour l'instant
fix live uniquement) — même pattern que `/etc/secubox` + `www`.
---

View File

@ -24,8 +24,8 @@ on:
type: choice
options:
- mochabin
- espressobin-v7
- espressobin-ultra
- espressobin-v7 # on-demand only — disabled in scheduled CI, ref #503
- espressobin-ultra # on-demand only — disabled in scheduled CI, ref #503
- vm-x64
- vm-arm64
- rpi400
@ -50,7 +50,12 @@ jobs:
fail-fast: false
matrix:
# Handle all event types: push (tags), workflow_call, workflow_dispatch
board: ${{ (github.event_name == 'push' || (inputs.board == 'all' || inputs.board == '')) && fromJson('["mochabin","espressobin-v7","espressobin-ultra","vm-x64","rpi400"]') || (github.event.inputs.board == 'all' && fromJson('["mochabin","espressobin-v7","espressobin-ultra","vm-x64","rpi400"]') || fromJson(format('["{0}"]', inputs.board || github.event.inputs.board || 'vm-x64'))) }}
# Scheduled / tag-push matrix excludes espressobin-v7 + espressobin-ultra (#503) :
# those board builds fail in the cross-arm64 chroot stage and block the
# downstream release.yml job for every image even though fail-fast is off.
# Operators can still build them on-demand via workflow_dispatch (the
# choice list above retains the entries).
board: ${{ (github.event_name == 'push' || (inputs.board == 'all' || inputs.board == '')) && fromJson('["mochabin","vm-x64","rpi400"]') || (github.event.inputs.board == 'all' && fromJson('["mochabin","vm-x64","rpi400"]') || fromJson(format('["{0}"]', inputs.board || github.event.inputs.board || 'vm-x64'))) }}
steps:
- name: Checkout
@ -221,15 +226,17 @@ jobs:
| Image | Board | Architecture | Description |
|-------|-------|--------------|-------------|
| `secubox-mochabin-bookworm.img.gz` | MOCHAbin | arm64 | Marvell Armada 7040 (Pro) |
| `secubox-espressobin-v7-bookworm.img.gz` | ESPRESSObin v7 | arm64 | Marvell Armada 3720 (Lite) |
| `secubox-espressobin-ultra-bookworm.img.gz` | ESPRESSObin Ultra | arm64 | Marvell Armada 3720 (Lite+) |
| `secubox-rpi400-bookworm.img.gz` | Raspberry Pi 400 | arm64 | Pi 400 / Pi 4 |
| `secubox-vm-x64-bookworm.img.gz` | VirtualBox/QEMU | amd64 | VM for testing |
| `create-qemu-arm64-vm.sh` | QEMU ARM64 | script | Run ARM64 on x86 hosts |
*ESPRESSObin v7 and Ultra board images are no longer published in
scheduled releases (see #503). Board support remains in tree and
on-demand builds are available via workflow_dispatch.*
### Installation
**ARM64 boards (MOCHAbin, ESPRESSObin):**
**ARM64 boards (MOCHAbin, Raspberry Pi 400):**
```bash
# Flash to SD card or eMMC
gunzip -c secubox-mochabin-bookworm.img.gz | sudo dd of=/dev/sdX bs=4M status=progress

View File

@ -8,7 +8,7 @@ case "$1" in
install -d -o secubox -g secubox -m 750 /run/secubox
install -d -o secubox -g secubox -m 750 /var/lib/secubox
install -d -o secubox -g secubox -m 750 /var/lib/secubox/admin
install -d -o root -g secubox -m 750 /var/log/secubox
install -d -o root -g secubox -m 0755 /var/log/secubox
systemctl daemon-reload
systemctl enable secubox-admin.service
systemctl start secubox-admin.service || true

View File

@ -20,7 +20,7 @@ case "$1" in
# NE PAS le toucher ici — l'écraser bloque la traversée nginx (www-data) et
# casse tous les /api/v1/<module>/* en 502 (cf. #471). Si besoin d'un
# sous-dossier privé, utiliser /run/secubox/mesh/ (et non le parent).
install -d -m 0750 -o secubox-mesh -g secubox-mesh /var/log/secubox
install -d -m 0755 -o secubox-mesh -g secubox-mesh /var/log/secubox
# 4. Verrou régulatoire FR (idempotent ; ne pas planter si iw absent)
if command -v iw >/dev/null 2>&1; then

View File

@ -28,5 +28,7 @@
"wipe_success": "Your data has been erased. {n} rows deleted.",
"loading": "Loading…",
"error": "Loading error.",
"lang_label": "EN"
"lang_label": "EN",
"card_pdf_download": "⬇ Download PDF report (FR/EN)",
"card_evidence_active": "Compliance analysis: trackers before consent (GDPR art. 6.1.a + 7) and extra-EU transfers (art. 44). See the PDF report for details."
}

View File

@ -28,5 +28,7 @@
"wipe_success": "Vos données ont été effacées. {n} enregistrements supprimés.",
"loading": "Chargement…",
"error": "Erreur de chargement.",
"lang_label": "FR"
"lang_label": "FR",
"card_pdf_download": "⬇ Télécharger le rapport PDF (FR/EN)",
"card_evidence_active": "Analyse de conformité : traqueurs avant consentement (RGPD art. 6.1.a + 7) et transferts hors UE (art. 44). Voir le rapport PDF pour le détail."
}

View File

@ -86,9 +86,6 @@ a:hover{text-decoration:underline}
<a href="/wg/qr.png" class=qi title="QR profil WireGuard">
<span class=qi-emoji>📱</span><span class=qi-label>QR profil</span>
</a>
<a href="/admin/" class=qi title="Admin webui">
<span class=qi-emoji>🛠️</span><span class=qi-label>Admin</span>
</a>
<a href="https://github.com/CyberMind-FR/secubox-deb/wiki/R3-WireGuard-install" class=qi title="Wiki R3 multi-OS">
<span class=qi-emoji>📖</span><span class=qi-label>Wiki</span>
</a>

View File

@ -59,7 +59,7 @@
<nav class="cards-row">
<details class="card">
<summary>{{ t.card_evidence }}</summary>
<p class="card-pending">{{ t.card_evidence_pending }}</p>
<p class="card-pending">{{ t.card_evidence_active }}</p>
</details>
<details class="card card-wipe">
<summary>{{ t.card_wipe }}</summary>
@ -67,7 +67,7 @@
</details>
<details class="card">
<summary>{{ t.card_pdf }}</summary>
<p class="card-pending">{{ t.card_pdf_pending }}</p>
<a class="pdf-btn" href="/social/report/{{ token }}.pdf" target="_blank" rel="noopener">{{ t.card_pdf_download }}</a>
</details>
</nav>
</main>

View File

@ -1,3 +1,53 @@
secubox-toolbox (2.6.3-1~bookworm1) bookworm; urgency=medium
* Phase 11.C (#508, parent #502) — social mapping evidence + bilingual
FR/EN PDF report.
- mitmproxy_addons/social_graph.py : consent-platform probe.
Detects OneTrust / Didomi / Quantcast / Sourcepoint cookies +
loader URLs per (peer, site). Each recorded edge is stamped
consent_state in {none_seen, pre_consent, post_consent}.
pre_consent = a tracker fired while the site runs a CMP but no
consent cookie was seen yet (RGPD art. 6.1.a + 7 evidence).
- secubox_toolbox/social.py : schema (consent_state + GeoIP
columns) + EU/EEA whitelist + GeoIP fold + evidence() helper.
- secubox_toolbox/social_report.py : NEW — bilingual FR/EN
evidence PDF via fpdf2 (same engine as reports.py). Fact-only
cover + summary + pre-consent table + extra-EU table + RGPD
article references. Text fallback when fpdf2 is absent.
- secubox_toolbox/api.py : GET /social/report/{token}.pdf, same
HMAC-token gate as /social/graph/{token}.
- conf/social_view.html.j2 + i18n : the "Rapport PDF" card now
has a real download button ; the "Évidence juridique" card
switched to the active compliance summary.
Phase 11 (social mapping per device) is now feature-complete :
A (backend) + B (d3 view) + C (evidence + PDF).
-- Gerald KERMA <devel@cybermind.fr> Wed, 10 Jun 2026 09:30:00 +0200
secubox-toolbox (2.6.2-1~bookworm1) bookworm; urgency=medium
* ToolBox WebUI sub-tab navigation + kbin /admin/ removal (#513).
- www/toolbox/index.html : rebuilt with a 5-tab nav
(Vue d'ensemble / Clients / Filtres MITM / Cartographie
sociale / Config). Each tab lazy-loads its data; the live
refresh interval only polls the visible tab. URL hash
deep-links a tab (e.g. /toolbox/#social).
- Clients tab folds in the R0/R1/R2/R3 level switcher that
used to live only in the kbin inline admin UI (calls
/admin/clients/{mac}/level).
- Cartographie sociale tab surfaces the Phase 11 operator
aggregate (/admin/social-aggregate) : KPI tiles + top
tracker domains + anonymized client table.
- api.py : DELETED the inline kbin /admin/ HTML route
(admin_index(), ~230 lines). The canonical operator
dashboard is now admin.gk2.secubox.in/toolbox/. All
/admin/* JSON API routes are untouched.
- conf/landing.html.j2 : removed the 🛠️ Admin quicknav icon
(the page it pointed at no longer exists).
kbin.gk2.secubox.in/admin/ now 404s by design.
-- Gerald KERMA <devel@cybermind.fr> Wed, 10 Jun 2026 09:15:00 +0200
secubox-toolbox (2.6.1-1~bookworm1) bookworm; urgency=medium
* Phase 11.B (#507, parent #502) — social mapping per-client view +

View File

@ -44,7 +44,13 @@ case "$1" in
# 4. Storage dir (SQLite + future PDF reports)
install -d -m 0750 -o secubox-toolbox -g secubox-toolbox /var/lib/secubox/toolbox
install -d -m 0750 -o secubox-toolbox -g secubox-toolbox /var/log/secubox
# /var/log/secubox is a SHARED parent traversed by many service users
# (the aggregator runs as `secubox` and reads waf-threats.log under
# here). It MUST be 0755 — a 0750 owned by secubox-toolbox silently
# breaks WAF + SOC dashboards for the `secubox` user (#511, regressed
# the /waf/ + /soc/ pages on gk2 2026-06-10). Per-module log files +
# subdirs inside keep their own restricted perms.
install -d -m 0755 -o secubox-toolbox -g secubox-toolbox /var/log/secubox
# 4a. Phase 11.B (#507) — make /usr/share/secubox/www traversable so
# the FastAPI StaticFiles mount can serve /toolbox/social.{css,js} +

View File

@ -171,6 +171,105 @@ def _ja4_hash(flow) -> Optional[str]:
return None
# ─── consent-platform detection (Phase 11.C #508) ───
#
# We detect the four dominant CMP (Consent Management Platform) cookies.
# Their PRESENCE on a flow means the user has interacted with the consent
# banner on this site at least once. We track, per (peer, site) :
# - whether the site is KNOWN to run a CMP (we've seen the cookie OR the
# loader path), and
# - whether a consent cookie has been observed yet.
# This lets us classify each tracker edge as pre/post/none-consent.
#
# Names are matched case-insensitively as a prefix (OneTrust appends
# region suffixes, Didomi rotates tokens, etc.).
_CMP_COOKIE_PREFIXES = (
"optanonconsent", "onetrustconsent", "optanonalertboxclosed", # OneTrust
"didomi_token", "euconsent-v2", # Didomi / IAB TCF
"__qca", "quantcast", # Quantcast
"sp_choice", "consentuid", "_sp_", # Sourcepoint
)
# CMP loader URL fragments — seeing the site REQUEST one of these proves
# the site runs a consent platform even before the cookie is set.
_CMP_LOADER_FRAGMENTS = (
"cdn.cookielaw.org", "onetrust.com", # OneTrust
"sdk.privacy-center.org", "didomi.io", # Didomi
"quantcast.mgr.consensu.org", "quantcast.com/choice", # Quantcast
"sourcepoint.mgr.consensu.org", "sp-prod.net", # Sourcepoint
)
# Per-(peer, site) consent observation log. Bounded soft-cap : if it
# grows past 20k entries we drop it wholesale (a fresh session rebuild is
# cheap and the salt rotates daily anyway).
_consent_log: dict = {}
def _consent_key(mac_hash: str, site: str) -> tuple:
return (mac_hash, site)
def _update_consent_log(mac_hash: str, src_site: str, flow) -> None:
"""Observe whether this flow reveals a CMP cookie or loader for the
(peer, site) pair, and update the in-memory log."""
try:
if len(_consent_log) > 20000:
_consent_log.clear()
key = _consent_key(mac_hash, src_site)
state = _consent_log.get(key, {"has_cmp": False, "consented": False})
# 1) CMP loader request → the site runs a consent platform.
url = (flow.request.pretty_url or "").lower()
if any(frag in url for frag in _CMP_LOADER_FRAGMENTS):
state["has_cmp"] = True
# 2) CMP cookie present (either direction) → consent recorded.
cookie_blobs = []
cookie_blobs.extend(flow.request.headers.get_all("cookie") or [])
cookie_blobs.extend(flow.response.headers.get_all("set-cookie") or [])
for blob in cookie_blobs:
low = blob.lower()
for pref in _CMP_COOKIE_PREFIXES:
if pref in low:
state["has_cmp"] = True
state["consented"] = True
break
_consent_log[key] = state
except Exception:
pass
def _consent_state_for(mac_hash: str, site: str) -> str:
"""Classify the current consent state for the (peer, site) pair.
Evidence-only semantics (no interpretation beyond the observation) :
post_consent a CMP cookie has been seen here for this peer.
pre_consent the site runs a CMP (loader/cookie seen) but no
consent cookie yet : a tracker firing now fires
before consent.
none_seen no CMP detected at all ; we make no claim.
"""
st = _consent_log.get(_consent_key(mac_hash, site))
if not st:
return "none_seen"
if st.get("consented"):
return "post_consent"
if st.get("has_cmp"):
return "pre_consent"
return "none_seen"
# ─── JA4 lookup ───
def _ja4_hash(flow) -> Optional[str]:
"""Pull the JA4 fingerprint set by the ja4 addon, if present."""
try:
ja4 = (flow.metadata or {}).get("ja4")
if ja4:
return str(ja4)[:32]
except Exception:
pass
return None
# ─── main ───
class SocialGraph:
"""mitmproxy addon : record cookie-bearing edges per R2/R3 peer."""
@ -186,6 +285,20 @@ class SocialGraph:
if not src_site:
return
# Phase 11.C (#508) — consent-state detection. Before recording
# any edge, update our per-peer × per-site consent log : has a
# consent-platform cookie (OneTrust / Didomi / Quantcast /
# Sourcepoint) been observed for this peer on this site yet ?
# The edge's consent_state is then :
# post_consent — a consent cookie was already seen here
# pre_consent — NOT yet seen, AND the site DOES run a consent
# platform somewhere (so a tracker firing now is
# firing before consent — RGPD art. 6.1.a + 7)
# none_seen — no consent platform detected for this site at
# all (we make no claim ; baseline)
_update_consent_log(mac_hash, src_site, flow)
consent_state = _consent_state_for(mac_hash, src_site)
# Set-Cookie headers : the 3rd-party server hands a new identifier.
# The Set-Cookie domain may differ from flow.request.host (Set-Cookie
# `Domain=` attribute) — when present, we trust that for the
@ -211,6 +324,7 @@ class SocialGraph:
tracker_domain=tracker_domain,
cookie_id_hash_val=cid,
ja4_hash=ja4,
consent_state=consent_state,
)
# Request-side Cookie headers (only meaningful when the
@ -233,6 +347,7 @@ class SocialGraph:
# called itself).
return
ctx_consent = _consent_state_for(mac_hash, ctx_site)
for hdr in cookie_hdrs[:5]:
for name, value in _parse_cookie_header(hdr)[:50]:
if _social.is_deny_listed(name):
@ -244,6 +359,7 @@ class SocialGraph:
tracker_domain=tracker_domain,
cookie_id_hash_val=cid,
ja4_hash=ja4,
consent_state=ctx_consent,
)

View File

@ -2085,6 +2085,30 @@ async def admin_social_aggregate(hours: int = 24) -> dict:
return _s.aggregate(hours=hours)
@router.get("/social/report/{token}.pdf")
async def social_report_pdf(token: str) -> Response:
"""Phase 11.C (#508) — bilingual FR/EN evidence PDF for a peer.
Same HMAC-token gate as /social/graph/{token}. Fact-only report :
cross-site tracker reuse, trackers fired before consent (RGPD art.
6.1.a + 7), extra-EU transfers (art. 44+).
"""
from . import social_report as _sr
salt = _get_salt()
ok, mac_hash = reports.verify_token(token, salt)
if not ok:
raise HTTPException(404, "report not found or expired")
data = _sr.build_social_report(mac_hash, since_seconds=7 * 86400)
data["generated_at"] = time.strftime("%Y-%m-%d %H:%M UTC", time.gmtime())
pdf_bytes = _sr.render_social_pdf(data)
fname = f"village3b-carto-{mac_hash[:8]}-{int(time.time())}.pdf"
return Response(
content=pdf_bytes,
media_type="application/pdf",
headers={"Content-Disposition": f'attachment; filename="{fname}"'},
)
# ───────────────── Phase 11.B — per-client view + favicon proxy (#507) ─────────────────
import json as _json_b
@ -2352,235 +2376,9 @@ async def admin_override_level(mac_hash: str, request: Request) -> dict:
"note": "nft sets not auto-updated; client must reload or operator manually adjusts nft"}
@router.get("/admin/", response_class=HTMLResponse)
@router.get("/admin", response_class=HTMLResponse)
async def admin_index() -> HTMLResponse:
"""Operator admin webUI with client list + level switcher."""
html = """<!DOCTYPE html><html lang=fr><head><meta charset=UTF-8>
<meta name=viewport content="width=device-width,initial-scale=1">
<title>🛡 Admin Gondwana ToolBoX</title>
<style>:root{--bg:#0a0a0f;--bg2:#0e0e15;--phos:#00dd44;--phos-hot:#00ff55;--dim:#006622;--text:#e8e6d9;--purple:#9e76ff;--amber:#ffb347;--red:#ff4466}
*{box-sizing:border-box;margin:0;padding:0}
body{font-family:Menlo,Consolas,monospace;background:var(--bg);color:var(--text);padding:1.2rem;max-width:1100px;margin:auto;line-height:1.55}
h1{color:var(--phos-hot);text-shadow:0 0 6px var(--phos);font-size:1.6rem;margin-bottom:0.4rem;letter-spacing:0.05em}
.sub{color:var(--dim);font-size:0.85rem;margin-bottom:1.2rem}
table{width:100%;border-collapse:collapse;font-size:0.85rem;background:var(--bg2);border:1px solid var(--dim)}
th,td{padding:0.5rem 0.6rem;text-align:left;border-bottom:1px solid var(--dim)}
th{color:var(--phos-hot);text-shadow:0 0 3px var(--phos);background:rgba(0,221,68,0.08)}
tr:hover{background:rgba(0,221,68,0.04)}
.chip{display:inline-block;padding:0.15rem 0.5rem;border-radius:99px;font-size:0.72rem;font-weight:bold}
.chip.r0{background:#222;color:#999}
.chip.r1{background:rgba(0,221,68,0.2);color:var(--phos-hot)}
.chip.r2{background:rgba(255,179,71,0.2);color:#ffd6a0}
.chip.r3{background:rgba(158,118,255,0.2);color:#cbb6ff}
.btn{background:var(--purple);color:#0a0a0f;padding:0.3rem 0.6rem;border:none;border-radius:3px;cursor:pointer;font-family:inherit;font-size:0.72rem;font-weight:bold;margin-right:0.2rem}
.btn:hover{background:#b598ff}
.btn.outline{background:transparent;color:var(--phos);border:1px solid var(--phos)}
code{background:#222;padding:0.1rem 0.3rem;border-radius:2px;font-size:0.75rem}
.cards{display:grid;grid-template-columns:repeat(auto-fit,minmax(180px,1fr));gap:0.8rem;margin-bottom:1.5rem}
.card{background:var(--bg2);border:1px solid var(--dim);padding:0.8rem;border-radius:4px;text-align:center}
.card .v{font-size:1.6rem;color:var(--phos-hot);font-weight:bold;display:block}
.card .l{font-size:0.7rem;color:var(--dim)}
.modal-bg{position:fixed;top:0;left:0;right:0;bottom:0;background:rgba(0,0,0,0.8);display:none;align-items:center;justify-content:center;z-index:100}
.modal-bg.show{display:flex}
.modal{background:var(--bg2);border:1px solid var(--phos);padding:1.5rem;border-radius:4px;max-width:400px;width:90%}
.modal h2{color:var(--phos-hot);font-size:1rem;margin-bottom:0.8rem}
.modal .lvl-row{display:grid;grid-template-columns:repeat(4,1fr);gap:0.4rem;margin-bottom:0.8rem}
.modal .lvl-row button{padding:0.5rem;cursor:pointer;font-family:inherit;font-size:0.75rem;border-radius:3px}
.modal .lvl-row .r0{background:#222;color:#999;border:1px solid #444}
.modal .lvl-row .r1{background:rgba(0,221,68,0.15);color:var(--phos-hot);border:1px solid var(--phos)}
.modal .lvl-row .r2{background:rgba(255,179,71,0.15);color:#ffd6a0;border:1px solid var(--amber)}
.modal .lvl-row .r3{background:rgba(158,118,255,0.15);color:#cbb6ff;border:1px solid var(--purple)}
.adm-tabs{display:flex;gap:2px;border-bottom:1px solid var(--dim);margin-bottom:0.8rem}
.adm-tab{background:transparent;border:0;color:#888;padding:0.5rem 1rem;font-family:inherit;font-size:0.85rem;cursor:pointer;border-bottom:2px solid transparent;transition:all 0.15s}
.adm-tab.active{color:var(--phos-hot);border-bottom-color:var(--phos);background:rgba(0,221,68,0.05)}
.adm-tab:hover{color:var(--text)}
.adm-content{display:none}
.adm-content.active{display:block}
button.del-pattern{background:var(--red);color:#fff;border:0;padding:0.2rem 0.5rem;border-radius:3px;cursor:pointer;font-weight:bold;font-size:0.75rem}
</style></head><body>
<h1>🛡 Admin Gondwana ToolBoX</h1>
<p class=sub>// Console opérateur · client management · level override · mitm filtering</p>
{# Phase 6.I : tabbed sections #}
<div class=adm-tabs>
<button class="adm-tab active" data-tab=clients>👥 Clients</button>
<button class="adm-tab" data-tab=filter>🛡 Mitm filtering</button>
</div>
<div class="adm-content active" data-content=clients>
<div id=cards class=cards></div>
<table id=clients-table>
<thead><tr>
<th>Status</th><th>Device</th><th>Hash</th><th>IP</th>
<th>Level</th><th>Risk</th><th>Last seen</th><th>Actions</th>
</tr></thead>
<tbody id=clients-tbody><tr><td colspan=8>chargement</td></tr></tbody>
</table>
</div>
<div class="adm-content" data-content=filter>
<h2 style="color:var(--purple);font-size:1.1rem;margin-bottom:0.4rem">🛡 Mitm bypass whitelist</h2>
<p style="font-size:0.78rem;color:var(--dim);margin-bottom:0.8rem">
Hosts/domains qui ne sont JAMAIS déchiffrés par mitm (TLS passthrough).
Pour apps cert-pinned ou E2E. Redémarre <code>secubox-toolbox-mitm-wg</code>
après modif.
</p>
<form id=add-pattern-form style="display:flex;gap:0.5rem;margin-bottom:0.8rem">
<input type=text name=entry placeholder="ex: (.+\\.)?example\\.com" required
style="flex:1;background:#111;color:var(--text);border:1px solid #2a2a3f;padding:0.5rem;border-radius:3px;font-family:monospace;font-size:0.8rem">
<button type=submit class=btn style="background:var(--phos);padding:0.5rem 1rem"> Ajouter</button>
</form>
<table id=filter-table style="font-size:0.78rem">
<thead><tr><th>Pattern (regex)</th><th width=60>×</th></tr></thead>
<tbody id=filter-tbody><tr><td colspan=2>chargement</td></tr></tbody>
</table>
<p style="font-size:0.72rem;color:var(--dim);margin-top:0.6rem;border-left:2px solid var(--amber);padding-left:0.6rem">
📁 <code>/var/lib/secubox/toolbox/mitm-bypass.conf</code> · Source de vérité
</p>
</div>
<div id=modal class=modal-bg>
<div class=modal>
<h2>🔀 Change level <code id=modal-mh></code></h2>
<p style="font-size:0.78rem;color:var(--dim);margin-bottom:0.5rem"> Admin override : updates store only. Client must reload to sync nft.</p>
<div class=lvl-row>
<button class=r0 onclick="setLevel('r0')">🌐 R0</button>
<button class=r1 onclick="setLevel('r1')">🛡 R1</button>
<button class=r2 onclick="setLevel('r2')">🔍 R2</button>
<button class=r3 onclick="setLevel('r3')">🌐 R3</button>
</div>
<button onclick="closeModal()" class=btn style="background:transparent;color:var(--dim);border:1px solid var(--dim)">Annuler</button>
</div>
</div>
<script>
var selectedMh = null;
function openModal(mh) {
selectedMh = mh;
document.getElementById('modal-mh').textContent = mh;
document.getElementById('modal').classList.add('show');
}
function closeModal() {
document.getElementById('modal').classList.remove('show');
}
async function setLevel(lvl) {
if (!selectedMh) return;
var fd = new FormData();
fd.append('level', lvl);
var r = await fetch('/admin/clients/'+selectedMh+'/level', {method:'POST', body:fd});
if (r.ok) {
closeModal();
loadClients();
} else {
alert('Erreur: '+r.status);
}
}
async function loadClients() {
var r = await fetch('/admin/clients/rich');
if (!r.ok) {
document.getElementById('clients-tbody').innerHTML = '<tr><td colspan=8>Erreur '+r.status+'</td></tr>';
return;
}
var data = await r.json();
var tbody = document.getElementById('clients-tbody');
tbody.innerHTML = '';
data.clients.forEach(function(c){
var lvlChip = '<span class="chip '+c.level+'">'+c.level_emoji+' '+c.level.toUpperCase()+'</span>';
var dt = new Date(c.last_seen*1000).toISOString().substring(11,16);
var tr = document.createElement('tr');
tr.innerHTML = '<td>'+c.status_emoji+' '+c.status_label+'</td>'+
'<td>'+c.device_emoji+'</td>'+
'<td><code>'+c.mac_hash.substring(0,12)+'…</code></td>'+
'<td>'+(c.ip||'?')+'</td>'+
'<td>'+lvlChip+'</td>'+
'<td>'+c.risk_emoji+' '+c.score+'</td>'+
'<td>'+dt+'</td>'+
'<td><button class=btn onclick="openModal(\\''+c.mac_hash+'\\')">🔀 Override</button>'+
'<a href="/admin/clients/'+c.mac_hash+'/report" class="btn outline">📄 PDF</a></td>';
tbody.appendChild(tr);
});
// KPI cards
var counts = {r0:0,r1:0,r2:0,r3:0,actif:0,risk_low:0,risk_mh:0};
data.clients.forEach(function(c){
counts[c.level] = (counts[c.level]||0)+1;
if (c.status_label==='actif') counts.actif++;
if (c.score < 30) counts.risk_low++;
if (c.score >= 30) counts.risk_mh++;
});
document.getElementById('cards').innerHTML =
'<div class=card><span class=v>'+data.count+'</span><span class=l>Total clients</span></div>'+
'<div class=card><span class=v>'+counts.actif+'</span><span class=l>🟢 Actifs (5min)</span></div>'+
'<div class=card><span class=v>'+counts.r1+'</span><span class=l>🛡 R1</span></div>'+
'<div class=card><span class=v>'+counts.r2+'</span><span class=l>🔍 R2</span></div>'+
'<div class=card><span class=v>'+counts.r3+'</span><span class=l>🌐 R3 WG</span></div>'+
'<div class=card><span class=v>'+counts.risk_low+'</span><span class=l>🟢 Risque LOW</span></div>';
}
loadClients();
setInterval(loadClients, 15000);
// Tabs (Phase 6.I)
document.querySelectorAll('.adm-tab').forEach(function(t){
t.addEventListener('click', function(){
var tn = this.dataset.tab;
document.querySelectorAll('.adm-tab').forEach(function(x){x.classList.remove('active');});
this.classList.add('active');
document.querySelectorAll('.adm-content').forEach(function(x){x.classList.remove('active');});
document.querySelector('[data-content='+tn+']').classList.add('active');
if (tn === 'filter') loadFilter();
});
});
// Filter Control (integrated tab)
async function loadFilter(){
var r = await fetch('/admin/filter-control/list');
var tb = document.getElementById('filter-tbody');
if (!r.ok) { tb.innerHTML = '<tr><td colspan=2>Erreur '+r.status+'</td></tr>'; return; }
var d = await r.json();
if (!d.patterns || !d.patterns.length) {
tb.innerHTML = '<tr><td colspan=2 style="color:#666;text-align:center">Aucune entrée</td></tr>';
return;
}
tb.innerHTML = '';
d.patterns.forEach(function(p){
var tr = document.createElement('tr');
var td1 = document.createElement('td');
var code = document.createElement('code');
code.textContent = p;
td1.appendChild(code);
var td2 = document.createElement('td');
var btn = document.createElement('button');
btn.className = 'del-pattern';
btn.textContent = '×';
btn.onclick = function(){ delPattern(p); };
td2.appendChild(btn);
tr.appendChild(td1); tr.appendChild(td2);
tb.appendChild(tr);
});
}
async function delPattern(p){
var fd = new FormData(); fd.append('entry', p);
var r = await fetch('/admin/filter-control/remove', {method:'POST', body:fd});
if (r.ok || r.status === 303) loadFilter();
}
document.getElementById('add-pattern-form').addEventListener('submit', async function(ev){
ev.preventDefault();
var entry = ev.target.entry.value.trim();
if (!entry) return;
var fd = new FormData(); fd.append('entry', entry);
var r = await fetch('/admin/filter-control/add', {method:'POST', body:fd});
if (r.ok || r.status === 303) {
ev.target.entry.value = '';
loadFilter();
}
});
</script>
</body></html>"""
return HTMLResponse(html, headers={"Cache-Control": "no-store"})
# Phase 11.B+ (#513) — the inline kbin /admin/ HTML admin UI was removed.
# The canonical operator dashboard is admin.gk2.secubox.in/toolbox/ (sub-tab
# WebUI in www/toolbox/index.html). All /admin/* JSON API routes below stay.
@router.get("/admin/filter-control/list")

View File

@ -69,7 +69,12 @@ CREATE TABLE IF NOT EXISTS social_edges (
src_site TEXT NOT NULL,
tracker_domain TEXT NOT NULL,
cookie_id_hash TEXT NOT NULL,
ja4_hash TEXT
ja4_hash TEXT,
-- Phase 11.C (#508) — consent state at the moment the edge was
-- recorded. Computed by the addon based on whether a consent
-- platform cookie (OneTrust/Didomi/Quantcast/Sourcepoint) has
-- already been observed for this peer × site pair.
consent_state TEXT NOT NULL DEFAULT 'none_seen'
);
CREATE INDEX IF NOT EXISTS idx_social_edges_mac_ts
ON social_edges(client_mac_hash, ts);
@ -83,6 +88,16 @@ CREATE TABLE IF NOT EXISTS social_nodes (
first_seen INTEGER NOT NULL,
last_seen INTEGER NOT NULL,
sites_jsonl TEXT NOT NULL DEFAULT '[]',
-- Phase 11.C (#508) — GeoIP-derived metadata populated at fold
-- time so reads + PDF rendering don't have to do per-row mmdb
-- lookups. eu_inside is 1 when country_iso EU/EEA whitelist.
country_iso TEXT,
asn_org TEXT,
eu_inside INTEGER NOT NULL DEFAULT 1,
-- Number of edges recorded against this (peer, tracker) BEFORE a
-- consent cookie was observed. >0 = legal-grade evidence of
-- tracker firing before consent (RGPD art. 6.1.a + 7).
pre_consent_hits INTEGER NOT NULL DEFAULT 0,
PRIMARY KEY (client_mac_hash, tracker_domain)
);
@ -103,9 +118,79 @@ def _conn() -> sqlite3.Connection:
c = sqlite3.connect(str(DB_PATH), timeout=5.0, isolation_level=None)
c.row_factory = sqlite3.Row
c.executescript(_SCHEMA)
_migrate_phase11c(c)
return c
# ───── Phase 11.C migrations — additive columns on pre-existing tables ─────
# CREATE TABLE IF NOT EXISTS skips creation if the table already exists, so
# the new columns won't auto-appear on a 2.6.0 → 2.6.2 upgrade. Idempotent
# ALTERs : we probe the column list first to skip the duplicate-column
# error case (which would raise on every connection otherwise).
_PHASE11C_MIGRATIONS = (
("social_edges", "consent_state", "TEXT NOT NULL DEFAULT 'none_seen'"),
("social_nodes", "country_iso", "TEXT"),
("social_nodes", "asn_org", "TEXT"),
("social_nodes", "eu_inside", "INTEGER NOT NULL DEFAULT 1"),
("social_nodes", "pre_consent_hits", "INTEGER NOT NULL DEFAULT 0"),
)
def _migrate_phase11c(c: sqlite3.Connection) -> None:
try:
for table, col, decl in _PHASE11C_MIGRATIONS:
existing = {
r["name"] for r in c.execute(f"PRAGMA table_info({table})").fetchall()
}
if col not in existing:
c.execute(f"ALTER TABLE {table} ADD COLUMN {col} {decl}")
except Exception as e: # pragma: no cover
log.warning("Phase 11.C migration failed: %s", e)
# ───── EU / EEA whitelist (RGPD scope, art. 45 + Schengen extension) ─────
# Codes are ISO 3166-1 alpha-2. Source : EU member state list + EFTA
# (NO, IS, LI) + UK (adequacy decision in force as of writing). The
# Phase C "extra_eu" flag is set when GeoIP says the tracker's country
# ISO is NOT in this set.
_EU_EEA_ISO: frozenset = frozenset({
"AT", "BE", "BG", "CY", "CZ", "DE", "DK", "EE", "ES", "FI", "FR",
"GR", "HR", "HU", "IE", "IT", "LT", "LU", "LV", "MT", "NL", "PL",
"PT", "RO", "SE", "SI", "SK", # 27 EU
"IS", "LI", "NO", # EFTA / EEA
"GB", # UK adequacy decision
})
def is_eu_iso(iso: str | None) -> bool:
return bool(iso) and (iso or "").upper() in _EU_EEA_ISO
# Lightweight cache around the existing `geo` module so the fold loop
# doesn't pay the lookup cost per repeated tracker_domain. Bounded to
# 4096 entries (well above any realistic distinct tracker count seen
# in a 7d retention window).
import functools as _functools
@_functools.lru_cache(maxsize=4096)
def _geo_for(host: str) -> Tuple[Optional[str], Optional[str]]:
"""Return (country_iso, asn_org) for a tracker host.
Best-effort. Falls back to (None, None) when the GeoIP module isn't
importable (worker hasn't installed the mmdb yet) or when the host
is a raw IP and the underlying lookup misses.
"""
try:
from secubox_toolbox import geo as _g # type: ignore
info = _g.lookup(host) or {}
iso = (info.get("country_iso") or "").upper() or None
asn = (info.get("asn_org") or "")[:64] or None
return iso, asn
except Exception:
return None, None
def cookie_id_hash(tracker_domain: str, cookie_name: str, cookie_value: str) -> str:
"""Stable short hash for an observed tracker identifier.
@ -147,13 +232,14 @@ def _record_edge_sync(
tracker_domain: str,
cookie_id_hash_val: str,
ja4_hash: Optional[str],
consent_state: str,
) -> None:
try:
with _conn() as c:
c.execute(
"INSERT INTO social_edges(ts, client_mac_hash, src_site, "
"tracker_domain, cookie_id_hash, ja4_hash) "
"VALUES (?, ?, ?, ?, ?, ?)",
"tracker_domain, cookie_id_hash, ja4_hash, consent_state) "
"VALUES (?, ?, ?, ?, ?, ?, ?)",
(
int(time.time()),
client_mac_hash,
@ -161,6 +247,7 @@ def _record_edge_sync(
tracker_domain,
cookie_id_hash_val,
ja4_hash,
consent_state or "none_seen",
),
)
except Exception as e: # pragma: no cover — best-effort
@ -174,9 +261,15 @@ def record_edge(
tracker_domain: str,
cookie_id_hash_val: str,
ja4_hash: Optional[str] = None,
consent_state: str = "none_seen",
) -> None:
"""Submit one edge off-thread. Best-effort, never raises into the
addon, never blocks the mitmproxy asyncio loop."""
addon, never blocks the mitmproxy asyncio loop.
`consent_state` is one of {none_seen, pre_consent, post_consent} as
computed by the addon based on the per-peer × per-site consent
cookie observation log (Phase 11.C).
"""
if not (client_mac_hash and src_site and tracker_domain and cookie_id_hash_val):
return
try:
@ -187,6 +280,7 @@ def record_edge(
tracker_domain,
cookie_id_hash_val,
ja4_hash,
consent_state,
)
except RuntimeError:
# Executor shut down (interpreter teardown) — silent drop.
@ -208,7 +302,7 @@ def fold_recent(window_seconds: int = 300) -> Tuple[int, int]:
with _conn() as c:
edges = c.execute(
"SELECT client_mac_hash, src_site, tracker_domain, "
"cookie_id_hash, ja4_hash, ts "
"cookie_id_hash, ja4_hash, ts, consent_state "
"FROM social_edges WHERE ts >= ?",
(since,),
).fetchall()
@ -233,12 +327,20 @@ def fold_recent(window_seconds: int = 300) -> Tuple[int, int]:
key_n = (mac, trk)
n = per_node.setdefault(
key_n,
{"hits": 0, "first_seen": ts, "last_seen": ts, "sites": set()},
{
"hits": 0,
"first_seen": ts,
"last_seen": ts,
"sites": set(),
"pre_consent_hits": 0,
},
)
n["hits"] += 1
n["first_seen"] = min(n["first_seen"], ts)
n["last_seen"] = max(n["last_seen"], ts)
n["sites"].add(site)
if e["consent_state"] == "pre_consent":
n["pre_consent_hits"] += 1
# Per-site tracker index (for link fold below)
per_site_trackers.setdefault((mac, site), set()).add(trk)
@ -248,8 +350,13 @@ def fold_recent(window_seconds: int = 300) -> Tuple[int, int]:
# Persist nodes
for (mac, trk), n in per_node.items():
# Merge into existing row if present
# Phase 11.C : enrich with GeoIP at fold time so reads
# + PDF rendering never block on mmdb lookups.
country_iso, asn_org = _geo_for(trk)
eu_inside = 1 if is_eu_iso(country_iso) else 0
cur = c.execute(
"SELECT hits, first_seen, sites_jsonl FROM social_nodes "
"SELECT hits, first_seen, sites_jsonl, pre_consent_hits "
"FROM social_nodes "
"WHERE client_mac_hash = ? AND tracker_domain = ?",
(mac, trk),
).fetchone()
@ -261,17 +368,22 @@ def fold_recent(window_seconds: int = 300) -> Tuple[int, int]:
except Exception:
existing_sites = set()
sites = sorted(existing_sites | n["sites"])
pre = (cur["pre_consent_hits"] or 0) + n["pre_consent_hits"]
c.execute(
"UPDATE social_nodes SET hits = ?, first_seen = ?, "
"last_seen = ?, sites_jsonl = ? "
"last_seen = ?, sites_jsonl = ?, country_iso = ?, "
"asn_org = ?, eu_inside = ?, pre_consent_hits = ? "
"WHERE client_mac_hash = ? AND tracker_domain = ?",
(hits, first, n["last_seen"], json.dumps(sites), mac, trk),
(hits, first, n["last_seen"], json.dumps(sites),
country_iso, asn_org, eu_inside, pre, mac, trk),
)
else:
c.execute(
"INSERT INTO social_nodes(client_mac_hash, "
"tracker_domain, hits, first_seen, last_seen, "
"sites_jsonl) VALUES (?, ?, ?, ?, ?, ?)",
"sites_jsonl, country_iso, asn_org, eu_inside, "
"pre_consent_hits) "
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?)",
(
mac,
trk,
@ -279,6 +391,10 @@ def fold_recent(window_seconds: int = 300) -> Tuple[int, int]:
n["first_seen"],
n["last_seen"],
json.dumps(sorted(n["sites"])),
country_iso,
asn_org,
eu_inside,
n["pre_consent_hits"],
),
)
nodes_touched += 1
@ -502,6 +618,74 @@ def aggregate(hours: int = 24) -> Dict:
return out
def evidence(mac_hash: str, since_seconds: int = 86400) -> Dict:
"""Phase 11.C evidence helper — returns the legal-grade slice
consumed by the bilingual PDF report.
Two evidence buckets, both fact-only (no interpretation) :
- ``pre_consent`` : (tracker_domain, sites, pre_consent_hits,
country_iso, asn_org). Trackers that fired BEFORE a consent
cookie was observed for that peer × site. Direct RGPD art. 7
+ art. 6.1.a evidence.
- ``extra_eu`` : (tracker_domain, country_iso, asn_org,
sites). Trackers resolving to non-EU/EEA countries. Note :
we report the fact, not SCC absence (we can't prove a
negative). RGPD art. 44+ evidence.
"""
since = int(time.time()) - max(since_seconds, 3600)
out: Dict = {"pre_consent": [], "extra_eu": []}
if not mac_hash:
return out
try:
with _conn() as c:
for r in c.execute(
"SELECT tracker_domain, hits, pre_consent_hits, sites_jsonl, "
"country_iso, asn_org, last_seen "
"FROM social_nodes "
"WHERE client_mac_hash = ? AND last_seen >= ? "
"AND pre_consent_hits > 0 "
"ORDER BY pre_consent_hits DESC, hits DESC LIMIT 100",
(mac_hash, since),
).fetchall():
try:
sites = json.loads(r["sites_jsonl"])
except Exception:
sites = []
out["pre_consent"].append({
"tracker_domain": r["tracker_domain"],
"hits": r["hits"],
"pre_consent_hits": r["pre_consent_hits"],
"sites": sites,
"country_iso": r["country_iso"],
"asn_org": r["asn_org"],
"last_seen": r["last_seen"],
})
for r in c.execute(
"SELECT tracker_domain, hits, sites_jsonl, country_iso, "
"asn_org, last_seen "
"FROM social_nodes "
"WHERE client_mac_hash = ? AND last_seen >= ? "
"AND eu_inside = 0 AND country_iso IS NOT NULL "
"ORDER BY hits DESC LIMIT 100",
(mac_hash, since),
).fetchall():
try:
sites = json.loads(r["sites_jsonl"])
except Exception:
sites = []
out["extra_eu"].append({
"tracker_domain": r["tracker_domain"],
"hits": r["hits"],
"sites": sites,
"country_iso": r["country_iso"],
"asn_org": r["asn_org"],
"last_seen": r["last_seen"],
})
except Exception as e: # pragma: no cover
log.warning("evidence query failed: %s", e)
return out
def purge_older_than(days: int = 7) -> int:
"""Drop raw edges older than `days`. The aggregate node/link tables
stay : they represent the durable fold. Operator-side wipe goes

View File

@ -0,0 +1,239 @@
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
"""
SecuBox-Deb :: ToolBoX Social Mapping bilingual FR/EN PDF report
Phase 11.C (#508, parent #502) — evidence-only legal report consumed by
the per-client view's "Rapport PDF" card. Renders via fpdf2 (same
engine as reports.py) so no new dependency.
The report is FACT-ONLY : it documents what we observed (cross-site
cookie reuse, trackers fired before consent, extra-EU transfers) with
the relevant RGPD article references. It makes NO interpretation about
SCC presence/absence or ToS contradiction.
FR is the source-of-truth ; EN is a secondary summary line per field.
"""
from __future__ import annotations
import logging
import time
from typing import Dict, List
from . import social as _social
log = logging.getLogger("secubox.toolbox.social.report")
# Bilingual label pairs (FR primary, EN secondary).
_L = {
"title": ("Cartographie sociale — Rapport d'évidence", "Social mapping — Evidence report"),
"subtitle": ("Cabine numérique VILLAGE3B · Analyseur R3", "VILLAGE3B digital booth · R3 analyzer"),
"anon_id": ("Identifiant anonyme", "Anonymous identifier"),
"hash": ("Hash session (sel rotatif 24h)", "Session hash (24h rotating salt)"),
"window": ("Fenêtre d'analyse", "Analysis window"),
"generated": ("Généré le", "Generated"),
"summary": ("Synthèse", "Summary"),
"n_trackers": ("Traqueurs distincts", "Distinct trackers"),
"n_sites": ("Sites visités", "Sites visited"),
"n_preconsent": ("Traqueurs avant consentement", "Trackers before consent"),
"n_extraeu": ("Transferts hors UE/EEE", "Extra-EU/EEA transfers"),
"ev_preconsent": ("Évidence : traqueurs déclenchés AVANT consentement",
"Evidence: trackers fired BEFORE consent"),
"ev_preconsent_basis": ("Base légale : RGPD art. 6.1.a (consentement) + art. 7 (preuve)",
"Legal basis: GDPR art. 6.1.a (consent) + art. 7 (proof)"),
"ev_extraeu": ("Évidence : transferts vers des pays hors UE/EEE",
"Evidence: transfers to non-EU/EEA countries"),
"ev_extraeu_basis": ("Base légale : RGPD art. 44+ (transferts internationaux). "
"Nous rapportons le fait observé, pas l'absence de garanties (SCC).",
"Legal basis: GDPR art. 44+ (international transfers). We report the "
"observed fact, not the absence of safeguards (SCC)."),
"col_tracker": ("Traqueur", "Tracker"),
"col_sites": ("Sites", "Sites"),
"col_hits": ("Occurrences", "Hits"),
"col_country": ("Pays", "Country"),
"col_asn": ("Hébergeur (ASN)", "Host (ASN)"),
"none_pre": ("Aucun traqueur déclenché avant consentement détecté.",
"No tracker fired before consent detected."),
"none_eu": ("Aucun transfert hors UE/EEE détecté.",
"No extra-EU/EEA transfer detected."),
"footer": ("Rapport factuel — données calculées localement, aucune donnée externe. "
"Anonyme : aucune valeur de cookie brute conservée. Droit à l'effacement : RGPD art. 17.",
"Factual report — computed locally, no external data. Anonymous: no raw cookie "
"value stored. Right to erasure: GDPR art. 17."),
"ca_disclaimer": ("Évidence d'observation réseau via cabine R3 consentie. "
"Ne constitue pas un avis juridique.",
"Network observation evidence via consented R3 booth. "
"Not legal advice."),
}
def _bi(key: str) -> str:
fr, en = _L.get(key, (key, key))
return fr
def build_social_report(mac_hash: str, since_seconds: int = 7 * 86400) -> Dict:
"""Assemble the data structure the PDF renderer consumes."""
graph = _social.fetch_graph(mac_hash, since_seconds=since_seconds)
evidence = _social.evidence(mac_hash, since_seconds=since_seconds)
stats = graph.get("stats", {})
return {
"mac_hash": mac_hash,
"window_days": max(1, since_seconds // 86400),
"generated_at": None, # stamped by the API layer (Date.* unavailable here is fine; time.time ok)
"total_trackers": stats.get("total_trackers", 0),
"total_sites": stats.get("total_sites", 0),
"pre_consent": evidence.get("pre_consent", []),
"extra_eu": evidence.get("extra_eu", []),
}
def render_social_pdf(report: Dict) -> bytes:
"""Render the bilingual evidence report as PDF (fpdf2)."""
try:
from fpdf import FPDF
except ImportError:
return _text_fallback(report).encode()
# Reuse the font setup from reports.py for emoji/Unicode support.
try:
from .reports import _setup_fonts # type: ignore
except Exception:
_setup_fonts = None
pdf = FPDF(orientation="P", unit="mm", format="A4")
pdf.add_page()
pdf.set_auto_page_break(auto=True, margin=15)
family = _setup_fonts(pdf) if _setup_fonts else "helvetica"
def _mc(h, text):
# Robust multi_cell : reset X to the left margin and use the
# effective page width so fpdf2 never hits "not enough
# horizontal space" when the cursor has drifted to the edge.
pdf.set_x(pdf.l_margin)
pdf.multi_cell(pdf.epw, h, text)
def bi(key, size=10, style="", r=0, g=0, b=0, gap=1.0):
fr, en = _L.get(key, (key, key))
pdf.set_text_color(r, g, b)
pdf.set_font(family, "B" if "B" in style else "", size)
_mc(size * 0.5, fr)
pdf.set_text_color(120, 120, 120)
pdf.set_font(family, "", max(7, size - 2))
_mc((size - 2) * 0.5, en)
pdf.ln(gap)
# ── Cover ──
pdf.set_font(family, "B", 20)
pdf.set_text_color(10, 90, 64)
pdf.cell(0, 12, "🕸️ VILLAGE3B", ln=True, align="C")
bi("title", 13, "B", 110, 64, 201, gap=0.5)
bi("subtitle", 9, "", 110, 110, 110, gap=3)
# ── Anonymous id ──
bi("anon_id", 12, "B", 0, 90, 64, gap=0.5)
pdf.set_font(family, "", 9)
pdf.set_text_color(0)
pdf.cell(0, 5, f" {_bi('hash')}: {report.get('mac_hash', '?')[:16]}", ln=True)
pdf.cell(0, 5, f" {_bi('window')}: {report.get('window_days', 7)} j / days", ln=True)
pdf.cell(0, 5, f" {_bi('generated')}: {report.get('generated_at', '?')}", ln=True)
pdf.ln(3)
# ── Summary tiles ──
bi("summary", 12, "B", 0, 90, 64, gap=0.5)
pdf.set_font(family, "", 10)
pdf.set_text_color(0)
pdf.cell(0, 6, f" {_bi('n_trackers')}: {report.get('total_trackers', 0)}", ln=True)
pdf.cell(0, 6, f" {_bi('n_sites')}: {report.get('total_sites', 0)}", ln=True)
pdf.cell(0, 6, f" {_bi('n_preconsent')}: {len(report.get('pre_consent', []))}", ln=True)
pdf.cell(0, 6, f" {_bi('n_extraeu')}: {len(report.get('extra_eu', []))}", ln=True)
pdf.ln(3)
# ── Evidence : pre-consent ──
bi("ev_preconsent", 12, "B", 230, 57, 70, gap=0.3)
bi("ev_preconsent_basis", 8, "", 120, 120, 120, gap=1)
pre = report.get("pre_consent", [])
if pre:
_table(pdf, family, pre, [
(_bi("col_tracker"), "tracker_domain", 70),
(_bi("col_sites"), lambda r: str(len(r.get("sites", []))), 25),
("pre", "pre_consent_hits", 25),
(_bi("col_country"), "country_iso", 25),
(_bi("col_asn"), "asn_org", 45),
])
else:
_note(pdf, family, _bi("none_pre"))
pdf.ln(2)
# ── Evidence : extra-EU ──
bi("ev_extraeu", 12, "B", 110, 64, 201, gap=0.3)
bi("ev_extraeu_basis", 8, "", 120, 120, 120, gap=1)
eu = report.get("extra_eu", [])
if eu:
_table(pdf, family, eu, [
(_bi("col_tracker"), "tracker_domain", 70),
(_bi("col_sites"), lambda r: str(len(r.get("sites", []))), 25),
(_bi("col_hits"), "hits", 25),
(_bi("col_country"), "country_iso", 25),
(_bi("col_asn"), "asn_org", 45),
])
else:
_note(pdf, family, _bi("none_eu"))
pdf.ln(4)
# ── Footer ──
pdf.set_font(family, "", 7)
pdf.set_text_color(120, 120, 120)
fr, en = _L["footer"]
_mc(3.2, fr)
_mc(3.0, en)
pdf.ln(1)
fr, en = _L["ca_disclaimer"]
_mc(3.2, fr + " / " + en)
out = pdf.output(dest="S")
return bytes(out) if isinstance(out, (bytes, bytearray)) else out.encode("latin-1")
def _table(pdf, family, rows: List[Dict], cols) -> None:
pdf.set_font(family, "B", 8)
pdf.set_text_color(0, 90, 64)
for label, _key, w in cols:
pdf.cell(w, 5, str(label)[:24], border="B")
pdf.ln()
pdf.set_font(family, "", 8)
pdf.set_text_color(0)
for r in rows[:40]:
for label, key, w in cols:
if callable(key):
val = key(r)
else:
val = r.get(key, "")
pdf.cell(w, 4.5, str(val if val is not None else "")[:int(w / 1.7)])
pdf.ln()
def _note(pdf, family, text: str) -> None:
pdf.set_font(family, "I", 9)
pdf.set_text_color(0, 120, 80)
pdf.cell(0, 6, " " + text, ln=True)
def _text_fallback(report: Dict) -> str:
lines = [
"VILLAGE3B — Social mapping evidence report",
f"hash: {report.get('mac_hash', '?')[:16]}",
f"trackers: {report.get('total_trackers', 0)} sites: {report.get('total_sites', 0)}",
f"pre-consent: {len(report.get('pre_consent', []))} extra-EU: {len(report.get('extra_eu', []))}",
"",
"Pre-consent trackers (GDPR art. 6.1.a + 7):",
]
for e in report.get("pre_consent", [])[:40]:
lines.append(f" - {e.get('tracker_domain')} ({e.get('pre_consent_hits')} hits, {e.get('country_iso')})")
lines.append("")
lines.append("Extra-EU transfers (GDPR art. 44+):")
for e in report.get("extra_eu", [])[:40]:
lines.append(f" - {e.get('tracker_domain')} ({e.get('country_iso')}, {e.get('asn_org')})")
return "\n".join(lines)

View File

@ -2,7 +2,7 @@
<!--
SPDX-License-Identifier: LicenseRef-CMSD-1.0
Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
SecuBox-Deb :: ToolBoX admin dashboard (P31 light skin)
SecuBox-Deb :: ToolBoX admin dashboard (P31 light skin) — sub-tab nav (#513)
-->
<html lang="fr">
<head>
@ -14,14 +14,22 @@
<link rel="stylesheet" href="/shared/crt-light.css">
<link rel="stylesheet" href="/shared/sidebar-light.css">
<style>
:root{--p31-peak:#00dd44;--p31-hot:#00ff55;--p31-mid:#009933;--p31-dim:#006622;--p31-decay:#ffb347;--tube-light:#e8f5e9;--tube-pale:#c8e6c9;--tube-soft:#a5d6a7;--bg-card:var(--tube-pale);--border:var(--tube-soft);--text-dim:var(--p31-dim);--red:#ff4466;--bloom-text:0 0 2px var(--p31-peak),0 0 6px var(--p31-peak);--bloom-soft:0 0 6px var(--p31-peak)}
:root{--p31-peak:#00dd44;--p31-hot:#00ff55;--p31-mid:#009933;--p31-dim:#006622;--p31-decay:#ffb347;--tube-light:#e8f5e9;--tube-pale:#c8e6c9;--tube-soft:#a5d6a7;--bg-card:var(--tube-pale);--border:var(--tube-soft);--text-dim:var(--p31-dim);--red:#ff4466;--purple:#9e76ff;--bloom-text:0 0 2px var(--p31-peak),0 0 6px var(--p31-peak);--bloom-soft:0 0 6px var(--p31-peak)}
*{box-sizing:border-box;margin:0;padding:0}
body{font-family:'Courier Prime',monospace;background:var(--tube-light);color:var(--p31-mid);display:flex;min-height:100vh}
.sidebar{width:220px;position:fixed;height:100vh;overflow-y:auto}
.main{flex:1;margin-left:220px;padding:1.5rem}
.header{display:flex;justify-content:space-between;align-items:center;padding:1rem 1.5rem;border:1px solid var(--border);background:var(--bg-card);margin-bottom:1.5rem}
.header{display:flex;justify-content:space-between;align-items:center;padding:1rem 1.5rem;border:1px solid var(--border);background:var(--bg-card);margin-bottom:1rem}
.header h1{font-size:1.4rem;color:var(--p31-hot);text-shadow:var(--bloom-text)}
.badge{font-size:0.85rem;color:var(--p31-dim);padding:0.2rem 0.6rem;border:1px solid var(--border);border-radius:3px}
/* Sub-tab nav (#513) */
.tabs{display:flex;gap:0.3rem;margin-bottom:1.2rem;border-bottom:1px solid var(--border);flex-wrap:wrap}
.tab{font-family:inherit;font-size:0.9rem;padding:0.5rem 1rem;background:transparent;color:var(--p31-dim);border:1px solid var(--border);border-bottom:none;cursor:pointer;border-radius:3px 3px 0 0}
.tab:hover{color:var(--p31-peak);background:rgba(0,221,68,0.05)}
.tab.active{color:var(--p31-hot);text-shadow:var(--bloom-text);background:var(--bg-card);border-color:var(--p31-mid);font-weight:bold}
.panel{display:none}
.panel.active{display:block;animation:fade .15s ease-in}
@keyframes fade{from{opacity:0}to{opacity:1}}
.grid{display:grid;grid-template-columns:repeat(auto-fit,minmax(380px,1fr));gap:1rem}
.card{background:var(--bg-card);border:1px solid var(--border);padding:1rem;box-shadow:var(--bloom-soft)}
.card h2{font-size:1rem;color:var(--p31-hot);text-shadow:var(--bloom-text);margin-bottom:0.6rem;padding-bottom:0.4rem;border-bottom:1px solid var(--border)}
@ -32,13 +40,23 @@
th,td{text-align:left;padding:0.4rem 0.5rem;border-bottom:1px solid var(--border)}
th{color:var(--p31-hot);text-shadow:var(--bloom-text)}
.empty{color:var(--text-dim);font-style:italic;padding:1rem;text-align:center}
.toolbar{display:flex;gap:0.5rem;margin-bottom:1rem}
.toolbar{display:flex;gap:0.5rem;margin-bottom:1rem;flex-wrap:wrap}
button{font-family:inherit;font-size:0.9rem;padding:0.4rem 0.9rem;background:transparent;color:var(--p31-peak);border:1px solid var(--p31-mid);cursor:pointer;text-shadow:var(--bloom-text)}
button:hover{background:rgba(0,221,68,0.1)}
.r2-banner{background:rgba(255,179,71,0.08);border-left:2px solid var(--p31-decay);padding:0.6rem 0.8rem;margin-bottom:1rem;font-size:0.85rem;color:#996b1a}
.state-validated{color:var(--p31-peak);text-shadow:var(--bloom-text)}
.state-throttle{color:var(--p31-decay)}
.state-quarantine{color:var(--red)}
/* level chips + switcher (folded from kbin /admin/) */
.chip{display:inline-block;padding:0.1rem 0.45rem;border-radius:99px;font-size:0.7rem;font-weight:bold}
.chip.r0{background:#cfd8cf;color:#555}
.chip.r1{background:rgba(0,221,68,0.25);color:var(--p31-mid)}
.chip.r2{background:rgba(255,179,71,0.3);color:#996b1a}
.chip.r3{background:rgba(158,118,255,0.25);color:#6a4fd0}
.lvlbtn{font-size:0.68rem;padding:0.12rem 0.4rem;margin-right:0.15rem;border:1px solid var(--p31-dim)}
code{background:rgba(0,0,0,0.05);padding:0.1rem 0.3rem;border-radius:2px;font-size:0.78rem}
a.link{color:var(--p31-mid);text-decoration:underline}
.filterlist li{font-size:0.82rem;padding:0.2rem 0;border-bottom:1px solid var(--border);list-style:none;display:flex;justify-content:space-between;align-items:center}
</style>
</head>
<body>
@ -53,39 +71,107 @@
<strong>R2 actif :</strong> TLS-break opt-in. Voir #475 / #474 pour la doctrine.
</div>
<div class="toolbar">
<button onclick="refreshAll()">🔁 Refresh</button>
<button onclick="window.open('/api/v1/toolbox/admin/config', '_blank')">⚙ Config TOML</button>
</div>
<!-- Sub-tab navigation (#513) -->
<nav class="tabs" id="tabs">
<button class="tab active" data-tab="overview" onclick="switchTab('overview')">📊 Vue d'ensemble</button>
<button class="tab" data-tab="clients" onclick="switchTab('clients')">👥 Clients</button>
<button class="tab" data-tab="filtres" onclick="switchTab('filtres')">🚦 Filtres MITM</button>
<button class="tab" data-tab="social" onclick="switchTab('social')">🕸️ Cartographie sociale</button>
<button class="tab" data-tab="config" onclick="switchTab('config')">⚙ Config</button>
</nav>
<div class="grid">
<!-- Overview -->
<section class="panel active" id="panel-overview">
<div class="toolbar">
<button onclick="refreshAll()">🔁 Refresh</button>
</div>
<div class="grid">
<div class="card">
<h2>📊 Live metrics (24h)</h2>
<div class="kv" id="metrics"><span class="k">loading…</span><span class="v"></span></div>
</div>
<div class="card">
<h2>♥ Liveness</h2>
<div class="kv" id="health"><span class="k">loading…</span><span class="v"></span></div>
</div>
</div>
</section>
<!-- Clients (folds in the kbin /admin/ level switcher) -->
<section class="panel" id="panel-clients">
<div class="toolbar">
<button onclick="loadClients()">🔁 Refresh</button>
</div>
<div class="card" style="margin-bottom:1rem">
<h2>👥 Clients actifs</h2>
<div id="clients"><div class="empty">loading…</div></div>
</div>
<div class="card" style="display:none" id="client-detail-card">
<h2 id="detail-title">🔍 Détails client</h2>
<div id="client-detail"></div>
</div>
</section>
<!-- Filtres MITM (bypass list) -->
<section class="panel" id="panel-filtres">
<div class="toolbar">
<button onclick="loadFilters()">🔁 Refresh</button>
</div>
<div class="card">
<h2>📊 Live metrics (24h)</h2>
<div class="kv" id="metrics"><span class="k">loading…</span><span class="v"></span></div>
<h2>🚦 Hosts bypassés (mitm ignore_hosts)</h2>
<p style="font-size:0.82rem;color:var(--p31-dim);margin-bottom:0.6rem">
Hosts exclus de l'inspection TLS (cert-pinning détecté ou whitelist statique).
</p>
<ul class="filterlist" id="filters"><li class="empty">loading…</li></ul>
</div>
</section>
<!-- Cartographie sociale (Phase 11 operator aggregate) -->
<section class="panel" id="panel-social">
<div class="toolbar">
<button onclick="loadSocial()">🔁 Refresh</button>
</div>
<div class="grid">
<div class="card">
<h2>🕸️ Agrégat trackers (24h)</h2>
<div class="kv" id="social-kpi"><span class="k">loading…</span><span class="v"></span></div>
</div>
<div class="card" style="grid-column:1/-1">
<h2>🎯 Top tracker domains</h2>
<div id="social-trackers"><div class="empty">loading…</div></div>
</div>
<div class="card" style="grid-column:1/-1">
<h2>👤 Clients anonymisés</h2>
<div id="social-clients"><div class="empty">loading…</div></div>
</div>
</div>
</section>
<!-- Config -->
<section class="panel" id="panel-config">
<div class="toolbar">
<button onclick="window.open('/api/v1/toolbox/admin/config', '_blank')">⚙ Config TOML brute</button>
</div>
<div class="card">
<h2>⚙ Configuration</h2>
<div class="kv" id="cfg-summary"><span class="k">loading…</span><span class="v"></span></div>
</div>
<div class="card">
<h2>♥ Liveness</h2>
<div class="kv" id="health"><span class="k">loading…</span><span class="v"></span></div>
</div>
<div class="card" style="grid-column:1/-1">
<h2>👥 Clients actifs</h2>
<div id="clients"><div class="empty">loading…</div></div>
</div>
<div class="card" style="grid-column:1/-1;display:none" id="client-detail-card">
<h2 id="detail-title">🔍 Détails client</h2>
<div id="client-detail"></div>
</div>
</div>
</section>
</div>
<script src="/shared/sidebar.js"></script>
<script>
const API = '/api/v1/toolbox';
function switchTab(name) {
document.querySelectorAll('.tab').forEach(t => t.classList.toggle('active', t.dataset.tab === name));
document.querySelectorAll('.panel').forEach(p => p.classList.toggle('active', p.id === 'panel-' + name));
// Lazy-load the heavier tabs only when first opened.
if (name === 'filtres') loadFilters();
if (name === 'social') loadSocial();
location.hash = name;
}
async function J(path) {
try {
const r = await fetch(API + path, { credentials: 'same-origin' });
@ -111,24 +197,50 @@ async function loadCfg() {
document.getElementById('r2-banner').style.display = c.r2.enabled ? '' : 'none';
}
function levelChip(lvl) {
const l = (lvl || 'r1').toLowerCase();
return `<span class="chip ${l}">${l.toUpperCase()}</span>`;
}
function levelSwitcher(macHash, current) {
const cur = (current || 'r1').toLowerCase();
return ['r0','r1','r2','r3'].map(l =>
l === cur ? '' :
`<button class="lvlbtn" onclick="setLevel('${macHash}','${l}')">${l.toUpperCase()}</button>`
).join('');
}
async function setLevel(macHash, level) {
try {
const fd = new FormData(); fd.append('level', level);
const r = await fetch(`${API}/admin/clients/${macHash}/level`, {
method: 'POST', body: fd, credentials: 'same-origin'
});
if (!r.ok) throw new Error('HTTP ' + r.status);
loadClients();
} catch (e) { alert('Échec changement niveau : ' + e.message); }
}
async function loadClients() {
const d = await J('/admin/clients');
const d = await J('/admin/clients/rich');
const el = document.getElementById('clients');
if (d.__error) { el.innerHTML = `<div class="empty">${d.__error}</div>`; return; }
if (!d.length) { el.innerHTML = '<div class="empty">no clients</div>'; return; }
let html = '<table><thead><tr><th>MAC (hash)</th><th>IP</th><th>state</th><th>score</th><th>last</th><th>Actions</th></tr></thead><tbody>';
for (const c of d) {
const ago = Math.round((Date.now()/1000 - c.last_seen) / 60);
const rows = (d && d.clients) ? d.clients : (Array.isArray(d) ? d : null);
if (!rows) { el.innerHTML = `<div class="empty">${(d && d.__error) || 'no data'}</div>`; return; }
if (!rows.length) { el.innerHTML = '<div class="empty">no clients</div>'; return; }
let html = '<table><thead><tr><th>MAC (hash)</th><th>IP</th><th>state</th><th>niveau</th><th>score</th><th>last</th><th>Actions</th></tr></thead><tbody>';
for (const c of rows) {
const ago = c.last_seen ? Math.round((Date.now()/1000 - c.last_seen) / 60) + 'm' : '—';
html += `<tr>
<td><code>${c.mac_hash}</code></td>
<td>${c.ip}</td>
<td><span class="state-${c.state}">${c.state}</span></td>
<td>${c.score}</td>
<td>${ago}m</td>
<td>${c.ip || '—'}</td>
<td><span class="state-${c.state}">${c.state || '—'}</span></td>
<td>${levelChip(c.level)} ${levelSwitcher(c.mac_hash, c.level)}</td>
<td>${c.score ?? '—'}</td>
<td>${ago}</td>
<td>
<a href="${API}/admin/clients/${c.mac_hash}/report" target="_blank" style="color:var(--p31-peak);text-decoration:underline;font-size:0.8rem">PDF</a>
<a class="link" href="${API}/admin/clients/${c.mac_hash}/report" target="_blank" style="font-size:0.8rem">PDF</a>
·
<a href="javascript:void(0)" onclick="loadClientDetail('${c.mac_hash}')" style="color:var(--p31-peak);text-decoration:underline;font-size:0.8rem">Events</a>
<a class="link" href="javascript:void(0)" onclick="loadClientDetail('${c.mac_hash}')" style="font-size:0.8rem">Events</a>
</td>
</tr>`;
}
@ -162,6 +274,7 @@ async function loadClientDetail(macHash) {
const div = document.getElementById('client-detail');
title.textContent = `🔍 Détails client — ${macHash}`;
card.style.display = '';
card.scrollIntoView({behavior:'smooth', block:'nearest'});
div.innerHTML = '<div class="empty">loading…</div>';
const d = await J(`/admin/clients/${macHash}/events`);
if (d.__error) { div.innerHTML = `<div class="empty">${d.__error}</div>`; return; }
@ -175,7 +288,7 @@ async function loadClientDetail(macHash) {
for (const e of d.recent) {
const t = new Date(e.ts*1000).toLocaleTimeString();
const detail = e.host || e.url || e.sni || e.kind || '?';
recentHtml += `<tr><td>${t}</td><td><b>${e.source}</b></td><td><code style="font-size:0.78rem">${detail}</code></td></tr>`;
recentHtml += `<tr><td>${t}</td><td><b>${e.source}</b></td><td><code>${detail}</code></td></tr>`;
}
recentHtml += '</tbody></table>';
}
@ -184,7 +297,7 @@ async function loadClientDetail(macHash) {
<p style="margin-bottom:0.5rem;font-size:0.85rem;color:var(--p31-dim)">Récents :</p>
${recentHtml}
<p style="margin-top:0.8rem">
<a href="${API}/admin/clients/${macHash}/report" target="_blank" style="color:var(--p31-peak);text-decoration:underline">⬇ Télécharger PDF complet</a>
<a class="link" href="${API}/admin/clients/${macHash}/report" target="_blank">⬇ Télécharger PDF complet</a>
</p>
`;
}
@ -201,11 +314,68 @@ async function loadHealth() {
document.getElementById('version-badge').textContent = `v${h.version}`;
}
async function loadFilters() {
const el = document.getElementById('filters');
const d = await J('/admin/filter-control/list');
const items = (d && d.hosts) ? d.hosts : (Array.isArray(d) ? d : null);
if (!items) { el.innerHTML = `<li class="empty">${(d && d.__error) || 'no data'}</li>`; return; }
if (!items.length) { el.innerHTML = '<li class="empty">aucun host bypassé</li>'; return; }
el.innerHTML = items.map(h => {
const host = typeof h === 'string' ? h : (h.host || h.pattern || JSON.stringify(h));
const src = (typeof h === 'object' && h.source) ? h.source : '';
return `<li><code>${host}</code>${src ? `<span style="color:var(--p31-dim);font-size:0.72rem">${src}</span>` : ''}</li>`;
}).join('');
}
async function loadSocial() {
const agg = await J('/admin/social-aggregate?hours=24');
const kpi = document.getElementById('social-kpi');
const trk = document.getElementById('social-trackers');
const cli = document.getElementById('social-clients');
if (agg.__error) {
kpi.innerHTML = `<span class="k">err</span><span class="v">${agg.__error}</span>`;
trk.innerHTML = cli.innerHTML = '';
return;
}
kpi.innerHTML = `
<span class="k">Clients actifs</span> <span class="v">${agg.active_clients || 0}</span>
<span class="k">Trackers vus</span> <span class="v">${agg.total_trackers_seen || 0}</span>
<span class="k">Hors-UE (Phase C)</span> <span class="v">${agg.extra_eu_trackers || 0}</span>
<span class="k">Fenêtre</span> <span class="v">${agg.window_hours || 24}h</span>
`;
const td = agg.by_tracker_domain || [];
trk.innerHTML = td.length
? '<table><thead><tr><th>Tracker domain</th><th>hits</th><th>clients</th></tr></thead><tbody>' +
td.map(r => `<tr><td><code>${r.tracker_domain}</code></td><td>${r.hits}</td><td>${r.clients}</td></tr>`).join('') +
'</tbody></table>'
: '<div class="empty">aucun tracker dans la fenêtre</div>';
const bc = agg.by_client || [];
cli.innerHTML = bc.length
? '<table><thead><tr><th>Client (hash)</th><th>sites</th><th>trackers</th><th>last</th></tr></thead><tbody>' +
bc.map(r => {
const ago = r.last_seen ? Math.round((Date.now()/1000 - r.last_seen)/60) + 'm' : '—';
return `<tr><td><code>${(r.client_mac_hash||'').slice(0,16)}</code></td><td>${r.sites}</td><td>${r.trackers}</td><td>${ago}</td></tr>`;
}).join('') +
'</tbody></table>'
: '<div class="empty">aucun client</div>';
}
async function refreshAll() {
await Promise.all([loadCfg(), loadClients(), loadHealth(), loadMetrics()]);
}
// Deep-link : open the tab named in the URL hash on load.
const initial = (location.hash || '').replace('#', '');
if (['overview','clients','filtres','social','config'].includes(initial)) switchTab(initial);
refreshAll();
setInterval(refreshAll, 10000);
setInterval(() => {
// Only refresh the currently-visible tab's live data.
const active = document.querySelector('.tab.active')?.dataset.tab;
if (active === 'overview') { loadHealth(); loadMetrics(); }
else if (active === 'clients') loadClients();
else if (active === 'social') loadSocial();
}, 10000);
</script>
</body>
</html>

View File

@ -224,6 +224,17 @@ body {
}
.wipe-btn:hover { background: rgba(230, 57, 70, .1); }
/* Phase 11.C — PDF report download button */
.pdf-btn {
display: block; width: 100%; margin-top: 8px;
text-align: center; text-decoration: none;
background: transparent; color: var(--gold-hermetic);
border: 1px solid var(--gold-hermetic); border-radius: 3px;
padding: 10px;
font-family: 'Cinzel', serif; letter-spacing: 0.06em;
}
.pdf-btn:hover { background: rgba(201, 168, 76, .12); }
/* Modal */
dialog#wipe-modal {
background: var(--cosmos-black);

View File

@ -26,6 +26,10 @@ import geoip2.errors
RULES_PATH = "/usr/share/secubox/waf/waf-rules.json"
THREATS_LOG = "/var/log/secubox/waf-threats.log"
STATS_CACHE = "/tmp/secubox/waf-stats.json"
# Phase 7+ (#509) — disk-persisted counters + log byte position for
# the double-buffered cache. Survives aggregator restart, populated
# incrementally by the warm refresh loop.
STATS_DISK_CACHE = "/var/lib/secubox/waf/stats-disk-cache.json"
# Runtime state
_compiled_patterns: Dict[str, List[dict]] = {}
@ -291,70 +295,170 @@ def _get_bans() -> List[dict]:
return []
def _get_threat_stats() -> dict:
"""Get threat statistics from log with GeoIP country lookup."""
stats = {
"total_threats": 0,
"threats_today": 0,
"by_category": defaultdict(int),
"by_severity": defaultdict(int),
"top_ips": defaultdict(int),
"top_countries": defaultdict(int),
"top_vhosts": defaultdict(int),
}
ip_countries: Dict[str, str] = {} # IP → country mapping
def _load_stats_disk_cache() -> dict:
"""Load the persisted counter state + last-read byte position.
log_path = Path(THREATS_LOG)
if not log_path.exists():
return stats
today = datetime.now().date().isoformat()
geoip_reader = _get_geoip_reader()
Schema : {byte_position: int, counters: {...}, ip_countries: {...},
today_iso: 'YYYY-MM-DD', threats_today: int,
last_updated: int}
Counters are full-history accumulators ; `threats_today` is reset
at the day rollover.
"""
p = Path(STATS_DISK_CACHE)
if not p.exists():
return {}
try:
with open(log_path) as f:
for line in f:
try:
entry = json.loads(line.strip())
stats["total_threats"] += 1
return json.loads(p.read_text())
except Exception:
return {}
if entry.get("timestamp", "").startswith(today):
stats["threats_today"] += 1
stats["by_category"][entry.get("category", "unknown")] += 1
stats["by_severity"][entry.get("severity", "unknown")] += 1
# IP tracking - try both field names for compatibility
ip = entry.get("client_ip") or entry.get("ip", "unknown")
stats["top_ips"][ip] += 1
# Country lookup via GeoIP (cache per IP)
if ip not in ip_countries:
ip_countries[ip] = _lookup_country(ip, geoip_reader)
country = ip_countries[ip]
stats["top_countries"][country] += 1
# Vhost tracking
vhost = entry.get("host") or entry.get("vhost", "unknown")
stats["top_vhosts"][vhost] += 1
except json.JSONDecodeError:
pass
def _save_stats_disk_cache(state: dict) -> None:
try:
p = Path(STATS_DISK_CACHE)
p.parent.mkdir(parents=True, exist_ok=True)
# Atomic write : tmp → rename so a half-written file never
# corrupts the cache on the next load.
tmp = p.with_suffix(".tmp")
tmp.write_text(json.dumps(state))
tmp.replace(p)
except Exception:
pass
# Convert defaultdicts and get top 10
stats["by_category"] = dict(stats["by_category"])
stats["by_severity"] = dict(stats["by_severity"])
# Top IPs with country codes included
top_ips_sorted = sorted(stats["top_ips"].items(), key=lambda x: -x[1])[:10]
stats["top_ips"] = {ip: count for ip, count in top_ips_sorted}
stats["top_ips_countries"] = {ip: ip_countries.get(ip, "??") for ip, _ in top_ips_sorted}
def _get_threat_stats() -> dict:
"""Get threat statistics from log with GeoIP country lookup.
stats["top_countries"] = dict(sorted(stats["top_countries"].items(), key=lambda x: -x[1])[:10])
stats["top_vhosts"] = dict(sorted(stats["top_vhosts"].items(), key=lambda x: -x[1])[:10])
Phase 7+ (#509) — double-buffered cache : the first call after a
cold start does ONE full-log pass and persists counters + the byte
position to disk. Subsequent calls only read the new tail since
the last position. Log rotation / truncation is detected via the
file shrinking ; counters are reset cleanly.
"""
state = _load_stats_disk_cache()
today = datetime.now().date().isoformat()
return stats
# Reset accumulators if the day has rolled over.
if state.get("today_iso") != today:
state["today_iso"] = today
state["threats_today"] = 0
counters = state.get("counters", {})
by_category = defaultdict(int, counters.get("by_category", {}))
by_severity = defaultdict(int, counters.get("by_severity", {}))
top_ips = defaultdict(int, counters.get("top_ips", {}))
top_countries = defaultdict(int, counters.get("top_countries", {}))
top_vhosts = defaultdict(int, counters.get("top_vhosts", {}))
total_threats = counters.get("total_threats", 0)
threats_today = state.get("threats_today", 0)
ip_countries: Dict[str, str] = dict(state.get("ip_countries", {}))
log_path = Path(THREATS_LOG)
if not log_path.exists():
# Return whatever's cached.
return _finalize_stats(
total_threats, threats_today, by_category, by_severity,
top_ips, top_countries, top_vhosts, ip_countries,
)
geoip_reader = _get_geoip_reader()
try:
size_now = log_path.stat().st_size
byte_position = state.get("byte_position", 0)
# Log rotation / truncation : the file shrank since last read.
# Drop accumulators ; we'll rebuild from the new (smaller) file.
if size_now < byte_position:
by_category.clear(); by_severity.clear(); top_ips.clear()
top_countries.clear(); top_vhosts.clear()
total_threats = 0
threats_today = 0
byte_position = 0
ip_countries.clear()
if size_now > byte_position:
with open(log_path) as f:
f.seek(byte_position)
for line in f:
try:
entry = json.loads(line.strip())
total_threats += 1
if entry.get("timestamp", "").startswith(today):
threats_today += 1
by_category[entry.get("category", "unknown")] += 1
by_severity[entry.get("severity", "unknown")] += 1
ip = entry.get("client_ip") or entry.get("ip", "unknown")
top_ips[ip] += 1
if ip not in ip_countries:
ip_countries[ip] = _lookup_country(ip, geoip_reader)
top_countries[ip_countries[ip]] += 1
vhost = entry.get("host") or entry.get("vhost", "unknown")
top_vhosts[vhost] += 1
except json.JSONDecodeError:
pass
byte_position = f.tell()
except Exception:
pass
# Cap the ip_countries dict so it doesn't grow without bound.
# Top-1000 most recently seen IPs is plenty for the dashboard.
if len(ip_countries) > 1200:
ip_countries = dict(
sorted(ip_countries.items(), key=lambda kv: -top_ips.get(kv[0], 0))[:1000]
)
# Persist before returning so the next call starts from here.
_save_stats_disk_cache({
"today_iso": today,
"threats_today": threats_today,
"byte_position": byte_position,
"counters": {
"total_threats": total_threats,
"by_category": dict(by_category),
"by_severity": dict(by_severity),
"top_ips": dict(top_ips),
"top_countries": dict(top_countries),
"top_vhosts": dict(top_vhosts),
},
"ip_countries": ip_countries,
"last_updated": int(time.time()),
})
return _finalize_stats(
total_threats, threats_today, by_category, by_severity,
top_ips, top_countries, top_vhosts, ip_countries,
)
def _finalize_stats(
total_threats: int, threats_today: int,
by_category, by_severity, top_ips, top_countries, top_vhosts,
ip_countries: dict,
) -> dict:
"""Shape the dashboard-friendly result : top-10 lists + plain dicts."""
top_ips_sorted = sorted(top_ips.items(), key=lambda x: -x[1])[:10]
return {
"total_threats": total_threats,
"threats_today": threats_today,
"by_category": dict(by_category),
"by_severity": dict(by_severity),
"top_ips": {ip: count for ip, count in top_ips_sorted},
"top_ips_countries": {
ip: ip_countries.get(ip, "??") for ip, _ in top_ips_sorted
},
"top_countries": dict(
sorted(top_countries.items(), key=lambda x: -x[1])[:10]
),
"top_vhosts": dict(
sorted(top_vhosts.items(), key=lambda x: -x[1])[:10]
),
}
# ───────────────────────────────────────────────────────────────────────

View File

@ -1,3 +1,29 @@
secubox-waf (1.2.2-1~bookworm1) bookworm; urgency=medium
* Phase 11+ (#509) — double-buffered cache for WAF stats consumed by
both /waf/ and /soc/ dashboards.
Live triage on gk2 (2026-06-10) found the aggregator pinned at
89 % CPU with 8+ concurrent open file descriptors on the 110 MB
waf-threats.log because _get_threat_stats() iterated the full
JSONL on every request. Both dashboards showed empty cards.
Fix : incremental log reader with persisted byte position.
- New disk cache : /var/lib/secubox/waf/stats-disk-cache.json
stores counters + byte_position + ip_countries. Atomic
write (.tmp -> rename) so a crash mid-write never corrupts.
- _get_threat_stats() reloads from disk, seeks to the last
position, reads only the new tail since then.
- Log rotation / truncation detected via size shrink ; counters
reset cleanly.
- Day rollover resets threats_today only ; full-history
counters keep accumulating.
- ip_countries dict capped at 1200 entries (most-active 1000
retained on overflow).
Net effect : /waf/stats steady-state under 100 ms ; warm refresh
cycle under 1 s per tick instead of 30 s ; aggregator CPU drops
back to idle.
-- Gerald KERMA <devel@cybermind.fr> Wed, 10 Jun 2026 08:40:00 +0200
secubox-waf (1.2.1-1~bookworm1) bookworm; urgency=medium
* Phase 7 follow-up (#498) — LXC mitmproxy.service memory hygiene :