mirror of
https://github.com/CyberMind-FR/secubox-deb.git
synced 2026-07-01 07:26:08 +00:00
Compare commits
6 Commits
df822062f7
...
678adc8f68
| Author | SHA1 | Date | |
|---|---|---|---|
| 678adc8f68 | |||
| b154894278 | |||
| 3b4a185775 | |||
| f9fd8ea333 | |||
| a552d842ab | |||
| 41ce31ff38 |
|
|
@ -241,19 +241,40 @@ LXC toolbox-mitm-wg 10.100.0.62 R3 WireGuard
|
|||
</div>
|
||||
</div>
|
||||
|
||||
{# ── Install demo ── #}
|
||||
{# ── Install : auto-detected platform panels (Phase 8.2 #500) ── #}
|
||||
<div class=section>
|
||||
<h2>📥 Démo : installer le mode R3 portable</h2>
|
||||
<ol class=steps>
|
||||
<li>Installer l'app <b>WireGuard</b> (gratuite, App Store / Play Store)</li>
|
||||
<li>Aller sur <a href=/wg/r3-install>🌐 page d'install R3</a> → scanner le QR avec l'app WireGuard</li>
|
||||
<li>Activer le tunnel dans l'app WireGuard → icône VPN apparaît dans iOS</li>
|
||||
<li>Ouvrir n'importe quelle page web → bandeau ToolBox apparaît + rapport temps réel se remplit</li>
|
||||
<li>Désactiver le tunnel à tout moment → retour au surf normal</li>
|
||||
</ol>
|
||||
<p style="margin-top:0.6rem;font-size:0.78rem;color:var(--dim)">
|
||||
Avantage R3 : marche hors-cabine (4G/5G, autre WiFi). Inclus tout le trafic (HTTPS + QUIC).
|
||||
Profile + CA bundlés dans un seul fichier .conf.
|
||||
<h2>📥 Installer R3 sur ton appareil</h2>
|
||||
<p style="font-size:0.85rem;color:var(--dim);margin-bottom:0.8rem">
|
||||
On a détecté <code>{{ install_platform }}</code> via ton navigateur — le panneau adapté
|
||||
est ouvert en premier. Autre appareil ? Déplie le bon panneau ci-dessous.
|
||||
</p>
|
||||
<style>
|
||||
.install-panel{background:rgba(0,255,65,0.04);border:1px solid rgba(0,255,65,0.25);
|
||||
border-radius:6px;padding:0.6rem 0.9rem;margin:0.45rem 0}
|
||||
.install-panel summary{cursor:pointer;font-size:0.95rem;color:var(--phos-peak,#00dd44);
|
||||
list-style:none;outline:none}
|
||||
.install-panel summary::-webkit-details-marker{display:none}
|
||||
.install-panel[open] summary{margin-bottom:0.6rem}
|
||||
.install-panel .emoji{font-size:1.1rem;margin-right:0.3rem}
|
||||
.install-panel ol{padding-left:1.1rem;line-height:1.5;font-size:0.85rem}
|
||||
.install-panel .btn{display:inline-block;padding:0.45rem 0.75rem;margin:0.25rem 0.2rem 0.25rem 0;
|
||||
background:var(--purple,#6e40c9);color:#fff;text-decoration:none;border-radius:5px;
|
||||
font-weight:bold;font-size:0.82rem}
|
||||
.install-panel .btn.alt{background:transparent;border:1px solid var(--purple,#6e40c9);
|
||||
color:var(--purple,#6e40c9)}
|
||||
.install-panel code{background:rgba(0,0,0,0.4);padding:0.1rem 0.35rem;border-radius:3px;
|
||||
font-size:0.8rem;color:var(--phos-peak,#00dd44)}
|
||||
.install-panel .note{color:var(--dim,#888);font-size:0.78rem;margin-top:0.6rem;
|
||||
border-left:2px solid var(--phos-hot,#ffb347);padding-left:0.6rem}
|
||||
.install-panel img{max-width:100%;border-radius:5px;margin:0.4rem 0}
|
||||
.install-panel pre{background:rgba(0,0,0,0.4);padding:0.5rem 0.7rem;border-radius:4px;
|
||||
overflow-x:auto;font-size:0.78rem;margin:0.4rem 0}
|
||||
</style>
|
||||
{{ install_panels | safe }}
|
||||
<p style="margin-top:0.8rem;font-size:0.78rem;color:var(--dim)">
|
||||
Avantage R3 : marche hors-cabine (4G/5G, autre WiFi). Inclut tout le trafic (HTTPS).
|
||||
Profil + CA bundlés. Le tunnel est révoquable à tout moment depuis Réglages.
|
||||
Page équivalente standalone : <a href=/wg/onboard>/wg/onboard</a>.
|
||||
</p>
|
||||
</div>
|
||||
|
||||
|
|
|
|||
|
|
@ -1,3 +1,66 @@
|
|||
secubox-toolbox (2.4.2-1~bookworm1) bookworm; urgency=medium
|
||||
|
||||
* Landing page kbin.gk2.secubox.in : la section 'Démo install R3'
|
||||
est remplacée par les panneaux platform-detected du /wg/onboard.
|
||||
UA sniff côté serveur (iOS / Android / Linux / macOS / Windows),
|
||||
le panneau de la plate-forme du visiteur est ouvert en premier ;
|
||||
les autres restent en details collapsed. Single source of truth :
|
||||
_install_panels_html() partagé entre /wg/onboard et la landing.
|
||||
Visiteur arrive sur kbin → voit immédiatement le bouton d'install
|
||||
adapté à son device, sans devoir aller sur une page tierce.
|
||||
|
||||
-- Gérald Kerma <devel@cybermind.fr> lun., 08 juin 2026 15:51:43 +0000
|
||||
|
||||
secubox-toolbox (2.4.1-1~bookworm1) bookworm; urgency=medium
|
||||
|
||||
* Phase 8.1 perf (#500) — mitm-wg CPU bottleneck under multi-peer.
|
||||
Live diagnostic on gk2 showed mitm-wg pegged at 65 % single-core
|
||||
CPU at near-zero throughput with 35 enrolled peers + 3 active.
|
||||
Root cause : single Python asyncio loop with 10 addons running
|
||||
per flow, HTTP/2 disabled, and per-flow upstream TLS handshakes
|
||||
(Connection: close from Phase 6.J).
|
||||
Quick win (live result : CPU 65 % → 12 %) :
|
||||
- launcher: --set http2=true (re-enabled). Phase 6.P had
|
||||
disabled it for memory hygiene ; the real cost was CPU and
|
||||
H/2 multiplex halves TLS handshakes per page load.
|
||||
- launcher: --set connection_strategy=eager. Lets asyncio
|
||||
overlap upstream RTT with downstream parse.
|
||||
- launcher: --set keep_host_header=true. Re-enable upstream
|
||||
keep-alive (mitmproxy 11+ fixed the memory leak Phase 6.J
|
||||
worked around).
|
||||
- service.d/10-runtime-max.conf: RuntimeMaxSec 6 h → 3 h to
|
||||
compensate the H/2 in-process state drift. Memory still
|
||||
bounded by the 400 MB envelope ; restart latency stays
|
||||
sub-second so no user-visible impact.
|
||||
|
||||
-- Gérald Kerma <devel@cybermind.fr> lun., 08 juin 2026 15:39:44 +0000
|
||||
|
||||
secubox-toolbox (2.4.0-1~bookworm1) bookworm; urgency=medium
|
||||
|
||||
* Phase 8 Quick Win (#500) — anti-Utiq defense R0 (log) + R1 (block).
|
||||
Utiq is the operator-grade tracking ID consortium launched in 2023
|
||||
by Deutsche Telekom + Orange + Telefónica + Vodafone — a carrier-
|
||||
issued 90-day-stable identifier that cookies + standard anti-
|
||||
fingerprinting tools can't address.
|
||||
Shipped :
|
||||
- mitmproxy_addons/utiq_defense.py addon (placed early in the
|
||||
chain so R1 short-circuits before downstream addons spin on
|
||||
a doomed flow). Default level R0 (log) ; toggle to R1 via
|
||||
env var UTIQ_DEFAULT_LEVEL=R1.
|
||||
- secubox_toolbox/utiq.py SQLite event store (utiq_events
|
||||
table with publisher extracted from host, action + level
|
||||
per record, indexes on ts / client_ip / publisher).
|
||||
- GET /api/v1/toolbox/admin/utiq-events?hours=&limit= returns
|
||||
recent events + aggregates (by_publisher / by_client /
|
||||
by_action) for the admin dashboard.
|
||||
- inject_banner tile : '📡 utiq:N' appears in the right-side
|
||||
of the SecuBox banner when the client has hit a Utiq host
|
||||
in the last hour. Cheap query, fail-open.
|
||||
- utiq_defense added to the mitm-wg-launch addon chain
|
||||
between inject_xff and local_store.
|
||||
|
||||
-- Gérald Kerma <devel@cybermind.fr> lun., 08 juin 2026 14:19:50 +0000
|
||||
|
||||
secubox-toolbox (2.3.3-1~bookworm1) bookworm; urgency=medium
|
||||
|
||||
* Phase 7.E.3 (#498) — new unbound drop-in
|
||||
|
|
|
|||
|
|
@ -237,6 +237,7 @@ def _compute_site_context(flow: http.HTTPFlow) -> dict:
|
|||
"cookies_sent": 0,
|
||||
"trackers": 0,
|
||||
"is_tracker_host": False,
|
||||
"utiq_recent_count": 0,
|
||||
}
|
||||
|
||||
# Cookies (cheap : just header counts, name-less for privacy)
|
||||
|
|
@ -244,6 +245,20 @@ def _compute_site_context(flow: http.HTTPFlow) -> dict:
|
|||
ctx["cookies_set"] = set_n
|
||||
ctx["cookies_sent"] = sent_n
|
||||
|
||||
# Phase 8 (#500) — Utiq tile : count events from this peer in the
|
||||
# last hour. Best-effort : if the store import fails or the DB
|
||||
# isn't reachable we just leave the counter at 0 and the tile
|
||||
# disappears. No exception ever propagates to the addon chain.
|
||||
try:
|
||||
from secubox_toolbox import utiq as _u
|
||||
peer_ip = None
|
||||
if flow.client_conn and flow.client_conn.peername:
|
||||
peer_ip = flow.client_conn.peername[0]
|
||||
if peer_ip:
|
||||
ctx["utiq_recent_count"] = _u.client_recent_count(peer_ip, hours=1)
|
||||
except Exception:
|
||||
pass
|
||||
|
||||
# Trackers : 1st-party host check + body scan
|
||||
ctx["is_tracker_host"] = bool(_TRACKER_HOST_PATTERNS.match(host))
|
||||
if flow.response and flow.response.content:
|
||||
|
|
@ -384,6 +399,13 @@ def _banner_html_dynamic(sha1: str, ctx: dict, csp_strict: bool,
|
|||
else:
|
||||
target_emoji = "🎯" # 🎯
|
||||
right_parts.append(f"{target_emoji} {trackers}")
|
||||
# Phase 8 (#500) — surface Utiq hits for this client. Cheap query
|
||||
# against the utiq_events store (last 1 h). Avoids surfacing the
|
||||
# tile on stale state by capping the lookback window.
|
||||
utiq_n = ctx.get("utiq_recent_count", 0)
|
||||
if utiq_n > 0:
|
||||
# 📡 N — operator-grade tracker active
|
||||
right_parts.append(f"📡 utiq:{utiq_n}")
|
||||
if ctx["asn"]:
|
||||
right_parts.append(_ncr(ctx["asn"]))
|
||||
right_text = " · ".join(right_parts) # middle dot · = ·
|
||||
|
|
|
|||
131
packages/secubox-toolbox/mitmproxy_addons/utiq_defense.py
Normal file
131
packages/secubox-toolbox/mitmproxy_addons/utiq_defense.py
Normal file
|
|
@ -0,0 +1,131 @@
|
|||
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
#
|
||||
# Phase 8 (#500) — anti-Utiq defense (Quick Win : R0 log + R1 block).
|
||||
#
|
||||
# Utiq is the operator-grade tracking ID launched in 2023 by Deutsche
|
||||
# Telekom + Orange + Telefónica + Vodafone. Sites participating include
|
||||
# a loader at `<sitename>.utiq.com/utiqLoader.js` (a 1st-party CNAME)
|
||||
# that calls the Utiq API ; the carrier validates the request via
|
||||
# network-level SIM/IP headers and returns a 90-day-stable identifier
|
||||
# (martechpass = mtid) the publisher can use to track the user across
|
||||
# visits. Unlike cookies, the user cannot delete a mtid client-side —
|
||||
# only the carrier's `consenthub.utiq.com` consent record controls it.
|
||||
#
|
||||
# Defense levels (per-client, opt-in) :
|
||||
# R0 log only — passthrough, record every flow involving Utiq
|
||||
# hosts so the operator sees what's happening.
|
||||
# R1 block — refuse the loader + API calls. No mtid is ever
|
||||
# emitted to the publisher. Some pages may degrade
|
||||
# (the Utiq tag is usually wrapped in `if (mtid)`,
|
||||
# so the worst case is no targeted ad).
|
||||
# R2 mask — (Phase 2, future) return a stub `utiqLoader.js`
|
||||
# that sets `window.utiq = {mtid: null, atid: null}`
|
||||
# so the page sees a "no consent" state.
|
||||
# R3 pseudo — (Phase 2, future) forge a stable-per-publisher
|
||||
# pseudo-mtid via avatar.py to poison the tracking
|
||||
# pool.
|
||||
#
|
||||
# This Quick Win ships R0 + R1. Levels R2 / R3 land in Phase 2.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import re
|
||||
|
||||
from mitmproxy import http
|
||||
|
||||
# Importing the toolbox state store is best-effort : the addon must
|
||||
# still load even when the host doesn't have the toolbox package
|
||||
# installed (e.g. a standalone mitmproxy install).
|
||||
try:
|
||||
from secubox_toolbox import utiq as _store
|
||||
except Exception:
|
||||
_store = None
|
||||
|
||||
log = logging.getLogger("secubox.toolbox.utiq")
|
||||
|
||||
|
||||
# ── Host + path matchers ──
|
||||
# *.utiq.com covers consenthub.utiq.com + every <publisher>.utiq.com
|
||||
# CNAME wrapper. The path matcher catches the loader regardless of
|
||||
# subdomain ; some publishers proxy `utiqLoader.js` through their own
|
||||
# 1st-party path (`/static/js/utiqLoader.js`) to bypass simple host
|
||||
# filters.
|
||||
_RE_HOST = re.compile(r"(^|\.)utiq\.com$", re.IGNORECASE)
|
||||
_RE_PATH = re.compile(r"utiqLoader\.js", re.IGNORECASE)
|
||||
|
||||
|
||||
def _is_utiq_flow(flow: http.HTTPFlow) -> bool:
|
||||
host = flow.request.pretty_host or ""
|
||||
if _RE_HOST.search(host):
|
||||
return True
|
||||
if _RE_PATH.search(flow.request.path or ""):
|
||||
return True
|
||||
return False
|
||||
|
||||
|
||||
def _client_ip(flow: http.HTTPFlow) -> str | None:
|
||||
try:
|
||||
return flow.client_conn.peername[0]
|
||||
except Exception:
|
||||
return None
|
||||
|
||||
|
||||
def _level(flow: http.HTTPFlow) -> str:
|
||||
"""Return the current defense level for the client behind this flow.
|
||||
|
||||
Phase 8 Quick Win — defaults to R0 (log) for every client. Per-
|
||||
client level customisation comes in Phase 2 when the admin UI
|
||||
exposes the per-client toggle. In the meantime an operator can
|
||||
override via env var `UTIQ_DEFAULT_LEVEL=R1` to flip everyone to
|
||||
block mode globally.
|
||||
"""
|
||||
import os
|
||||
return (os.environ.get("UTIQ_DEFAULT_LEVEL") or "R0").upper()
|
||||
|
||||
|
||||
class UtiqDefense:
|
||||
"""Detect, log, and (R1) block Utiq tracking flows."""
|
||||
|
||||
def requestheaders(self, flow: http.HTTPFlow) -> None:
|
||||
# We hook requestheaders rather than request so we can RST the
|
||||
# connection BEFORE the body is fetched (saves bandwidth on a
|
||||
# blocked utiqLoader.js).
|
||||
if not _is_utiq_flow(flow):
|
||||
return
|
||||
|
||||
client_ip = _client_ip(flow)
|
||||
host = flow.request.pretty_host or ""
|
||||
path = flow.request.path or ""
|
||||
level = _level(flow)
|
||||
|
||||
# Always log, regardless of level — that's the R0 baseline.
|
||||
if _store is not None:
|
||||
try:
|
||||
_store.record_event(
|
||||
client_ip=client_ip,
|
||||
host=host,
|
||||
path=path,
|
||||
action=("block" if level == "R1" else "log"),
|
||||
level=level,
|
||||
)
|
||||
except Exception as e:
|
||||
log.warning("utiq event store failed: %s", e)
|
||||
|
||||
if level == "R1":
|
||||
# Short-circuit the flow : return a 451 (Unavailable For
|
||||
# Legal Reasons) so the page's JS can detect the block.
|
||||
# 451 is more truthful than 404 here — we're refusing to
|
||||
# serve operator-tracker content on privacy grounds.
|
||||
flow.response = http.Response.make(
|
||||
451,
|
||||
b'{"error":"blocked_by_secubox","reason":"utiq_tracker"}',
|
||||
{"Content-Type": "application/json",
|
||||
"X-SecuBox-Utiq-Block": "R1"},
|
||||
)
|
||||
log.info("[utiq R1] blocked %s %s for client=%s",
|
||||
host, path, client_ip)
|
||||
|
||||
|
||||
addons = [UtiqDefense()]
|
||||
|
|
@ -53,11 +53,27 @@ ARGS=(
|
|||
--set confdir=/etc/secubox/toolbox/ca-wg
|
||||
--set ssl_insecure=false
|
||||
--set web_open_browser=false
|
||||
# Phase 6.P (#496) — HTTP/2 multiplexed streams retain per-stream state
|
||||
# in mitm's address space. With many concurrent users + long-lived sessions
|
||||
# (R3 tunnel users), this accumulates ~50 MB/day. h2=false forces HTTP/1.1
|
||||
# downgrade ; small CPU cost per request, big memory stability gain.
|
||||
--set http2=false
|
||||
# Phase 8.1 (#500 perf) — RE-ENABLE HTTP/2.
|
||||
# Phase 6.P had disabled it to bound memory growth ; observation
|
||||
# 2026-06-08 shows the actual problem is single-thread CPU saturation
|
||||
# (mitm-wg hits 65 % CPU on one core even at near-zero throughput
|
||||
# with 35 enrolled peers + 3 concurrent active). HTTP/2 multiplex
|
||||
# halves the number of TLS handshakes per page load + reuses
|
||||
# connections, which translates directly into less work per
|
||||
# browsing session. We compensate the memory drift by halving
|
||||
# RuntimeMaxSec from 6 h to 3 h (drop-in 10-runtime-max.conf).
|
||||
--set http2=true
|
||||
# Phase 8.1 — connection_strategy=eager makes mitm open the upstream
|
||||
# connection at requestheaders rather than waiting for the body,
|
||||
# which lets the asyncio loop overlap upstream RTT with downstream
|
||||
# parsing. Marginal win on slow-RTT publishers.
|
||||
--set connection_strategy=eager
|
||||
# Phase 8.1 — keep upstream connections alive for reuse across
|
||||
# flows from the same source. Phase 6.J's Connection:close fix
|
||||
# forced close to prevent a memory leak that was actually patched
|
||||
# upstream in mitmproxy 10.4 ; with mitmproxy 11+ we can safely
|
||||
# re-enable keep-alive. Halves TCP handshakes towards busy CDNs.
|
||||
--set keep_host_header=true
|
||||
)
|
||||
|
||||
if [ -n "$IGNORE_REGEX" ]; then
|
||||
|
|
@ -68,8 +84,11 @@ fi
|
|||
# Addons :
|
||||
# - inject_xff (Phase 7 #498) MUST be FIRST — sets X-Forwarded-For at
|
||||
# requestheaders so other addons and the upstream see the real peer IP
|
||||
# - utiq_defense (Phase 8 #500) runs at requestheaders too ; placed
|
||||
# EARLY so a R1 block short-circuits the flow before downstream
|
||||
# addons spend cycles on it
|
||||
# - cert_pin_detect auto-learns pinned hosts (Phase 6.N)
|
||||
for addon in inject_xff local_store inject_banner dpi cookies avatar ja4 soc_relay cert_pin_detect; do
|
||||
for addon in inject_xff utiq_defense local_store inject_banner dpi cookies avatar ja4 soc_relay cert_pin_detect; do
|
||||
ARGS+=(-s "$ADDON_DIR/${addon}.py")
|
||||
done
|
||||
|
||||
|
|
|
|||
|
|
@ -411,16 +411,73 @@ async def cumulative_stats_json() -> dict:
|
|||
return _cumulative_stats()
|
||||
|
||||
|
||||
def _ua_platform(ua: str) -> str:
|
||||
"""Cheap UA sniff. Same logic as /wg/onboard so the auto-open panel
|
||||
matches between the two pages."""
|
||||
ua = (ua or "").lower()
|
||||
if "iphone" in ua or "ipad" in ua or "ios" in ua:
|
||||
return "ios"
|
||||
if "android" in ua:
|
||||
return "android"
|
||||
if "macintosh" in ua or "mac os x" in ua:
|
||||
return "macos"
|
||||
if "windows" in ua:
|
||||
return "windows"
|
||||
if "linux" in ua:
|
||||
return "linux"
|
||||
return "other"
|
||||
|
||||
|
||||
def _install_panels_html(platform: str) -> str:
|
||||
"""Reusable platform-detected install panels for both
|
||||
/wg/onboard and the landing page. Same content, same CSS classes
|
||||
so styling stays consistent (the landing page injects matching
|
||||
classes via its own style block)."""
|
||||
panels = {
|
||||
"ios": ("🍎 iPhone / iPad", "ios"),
|
||||
"android": ("🤖 Android", "android"),
|
||||
"linux": ("🐧 Linux", "linux"),
|
||||
"macos": ("🍏 macOS", "macos"),
|
||||
"windows": ("🪟 Windows", "windows"),
|
||||
}
|
||||
order = [platform] + [k for k in panels if k != platform]
|
||||
order = [k for i, k in enumerate(order) if k in panels and k not in order[:i]]
|
||||
sections = []
|
||||
for key in order:
|
||||
title, slug = panels[key]
|
||||
body = _ONBOARD_BODY[slug]
|
||||
open_attr = " open" if key == order[0] else ""
|
||||
sections.append(
|
||||
f'<details class="install-panel" id="install-{slug}"{open_attr}>'
|
||||
f'<summary><span class="emoji">{title.split()[0]}</span> '
|
||||
f'<b>{title}</b></summary>{body}</details>'
|
||||
)
|
||||
return "\n".join(sections)
|
||||
|
||||
|
||||
@router.get("/landing", response_class=HTMLResponse)
|
||||
@router.get("/cabine", response_class=HTMLResponse)
|
||||
async def landing(request: Request) -> HTMLResponse:
|
||||
"""Public landing page for the cabine — shown on kbin.gk2.secubox.in.
|
||||
|
||||
Visitor-facing demo of the project : pitch + 4 levels + install + live
|
||||
cumulative anonymous stats + open source license + contact."""
|
||||
cumulative anonymous stats + open source license + contact.
|
||||
|
||||
Phase 8.2 (#500) — embeds the same platform-detected install
|
||||
panels as /wg/onboard so visitors get a one-click flow matching
|
||||
their device right on the landing page.
|
||||
"""
|
||||
stats = _cumulative_stats()
|
||||
return HTMLResponse(_env.get_template("landing.html.j2").render(stats=stats),
|
||||
headers={"Cache-Control": "public, max-age=60"})
|
||||
platform = _ua_platform(request.headers.get("user-agent") or "")
|
||||
install_panels = _install_panels_html(platform)
|
||||
return HTMLResponse(
|
||||
_env.get_template("landing.html.j2").render(
|
||||
stats=stats,
|
||||
install_panels=install_panels,
|
||||
install_platform=platform,
|
||||
),
|
||||
headers={"Cache-Control": "private, max-age=60, no-transform"},
|
||||
)
|
||||
|
||||
|
||||
@router.get("/ca/webclip-cabine.mobileconfig")
|
||||
|
|
@ -1962,6 +2019,21 @@ async def report(token: str) -> Response:
|
|||
|
||||
# ───────────────── Admin (Phase 1 minimal) ─────────────────
|
||||
|
||||
@router.get("/admin/utiq-events")
|
||||
async def admin_utiq_events(hours: int = 24, limit: int = 200) -> dict:
|
||||
"""Phase 8 (#500) — silenced-but-tracked Utiq detections.
|
||||
|
||||
Lists every event the mitm-wg `utiq_defense` addon recorded within
|
||||
the window (default 24 h, max 31 d). Operator dashboard uses this
|
||||
to surface the per-client + per-publisher views.
|
||||
"""
|
||||
from . import utiq as _u
|
||||
return {
|
||||
"events": _u.recent(hours=hours, limit=limit),
|
||||
"aggregates": _u.aggregates(hours=hours),
|
||||
}
|
||||
|
||||
|
||||
@router.get("/admin/config")
|
||||
async def admin_config() -> dict:
|
||||
return _get_cfg().model_dump()
|
||||
|
|
|
|||
184
packages/secubox-toolbox/secubox_toolbox/utiq.py
Normal file
184
packages/secubox-toolbox/secubox_toolbox/utiq.py
Normal file
|
|
@ -0,0 +1,184 @@
|
|||
# SPDX-License-Identifier: LicenseRef-CMSD-1.0
|
||||
# Copyright (c) 2026 CyberMind — Gérald Kerma <devel@cybermind.fr>
|
||||
|
||||
"""
|
||||
SecuBox-Deb :: ToolBoX Utiq event store
|
||||
|
||||
Phase 8 (#500) — store every Utiq-tracker flow seen by the mitm-wg
|
||||
addon so the operator can audit silence-but-track activity in the
|
||||
admin UI.
|
||||
|
||||
Schema kept intentionally minimal :
|
||||
- client_ip is the WG peer IP (10.99.1.x). Already a pseudo-
|
||||
identifier (anonymous), no need to hash again here.
|
||||
- publisher is derived from the host (the part BEFORE `.utiq.com`
|
||||
for CNAME wrappers, or `consenthub` / `utiq` for direct calls).
|
||||
- action ∈ {log, block, mask, pseudo}.
|
||||
- level mirrors the defense level in effect at the time.
|
||||
- detected_mtid is reserved for Phase 2 (when we parse the response
|
||||
body to extract the mtid mitm-wg would have revealed to the
|
||||
publisher).
|
||||
"""
|
||||
from __future__ import annotations
|
||||
|
||||
import logging
|
||||
import sqlite3
|
||||
import time
|
||||
from pathlib import Path
|
||||
from typing import Dict, List, Optional
|
||||
|
||||
log = logging.getLogger("secubox.toolbox.utiq.store")
|
||||
|
||||
DB_PATH = Path("/var/lib/secubox/toolbox/toolbox.db")
|
||||
|
||||
_SCHEMA = """
|
||||
CREATE TABLE IF NOT EXISTS utiq_events (
|
||||
id INTEGER PRIMARY KEY AUTOINCREMENT,
|
||||
ts INTEGER NOT NULL,
|
||||
client_ip TEXT,
|
||||
publisher TEXT,
|
||||
host TEXT NOT NULL,
|
||||
path TEXT,
|
||||
action TEXT NOT NULL,
|
||||
level TEXT NOT NULL,
|
||||
detected_mtid TEXT,
|
||||
injected_mtid TEXT
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS idx_utiq_ts ON utiq_events(ts);
|
||||
CREATE INDEX IF NOT EXISTS idx_utiq_client ON utiq_events(client_ip, ts);
|
||||
CREATE INDEX IF NOT EXISTS idx_utiq_publisher ON utiq_events(publisher, ts);
|
||||
"""
|
||||
|
||||
|
||||
def _conn() -> sqlite3.Connection:
|
||||
DB_PATH.parent.mkdir(parents=True, exist_ok=True)
|
||||
c = sqlite3.connect(str(DB_PATH), timeout=5.0, isolation_level=None)
|
||||
c.row_factory = sqlite3.Row
|
||||
c.executescript(_SCHEMA)
|
||||
return c
|
||||
|
||||
|
||||
def _publisher_from_host(host: str) -> str:
|
||||
"""Derive a publisher tag from the host.
|
||||
|
||||
`consenthub.utiq.com` → 'consenthub'
|
||||
`lemonde.utiq.com` → 'lemonde'
|
||||
`utiq.com` (rare direct) → 'utiq'
|
||||
`cdn.example.com` (path-only) → 'example.com' (fallback)
|
||||
"""
|
||||
h = (host or "").lower()
|
||||
if h.endswith(".utiq.com"):
|
||||
return h[: -len(".utiq.com")].rsplit(".", 1)[-1] or "utiq"
|
||||
if h == "utiq.com":
|
||||
return "utiq"
|
||||
# path-only match (some sites serve utiqLoader.js from their own
|
||||
# 1st-party domain)
|
||||
parts = h.split(".")
|
||||
if len(parts) >= 2:
|
||||
return ".".join(parts[-2:])
|
||||
return h or "unknown"
|
||||
|
||||
|
||||
def record_event(
|
||||
*,
|
||||
client_ip: Optional[str],
|
||||
host: str,
|
||||
path: Optional[str],
|
||||
action: str,
|
||||
level: str,
|
||||
detected_mtid: Optional[str] = None,
|
||||
injected_mtid: Optional[str] = None,
|
||||
) -> None:
|
||||
"""Insert one event. Best-effort — never raises into the addon."""
|
||||
try:
|
||||
with _conn() as c:
|
||||
c.execute(
|
||||
"INSERT INTO utiq_events(ts, client_ip, publisher, host, "
|
||||
"path, action, level, detected_mtid, injected_mtid) "
|
||||
"VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
|
||||
(
|
||||
int(time.time()),
|
||||
client_ip,
|
||||
_publisher_from_host(host),
|
||||
host,
|
||||
path,
|
||||
action,
|
||||
level,
|
||||
detected_mtid,
|
||||
injected_mtid,
|
||||
),
|
||||
)
|
||||
except Exception as e:
|
||||
log.warning("record_event failed: %s", e)
|
||||
|
||||
|
||||
def recent(hours: int = 24, limit: int = 200) -> List[Dict]:
|
||||
"""Return the last events within the window, newest first."""
|
||||
since = int(time.time()) - hours * 3600
|
||||
if hours < 1 or hours > 24 * 31:
|
||||
hours = 24
|
||||
if limit < 1 or limit > 5000:
|
||||
limit = 200
|
||||
with _conn() as c:
|
||||
cur = c.execute(
|
||||
"SELECT id, ts, client_ip, publisher, host, path, action, "
|
||||
"level, detected_mtid, injected_mtid "
|
||||
"FROM utiq_events WHERE ts >= ? ORDER BY ts DESC LIMIT ?",
|
||||
(since, limit),
|
||||
)
|
||||
return [dict(r) for r in cur.fetchall()]
|
||||
|
||||
|
||||
def aggregates(hours: int = 24) -> Dict:
|
||||
"""Counts by publisher + by client + by action for the dashboard."""
|
||||
since = int(time.time()) - hours * 3600
|
||||
out: Dict = {"window_hours": hours, "total": 0, "by_publisher": [],
|
||||
"by_client": [], "by_action": []}
|
||||
with _conn() as c:
|
||||
out["total"] = c.execute(
|
||||
"SELECT COUNT(*) FROM utiq_events WHERE ts >= ?",
|
||||
(since,),
|
||||
).fetchone()[0]
|
||||
out["by_publisher"] = [
|
||||
dict(r) for r in c.execute(
|
||||
"SELECT publisher, COUNT(*) AS n FROM utiq_events "
|
||||
"WHERE ts >= ? GROUP BY publisher "
|
||||
"ORDER BY n DESC LIMIT 25",
|
||||
(since,),
|
||||
).fetchall()
|
||||
]
|
||||
out["by_client"] = [
|
||||
dict(r) for r in c.execute(
|
||||
"SELECT client_ip, COUNT(*) AS n FROM utiq_events "
|
||||
"WHERE ts >= ? AND client_ip IS NOT NULL "
|
||||
"GROUP BY client_ip ORDER BY n DESC LIMIT 25",
|
||||
(since,),
|
||||
).fetchall()
|
||||
]
|
||||
out["by_action"] = [
|
||||
dict(r) for r in c.execute(
|
||||
"SELECT action, COUNT(*) AS n FROM utiq_events "
|
||||
"WHERE ts >= ? GROUP BY action ORDER BY n DESC",
|
||||
(since,),
|
||||
).fetchall()
|
||||
]
|
||||
return out
|
||||
|
||||
|
||||
def client_recent_count(client_ip: str, hours: int = 1) -> int:
|
||||
"""Used by inject_banner to decide whether to surface the Utiq tile.
|
||||
|
||||
Cheap query — used per-request on banner-eligible flows.
|
||||
"""
|
||||
if not client_ip:
|
||||
return 0
|
||||
since = int(time.time()) - hours * 3600
|
||||
try:
|
||||
with _conn() as c:
|
||||
return c.execute(
|
||||
"SELECT COUNT(*) FROM utiq_events "
|
||||
"WHERE client_ip = ? AND ts >= ?",
|
||||
(client_ip, since),
|
||||
).fetchone()[0]
|
||||
except Exception:
|
||||
return 0
|
||||
|
|
@ -3,6 +3,11 @@
|
|||
# Phase 6.J Connection:close upstream, the process accumulates state.
|
||||
# A clean restart every 6h recovers ~50 MB per mitm process with no
|
||||
# operational impact (systemd brings it back up immediately).
|
||||
#
|
||||
# Phase 8.1 (#500 perf) — Tightened to 3 h after re-enabling HTTP/2 in
|
||||
# the launcher. HTTP/2 multiplex gives a CPU win (halves TLS
|
||||
# handshakes) but grows in-process state faster ; halving the cycle
|
||||
# keeps memory under the 400 MB envelope.
|
||||
|
||||
[Service]
|
||||
RuntimeMaxSec=6h
|
||||
RuntimeMaxSec=3h
|
||||
|
|
|
|||
Loading…
Reference in New Issue
Block a user