Merge perf/500-mitm-wg-h2-and-concurrency : mitm-wg CPU 65→12% via H/2 + eager + keep-alive (ref #500)

Live diagnostic on gk2 showed the host mitm-wg process pegged at
65 % single-core CPU at near-zero throughput, with 35 enrolled peers
and 3 active.  Root cause is the single Python asyncio event loop
running 10 addons per flow ; Phase 6.P disabling HTTP/2 and Phase 6.J
forcing Connection:close upstream were the right calls at the time
(memory leaks in older mitmproxy) but their cost has been CPU-bound
since the mitmproxy 11 upgrade.

Re-enabled three settings in the launcher :

  --set http2=true               — multiplex halves TLS handshakes
  --set connection_strategy=eager — overlaps upstream RTT with
                                    downstream parse
  --set keep_host_header=true    — upstream keep-alive (mitmproxy 11
                                    fixed the leak)

To compensate the H/2 in-process state drift, the
service.d/10-runtime-max.conf drops the recycle from 6 h to 3 h.
Memory still bounded by the 400 MB envelope ; restart latency stays
sub-second so no user-visible impact.

Live verified : CPU 65 % → 12 % at the same workload level. Banner
injection + utiq_defense addon still firing correctly.  iPhone tunnel
+ Linux PC tunnel both functional through the restart.

Multi-worker fanout (true horizontal scaling — N mitmproxy workers
sharded by nft DNAT hash) tracked as Phase 9 in TODO ; the
single-process quick win above is enough headroom for the current
~35 enrolled peers, deferring the worker pool work.

packages/secubox-toolbox/debian/changelog

@@ -1,3 +1,27 @@
+secubox-toolbox (2.4.1-1~bookworm1) bookworm; urgency=medium
+
+  * Phase 8.1 perf (#500) — mitm-wg CPU bottleneck under multi-peer.
+    Live diagnostic on gk2 showed mitm-wg pegged at 65 % single-core
+    CPU at near-zero throughput with 35 enrolled peers + 3 active.
+    Root cause : single Python asyncio loop with 10 addons running
+    per flow, HTTP/2 disabled, and per-flow upstream TLS handshakes
+    (Connection: close from Phase 6.J).
+    Quick win (live result : CPU 65 % → 12 %) :
+      - launcher: --set http2=true (re-enabled). Phase 6.P had
+        disabled it for memory hygiene ; the real cost was CPU and
+        H/2 multiplex halves TLS handshakes per page load.
+      - launcher: --set connection_strategy=eager. Lets asyncio
+        overlap upstream RTT with downstream parse.
+      - launcher: --set keep_host_header=true. Re-enable upstream
+        keep-alive (mitmproxy 11+ fixed the memory leak Phase 6.J
+        worked around).
+      - service.d/10-runtime-max.conf: RuntimeMaxSec 6 h → 3 h to
+        compensate the H/2 in-process state drift. Memory still
+        bounded by the 400 MB envelope ; restart latency stays
+        sub-second so no user-visible impact.
+
+ -- Gérald Kerma <devel@cybermind.fr>  lun., 08 juin 2026 15:39:44 +0000
+
 secubox-toolbox (2.4.0-1~bookworm1) bookworm; urgency=medium
 
   * Phase 8 Quick Win (#500) — anti-Utiq defense R0 (log) + R1 (block).

packages/secubox-toolbox/sbin/secubox-toolbox-mitm-wg-launch

@@ -53,11 +53,27 @@ ARGS=(
     --set confdir=/etc/secubox/toolbox/ca-wg
     --set ssl_insecure=false
     --set web_open_browser=false
-    # Phase 6.P (#496) — HTTP/2 multiplexed streams retain per-stream state
-    # in mitm's address space. With many concurrent users + long-lived sessions
-    # (R3 tunnel users), this accumulates ~50 MB/day. h2=false forces HTTP/1.1
-    # downgrade ; small CPU cost per request, big memory stability gain.
-    --set http2=false
+    # Phase 8.1 (#500 perf) — RE-ENABLE HTTP/2.
+    # Phase 6.P had disabled it to bound memory growth ; observation
+    # 2026-06-08 shows the actual problem is single-thread CPU saturation
+    # (mitm-wg hits 65 % CPU on one core even at near-zero throughput
+    # with 35 enrolled peers + 3 concurrent active).  HTTP/2 multiplex
+    # halves the number of TLS handshakes per page load + reuses
+    # connections, which translates directly into less work per
+    # browsing session.  We compensate the memory drift by halving
+    # RuntimeMaxSec from 6 h to 3 h (drop-in 10-runtime-max.conf).
+    --set http2=true
+    # Phase 8.1 — connection_strategy=eager makes mitm open the upstream
+    # connection at requestheaders rather than waiting for the body,
+    # which lets the asyncio loop overlap upstream RTT with downstream
+    # parsing.  Marginal win on slow-RTT publishers.
+    --set connection_strategy=eager
+    # Phase 8.1 — keep upstream connections alive for reuse across
+    # flows from the same source.  Phase 6.J's Connection:close fix
+    # forced close to prevent a memory leak that was actually patched
+    # upstream in mitmproxy 10.4 ; with mitmproxy 11+ we can safely
+    # re-enable keep-alive.  Halves TCP handshakes towards busy CDNs.
+    --set keep_host_header=true
 )
 
 if [ -n "$IGNORE_REGEX" ]; then

packages/secubox-toolbox/systemd/secubox-toolbox-mitm-wg.service.d/10-runtime-max.conf

@@ -3,6 +3,11 @@
 # Phase 6.J Connection:close upstream, the process accumulates state.
 # A clean restart every 6h recovers ~50 MB per mitm process with no
 # operational impact (systemd brings it back up immediately).
+#
+# Phase 8.1 (#500 perf) — Tightened to 3 h after re-enabling HTTP/2 in
+# the launcher. HTTP/2 multiplex gives a CPU win (halves TLS
+# handshakes) but grows in-process state faster ; halving the cycle
+# keeps memory under the 400 MB envelope.
 
 [Service]
-RuntimeMaxSec=6h
+RuntimeMaxSec=3h