feat(core): Add 3-tier stats persistence and LuCI tree navigation
Stats Persistence Layer: - Add secubox-stats-persist daemon for never-trashed stats - 3-tier caching: RAM (/tmp) → buffer → persistent (/srv) - Hourly snapshots (24h), daily aggregates (30d) - Boot recovery from persistent storage - Heartbeat line: real-time 60-sample buffer (3min window) - Evolution view: combined influence score over time RPCD Stats Module: - get_timeline: 24h evolution for all collectors - get_evolution: combined influence score timeline - get_heartbeat_line: real-time 3min buffer - get_stats_status: persistence status and current values - get_history: historical data per collector - get_collector_cache: current cache value LuCI Tree Navigation: - Add clickable tree of all 60+ SecuBox LuCI apps - Organized by category: Security, Network, Monitoring, Services, etc. - Real-time search filter - Available at /secubox-public/luci-tree and /admin/secubox/luci-tree Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
This commit is contained in:
parent
8055bca368
commit
13c1e596d2
@ -1,6 +1,6 @@
|
|||||||
# SecuBox UI & Theme History
|
# SecuBox UI & Theme History
|
||||||
|
|
||||||
_Last updated: 2026-02-10_
|
_Last updated: 2026-02-11_
|
||||||
|
|
||||||
1. **Unified Dashboard Refresh (2025-12-20)**
|
1. **Unified Dashboard Refresh (2025-12-20)**
|
||||||
- Dashboard received the "sh-page-header" layout, hero stats, and SecuNav top tabs.
|
- Dashboard received the "sh-page-header" layout, hero stats, and SecuNav top tabs.
|
||||||
@ -1122,3 +1122,71 @@ _Last updated: 2026-02-10_
|
|||||||
- Settings: UCI form for configuration
|
- Settings: UCI form for configuration
|
||||||
- **RPCD Handler**: 11 methods (status, get_devices, get_device, get_anomalies, scan, isolate/trust/block_device, get_vendor_rules, add/delete_vendor_rule, get_cloud_map)
|
- **RPCD Handler**: 11 methods (status, get_devices, get_device, get_anomalies, scan, isolate/trust/block_device, get_vendor_rules, add/delete_vendor_rule, get_cloud_map)
|
||||||
- **ACL**: Public access for status and device list via `unauthenticated` group
|
- **ACL**: Public access for status and device list via `unauthenticated` group
|
||||||
|
|
||||||
|
59. **InterceptoR "Gandalf Proxy" Implementation (2026-02-11)**
|
||||||
|
- Created `luci-app-interceptor` — unified dashboard for 5-pillar transparent traffic interception.
|
||||||
|
- **Dashboard Features**:
|
||||||
|
- Health Score (0-100%) with color-coded display
|
||||||
|
- 5 Pillar Status Cards: WPAD Redirector, MITM Proxy, CDN Cache, Cookie Tracker, API Failover
|
||||||
|
- Per-pillar stats: threats, connections, hit ratio, trackers, stale serves
|
||||||
|
- Quick links to individual module dashboards
|
||||||
|
- **RPCD Handler** (`luci.interceptor`):
|
||||||
|
- `status`: Aggregates status from all 5 pillars
|
||||||
|
- `getPillarStatus`: Individual pillar details
|
||||||
|
- Health score calculation: 20 points per active pillar
|
||||||
|
- Checks: WPAD PAC file, mitmproxy LXC, Squid process, Cookie Tracker UCI, API Failover UCI
|
||||||
|
- Created `secubox-cookie-tracker` package — Cookie classification database + mitmproxy addon.
|
||||||
|
- **SQLite database** (`/var/lib/cookie-tracker/cookies.db`): domain, name, category, seen times, blocked status
|
||||||
|
- **Categories**: essential, functional, analytics, advertising, tracking
|
||||||
|
- **mitmproxy addon** (`mitmproxy-addon.py`): Real-time cookie extraction from Set-Cookie headers
|
||||||
|
- **Known trackers** (`known-trackers.tsv`): 100+ tracker domains (Google Analytics, Facebook, DoubleClick, etc.)
|
||||||
|
- **CLI** (`cookie-trackerctl`): status, list, classify, block, report --json
|
||||||
|
- **Init script**: procd service with SQLite database initialization
|
||||||
|
- Enhanced `luci-app-network-tweaks` with WPAD safety net:
|
||||||
|
- Added `setWpadEnforce`/`getWpadEnforce` RPCD methods
|
||||||
|
- Added `setup_wpad_enforce()` iptables function for non-compliant clients
|
||||||
|
- Redirect TCP 80/443 to Squid proxy for WPAD-ignoring clients
|
||||||
|
- Enhanced `luci-app-cdn-cache` with API failover config:
|
||||||
|
- Added `api_failover` UCI section: stale_if_error, offline_mode, collapsed_forwarding
|
||||||
|
- Modified init.d to generate API failover Squid config (refresh_pattern with stale-if-error)
|
||||||
|
- Created `/etc/hotplug.d/iface/99-cdn-offline` for WAN up/down detection
|
||||||
|
- Automatic offline mode on WAN down, disable on WAN up
|
||||||
|
- Configured `.sblocal` mesh domain via BIND zone file:
|
||||||
|
- Created `/etc/bind/zones/sblocal.zone` for internal service discovery
|
||||||
|
- Added c3box.sblocal A record pointing to 192.168.255.1
|
||||||
|
- Part of InterceptoR transparent proxy architecture (Peek/Poke/Emancipate model).
|
||||||
|
|
||||||
|
60. **3-Tier Stats Persistence & Evolution (2026-02-11)**
|
||||||
|
- Created `secubox-stats-persist` — 3-tier caching for never-trashed stats.
|
||||||
|
- **3-Tier Cache Architecture**:
|
||||||
|
- Tier 1: RAM cache (`/tmp/secubox/*.json`) — 3-30 second updates
|
||||||
|
- Tier 2: Volatile buffer — atomic writes with tmp+mv pattern
|
||||||
|
- Tier 3: Persistent storage (`/srv/secubox/stats/`) — survives reboot
|
||||||
|
- **Time-Series Evolution**:
|
||||||
|
- Hourly snapshots (24h retention) per collector
|
||||||
|
- Daily aggregates (30d retention) with min/max/avg
|
||||||
|
- Combined timeline JSON with all collectors
|
||||||
|
- **Heartbeat Line**:
|
||||||
|
- Real-time 60-sample buffer (3min window)
|
||||||
|
- Combined "influence" score: (health×40 + inv_threat×30 + inv_capacity×30)/100
|
||||||
|
- Updated every 3 seconds via daemon loop
|
||||||
|
- **Evolution View**:
|
||||||
|
- 48-hour combined metrics graph
|
||||||
|
- Health, Threat, Capacity, and Influence scores per hour
|
||||||
|
- JSON output for dashboard sparklines
|
||||||
|
- **Boot Recovery**:
|
||||||
|
- On daemon start, recovers cache from persistent storage
|
||||||
|
- Ensures stats continuity across reboots
|
||||||
|
- **RPCD Methods**:
|
||||||
|
- `get_timeline`: 24h evolution for all collectors
|
||||||
|
- `get_evolution`: Combined influence score timeline
|
||||||
|
- `get_heartbeat_line`: Real-time 3min buffer
|
||||||
|
- `get_stats_status`: Persistence status and current values
|
||||||
|
- `get_history`: Historical data for specific collector
|
||||||
|
- `get_collector_cache`: Current cache value for collector
|
||||||
|
- **Cron Jobs**:
|
||||||
|
- Every 5min: Persist cache to /srv (backup)
|
||||||
|
- Every hour: Generate timeline and evolution
|
||||||
|
- Daily: Aggregate hourly to daily, cleanup old data
|
||||||
|
- Integrated into `secubox-core` daemon startup (r16).
|
||||||
|
- Bumped `secubox-core` version to 0.10.0-r16.
|
||||||
|
|||||||
@ -11,7 +11,7 @@ LUCI_DESCRIPTION:=Unified entry point for all SecuBox applications with tabbed n
|
|||||||
LUCI_DEPENDS:=+luci-base +luci-theme-secubox
|
LUCI_DEPENDS:=+luci-base +luci-theme-secubox
|
||||||
LUCI_PKGARCH:=all
|
LUCI_PKGARCH:=all
|
||||||
PKG_VERSION:=0.7.0
|
PKG_VERSION:=0.7.0
|
||||||
PKG_RELEASE:=2
|
PKG_RELEASE:=3
|
||||||
PKG_LICENSE:=GPL-3.0-or-later
|
PKG_LICENSE:=GPL-3.0-or-later
|
||||||
PKG_MAINTAINER:=SecuBox Team <secubox@example.com>
|
PKG_MAINTAINER:=SecuBox Team <secubox@example.com>
|
||||||
|
|
||||||
|
|||||||
@ -0,0 +1,329 @@
|
|||||||
|
'use strict';
|
||||||
|
'require view';
|
||||||
|
'require dom';
|
||||||
|
'require poll';
|
||||||
|
|
||||||
|
// SecuBox LuCI Tree - Clickable navigation map
|
||||||
|
var LUCI_TREE = {
|
||||||
|
"SecuBox": {
|
||||||
|
path: "admin/secubox",
|
||||||
|
icon: "shield",
|
||||||
|
children: {
|
||||||
|
"Dashboard": { path: "admin/secubox/dashboard", icon: "dashboard" },
|
||||||
|
"App Store": { path: "admin/secubox/apps", icon: "store" },
|
||||||
|
"Modules": { path: "admin/secubox/modules", icon: "cubes" },
|
||||||
|
"Alerts": { path: "admin/secubox/alerts", icon: "bell" },
|
||||||
|
"Settings": { path: "admin/secubox/settings", icon: "cog" },
|
||||||
|
"Help": { path: "admin/secubox/help", icon: "question" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Admin Control": {
|
||||||
|
path: "admin/secubox/admin",
|
||||||
|
icon: "user-shield",
|
||||||
|
children: {
|
||||||
|
"Control Panel": { path: "admin/secubox/admin/dashboard", icon: "sliders" },
|
||||||
|
"Cyber Console": { path: "admin/secubox/admin/cyber-dashboard", icon: "terminal" },
|
||||||
|
"Apps Manager": { path: "admin/secubox/admin/apps", icon: "boxes" },
|
||||||
|
"Profiles": { path: "admin/secubox/admin/profiles", icon: "id-card" },
|
||||||
|
"Skills": { path: "admin/secubox/admin/skills", icon: "magic" },
|
||||||
|
"System Health": { path: "admin/secubox/admin/health", icon: "heartbeat" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Security": {
|
||||||
|
path: "admin/secubox/security",
|
||||||
|
icon: "lock",
|
||||||
|
children: {
|
||||||
|
"CrowdSec": {
|
||||||
|
path: "admin/secubox/security/crowdsec",
|
||||||
|
icon: "shield-alt",
|
||||||
|
children: {
|
||||||
|
"Overview": { path: "admin/secubox/security/crowdsec/overview" },
|
||||||
|
"Decisions": { path: "admin/secubox/security/crowdsec/decisions" },
|
||||||
|
"Alerts": { path: "admin/secubox/security/crowdsec/alerts" },
|
||||||
|
"Bouncers": { path: "admin/secubox/security/crowdsec/bouncers" },
|
||||||
|
"Setup": { path: "admin/secubox/security/crowdsec/setup" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"mitmproxy": {
|
||||||
|
path: "admin/secubox/security/mitmproxy",
|
||||||
|
icon: "eye",
|
||||||
|
children: {
|
||||||
|
"Status": { path: "admin/secubox/security/mitmproxy/status" },
|
||||||
|
"Settings": { path: "admin/secubox/security/mitmproxy/settings" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Client Guardian": { path: "admin/secubox/security/guardian", icon: "users" },
|
||||||
|
"DNS Guard": { path: "admin/secubox/security/dnsguard", icon: "dns" },
|
||||||
|
"Threat Analyst": { path: "admin/secubox/security/threat-analyst", icon: "brain" },
|
||||||
|
"Network Anomaly": { path: "admin/secubox/security/network-anomaly", icon: "chart-line" },
|
||||||
|
"Auth Guardian": { path: "admin/secubox/security/auth-guardian", icon: "key" },
|
||||||
|
"Key Storage": { path: "admin/secubox/security/ksm-manager", icon: "vault" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"AI Gateway": {
|
||||||
|
path: "admin/secubox/ai",
|
||||||
|
icon: "robot",
|
||||||
|
children: {
|
||||||
|
"AI Insights": { path: "admin/secubox/ai/insights", icon: "lightbulb" },
|
||||||
|
"LocalRecall": { path: "admin/secubox/ai/localrecall", icon: "memory" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"MirrorBox": {
|
||||||
|
path: "admin/secubox/mirrorbox",
|
||||||
|
icon: "network-wired",
|
||||||
|
children: {
|
||||||
|
"Overview": { path: "admin/secubox/mirrorbox/overview", icon: "home" },
|
||||||
|
"P2P Hub": { path: "admin/secubox/mirrorbox/hub", icon: "hubspot" },
|
||||||
|
"Peers": { path: "admin/secubox/mirrorbox/peers", icon: "users" },
|
||||||
|
"Services": { path: "admin/secubox/mirrorbox/services", icon: "server" },
|
||||||
|
"Factory": { path: "admin/secubox/mirrorbox/factory", icon: "industry" },
|
||||||
|
"App Store": { path: "admin/secubox/mirrorbox/packages", icon: "store" },
|
||||||
|
"Dev Status": { path: "admin/secubox/mirrorbox/devstatus", icon: "code" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Network": {
|
||||||
|
path: "admin/secubox/network",
|
||||||
|
icon: "sitemap",
|
||||||
|
children: {
|
||||||
|
"Network Modes": { path: "admin/secubox/network/modes", icon: "random" },
|
||||||
|
"DNS Providers": { path: "admin/secubox/network/dns-provider", icon: "globe" },
|
||||||
|
"Service Exposure": { path: "admin/secubox/network/exposure", icon: "broadcast-tower" },
|
||||||
|
"Bandwidth Manager": { path: "admin/secubox/network/bandwidth-manager", icon: "tachometer-alt" },
|
||||||
|
"Traffic Shaper": { path: "admin/secubox/network/traffic-shaper", icon: "filter" },
|
||||||
|
"MQTT Bridge": { path: "admin/secubox/network/mqtt-bridge", icon: "exchange-alt" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Monitoring": {
|
||||||
|
path: "admin/secubox/monitoring",
|
||||||
|
icon: "chart-bar",
|
||||||
|
children: {
|
||||||
|
"Netdata": { path: "admin/secubox/monitoring/netdata", icon: "chart-area" },
|
||||||
|
"Glances": { path: "admin/secubox/monitoring/glances", icon: "eye" },
|
||||||
|
"Media Flow": { path: "admin/secubox/monitoring/mediaflow", icon: "film" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"System": {
|
||||||
|
path: "admin/secubox/system",
|
||||||
|
icon: "server",
|
||||||
|
children: {
|
||||||
|
"System Hub": { path: "admin/secubox/system/system-hub", icon: "cogs" },
|
||||||
|
"Cloning Station": { path: "admin/secubox/system/cloner", icon: "clone" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Device Intel": {
|
||||||
|
path: "admin/secubox/device-intel",
|
||||||
|
icon: "microchip",
|
||||||
|
children: {
|
||||||
|
"Dashboard": { path: "admin/secubox/device-intel/dashboard" },
|
||||||
|
"Devices": { path: "admin/secubox/device-intel/devices" },
|
||||||
|
"Mesh": { path: "admin/secubox/device-intel/mesh" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"InterceptoR": {
|
||||||
|
path: "admin/secubox/interceptor",
|
||||||
|
icon: "filter",
|
||||||
|
children: {
|
||||||
|
"Overview": { path: "admin/secubox/interceptor/overview" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"Services (LuCI)": {
|
||||||
|
path: "admin/services",
|
||||||
|
icon: "puzzle-piece",
|
||||||
|
children: {
|
||||||
|
"Service Registry": { path: "admin/services/service-registry", icon: "list" },
|
||||||
|
"HAProxy": { path: "admin/services/haproxy", icon: "random" },
|
||||||
|
"WireGuard": { path: "admin/services/wireguard", icon: "shield-alt" },
|
||||||
|
"Tor Shield": { path: "admin/services/tor-shield", icon: "user-secret" },
|
||||||
|
"VHost Manager": { path: "admin/services/vhosts", icon: "server" },
|
||||||
|
"CDN Cache": { path: "admin/services/cdn-cache", icon: "database" },
|
||||||
|
"LocalAI": { path: "admin/services/localai", icon: "brain" },
|
||||||
|
"Ollama": { path: "admin/services/ollama", icon: "comment-dots" },
|
||||||
|
"Nextcloud": { path: "admin/services/nextcloud", icon: "cloud" },
|
||||||
|
"Jellyfin": { path: "admin/services/jellyfin", icon: "film" },
|
||||||
|
"Jitsi Meet": { path: "admin/services/jitsi", icon: "video" },
|
||||||
|
"SimpleX Chat": { path: "admin/services/simplex", icon: "comments" },
|
||||||
|
"Domoticz": { path: "admin/services/domoticz", icon: "home" },
|
||||||
|
"Lyrion": { path: "admin/services/lyrion", icon: "music" },
|
||||||
|
"MagicMirror": { path: "admin/services/magicmirror2", icon: "desktop" },
|
||||||
|
"MAC Guardian": { path: "admin/services/mac-guardian", icon: "wifi" },
|
||||||
|
"Mail Server": { path: "admin/services/mailserver", icon: "envelope" },
|
||||||
|
"Mesh Link": { path: "admin/services/secubox-mesh", icon: "project-diagram" },
|
||||||
|
"MirrorNet": { path: "admin/services/mirrornet", icon: "network-wired" },
|
||||||
|
"Gitea": { path: "admin/services/gitea", icon: "code-branch" },
|
||||||
|
"Hexo CMS": { path: "admin/services/hexojs", icon: "blog" },
|
||||||
|
"MetaBlogizer": { path: "admin/services/metablogizer", icon: "rss" },
|
||||||
|
"Streamlit": { path: "admin/services/streamlit", icon: "stream" },
|
||||||
|
"PicoBrew": { path: "admin/services/picobrew", icon: "beer" },
|
||||||
|
"CyberFeed": { path: "admin/services/cyberfeed", icon: "newspaper" },
|
||||||
|
"Vortex DNS": { path: "admin/services/vortex-dns", icon: "globe" },
|
||||||
|
"Vortex Firewall": { path: "admin/services/vortex-firewall", icon: "fire" },
|
||||||
|
"Config Advisor": { path: "admin/services/config-advisor", icon: "clipboard-check" },
|
||||||
|
"Threat Monitor": { path: "admin/services/threat-monitor", icon: "exclamation-triangle" },
|
||||||
|
"Network Diagnostics": { path: "admin/services/network-diagnostics", icon: "stethoscope" },
|
||||||
|
"Backup Manager": { path: "admin/system/backup", icon: "save" }
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"IoT & Automation": {
|
||||||
|
path: "admin/secubox/services",
|
||||||
|
icon: "microchip",
|
||||||
|
children: {
|
||||||
|
"IoT Guard": { path: "admin/secubox/services/iot-guard", icon: "shield-alt" },
|
||||||
|
"Zigbee2MQTT": { path: "admin/secubox/zigbee2mqtt", icon: "broadcast-tower" },
|
||||||
|
"nDPId": { path: "admin/secubox/ndpid", icon: "search" },
|
||||||
|
"Netifyd": { path: "admin/secubox/netifyd", icon: "chart-network" }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
};
|
||||||
|
|
||||||
|
return view.extend({
|
||||||
|
render: function() {
|
||||||
|
var container = E('div', { 'class': 'cbi-map', 'style': 'background:#111;min-height:100vh;padding:20px;' }, [
|
||||||
|
E('style', {}, `
|
||||||
|
.luci-tree { font-family: monospace; color: #0f0; }
|
||||||
|
.luci-tree a { color: #0ff; text-decoration: none; }
|
||||||
|
.luci-tree a:hover { color: #fff; text-decoration: underline; }
|
||||||
|
.tree-section { margin: 15px 0; padding: 10px; background: #1a1a1a; border-left: 3px solid #0f0; border-radius: 4px; }
|
||||||
|
.tree-section-title { font-size: 18px; color: #0f0; margin-bottom: 10px; cursor: pointer; }
|
||||||
|
.tree-section-title:hover { color: #0ff; }
|
||||||
|
.tree-item { padding: 3px 0 3px 20px; border-left: 1px dashed #333; }
|
||||||
|
.tree-item:last-child { border-left-color: transparent; }
|
||||||
|
.tree-item::before { content: "├── "; color: #555; }
|
||||||
|
.tree-item:last-child::before { content: "└── "; }
|
||||||
|
.tree-nested { margin-left: 20px; }
|
||||||
|
.tree-icon { margin-right: 8px; opacity: 0.7; }
|
||||||
|
.tree-header { text-align: center; margin-bottom: 30px; }
|
||||||
|
.tree-header h1 { color: #0f0; font-size: 28px; margin: 0; }
|
||||||
|
.tree-header p { color: #888; }
|
||||||
|
.tree-stats { display: flex; justify-content: center; gap: 30px; margin: 20px 0; }
|
||||||
|
.tree-stat { text-align: center; padding: 10px 20px; background: #222; border-radius: 8px; }
|
||||||
|
.tree-stat-value { font-size: 24px; color: #0ff; }
|
||||||
|
.tree-stat-label { font-size: 12px; color: #888; }
|
||||||
|
.tree-search { margin: 20px auto; max-width: 400px; }
|
||||||
|
.tree-search input { width: 100%; padding: 10px; background: #222; border: 1px solid #333; color: #fff; border-radius: 4px; }
|
||||||
|
.tree-search input:focus { outline: none; border-color: #0f0; }
|
||||||
|
.tree-grid { display: grid; grid-template-columns: repeat(auto-fill, minmax(350px, 1fr)); gap: 15px; }
|
||||||
|
`),
|
||||||
|
|
||||||
|
E('div', { 'class': 'tree-header' }, [
|
||||||
|
E('h1', {}, 'SecuBox LuCI Navigation Tree'),
|
||||||
|
E('p', {}, 'Clickable map of all LuCI dashboards and modules')
|
||||||
|
]),
|
||||||
|
|
||||||
|
E('div', { 'class': 'tree-stats' }, [
|
||||||
|
E('div', { 'class': 'tree-stat' }, [
|
||||||
|
E('div', { 'class': 'tree-stat-value' }, Object.keys(LUCI_TREE).length.toString()),
|
||||||
|
E('div', { 'class': 'tree-stat-label' }, 'Categories')
|
||||||
|
]),
|
||||||
|
E('div', { 'class': 'tree-stat' }, [
|
||||||
|
E('div', { 'class': 'tree-stat-value', 'id': 'total-links' }, '...'),
|
||||||
|
E('div', { 'class': 'tree-stat-label' }, 'Total Links')
|
||||||
|
]),
|
||||||
|
E('div', { 'class': 'tree-stat' }, [
|
||||||
|
E('div', { 'class': 'tree-stat-value' }, '60+'),
|
||||||
|
E('div', { 'class': 'tree-stat-label' }, 'LuCI Apps')
|
||||||
|
])
|
||||||
|
]),
|
||||||
|
|
||||||
|
E('div', { 'class': 'tree-search' }, [
|
||||||
|
E('input', {
|
||||||
|
'type': 'text',
|
||||||
|
'placeholder': 'Search modules...',
|
||||||
|
'id': 'tree-search-input',
|
||||||
|
'oninput': 'filterTree(this.value)'
|
||||||
|
})
|
||||||
|
]),
|
||||||
|
|
||||||
|
E('div', { 'class': 'luci-tree tree-grid', 'id': 'tree-container' })
|
||||||
|
]);
|
||||||
|
|
||||||
|
// Build tree
|
||||||
|
var treeContainer = container.querySelector('#tree-container');
|
||||||
|
var totalLinks = 0;
|
||||||
|
|
||||||
|
function buildTreeNode(name, node, level) {
|
||||||
|
var items = [];
|
||||||
|
totalLinks++;
|
||||||
|
|
||||||
|
var link = E('a', {
|
||||||
|
'href': '/cgi-bin/luci/' + node.path,
|
||||||
|
'target': '_blank',
|
||||||
|
'class': 'tree-link'
|
||||||
|
}, name);
|
||||||
|
|
||||||
|
if (node.children) {
|
||||||
|
var nested = E('div', { 'class': 'tree-nested' });
|
||||||
|
Object.keys(node.children).forEach(function(childName) {
|
||||||
|
var childItems = buildTreeNode(childName, node.children[childName], level + 1);
|
||||||
|
childItems.forEach(function(item) {
|
||||||
|
nested.appendChild(E('div', { 'class': 'tree-item', 'data-name': childName.toLowerCase() }, [item]));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
items.push(link);
|
||||||
|
items.push(nested);
|
||||||
|
} else {
|
||||||
|
items.push(link);
|
||||||
|
}
|
||||||
|
|
||||||
|
return items;
|
||||||
|
}
|
||||||
|
|
||||||
|
Object.keys(LUCI_TREE).forEach(function(sectionName) {
|
||||||
|
var section = LUCI_TREE[sectionName];
|
||||||
|
var sectionDiv = E('div', { 'class': 'tree-section', 'data-section': sectionName.toLowerCase() });
|
||||||
|
|
||||||
|
var titleLink = E('a', {
|
||||||
|
'href': '/cgi-bin/luci/' + section.path,
|
||||||
|
'target': '_blank',
|
||||||
|
'class': 'tree-section-title'
|
||||||
|
}, sectionName);
|
||||||
|
|
||||||
|
sectionDiv.appendChild(titleLink);
|
||||||
|
|
||||||
|
if (section.children) {
|
||||||
|
Object.keys(section.children).forEach(function(childName) {
|
||||||
|
var childItems = buildTreeNode(childName, section.children[childName], 1);
|
||||||
|
childItems.forEach(function(item) {
|
||||||
|
sectionDiv.appendChild(E('div', { 'class': 'tree-item', 'data-name': childName.toLowerCase() }, [item]));
|
||||||
|
});
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
treeContainer.appendChild(sectionDiv);
|
||||||
|
});
|
||||||
|
|
||||||
|
container.querySelector('#total-links').textContent = totalLinks.toString();
|
||||||
|
|
||||||
|
// Add search filter script
|
||||||
|
var script = E('script', {}, `
|
||||||
|
function filterTree(query) {
|
||||||
|
query = query.toLowerCase();
|
||||||
|
var sections = document.querySelectorAll('.tree-section');
|
||||||
|
sections.forEach(function(section) {
|
||||||
|
var sectionName = section.dataset.section;
|
||||||
|
var items = section.querySelectorAll('.tree-item');
|
||||||
|
var hasMatch = sectionName.includes(query);
|
||||||
|
|
||||||
|
items.forEach(function(item) {
|
||||||
|
var name = item.dataset.name || '';
|
||||||
|
var text = item.textContent.toLowerCase();
|
||||||
|
if (text.includes(query) || name.includes(query)) {
|
||||||
|
item.style.display = '';
|
||||||
|
hasMatch = true;
|
||||||
|
} else {
|
||||||
|
item.style.display = 'none';
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
section.style.display = hasMatch ? '' : 'none';
|
||||||
|
});
|
||||||
|
}
|
||||||
|
`);
|
||||||
|
container.appendChild(script);
|
||||||
|
|
||||||
|
return container;
|
||||||
|
},
|
||||||
|
|
||||||
|
handleSaveApply: null,
|
||||||
|
handleSave: null,
|
||||||
|
handleReset: null
|
||||||
|
});
|
||||||
@ -104,6 +104,25 @@
|
|||||||
"path": "secubox-portal/devstatus"
|
"path": "secubox-portal/devstatus"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"secubox-public/luci-tree": {
|
||||||
|
"title": "LuCI Tree",
|
||||||
|
"order": 40,
|
||||||
|
"action": {
|
||||||
|
"type": "view",
|
||||||
|
"path": "secubox-portal/luci-tree"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"admin/secubox/luci-tree": {
|
||||||
|
"title": "LuCI Tree",
|
||||||
|
"order": 95,
|
||||||
|
"action": {
|
||||||
|
"type": "view",
|
||||||
|
"path": "secubox-portal/luci-tree"
|
||||||
|
},
|
||||||
|
"depends": {
|
||||||
|
"acl": ["luci-app-secubox-portal"]
|
||||||
|
}
|
||||||
|
},
|
||||||
"secubox-public/login": {
|
"secubox-public/login": {
|
||||||
"title": "Connexion",
|
"title": "Connexion",
|
||||||
"order": 99,
|
"order": 99,
|
||||||
|
|||||||
@ -45,7 +45,13 @@
|
|||||||
"p2p_discover",
|
"p2p_discover",
|
||||||
"p2p_get_catalog",
|
"p2p_get_catalog",
|
||||||
"p2p_get_peer_catalog",
|
"p2p_get_peer_catalog",
|
||||||
"p2p_get_shared_services"
|
"p2p_get_shared_services",
|
||||||
|
"get_timeline",
|
||||||
|
"get_evolution",
|
||||||
|
"get_heartbeat_line",
|
||||||
|
"get_stats_status",
|
||||||
|
"get_history",
|
||||||
|
"get_collector_cache"
|
||||||
],
|
],
|
||||||
"luci.service-registry": [
|
"luci.service-registry": [
|
||||||
"list_services",
|
"list_services",
|
||||||
|
|||||||
@ -6,7 +6,7 @@ include $(TOPDIR)/rules.mk
|
|||||||
|
|
||||||
PKG_NAME:=secubox-core
|
PKG_NAME:=secubox-core
|
||||||
PKG_VERSION:=0.10.0
|
PKG_VERSION:=0.10.0
|
||||||
PKG_RELEASE:=15
|
PKG_RELEASE:=16
|
||||||
PKG_ARCH:=all
|
PKG_ARCH:=all
|
||||||
PKG_LICENSE:=GPL-2.0
|
PKG_LICENSE:=GPL-2.0
|
||||||
PKG_MAINTAINER:=SecuBox Team
|
PKG_MAINTAINER:=SecuBox Team
|
||||||
@ -88,6 +88,7 @@ define Package/secubox-core/install
|
|||||||
$(INSTALL_BIN) ./root/usr/sbin/secubox-feedback $(1)/usr/sbin/
|
$(INSTALL_BIN) ./root/usr/sbin/secubox-feedback $(1)/usr/sbin/
|
||||||
$(INSTALL_BIN) ./root/usr/sbin/secubox-tftp-recovery $(1)/usr/sbin/
|
$(INSTALL_BIN) ./root/usr/sbin/secubox-tftp-recovery $(1)/usr/sbin/
|
||||||
$(INSTALL_BIN) ./root/usr/sbin/secubox-vhost $(1)/usr/sbin/
|
$(INSTALL_BIN) ./root/usr/sbin/secubox-vhost $(1)/usr/sbin/
|
||||||
|
$(INSTALL_BIN) ./root/usr/sbin/secubox-stats-persist $(1)/usr/sbin/
|
||||||
|
|
||||||
$(INSTALL_DIR) $(1)/usr/bin
|
$(INSTALL_DIR) $(1)/usr/bin
|
||||||
$(INSTALL_BIN) ./root/usr/bin/secubox-services-status $(1)/usr/bin/
|
$(INSTALL_BIN) ./root/usr/bin/secubox-services-status $(1)/usr/bin/
|
||||||
@ -95,9 +96,10 @@ define Package/secubox-core/install
|
|||||||
# TFTP Recovery init script
|
# TFTP Recovery init script
|
||||||
$(INSTALL_BIN) ./root/etc/init.d/secubox-tftp-recovery $(1)/etc/init.d/
|
$(INSTALL_BIN) ./root/etc/init.d/secubox-tftp-recovery $(1)/etc/init.d/
|
||||||
|
|
||||||
# File integrity monitoring cron job
|
# Cron jobs: integrity monitoring and stats persistence
|
||||||
$(INSTALL_DIR) $(1)/etc/cron.d
|
$(INSTALL_DIR) $(1)/etc/cron.d
|
||||||
$(INSTALL_DATA) ./root/etc/cron.d/secubox-integrity $(1)/etc/cron.d/
|
$(INSTALL_DATA) ./root/etc/cron.d/secubox-integrity $(1)/etc/cron.d/
|
||||||
|
$(INSTALL_DATA) ./root/etc/cron.d/secubox-stats-persist $(1)/etc/cron.d/
|
||||||
|
|
||||||
# TFTP Mesh library
|
# TFTP Mesh library
|
||||||
$(INSTALL_DIR) $(1)/usr/lib/secubox
|
$(INSTALL_DIR) $(1)/usr/lib/secubox
|
||||||
|
|||||||
@ -0,0 +1,13 @@
|
|||||||
|
# SecuBox Stats Persistence
|
||||||
|
# Periodic cache persistence and evolution generation
|
||||||
|
# Daemon runs its own loops but this ensures recovery on daemon restart
|
||||||
|
|
||||||
|
# Every 5 minutes: persist cache to /srv (backup in case daemon dies)
|
||||||
|
*/5 * * * * root /usr/sbin/secubox-stats-persist persist >/dev/null 2>&1
|
||||||
|
|
||||||
|
# Every hour: generate timeline and evolution
|
||||||
|
0 * * * * root /usr/sbin/secubox-stats-persist timeline >/dev/null 2>&1
|
||||||
|
5 * * * * root /usr/sbin/secubox-stats-persist evolution >/dev/null 2>&1
|
||||||
|
|
||||||
|
# Daily: aggregate and cleanup old history
|
||||||
|
0 1 * * * root /usr/sbin/secubox-stats-persist aggregate >/dev/null 2>&1
|
||||||
@ -0,0 +1,219 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
#
|
||||||
|
# SecuBox RPCD - Stats Evolution & Timeline
|
||||||
|
# Persistent stats, time-series evolution, combined heartbeat line
|
||||||
|
#
|
||||||
|
|
||||||
|
PERSIST_DIR="/srv/secubox/stats"
|
||||||
|
CACHE_DIR="/tmp/secubox"
|
||||||
|
|
||||||
|
# Register methods
|
||||||
|
list_methods_stats() {
|
||||||
|
add_method "get_timeline"
|
||||||
|
add_method "get_evolution"
|
||||||
|
add_method "get_heartbeat_line"
|
||||||
|
add_method "get_stats_status"
|
||||||
|
add_method "get_history"
|
||||||
|
add_method_str "get_collector_cache" "collector"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Handle method calls
|
||||||
|
handle_stats() {
|
||||||
|
local method="$1"
|
||||||
|
case "$method" in
|
||||||
|
get_timeline)
|
||||||
|
_do_get_timeline
|
||||||
|
;;
|
||||||
|
get_evolution)
|
||||||
|
_do_get_evolution
|
||||||
|
;;
|
||||||
|
get_heartbeat_line)
|
||||||
|
_do_get_heartbeat_line
|
||||||
|
;;
|
||||||
|
get_stats_status)
|
||||||
|
_do_get_stats_status
|
||||||
|
;;
|
||||||
|
get_history)
|
||||||
|
read_input_json
|
||||||
|
local collector=$(get_input "collector")
|
||||||
|
local period=$(get_input "period")
|
||||||
|
_do_get_history "$collector" "$period"
|
||||||
|
;;
|
||||||
|
get_collector_cache)
|
||||||
|
read_input_json
|
||||||
|
local collector=$(get_input "collector")
|
||||||
|
_do_get_collector_cache "$collector"
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
return 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get timeline (24h evolution for all collectors)
|
||||||
|
_do_get_timeline() {
|
||||||
|
local timeline_file="$PERSIST_DIR/timeline.json"
|
||||||
|
|
||||||
|
if [ -f "$timeline_file" ]; then
|
||||||
|
cat "$timeline_file"
|
||||||
|
else
|
||||||
|
json_init
|
||||||
|
json_add_string "error" "Timeline not generated yet"
|
||||||
|
json_add_string "hint" "Run: secubox-stats-persist timeline"
|
||||||
|
json_dump
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get evolution (combined influence score)
|
||||||
|
_do_get_evolution() {
|
||||||
|
local evolution_file="$PERSIST_DIR/evolution.json"
|
||||||
|
|
||||||
|
if [ -f "$evolution_file" ]; then
|
||||||
|
cat "$evolution_file"
|
||||||
|
else
|
||||||
|
json_init
|
||||||
|
json_add_string "error" "Evolution not generated yet"
|
||||||
|
json_add_string "hint" "Run: secubox-stats-persist evolution"
|
||||||
|
json_dump
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get heartbeat line (real-time 3min buffer)
|
||||||
|
_do_get_heartbeat_line() {
|
||||||
|
local heartbeat_file="$PERSIST_DIR/heartbeat-line.json"
|
||||||
|
|
||||||
|
if [ -f "$heartbeat_file" ]; then
|
||||||
|
cat "$heartbeat_file"
|
||||||
|
else
|
||||||
|
# Generate on-demand if daemon not running
|
||||||
|
if [ -x /usr/sbin/secubox-stats-persist ]; then
|
||||||
|
/usr/sbin/secubox-stats-persist heartbeat
|
||||||
|
else
|
||||||
|
json_init
|
||||||
|
json_add_string "error" "Heartbeat line not available"
|
||||||
|
json_dump
|
||||||
|
fi
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get stats persistence status
|
||||||
|
_do_get_stats_status() {
|
||||||
|
json_init
|
||||||
|
|
||||||
|
# Check persistence directory
|
||||||
|
local persist_ok=0
|
||||||
|
[ -d "$PERSIST_DIR" ] && persist_ok=1
|
||||||
|
json_add_boolean "persistence_enabled" "$persist_ok"
|
||||||
|
json_add_string "persist_dir" "$PERSIST_DIR"
|
||||||
|
json_add_string "cache_dir" "$CACHE_DIR"
|
||||||
|
|
||||||
|
# Count cached files
|
||||||
|
local cache_count=$(ls "$CACHE_DIR"/*.json 2>/dev/null | wc -l)
|
||||||
|
json_add_int "cached_files" "${cache_count:-0}"
|
||||||
|
|
||||||
|
# Count persisted files
|
||||||
|
local persist_count=$(ls "$PERSIST_DIR"/*.json 2>/dev/null | wc -l)
|
||||||
|
json_add_int "persisted_files" "${persist_count:-0}"
|
||||||
|
|
||||||
|
# Last persist time
|
||||||
|
local last_persist=""
|
||||||
|
if [ -f "$PERSIST_DIR/.last_persist" ]; then
|
||||||
|
last_persist=$(cat "$PERSIST_DIR/.last_persist")
|
||||||
|
fi
|
||||||
|
json_add_string "last_persist" "${last_persist:-never}"
|
||||||
|
|
||||||
|
# History stats
|
||||||
|
json_add_object "history"
|
||||||
|
local hourly_total=0 daily_total=0
|
||||||
|
for collector in health threat capacity crowdsec mitmproxy; do
|
||||||
|
local hourly_count=$(ls "$PERSIST_DIR/history/hourly/$collector"/*.json 2>/dev/null | wc -l)
|
||||||
|
local daily_count=$(ls "$PERSIST_DIR/history/daily/$collector"/*.json 2>/dev/null | wc -l)
|
||||||
|
hourly_total=$((hourly_total + hourly_count))
|
||||||
|
daily_total=$((daily_total + daily_count))
|
||||||
|
done
|
||||||
|
json_add_int "hourly_snapshots" "$hourly_total"
|
||||||
|
json_add_int "daily_aggregates" "$daily_total"
|
||||||
|
json_close_object
|
||||||
|
|
||||||
|
# Current cache values
|
||||||
|
json_add_object "current"
|
||||||
|
local health=$(jsonfilter -i "$CACHE_DIR/health.json" -e '@.score' 2>/dev/null || echo 0)
|
||||||
|
local threat=$(jsonfilter -i "$CACHE_DIR/threat.json" -e '@.level' 2>/dev/null || echo 0)
|
||||||
|
local capacity=$(jsonfilter -i "$CACHE_DIR/capacity.json" -e '@.combined' 2>/dev/null || echo 0)
|
||||||
|
json_add_int "health" "$health"
|
||||||
|
json_add_int "threat" "$threat"
|
||||||
|
json_add_int "capacity" "$capacity"
|
||||||
|
|
||||||
|
# Calculate influence
|
||||||
|
local t_inv=$((100 - threat))
|
||||||
|
local c_inv=$((100 - capacity))
|
||||||
|
local influence=$(( (health * 40 + t_inv * 30 + c_inv * 30) / 100 ))
|
||||||
|
json_add_int "influence" "$influence"
|
||||||
|
json_close_object
|
||||||
|
|
||||||
|
json_dump
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get history for specific collector
|
||||||
|
_do_get_history() {
|
||||||
|
local collector="$1"
|
||||||
|
local period="$2"
|
||||||
|
|
||||||
|
[ -z "$period" ] && period="hourly"
|
||||||
|
|
||||||
|
json_init
|
||||||
|
json_add_string "collector" "$collector"
|
||||||
|
json_add_string "period" "$period"
|
||||||
|
|
||||||
|
local history_dir="$PERSIST_DIR/history/$period/$collector"
|
||||||
|
|
||||||
|
if [ ! -d "$history_dir" ]; then
|
||||||
|
json_add_string "error" "No history for $collector ($period)"
|
||||||
|
json_dump
|
||||||
|
return
|
||||||
|
fi
|
||||||
|
|
||||||
|
json_add_array "data"
|
||||||
|
for hfile in $(ls -t "$history_dir"/*.json 2>/dev/null | head -48); do
|
||||||
|
[ -f "$hfile" ] || continue
|
||||||
|
local filename=$(basename "$hfile" .json)
|
||||||
|
local content=$(cat "$hfile" 2>/dev/null)
|
||||||
|
|
||||||
|
# Extract key values based on collector
|
||||||
|
local ts=$(jsonfilter -i "$hfile" -e '@.timestamp' 2>/dev/null || echo 0)
|
||||||
|
local val
|
||||||
|
case "$collector" in
|
||||||
|
health) val=$(jsonfilter -i "$hfile" -e '@.score' 2>/dev/null) ;;
|
||||||
|
threat) val=$(jsonfilter -i "$hfile" -e '@.level' 2>/dev/null) ;;
|
||||||
|
capacity) val=$(jsonfilter -i "$hfile" -e '@.combined' 2>/dev/null) ;;
|
||||||
|
crowdsec*) val=$(jsonfilter -i "$hfile" -e '@.alerts_24h' 2>/dev/null) ;;
|
||||||
|
mitmproxy) val=$(jsonfilter -i "$hfile" -e '@.threats_today' 2>/dev/null) ;;
|
||||||
|
*) val=$(jsonfilter -i "$hfile" -e '@.total' 2>/dev/null) ;;
|
||||||
|
esac
|
||||||
|
[ -z "$val" ] && val=0
|
||||||
|
|
||||||
|
json_add_object ""
|
||||||
|
json_add_string "time" "$filename"
|
||||||
|
json_add_int "timestamp" "$ts"
|
||||||
|
json_add_int "value" "$val"
|
||||||
|
json_close_object
|
||||||
|
done
|
||||||
|
json_close_array
|
||||||
|
|
||||||
|
json_dump
|
||||||
|
}
|
||||||
|
|
||||||
|
# Get current collector cache
|
||||||
|
_do_get_collector_cache() {
|
||||||
|
local collector="$1"
|
||||||
|
|
||||||
|
local cache_file="$CACHE_DIR/${collector}.json"
|
||||||
|
|
||||||
|
if [ -f "$cache_file" ]; then
|
||||||
|
cat "$cache_file"
|
||||||
|
else
|
||||||
|
json_init
|
||||||
|
json_add_string "error" "Cache not found: $collector"
|
||||||
|
json_dump
|
||||||
|
fi
|
||||||
|
}
|
||||||
@ -67,6 +67,9 @@ _list_all_methods() {
|
|||||||
# P2P module
|
# P2P module
|
||||||
type list_methods_p2p >/dev/null 2>&1 && list_methods_p2p
|
type list_methods_p2p >/dev/null 2>&1 && list_methods_p2p
|
||||||
|
|
||||||
|
# Stats module (evolution, timeline, heartbeat line)
|
||||||
|
type list_methods_stats >/dev/null 2>&1 && list_methods_stats
|
||||||
|
|
||||||
json_dump
|
json_dump
|
||||||
}
|
}
|
||||||
|
|
||||||
@ -142,6 +145,11 @@ _call_method() {
|
|||||||
handle_p2p "$method" && return 0
|
handle_p2p "$method" && return 0
|
||||||
fi
|
fi
|
||||||
|
|
||||||
|
# Stats methods (evolution, timeline, heartbeat line)
|
||||||
|
if type handle_stats >/dev/null 2>&1; then
|
||||||
|
handle_stats "$method" && return 0
|
||||||
|
fi
|
||||||
|
|
||||||
# Unknown method
|
# Unknown method
|
||||||
json_init
|
json_init
|
||||||
json_add_boolean "error" true
|
json_add_boolean "error" true
|
||||||
|
|||||||
@ -1216,6 +1216,14 @@ daemon_mode() {
|
|||||||
STATUS_COLLECTOR_PID=$!
|
STATUS_COLLECTOR_PID=$!
|
||||||
log debug "Status collector started (PID: $STATUS_COLLECTOR_PID)"
|
log debug "Status collector started (PID: $STATUS_COLLECTOR_PID)"
|
||||||
|
|
||||||
|
# Start stats persistence layer (3-tier: RAM → /tmp → /srv)
|
||||||
|
if [ -x /usr/sbin/secubox-stats-persist ]; then
|
||||||
|
/usr/sbin/secubox-stats-persist recover 2>/dev/null
|
||||||
|
/usr/sbin/secubox-stats-persist daemon &
|
||||||
|
STATS_PERSIST_PID=$!
|
||||||
|
log debug "Stats persistence daemon started (PID: $STATS_PERSIST_PID)"
|
||||||
|
fi
|
||||||
|
|
||||||
# Wait for initial cache population
|
# Wait for initial cache population
|
||||||
sleep 1
|
sleep 1
|
||||||
|
|
||||||
|
|||||||
383
package/secubox/secubox-core/root/usr/sbin/secubox-stats-persist
Normal file
383
package/secubox/secubox-core/root/usr/sbin/secubox-stats-persist
Normal file
@ -0,0 +1,383 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
#
|
||||||
|
# SecuBox Stats Persistence & Evolution Layer
|
||||||
|
# 3-tier caching: RAM (/tmp) → Volatile Buffer → Persistent (/srv)
|
||||||
|
# Time-series: Hourly snapshots (24h), Daily aggregates (30d)
|
||||||
|
# Never-trashed stats with reboot recovery
|
||||||
|
#
|
||||||
|
|
||||||
|
PERSIST_DIR="/srv/secubox/stats"
|
||||||
|
CACHE_DIR="/tmp/secubox"
|
||||||
|
HISTORY_DIR="$PERSIST_DIR/history"
|
||||||
|
TIMELINE_FILE="$PERSIST_DIR/timeline.json"
|
||||||
|
EVOLUTION_FILE="$PERSIST_DIR/evolution.json"
|
||||||
|
HEARTBEAT_LINE="$PERSIST_DIR/heartbeat-line.json"
|
||||||
|
|
||||||
|
# Collectors to persist (must match cache file basenames)
|
||||||
|
COLLECTORS="health threat capacity crowdsec mitmproxy netifyd client-guardian mac-guardian netdiag crowdsec-overview"
|
||||||
|
|
||||||
|
# Initialize directories
|
||||||
|
init_persist() {
|
||||||
|
mkdir -p "$PERSIST_DIR" "$HISTORY_DIR/hourly" "$HISTORY_DIR/daily"
|
||||||
|
mkdir -p "$CACHE_DIR"
|
||||||
|
|
||||||
|
# Create evolution tracking files if missing
|
||||||
|
for collector in $COLLECTORS; do
|
||||||
|
local hourly_dir="$HISTORY_DIR/hourly/$collector"
|
||||||
|
local daily_dir="$HISTORY_DIR/daily/$collector"
|
||||||
|
mkdir -p "$hourly_dir" "$daily_dir"
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "Stats persistence initialized at $PERSIST_DIR"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Recover cache from persistent storage on boot
|
||||||
|
recover_cache() {
|
||||||
|
for collector in $COLLECTORS; do
|
||||||
|
local persist_file="$PERSIST_DIR/${collector}.json"
|
||||||
|
local cache_file="$CACHE_DIR/${collector}.json"
|
||||||
|
|
||||||
|
# Only recover if cache is missing but persistent exists
|
||||||
|
if [ ! -f "$cache_file" ] && [ -f "$persist_file" ]; then
|
||||||
|
cp "$persist_file" "$cache_file"
|
||||||
|
echo "Recovered $collector from persistent storage"
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# Persist current cache to storage (atomic writes)
|
||||||
|
persist_cache() {
|
||||||
|
local now=$(date +%s)
|
||||||
|
local hour=$(date +%Y%m%d%H)
|
||||||
|
local day=$(date +%Y%m%d)
|
||||||
|
|
||||||
|
for collector in $COLLECTORS; do
|
||||||
|
local cache_file="$CACHE_DIR/${collector}.json"
|
||||||
|
local persist_file="$PERSIST_DIR/${collector}.json"
|
||||||
|
|
||||||
|
# Skip if cache doesn't exist
|
||||||
|
[ -f "$cache_file" ] || continue
|
||||||
|
|
||||||
|
# Atomic persist: cache → tmp → persistent
|
||||||
|
local tmp_file="$PERSIST_DIR/.${collector}.tmp"
|
||||||
|
cp "$cache_file" "$tmp_file" 2>/dev/null && \
|
||||||
|
mv -f "$tmp_file" "$persist_file" 2>/dev/null
|
||||||
|
|
||||||
|
# Hourly snapshot (only once per hour)
|
||||||
|
local hourly_file="$HISTORY_DIR/hourly/$collector/${hour}.json"
|
||||||
|
if [ ! -f "$hourly_file" ]; then
|
||||||
|
cp "$cache_file" "$hourly_file" 2>/dev/null
|
||||||
|
fi
|
||||||
|
done
|
||||||
|
|
||||||
|
echo "$now" > "$PERSIST_DIR/.last_persist"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create hourly aggregate from snapshots
|
||||||
|
aggregate_hourly() {
|
||||||
|
local collector="$1"
|
||||||
|
local hour="$2" # Format: YYYYMMDDHH
|
||||||
|
local hourly_file="$HISTORY_DIR/hourly/$collector/${hour}.json"
|
||||||
|
|
||||||
|
[ -f "$hourly_file" ] || return 1
|
||||||
|
|
||||||
|
# Extract key numeric fields for aggregation
|
||||||
|
local data=$(cat "$hourly_file" 2>/dev/null)
|
||||||
|
echo "$data"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Create daily aggregate from 24 hourly snapshots
|
||||||
|
aggregate_daily() {
|
||||||
|
local day=$(date +%Y%m%d)
|
||||||
|
|
||||||
|
for collector in $COLLECTORS; do
|
||||||
|
local daily_file="$HISTORY_DIR/daily/$collector/${day}.json"
|
||||||
|
local hourly_dir="$HISTORY_DIR/hourly/$collector"
|
||||||
|
|
||||||
|
# Skip if already aggregated today
|
||||||
|
[ -f "$daily_file" ] && continue
|
||||||
|
|
||||||
|
# Count hourly files for this day (should be 24 at end of day)
|
||||||
|
local hourly_count=$(ls "$hourly_dir/${day}"*.json 2>/dev/null | wc -l)
|
||||||
|
[ "$hourly_count" -lt 6 ] && continue # Need at least 6 hours
|
||||||
|
|
||||||
|
# Create daily aggregate with min/max/avg
|
||||||
|
local min=999999 max=0 sum=0 count=0
|
||||||
|
|
||||||
|
for hfile in "$hourly_dir/${day}"*.json; do
|
||||||
|
[ -f "$hfile" ] || continue
|
||||||
|
|
||||||
|
# Extract primary metric based on collector type
|
||||||
|
local val
|
||||||
|
case "$collector" in
|
||||||
|
health) val=$(jsonfilter -i "$hfile" -e '@.score' 2>/dev/null) ;;
|
||||||
|
threat) val=$(jsonfilter -i "$hfile" -e '@.level' 2>/dev/null) ;;
|
||||||
|
capacity) val=$(jsonfilter -i "$hfile" -e '@.combined' 2>/dev/null) ;;
|
||||||
|
crowdsec*) val=$(jsonfilter -i "$hfile" -e '@.alerts_24h' 2>/dev/null) ;;
|
||||||
|
mitmproxy) val=$(jsonfilter -i "$hfile" -e '@.threats_today' 2>/dev/null) ;;
|
||||||
|
*) val=$(jsonfilter -i "$hfile" -e '@.total' 2>/dev/null) ;;
|
||||||
|
esac
|
||||||
|
|
||||||
|
[ -z "$val" ] && val=0
|
||||||
|
[ "$val" -lt "$min" ] && min=$val
|
||||||
|
[ "$val" -gt "$max" ] && max=$val
|
||||||
|
sum=$((sum + val))
|
||||||
|
count=$((count + 1))
|
||||||
|
done
|
||||||
|
|
||||||
|
[ "$count" -gt 0 ] || continue
|
||||||
|
local avg=$((sum / count))
|
||||||
|
|
||||||
|
printf '{"date":"%s","min":%d,"max":%d,"avg":%d,"samples":%d}\n' \
|
||||||
|
"$day" "$min" "$max" "$avg" "$count" > "$daily_file"
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# Cleanup old history files (keep 24h hourly, 30d daily)
|
||||||
|
cleanup_history() {
|
||||||
|
local now=$(date +%s)
|
||||||
|
local hourly_cutoff=$((now - 86400)) # 24 hours
|
||||||
|
local daily_cutoff=$((now - 2592000)) # 30 days
|
||||||
|
|
||||||
|
for collector in $COLLECTORS; do
|
||||||
|
# Cleanup hourly (older than 24h)
|
||||||
|
for hfile in "$HISTORY_DIR/hourly/$collector"/*.json; do
|
||||||
|
[ -f "$hfile" ] || continue
|
||||||
|
local mtime=$(stat -c %Y "$hfile" 2>/dev/null || echo 0)
|
||||||
|
[ "$mtime" -lt "$hourly_cutoff" ] && rm -f "$hfile"
|
||||||
|
done
|
||||||
|
|
||||||
|
# Cleanup daily (older than 30d)
|
||||||
|
for dfile in "$HISTORY_DIR/daily/$collector"/*.json; do
|
||||||
|
[ -f "$dfile" ] || continue
|
||||||
|
local mtime=$(stat -c %Y "$dfile" 2>/dev/null || echo 0)
|
||||||
|
[ "$mtime" -lt "$daily_cutoff" ] && rm -f "$dfile"
|
||||||
|
done
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# Generate combined timeline (last 24h evolution)
|
||||||
|
generate_timeline() {
|
||||||
|
local now=$(date +%s)
|
||||||
|
local tmp_file="$PERSIST_DIR/.timeline.tmp"
|
||||||
|
|
||||||
|
printf '{"generated":%d,"collectors":{' "$now" > "$tmp_file"
|
||||||
|
|
||||||
|
local first=1
|
||||||
|
for collector in $COLLECTORS; do
|
||||||
|
local hourly_dir="$HISTORY_DIR/hourly/$collector"
|
||||||
|
|
||||||
|
[ "$first" = "0" ] && printf ',' >> "$tmp_file"
|
||||||
|
first=0
|
||||||
|
|
||||||
|
printf '"%s":[' "$collector" >> "$tmp_file"
|
||||||
|
|
||||||
|
# Get last 24 hourly snapshots
|
||||||
|
local hfirst=1
|
||||||
|
for hfile in $(ls -t "$hourly_dir"/*.json 2>/dev/null | head -24); do
|
||||||
|
[ -f "$hfile" ] || continue
|
||||||
|
|
||||||
|
[ "$hfirst" = "0" ] && printf ',' >> "$tmp_file"
|
||||||
|
hfirst=0
|
||||||
|
|
||||||
|
# Extract timestamp and primary value
|
||||||
|
local ts=$(jsonfilter -i "$hfile" -e '@.timestamp' 2>/dev/null || echo 0)
|
||||||
|
local val
|
||||||
|
case "$collector" in
|
||||||
|
health) val=$(jsonfilter -i "$hfile" -e '@.score' 2>/dev/null) ;;
|
||||||
|
threat) val=$(jsonfilter -i "$hfile" -e '@.level' 2>/dev/null) ;;
|
||||||
|
capacity) val=$(jsonfilter -i "$hfile" -e '@.combined' 2>/dev/null) ;;
|
||||||
|
*) val=0 ;;
|
||||||
|
esac
|
||||||
|
[ -z "$val" ] && val=0
|
||||||
|
|
||||||
|
printf '{"t":%d,"v":%d}' "$ts" "$val" >> "$tmp_file"
|
||||||
|
done
|
||||||
|
|
||||||
|
printf ']' >> "$tmp_file"
|
||||||
|
done
|
||||||
|
|
||||||
|
printf '}}\n' >> "$tmp_file"
|
||||||
|
mv -f "$tmp_file" "$TIMELINE_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Generate evolution sparkline data (combined metrics beep line)
|
||||||
|
generate_evolution() {
|
||||||
|
local now=$(date +%s)
|
||||||
|
local tmp_file="$PERSIST_DIR/.evolution.tmp"
|
||||||
|
|
||||||
|
printf '{"generated":%d,"window":"24h","points":[' "$now" > "$tmp_file"
|
||||||
|
|
||||||
|
# Combine health, threat, capacity into single timeline
|
||||||
|
local health_dir="$HISTORY_DIR/hourly/health"
|
||||||
|
local threat_dir="$HISTORY_DIR/hourly/threat"
|
||||||
|
local capacity_dir="$HISTORY_DIR/hourly/capacity"
|
||||||
|
|
||||||
|
# Get timestamps from health (most reliable)
|
||||||
|
local first=1
|
||||||
|
for hfile in $(ls -t "$health_dir"/*.json 2>/dev/null | head -48 | tac); do
|
||||||
|
[ -f "$hfile" ] || continue
|
||||||
|
|
||||||
|
local hour=$(basename "$hfile" .json)
|
||||||
|
local ts=$(jsonfilter -i "$hfile" -e '@.timestamp' 2>/dev/null || echo 0)
|
||||||
|
|
||||||
|
# Get values from all three
|
||||||
|
local h=$(jsonfilter -i "$hfile" -e '@.score' 2>/dev/null || echo 100)
|
||||||
|
|
||||||
|
local tfile="$threat_dir/${hour}.json"
|
||||||
|
local t=$(jsonfilter -i "$tfile" -e '@.level' 2>/dev/null 2>/dev/null || echo 0)
|
||||||
|
|
||||||
|
local cfile="$capacity_dir/${hour}.json"
|
||||||
|
local c=$(jsonfilter -i "$cfile" -e '@.combined' 2>/dev/null 2>/dev/null || echo 0)
|
||||||
|
|
||||||
|
[ "$first" = "0" ] && printf ',' >> "$tmp_file"
|
||||||
|
first=0
|
||||||
|
|
||||||
|
# Combined "influence" score: weighted combination
|
||||||
|
# Health (40%), inverse Threat (30%), inverse Capacity (30%)
|
||||||
|
local t_inv=$((100 - t))
|
||||||
|
local c_inv=$((100 - c))
|
||||||
|
local influence=$(( (h * 40 + t_inv * 30 + c_inv * 30) / 100 ))
|
||||||
|
|
||||||
|
printf '{"t":%d,"h":%d,"th":%d,"c":%d,"i":%d}' \
|
||||||
|
"$ts" "$h" "$t" "$c" "$influence" >> "$tmp_file"
|
||||||
|
done
|
||||||
|
|
||||||
|
printf ']}\n' >> "$tmp_file"
|
||||||
|
mv -f "$tmp_file" "$EVOLUTION_FILE"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Generate heartbeat line (last 60 samples, ~3min of data)
|
||||||
|
generate_heartbeat_line() {
|
||||||
|
local now=$(date +%s)
|
||||||
|
local tmp_file="$PERSIST_DIR/.heartbeat.tmp"
|
||||||
|
|
||||||
|
# Read current cache values
|
||||||
|
local h=$(jsonfilter -i "$CACHE_DIR/health.json" -e '@.score' 2>/dev/null || echo 100)
|
||||||
|
local t=$(jsonfilter -i "$CACHE_DIR/threat.json" -e '@.level' 2>/dev/null || echo 0)
|
||||||
|
local c=$(jsonfilter -i "$CACHE_DIR/capacity.json" -e '@.combined' 2>/dev/null || echo 0)
|
||||||
|
|
||||||
|
# Calculate influence
|
||||||
|
local t_inv=$((100 - t))
|
||||||
|
local c_inv=$((100 - c))
|
||||||
|
local influence=$(( (h * 40 + t_inv * 30 + c_inv * 30) / 100 ))
|
||||||
|
|
||||||
|
# Append to rolling buffer (keep last 60)
|
||||||
|
local buffer_file="$PERSIST_DIR/.heartbeat_buffer"
|
||||||
|
|
||||||
|
# Read existing buffer
|
||||||
|
local buffer=""
|
||||||
|
[ -f "$buffer_file" ] && buffer=$(cat "$buffer_file")
|
||||||
|
|
||||||
|
# Append new point
|
||||||
|
local new_point=$(printf '{"t":%d,"h":%d,"th":%d,"c":%d,"i":%d}' "$now" "$h" "$t" "$c" "$influence")
|
||||||
|
|
||||||
|
if [ -z "$buffer" ]; then
|
||||||
|
buffer="[$new_point]"
|
||||||
|
else
|
||||||
|
# Parse existing, keep last 59, add new
|
||||||
|
local count=$(echo "$buffer" | tr ',' '\n' | grep -c '"t":')
|
||||||
|
if [ "$count" -ge 60 ]; then
|
||||||
|
# Remove first element
|
||||||
|
buffer=$(echo "$buffer" | sed 's/^\[{[^}]*},/[/')
|
||||||
|
fi
|
||||||
|
buffer=$(echo "$buffer" | sed 's/\]$//')
|
||||||
|
buffer="$buffer,$new_point]"
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "$buffer" > "$buffer_file"
|
||||||
|
|
||||||
|
# Write heartbeat line file
|
||||||
|
printf '{"generated":%d,"window":"3m","samples":60,"points":%s}\n' \
|
||||||
|
"$now" "$buffer" > "$tmp_file"
|
||||||
|
mv -f "$tmp_file" "$HEARTBEAT_LINE"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Main persistence loop (runs every 60s)
|
||||||
|
daemon_loop() {
|
||||||
|
init_persist
|
||||||
|
recover_cache
|
||||||
|
|
||||||
|
echo "Stats persistence daemon started"
|
||||||
|
|
||||||
|
while true; do
|
||||||
|
# Persist current cache atomically
|
||||||
|
persist_cache
|
||||||
|
|
||||||
|
# Generate aggregates and timelines
|
||||||
|
aggregate_daily
|
||||||
|
generate_timeline
|
||||||
|
generate_evolution
|
||||||
|
|
||||||
|
# Cleanup old data (hourly check)
|
||||||
|
local hour=$(date +%M)
|
||||||
|
[ "$hour" = "00" ] && cleanup_history
|
||||||
|
|
||||||
|
sleep 60
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# Fast heartbeat loop (runs every 3s for heartbeat line)
|
||||||
|
heartbeat_loop() {
|
||||||
|
while true; do
|
||||||
|
generate_heartbeat_line
|
||||||
|
sleep 3
|
||||||
|
done
|
||||||
|
}
|
||||||
|
|
||||||
|
# CLI
|
||||||
|
case "$1" in
|
||||||
|
init)
|
||||||
|
init_persist
|
||||||
|
;;
|
||||||
|
recover)
|
||||||
|
init_persist
|
||||||
|
recover_cache
|
||||||
|
;;
|
||||||
|
persist)
|
||||||
|
persist_cache
|
||||||
|
;;
|
||||||
|
aggregate)
|
||||||
|
aggregate_daily
|
||||||
|
;;
|
||||||
|
timeline)
|
||||||
|
generate_timeline
|
||||||
|
cat "$TIMELINE_FILE"
|
||||||
|
;;
|
||||||
|
evolution)
|
||||||
|
generate_evolution
|
||||||
|
cat "$EVOLUTION_FILE"
|
||||||
|
;;
|
||||||
|
heartbeat)
|
||||||
|
generate_heartbeat_line
|
||||||
|
cat "$HEARTBEAT_LINE"
|
||||||
|
;;
|
||||||
|
daemon)
|
||||||
|
daemon_loop &
|
||||||
|
heartbeat_loop
|
||||||
|
;;
|
||||||
|
status)
|
||||||
|
echo "=== Stats Persistence Status ==="
|
||||||
|
echo "Persist Dir: $PERSIST_DIR"
|
||||||
|
echo "Cache Dir: $CACHE_DIR"
|
||||||
|
echo ""
|
||||||
|
echo "Persisted Files:"
|
||||||
|
ls -la "$PERSIST_DIR"/*.json 2>/dev/null || echo " (none)"
|
||||||
|
echo ""
|
||||||
|
echo "Hourly History:"
|
||||||
|
for collector in $COLLECTORS; do
|
||||||
|
local count=$(ls "$HISTORY_DIR/hourly/$collector"/*.json 2>/dev/null | wc -l)
|
||||||
|
echo " $collector: $count snapshots"
|
||||||
|
done
|
||||||
|
echo ""
|
||||||
|
echo "Daily History:"
|
||||||
|
for collector in $COLLECTORS; do
|
||||||
|
local count=$(ls "$HISTORY_DIR/daily/$collector"/*.json 2>/dev/null | wc -l)
|
||||||
|
echo " $collector: $count days"
|
||||||
|
done
|
||||||
|
;;
|
||||||
|
*)
|
||||||
|
echo "Usage: $0 {init|recover|persist|aggregate|timeline|evolution|heartbeat|daemon|status}"
|
||||||
|
exit 1
|
||||||
|
;;
|
||||||
|
esac
|
||||||
Loading…
Reference in New Issue
Block a user