secubox-openwrt/secubox-tools/README.md

# SecuBox Development Tools

**Version:** 1.2.0
**Last Updated:** 2026-02-28
**Status:** Active

This directory contains utilities for validating, debugging, and maintaining SecuBox modules.

---

## See Also

- **Quick Commands & Rules:** [DOCS/QUICK-START.md](../DOCS/QUICK-START.md)
- **Automation Guardrails:** [DOCS/CODEX.md](../DOCS/CODEX.md)
- **Validation Guide:** [DOCS/VALIDATION-GUIDE.md](../DOCS/VALIDATION-GUIDE.md)
- **Deployment Procedures:** [DOCS/DEVELOPMENT-GUIDELINES.md §9](../DOCS/DEVELOPMENT-GUIDELINES.md#deployment-procedures)

## Tools Overview

### Build and Test Tools

#### local-build.sh

**NEW!** Local build system that replicates GitHub Actions workflows.

Build and test packages locally without pushing to GitHub. Automatically downloads and configures the OpenWrt SDK, builds packages, and collects artifacts.

**Usage:**
```bash
# Validate all packages
./secubox-tools/local-build.sh validate

# Build all packages (x86_64)
./secubox-tools/local-build.sh build

# Build single package
./secubox-tools/local-build.sh build luci-app-system-hub

# Build SecuBox Core package
./secubox-tools/local-build.sh build secubox-core

# Build for specific architecture
./secubox-tools/local-build.sh build --arch aarch64-cortex-a72

# Build firmware image for MOCHAbin
./secubox-tools/local-build.sh build-firmware mochabin

# Build firmware image for ESPRESSObin V7
./secubox-tools/local-build.sh build-firmware espressobin-v7

# Full validation + build
./secubox-tools/local-build.sh full

# Clean build artifacts
./secubox-tools/local-build.sh clean

# Clean everything including OpenWrt source
./secubox-tools/local-build.sh clean-all
```

**Supported Architectures (for package building):**
- `x86-64` - PC, VMs (default)
- `aarch64-cortex-a53` - ARM Cortex-A53 (ESPRESSObin)
- `aarch64-cortex-a72` - ARM Cortex-A72 (MOCHAbin, RPi4)
- `aarch64-generic` - Generic ARM64
- `mips-24kc` - MIPS 24Kc (TP-Link)
- `mipsel-24kc` - MIPS LE (Xiaomi, GL.iNet)

**Supported Devices (for firmware building):**
- `espressobin-v7` - ESPRESSObin V7 (1-2GB DDR4)
- `espressobin-ultra` - ESPRESSObin Ultra (PoE, WiFi)
- `sheeva64` - Sheeva64 (Plug computer)
- `mochabin` - MOCHAbin (Quad-core A72, 10G)
- `x86-64` - x86_64 Generic PC

**Environment Variables:**
- `OPENWRT_VERSION` - OpenWrt version (default: 24.10.5, also supports: 25.12.0-rc1, 23.05.5, SNAPSHOT)
- `SDK_DIR` - SDK directory (default: ./sdk)
- `BUILD_DIR` - Build output directory (default: ./build)
- `CACHE_DIR` - Download cache directory (default: ./cache)
- `OPENWRT_DIR` - OpenWrt source directory for firmware builds (default: ./openwrt)

**Output:**
- Built packages are placed in `build/<arch>/` with SHA256 checksums
- Firmware images are placed in `build/firmware/<device>/` with checksums and build info

**Dependencies:**
```bash
# Required for building
sudo apt-get install -y build-essential clang flex bison g++ gawk \
    gcc-multilib g++-multilib gettext git libncurses5-dev \
    libssl-dev python3-setuptools python3-dev rsync \
    swig unzip zlib1g-dev file wget curl jq ninja-build

# Optional for validation
sudo apt-get install -y shellcheck nodejs
```

**Features:**
- **Package Building**: Downloads and caches OpenWrt SDK for faster builds
- **Firmware Building**: Downloads full OpenWrt source and builds custom firmware images
- Configures feeds (packages, luci) automatically
- Validates packages before building
- Builds .ipk packages with verbose output
- Builds complete firmware images (.img.gz, *sysupgrade.bin, etc.)
- Collects artifacts with checksums
- Supports single package or all packages
- Multiple architecture and device support
- Device profile verification before building

**⚠️ IMPORTANT: SDK vs Full Toolchain Builds**

Some packages **MUST** be built with the full OpenWrt toolchain (`openwrt/`) instead of the SDK:

| Package | Reason |
|---------|--------|
| `crowdsec` | Go binary with CGO - SDK produces ARM64 LSE atomics that crash on some CPUs |
| `crowdsec-firewall-bouncer` | Go binary with CGO |
| `netifyd` | C++ native binary |
| `nodogsplash` | C native binary |

**To build these packages:**
```bash
cd secubox-tools/openwrt
make package/<package-name>/compile V=s
```

LuCI apps and shell/Lua packages can use the SDK via `local-build.sh`.

**Example Workflow - Toolchain Build (for Go/native packages):**
```bash
# 1. Navigate to the full OpenWrt build directory
cd secubox-tools/openwrt

# 2. Update feeds if needed
./scripts/feeds update -a
./scripts/feeds install -a

# 3. Build CrowdSec with full toolchain
make package/crowdsec/compile V=s

# 4. Build the firewall bouncer
make package/crowdsec-firewall-bouncer/compile V=s

# 5. Packages are in: bin/packages/aarch64_cortex-a72/packages/
```

**Example Workflow - Package Building (SDK):**
```bash
# 1. Make changes to a module
vim luci-app-system-hub/htdocs/luci-static/resources/view/system-hub/overview.js

# 2. Validate and build locally
./secubox-tools/local-build.sh full

# 3. Test on router
scp build/x86-64/*.ipk root@192.168.1.1:/tmp/
ssh root@192.168.1.1
opkg install /tmp/luci-app-system-hub*.ipk
/etc/init.d/rpcd restart
```

### Image & Deployment Tools

#### secubox-image.sh

Build SecuBox firmware images via the OpenWrt ASU (Attended SysUpgrade) API.

**Usage:**
```bash
# Build firmware image for device
./secubox-tools/secubox-image.sh build mochabin

# Generate firmware-selector config
./secubox-tools/secubox-image.sh firmware-selector mochabin

# Check build status
./secubox-tools/secubox-image.sh status <build-hash>

# Download completed build
./secubox-tools/secubox-image.sh download <build-hash>
```

**Features:**
- Uses firmware-selector.openwrt.org backend (ASU API)
- Supports MOCHAbin, ESPRESSObin V7/Ultra, x86-64
- Maximum rootfs partition (1024 MB)
- First-boot script auto-installs SecuBox packages
- Image resizing for full eMMC utilization

**Output:** Firmware images in `build/images/` with SHA256 checksums

#### secubox-sysupgrade.sh

Upgrade a running SecuBox device in-place while preserving packages.

**Usage:**
```bash
# Check current version and available upgrades
secubox-sysupgrade check

# Build sysupgrade image (without flashing)
secubox-sysupgrade build

# Build + download + flash (full upgrade)
secubox-sysupgrade upgrade

# Show device info
secubox-sysupgrade status
```

**Features:**
- Auto-detects device, version, and installed packages
- Requests custom image with all packages preserved
- Preserves /etc/config, /etc/secubox, /srv/ across upgrades
- Uses /etc/board.json for device detection

#### quick-deploy.sh

Fast development deployment to router.

**Usage:**
```bash
# Deploy IPK package
./secubox-tools/quick-deploy.sh --ipk /tmp/package.ipk

# Deploy from source directory
./secubox-tools/quick-deploy.sh --src package/secubox/luci-app-example

# Shortcut for LuCI apps
./secubox-tools/quick-deploy.sh --app system-hub

# Deploy from git repo
./secubox-tools/quick-deploy.sh --git https://github.com/user/repo --branch develop

# List available apps
./secubox-tools/quick-deploy.sh --list-apps
```

**Features:**
- Multiple source modes: IPK, APK, tar, git
- Automatic LuCI app detection
- Post-deploy verification and cache busting
- Backup and restore capability
- SSH multiplexing for faster transfers

#### c3box-vm-builder.sh

Build portable C3Box VM images for VMware/VirtualBox.

**Usage:**
```bash
# Build x86-64 firmware
./secubox-tools/c3box-vm-builder.sh build

# Convert to VM formats
./secubox-tools/c3box-vm-builder.sh convert

# Full build + convert
./secubox-tools/c3box-vm-builder.sh full

# Create distributable archive
./secubox-tools/c3box-vm-builder.sh package
```

**Output formats:** VMDK (VMware), OVA, VDI (VirtualBox), QCOW2 (KVM)

#### secubox-clone-station.sh

Orchestrate cloning of SecuBox devices via dual USB serial.

**Usage:**
```bash
# Detect serial devices
./secubox-tools/secubox-clone-station.sh detect

# Extract master config
./secubox-tools/secubox-clone-station.sh pull --master /dev/ttyUSB0

# Flash target device
./secubox-tools/secubox-clone-station.sh flash --target /dev/ttyUSB1

# Full clone workflow
./secubox-tools/secubox-clone-station.sh clone
```

**Features:**
- Extract config from master device
- Build clone image with ASU API
- Generate join token for mesh
- U-Boot automation via MOKATOOL
- TFTP-based flashing

---

### Logging & Debug Utilities

#### secubox-log.sh

Centralized logger/aggregator for SecuBox modules. Appends tagged events to `/var/log/seccubox.log`, captures snapshots that merge `dmesg` + `logread`, and can tail the aggregated file for troubleshooting.

```
# Append a message
secubox-log.sh --tag netdata --message "Netdata restarted"

# Add a snapshot with dmesg/logread tail
secubox-log.sh --snapshot

# Tail the aggregated log
secubox-log.sh --tail 100
```

The script is also installed on the router as `/usr/sbin/secubox-log` (via `luci-app-secubox`) so LuCI modules can log lifecycle events and collect debug bundles.

**Example Workflow - Firmware Building:**
```bash
# 1. Build firmware for MOCHAbin with SecuBox pre-installed
./secubox-tools/local-build.sh build-firmware mochabin

# 2. Flash to device
# Firmware images are in: build/firmware/mochabin/
# - openwrt-*-sysupgrade.bin (for upgrading existing OpenWrt)
# - openwrt-*-factory.bin (for initial installation)
# - SHA256SUMS (checksums for verification)
# - BUILD_INFO.txt (build details)
# - packages/ (SecuBox .ipk files)

# 3. Clean up after building (optional)
./secubox-tools/local-build.sh clean-all  # Removes OpenWrt source (saves ~20GB)
```

### Validation Tools

#### validate-modules.sh

Fast validation of all modules in the repository.

**Usage:**
```bash
./secubox-tools/validate-modules.sh
```

**Checks performed:**
1. **RPCD script names vs ubus objects** - Ensures RPCD script filenames match JavaScript ubus object declarations
2. **Menu paths vs view files** - Verifies menu.d JSON paths correspond to actual view files
3. **View files have menu entries** - Checks that all view files are referenced in menus
4. **RPCD script permissions** - Ensures scripts are executable
5. **JSON syntax validation** - Validates all menu.d and acl.d JSON files
6. **ubus naming convention** - Verifies all ubus objects use the `luci.` prefix

**Exit codes:**
- `0` - All checks passed (or only warnings)
- `1` - Critical errors found

**Example output:**
```
✓ luci-app-cdn-cache: RPCD script 'luci.cdn-cache' matches ubus object 'luci.cdn-cache'
✓ luci-app-cdn-cache: Menu path 'cdn-cache/overview' → file exists
❌ ERROR: luci-app-example: RPCD script 'example' does not match ubus object 'luci.example'
```

#### validate-module-generation.sh

**NEW!** Comprehensive validation for a single module during/after generation.

**Usage:**
```bash
./secubox-tools/validate-module-generation.sh luci-app-cdn-cache
```

**Checks performed:**
- Makefile completeness (all required fields)
- RPCD script naming convention (luci.* prefix)
- RPCD script structure and handlers
- ACL permissions coverage
- Menu path validation
- JavaScript view validation
- RPC method declarations vs RPCD implementations
- Security scans (hardcoded credentials, dangerous commands)
- Documentation presence

**When to use:**
- After generating a new module
- Before committing changes to a module
- When debugging module integration issues

#### pre-deploy-lint.sh

**NEW!** Comprehensive syntax validation before deployment. Catches JavaScript, JSON, shell, and CSS errors before they break production.

**Usage:**
```bash
# Validate a single package
./secubox-tools/pre-deploy-lint.sh luci-app-system-hub

# Validate by short name
./secubox-tools/pre-deploy-lint.sh system-hub

# Validate all packages
./secubox-tools/pre-deploy-lint.sh --all

# Automatically via quick-deploy.sh (default for LuCI apps)
./secubox-tools/quick-deploy.sh --app system-hub
```

**Checks performed:**
1. **JavaScript Validation:**
   - Full syntax checking via Node.js `--check` (when available)
   - Fallback pattern-based checks for common errors
   - Detects: debugger statements, console.log, missing 'use strict'
   - LuCI-specific: validates require statement format
2. **JSON Validation:**
   - Menu.d and acl.d syntax verification
   - Python json.tool for proper parsing
3. **Shell Script Validation:**
   - Bash/sh syntax checking via `-n` flag
   - shellcheck integration (when available)
   - RPCD-specific checks: JSON output, method dispatcher
4. **CSS Validation:**
   - Unclosed brace detection
   - Common typo detection

**Integration with quick-deploy.sh:**
```bash
# Lint runs automatically before deployment (default)
./secubox-tools/quick-deploy.sh --app cdn-cache

# Skip lint (not recommended)
./secubox-tools/quick-deploy.sh --app cdn-cache --no-lint

# Force lint even for non-LuCI deployments
./secubox-tools/quick-deploy.sh --src ./path --lint
```

**Exit codes:**
- `0` - All checks passed (or only warnings)
- `1` - Critical errors found (deployment blocked)

**Example output:**
```
✓ luci-app-cdn-cache: All files validated

❌ JS syntax error: htdocs/view/cdn-cache/overview.js
    SyntaxError: Unexpected token '}'
⚠️  console.log found in: htdocs/view/cdn-cache/debug.js
```

#### pre-push-validation.sh

**NEW!** Git pre-push hook that validates all modules before allowing push.

**Usage:**
```bash
# Automatic (via git hook):
git push  # validation runs automatically

# Manual:
./secubox-tools/pre-push-validation.sh
```

**Checks performed:**
- All validation from validate-modules.sh
- Git staged changes analysis
- Modified module detection
- Comprehensive security scans
- Full module validation on modified modules

**Exit codes:**
- `0` - Push allowed
- `1` - Push blocked (critical errors found)

**Installation:**
```bash
./secubox-tools/install-git-hooks.sh
```

### Apps & Plugin Registry

#### secubox-app

CLI helper for the nascent SecuBox App Store. It reads `plugins/*/manifest.json`, installs/removes the packages listed there, and runs optional shell actions (`install`, `check`, `update`, `status`) defined in the manifest.

```bash
# List manifests and install state
secubox-app list

# Install Zigbee2MQTT (packages + zigbee2mqttctl install)
secubox-app install zigbee2mqtt

# Show manifest or run status/update
secubox-app show zigbee2mqtt
secubox-app status zigbee2mqtt
secubox-app update zigbee2mqtt
```

Environment: set `SECUBOX_PLUGINS_DIR` to override the manifest directory (default `../plugins`). Requires `opkg` and `jsonfilter`, so run it on an OpenWrt system (or within the SDK chroot).

### Maintenance Tools

#### secubox-repair.sh

Auto-repair tool that fixes common issues in Makefiles and RPCD scripts.

**Usage:**
```bash
./secubox-tools/secubox-repair.sh
```

#### secubox-debug.sh

Validates individual package structure and dependencies on OpenWrt device.

**Usage:**
```bash
./secubox-tools/secubox-debug.sh luci-app-<module-name>
```

#### install-git-hooks.sh

**NEW!** Installs git hooks for automatic validation.

**Usage:**
```bash
./secubox-tools/install-git-hooks.sh
```

This creates a symbolic link from `.git/hooks/pre-push` to `pre-push-validation.sh`.

## Recommended Workflow

### When Generating a New Module

1. **Generate module files** (use Claude with module-prompts.md)

2. **Validate the module:**
   ```bash
   ./secubox-tools/validate-module-generation.sh luci-app-<module-name>
   ```

3. **Fix all ERRORS** (critical)

4. **Review and fix WARNINGS** (recommended)

5. **Build and test locally** (recommended):
   ```bash
   ./secubox-tools/local-build.sh build luci-app-<module-name>
   # Test on router if needed
   ```

6. **Commit changes:**
   ```bash
   git add luci-app-<module-name>
   git commit -m "feat: implement <module-name> module"
   git push  # Pre-push validation runs automatically
   ```

### When Modifying Existing Modules

1. **Make your changes**

2. **Run quick validation:**
   ```bash
   ./secubox-tools/validate-modules.sh
   ```

3. **For complex changes, run full validation:**
   ```bash
   ./secubox-tools/validate-module-generation.sh luci-app-<module-name>
   ```

4. **Build and test locally** (recommended):
   ```bash
   ./secubox-tools/local-build.sh build luci-app-<module-name>
   ```

5. **Commit and push** (validation runs automatically)

### Before Committing Changes

Always run at least one validation tool before committing:

1. **Run validation** (CRITICAL):
   ```bash
   ./secubox-tools/validate-modules.sh
   # Or use local-build.sh for validation + build:
   ./secubox-tools/local-build.sh full
   ```

2. Fix any errors reported

3. Run shellcheck on RPCD scripts:
   ```bash
   shellcheck luci-app-*/root/usr/libexec/rpcd/*
   ```

4. **Test build locally** (recommended):
   ```bash
   ./secubox-tools/local-build.sh build
   ```

5. Commit changes

## Common Fixes

### Fix RPCD naming mismatch

If validation reports RPCD script name doesn't match ubus object:

```bash
# Rename the script to include luci. prefix
cd luci-app-example/root/usr/libexec/rpcd
mv example luci.example
```

### Fix menu path mismatch

If validation reports menu path doesn't match view file:

```bash
# Option 1: Update menu.d JSON to match file location
# Edit: root/usr/share/luci/menu.d/luci-app-example.json
# Change: "path": "example/view" → "path": "example-dashboard/view"

# Option 2: Move view files to match menu path
mv htdocs/luci-static/resources/view/example-dashboard \
   htdocs/luci-static/resources/view/example
```

### Fix non-executable RPCD script

```bash
chmod +x luci-app-example/root/usr/libexec/rpcd/luci.example
```

## SecuBox Package Feed

The SecuBox feed provides custom OpenWrt packages installable via `opkg`. After building packages, they are synced to `/www/secubox-feed` on the router.

### Feed Structure

```
/www/secubox-feed/
├── Packages              # Package index (text)
├── Packages.gz           # Compressed package index
├── Packages.sig          # Optional signature
└── *.ipk                 # Package files
```

### Configuring opkg to Use the Feed

**Option 1: Local File Access (same device)**
```bash
echo 'src/gz secubox file:///www/secubox-feed' >> /etc/opkg/customfeeds.conf
opkg update
```

**Option 2: HTTP Access (network devices)**
```bash
# From other devices on the network (replace IP with your router's address)
echo 'src/gz secubox http://192.168.255.1/secubox-feed' >> /etc/opkg/customfeeds.conf
opkg update
```

**Option 3: HAProxy Published Feed (with SSL)**
```bash
# If published via HAProxy with domain
echo 'src/gz secubox https://feed.example.com' >> /etc/opkg/customfeeds.conf
opkg update
```

### Installing Packages from the Feed

```bash
# Update package lists
opkg update

# List available SecuBox packages
opkg list | grep -E '^(luci-app-|secubox-)'

# Install a package
opkg install luci-app-service-registry

# Install with dependencies
opkg install --force-depends luci-app-haproxy
```

### Regenerating the Package Index

After adding new .ipk files to the feed:

```bash
# On the router
cd /www/secubox-feed
/usr/libexec/opkg-make-index . > Packages
gzip -k Packages
```

Or use the deploy command:
```bash
# From development machine
./secubox-tools/local-build.sh deploy root@192.168.255.1 "luci-app-*"
```

### App Store Integration

The LuCI App Store reads from `apps-local.json` to list available packages:

```bash
# Generate apps manifest from feed
cat /www/secubox-feed/Packages | awk '
/^Package:/ { pkg=$2 }
/^Version:/ { ver=$2 }
/^Description:/ { desc=substr($0, 14); print pkg, ver, desc }
'
```

The Service Registry dashboard aggregates installed apps and their status.

### Exposing Feed via HAProxy

To publish the feed with HTTPS:

```bash
# Create HAProxy backend for the feed
ubus call luci.haproxy create_backend '{"name":"secubox-feed","mode":"http"}'
ubus call luci.haproxy create_server '{"backend":"secubox-feed","address":"127.0.0.1","port":80}'
ubus call luci.haproxy create_vhost '{"domain":"feed.example.com","backend":"secubox-feed","ssl":1,"acme":1}'

# Request certificate
ubus call luci.haproxy request_certificate '{"domain":"feed.example.com"}'
```

### Troubleshooting

**Feed not updating:**
```bash
# Check feed URL is accessible
curl -I http://192.168.255.1/secubox-feed/Packages

# Check opkg config
cat /etc/opkg/customfeeds.conf

# Force refresh
rm /var/opkg-lists/secubox
opkg update
```

**Package signature errors:**
```bash
# Skip signature verification (development only)
opkg update --no-check-certificate
opkg install --force-checksum <package>
```

---

## Integration with CI/CD

The validation script can be integrated into GitHub Actions workflows:

```yaml
- name: Validate modules
  run: |
    chmod +x secubox-tools/validate-modules.sh
    ./secubox-tools/validate-modules.sh    
```

## Critical Naming Rules

**These rules are MANDATORY** - violations will cause runtime errors:

1. **RPCD scripts** must be named `luci.<module-name>`
   - ✅ `luci.cdn-cache`
   - ❌ `cdn-cache`

2. **Menu paths** must match view file locations
   - Menu: `"path": "cdn-cache/overview"`
   - File: `view/cdn-cache/overview.js`

3. **ubus objects** must use `luci.` prefix
   - ✅ `object: 'luci.cdn-cache'`
   - ❌ `object: 'cdn-cache'`

See `CLAUDE.md` for complete documentation.