From 6da042505edfddfcd2caa05e5dad527c77eed4e2 Mon Sep 17 00:00:00 2001 From: CyberMind-FR Date: Wed, 24 Dec 2025 16:31:12 +0100 Subject: [PATCH] feat: add firmware image building to local build script MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Add comprehensive firmware building functionality to local-build.sh, replicating the GitHub Actions firmware build workflow. Users can now build complete OpenWrt firmware images with SecuBox pre-installed locally, in addition to building packages. New Features: - New 'build-firmware' command to build complete firmware images - Support for GlobalScale devices (ESPRESSObin, Sheeva64, MOCHAbin) - Device profile definitions with automatic target/subtarget configuration - Automatic OpenWrt source download and caching - Device-specific package configuration (10G for MOCHAbin, WiFi for Ultra) - Firmware configuration with same settings as GitHub Actions workflow: * CONFIG_TARGET_PER_DEVICE_ROOTFS=y for image generation * Explicit PROFILE parameter in make commands * Device profile verification before building - Artifact collection with firmware images, packages, and checksums - Build time tracking and verbose logging - 'clean-all' command to remove OpenWrt source (saves ~20GB) Device Profiles: - espressobin-v7 (mvebu/cortexa53) - espressobin-ultra (mvebu/cortexa53 with WiFi) - sheeva64 (mvebu/cortexa53) - mochabin (mvebu/cortexa72 with 10G networking) - x86-64 (x86/64 generic PC) Usage: ./secubox-tools/local-build.sh build-firmware mochabin ./secubox-tools/local-build.sh build-firmware espressobin-v7 Output: Firmware images placed in build/firmware// - *.img.gz, *sysupgrade.bin, *factory.bin files - SHA256SUMS checksums - BUILD_INFO.txt with build details - packages/ directory with SecuBox .ipk files Documentation: - Updated README.md with firmware building examples - Added device profiles and environment variables - Added example workflows for firmware building - Distinguished between package building (SDK) and firmware building (full source) Technical Details: - Uses full OpenWrt source (not SDK) for firmware builds - Configures feeds identically to GitHub Actions workflow - Includes same configuration fixes for image generation - Verifies device profile selection before lengthy build - Supports parallel and single-threaded builds with automatic retry - Downloads and caches OpenWrt source for reuse 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 --- secubox-tools/README.md | 49 +++- secubox-tools/local-build.sh | 468 ++++++++++++++++++++++++++++++++++- 2 files changed, 511 insertions(+), 6 deletions(-) diff --git a/secubox-tools/README.md b/secubox-tools/README.md index 6e6896c8..996fa1a5 100644 --- a/secubox-tools/README.md +++ b/secubox-tools/README.md @@ -26,14 +26,23 @@ Build and test packages locally without pushing to GitHub. Automatically downloa # Build for specific architecture ./secubox-tools/local-build.sh build --arch aarch64-cortex-a72 +# Build firmware image for MOCHAbin +./secubox-tools/local-build.sh build-firmware mochabin + +# Build firmware image for ESPRESSObin V7 +./secubox-tools/local-build.sh build-firmware espressobin-v7 + # Full validation + build ./secubox-tools/local-build.sh full # Clean build artifacts ./secubox-tools/local-build.sh clean + +# Clean everything including OpenWrt source +./secubox-tools/local-build.sh clean-all ``` -**Supported Architectures:** +**Supported Architectures (for package building):** - `x86-64` - PC, VMs (default) - `aarch64-cortex-a53` - ARM Cortex-A53 (ESPRESSObin) - `aarch64-cortex-a72` - ARM Cortex-A72 (MOCHAbin, RPi4) @@ -41,13 +50,23 @@ Build and test packages locally without pushing to GitHub. Automatically downloa - `mips-24kc` - MIPS 24Kc (TP-Link) - `mipsel-24kc` - MIPS LE (Xiaomi, GL.iNet) +**Supported Devices (for firmware building):** +- `espressobin-v7` - ESPRESSObin V7 (1-2GB DDR4) +- `espressobin-ultra` - ESPRESSObin Ultra (PoE, WiFi) +- `sheeva64` - Sheeva64 (Plug computer) +- `mochabin` - MOCHAbin (Quad-core A72, 10G) +- `x86-64` - x86_64 Generic PC + **Environment Variables:** - `OPENWRT_VERSION` - OpenWrt version (default: 23.05.5) - `SDK_DIR` - SDK directory (default: ./sdk) - `BUILD_DIR` - Build output directory (default: ./build) - `CACHE_DIR` - Download cache directory (default: ./cache) +- `OPENWRT_DIR` - OpenWrt source directory for firmware builds (default: ./openwrt) -**Output:** Built packages are placed in `build//` with SHA256 checksums. +**Output:** +- Built packages are placed in `build//` with SHA256 checksums +- Firmware images are placed in `build/firmware//` with checksums and build info **Dependencies:** ```bash @@ -62,15 +81,18 @@ sudo apt-get install -y shellcheck nodejs ``` **Features:** -- Downloads and caches OpenWrt SDK for faster subsequent builds +- **Package Building**: Downloads and caches OpenWrt SDK for faster builds +- **Firmware Building**: Downloads full OpenWrt source and builds custom firmware images - Configures feeds (packages, luci) automatically - Validates packages before building - Builds .ipk packages with verbose output +- Builds complete firmware images (.img.gz, *sysupgrade.bin, etc.) - Collects artifacts with checksums - Supports single package or all packages -- Multiple architecture support +- Multiple architecture and device support +- Device profile verification before building -**Example Workflow:** +**Example Workflow - Package Building:** ```bash # 1. Make changes to a module vim luci-app-system-hub/htdocs/luci-static/resources/view/system-hub/overview.js @@ -85,6 +107,23 @@ opkg install /tmp/luci-app-system-hub*.ipk /etc/init.d/rpcd restart ``` +**Example Workflow - Firmware Building:** +```bash +# 1. Build firmware for MOCHAbin with SecuBox pre-installed +./secubox-tools/local-build.sh build-firmware mochabin + +# 2. Flash to device +# Firmware images are in: build/firmware/mochabin/ +# - openwrt-*-sysupgrade.bin (for upgrading existing OpenWrt) +# - openwrt-*-factory.bin (for initial installation) +# - SHA256SUMS (checksums for verification) +# - BUILD_INFO.txt (build details) +# - packages/ (SecuBox .ipk files) + +# 3. Clean up after building (optional) +./secubox-tools/local-build.sh clean-all # Removes OpenWrt source (saves ~20GB) +``` + ### Validation Tools #### validate-modules.sh diff --git a/secubox-tools/local-build.sh b/secubox-tools/local-build.sh index da6c52f4..414a7404 100755 --- a/secubox-tools/local-build.sh +++ b/secubox-tools/local-build.sh @@ -26,12 +26,22 @@ OPENWRT_VERSION="${OPENWRT_VERSION:-23.05.5}" SDK_DIR="${SDK_DIR:-./sdk}" BUILD_DIR="${BUILD_DIR:-./build}" CACHE_DIR="${CACHE_DIR:-./cache}" +OPENWRT_DIR="${OPENWRT_DIR:-./openwrt}" # Default architecture ARCH="x86-64" ARCH_NAME="x86_64" SDK_PATH="x86/64" +# Device profiles for firmware building +declare -A DEVICE_PROFILES=( + ["espressobin-v7"]="mvebu:cortexa53:globalscale_espressobin:ESPRESSObin V7 (1-2GB DDR4)" + ["espressobin-ultra"]="mvebu:cortexa53:globalscale_espressobin-ultra:ESPRESSObin Ultra (PoE, WiFi)" + ["sheeva64"]="mvebu:cortexa53:globalscale_sheeva64:Sheeva64 (Plug computer)" + ["mochabin"]="mvebu:cortexa72:globalscale_mochabin:MOCHAbin (Quad-core A72, 10G)" + ["x86-64"]="x86:64:generic:x86_64 Generic PC" +) + # Helper functions print_header() { echo "" @@ -768,6 +778,426 @@ run_build() { return 0 } +# ============================================ +# Firmware Image Building Functions +# ============================================ + +# Parse device profile +parse_device_profile() { + local device="$1" + + if [[ -z "${DEVICE_PROFILES[$device]}" ]]; then + print_error "Unknown device: $device" + print_info "Available devices: ${!DEVICE_PROFILES[*]}" + return 1 + fi + + local profile="${DEVICE_PROFILES[$device]}" + IFS=':' read -r TARGET SUBTARGET PROFILE_NAME DESCRIPTION <<< "$profile" + + export FW_TARGET="$TARGET" + export FW_SUBTARGET="$SUBTARGET" + export FW_PROFILE="$PROFILE_NAME" + export FW_DESCRIPTION="$DESCRIPTION" + export FW_DEVICE="$device" + + return 0 +} + +# Download OpenWrt source +download_openwrt_source() { + print_header "Downloading OpenWrt Source" + + if [[ -d "$OPENWRT_DIR/.git" ]]; then + print_info "OpenWrt source already exists, checking version..." + cd "$OPENWRT_DIR" + local current_version=$(git describe --tags 2>/dev/null || echo "unknown") + if [[ "$current_version" == "v${OPENWRT_VERSION}" ]]; then + print_success "Using existing OpenWrt $OPENWRT_VERSION" + cd - > /dev/null + return 0 + else + print_info "Version mismatch (current: $current_version), re-cloning..." + cd - > /dev/null + rm -rf "$OPENWRT_DIR" + fi + fi + + print_info "Cloning OpenWrt $OPENWRT_VERSION..." + + if [[ "$OPENWRT_VERSION" == "SNAPSHOT" ]]; then + git clone --depth 1 https://github.com/openwrt/openwrt.git "$OPENWRT_DIR" + else + git clone --depth 1 --branch "v${OPENWRT_VERSION}" \ + https://github.com/openwrt/openwrt.git "$OPENWRT_DIR" + fi + + print_success "OpenWrt source downloaded" + return 0 +} + +# Setup OpenWrt feeds for firmware build +setup_openwrt_feeds() { + print_header "Setting up OpenWrt Feeds" + + cd "$OPENWRT_DIR" + + # Remove unwanted feeds + if [[ -f "feeds.conf.default" ]]; then + sed -i '/telephony/d' feeds.conf.default + sed -i '/routing/d' feeds.conf.default + print_success "Removed telephony and routing from feeds.conf.default" + fi + + # Update feeds + print_info "Updating feeds (this may take a few minutes)..." + if ! ./scripts/feeds update -a 2>&1 | tee feed-update.log; then + print_warning "Feed update had errors, continuing..." + fi + + # Install feeds + print_info "Installing feeds..." + if ! ./scripts/feeds install -a 2>&1 | tee feed-install.log; then + print_warning "Feed install had warnings, checking directories..." + fi + + # Verify feeds + for feed in packages luci; do + if [[ -d "feeds/$feed" ]]; then + local feed_size=$(du -sh "feeds/$feed" 2>/dev/null | cut -f1) + print_success "feeds/$feed ($feed_size)" + else + print_error "feeds/$feed missing!" + cd - > /dev/null + return 1 + fi + done + + cd - > /dev/null + print_success "OpenWrt feeds configured" + return 0 +} + +# Copy SecuBox packages to OpenWrt +copy_secubox_to_openwrt() { + print_header "Copying SecuBox Packages to OpenWrt" + + cd "$OPENWRT_DIR" + + mkdir -p package/secubox + + local pkg_count=0 + for pkg in ../../luci-app-*/; do + if [[ -d "$pkg" ]]; then + local pkg_name=$(basename "$pkg") + echo " ✅ $pkg_name" + cp -r "$pkg" package/secubox/ + + # Fix Makefile include path + if [[ -f "package/secubox/$pkg_name/Makefile" ]]; then + sed -i 's|include.*luci\.mk|include $(TOPDIR)/feeds/luci/luci.mk|' \ + "package/secubox/$pkg_name/Makefile" + fi + + pkg_count=$((pkg_count + 1)) + fi + done + + cd - > /dev/null + + print_success "Copied $pkg_count SecuBox packages" + return 0 +} + +# Generate firmware configuration +generate_firmware_config() { + print_header "Generating Firmware Configuration" + + cd "$OPENWRT_DIR" + + print_info "Device: $FW_DESCRIPTION" + print_info "Target: $FW_TARGET/$FW_SUBTARGET" + print_info "Profile: $FW_PROFILE" + + # Base configuration + cat > .config << EOF +# Target +CONFIG_TARGET_${FW_TARGET}=y +CONFIG_TARGET_${FW_TARGET}_${FW_SUBTARGET}=y +CONFIG_TARGET_${FW_TARGET}_${FW_SUBTARGET}_DEVICE_${FW_PROFILE}=y + +# Image building (REQUIRED for firmware generation) +CONFIG_TARGET_MULTI_PROFILE=n +CONFIG_TARGET_ALL_PROFILES=n +CONFIG_TARGET_PER_DEVICE_ROOTFS=y + +# Image settings +CONFIG_TARGET_ROOTFS_SQUASHFS=y +CONFIG_TARGET_ROOTFS_EXT4FS=y +CONFIG_TARGET_KERNEL_PARTSIZE=32 +CONFIG_TARGET_ROOTFS_PARTSIZE=512 + +# Base packages +CONFIG_PACKAGE_luci=y +CONFIG_PACKAGE_luci-ssl=y +CONFIG_PACKAGE_luci-app-opkg=y +CONFIG_PACKAGE_luci-theme-openwrt-2020=y + +# Networking essentials +CONFIG_PACKAGE_curl=y +CONFIG_PACKAGE_wget-ssl=y +CONFIG_PACKAGE_iptables=y +CONFIG_PACKAGE_ip6tables=y + +# USB support +CONFIG_PACKAGE_kmod-usb-core=y +CONFIG_PACKAGE_kmod-usb3=y +CONFIG_PACKAGE_kmod-usb-storage=y + +# Filesystem +CONFIG_PACKAGE_kmod-fs-ext4=y +CONFIG_PACKAGE_kmod-fs-vfat=y + +# SecuBox packages +CONFIG_PACKAGE_luci-app-system-hub=y +CONFIG_PACKAGE_luci-app-network-modes=y +CONFIG_PACKAGE_luci-app-wireguard-dashboard=y +CONFIG_PACKAGE_luci-app-crowdsec-dashboard=y +CONFIG_PACKAGE_luci-app-netdata-dashboard=y +CONFIG_PACKAGE_luci-app-netifyd-dashboard=y +CONFIG_PACKAGE_luci-app-client-guardian=y + +# WireGuard +CONFIG_PACKAGE_wireguard-tools=y +CONFIG_PACKAGE_kmod-wireguard=y +CONFIG_PACKAGE_qrencode=y +EOF + + # Device-specific packages + case "$FW_DEVICE" in + mochabin) + cat >> .config << EOF + +# MOCHAbin specific - 10G networking +CONFIG_PACKAGE_kmod-sfp=y +CONFIG_PACKAGE_kmod-phy-marvell-10g=y +EOF + ;; + espressobin-ultra|sheeva64) + cat >> .config << EOF + +# WiFi support +CONFIG_PACKAGE_kmod-mt76=y +CONFIG_PACKAGE_kmod-mac80211=y +EOF + ;; + esac + + # Run defconfig + make defconfig + + cd - > /dev/null + + print_success "Configuration generated" + return 0 +} + +# Verify firmware configuration +verify_firmware_config() { + print_header "Verifying Firmware Configuration" + + cd "$OPENWRT_DIR" + + # Check device profile + if grep -q "CONFIG_TARGET_${FW_TARGET}_${FW_SUBTARGET}_DEVICE_${FW_PROFILE}=y" .config; then + print_success "Device profile correctly configured" + else + print_error "Device profile not found in .config!" + print_info "Searching for available profiles..." + find "target/$FW_TARGET/$FW_SUBTARGET" -name "*.mk" -exec grep -l "DEVICE_NAME" {} \; 2>/dev/null | head -5 + cd - > /dev/null + return 1 + fi + + # Check image generation + if grep -q "CONFIG_TARGET_ROOTFS_SQUASHFS=y" .config; then + print_success "SQUASHFS image generation enabled" + fi + + if grep -q "CONFIG_TARGET_ROOTFS_EXT4FS=y" .config; then + print_success "EXT4 image generation enabled" + fi + + # Show relevant config + echo "" + print_info "Device configuration:" + grep "^CONFIG_TARGET_" .config | head -10 + + cd - > /dev/null + + print_success "Configuration verified" + return 0 +} + +# Build firmware image +build_firmware_image() { + print_header "Building Firmware Image" + + cd "$OPENWRT_DIR" + + print_info "Device: $FW_DESCRIPTION" + print_info "Target: $FW_TARGET/$FW_SUBTARGET" + print_info "Profile: $FW_PROFILE" + print_info "CPU Cores: $(nproc)" + echo "" + + local start_time=$(date +%s) + + # Download packages first + print_info "Downloading packages..." + if ! make download -j$(nproc) V=s; then + print_warning "Parallel download failed, retrying single-threaded..." + make download -j1 V=s + fi + + echo "" + print_header "Compiling Firmware (This may take 1-2 hours)" + echo "" + + # Build with explicit PROFILE + if make -j$(nproc) PROFILE="$FW_PROFILE" V=s 2>&1 | tee build.log; then + local end_time=$(date +%s) + local duration=$((end_time - start_time)) + local minutes=$((duration / 60)) + local seconds=$((duration % 60)) + + print_success "Build completed in ${minutes}m ${seconds}s" + else + print_error "Parallel build failed, retrying single-threaded..." + if make -j1 PROFILE="$FW_PROFILE" V=s 2>&1 | tee build-retry.log; then + local end_time=$(date +%s) + local duration=$((end_time - start_time)) + local minutes=$((duration / 60)) + local seconds=$((duration % 60)) + + print_success "Build completed in ${minutes}m ${seconds}s (after retry)" + else + print_error "Build failed!" + echo "" + echo "Last 50 lines of build log:" + tail -50 build-retry.log + cd - > /dev/null + return 1 + fi + fi + + cd - > /dev/null + return 0 +} + +# Collect firmware artifacts +collect_firmware_artifacts() { + print_header "Collecting Firmware Artifacts" + + local target_dir="$OPENWRT_DIR/bin/targets/$FW_TARGET/$FW_SUBTARGET" + local output_dir="$BUILD_DIR/firmware/$FW_DEVICE" + + mkdir -p "$output_dir" + + # Find and copy firmware images + local img_count=0 + if [[ -d "$target_dir" ]]; then + echo "📂 Files in target directory:" + ls -lh "$target_dir" | grep -v "^total" | grep -v "^d" + echo "" + + while IFS= read -r file; do + case "$(basename "$file")" in + *.ipk|*.manifest|*.json|sha256sums|*.buildinfo|packages) + continue + ;; + *) + cp "$file" "$output_dir/" + print_success "$(basename "$file") ($(du -h "$file" | cut -f1))" + img_count=$((img_count + 1)) + ;; + esac + done < <(find "$target_dir" -maxdepth 1 -type f 2>/dev/null) + fi + + # Copy packages + mkdir -p "$output_dir/packages" + find "$OPENWRT_DIR/bin/packages" -name "luci-app-*.ipk" -exec cp {} "$output_dir/packages/" \; 2>/dev/null || true + + local pkg_count=$(find "$output_dir/packages" -name "*.ipk" 2>/dev/null | wc -l) + + # Generate checksums + cd "$output_dir" + sha256sum *.* > SHA256SUMS 2>/dev/null || true + if [[ -d packages && -n "$(ls -A packages 2>/dev/null)" ]]; then + (cd packages && sha256sum *.ipk > SHA256SUMS 2>/dev/null || true) + fi + cd - > /dev/null + + # Create build info + cat > "$output_dir/BUILD_INFO.txt" << EOF +SecuBox Firmware Build +====================== +Device: $FW_DESCRIPTION +Profile: $FW_PROFILE +Target: $FW_TARGET/$FW_SUBTARGET +OpenWrt: $OPENWRT_VERSION +Built: $(date -u +%Y-%m-%dT%H:%M:%SZ) + +Firmware Images: $img_count +SecuBox Packages: $pkg_count +EOF + + echo "" + print_success "Firmware images: $img_count" + print_success "SecuBox packages: $pkg_count" + print_success "Artifacts saved to: $output_dir" + + echo "" + print_info "Contents:" + ls -lh "$output_dir" + + return 0 +} + +# Run firmware build +run_firmware_build() { + local device="$1" + + if [[ -z "$device" ]]; then + print_error "Device not specified" + print_info "Usage: $0 build-firmware " + print_info "Available devices: ${!DEVICE_PROFILES[*]}" + return 1 + fi + + # Parse device profile + parse_device_profile "$device" || return 1 + + # Check dependencies + check_dependencies + + # Build firmware + download_openwrt_source || return 1 + setup_openwrt_feeds || return 1 + copy_secubox_to_openwrt || return 1 + generate_firmware_config || return 1 + verify_firmware_config || return 1 + build_firmware_image || return 1 + collect_firmware_artifacts || return 1 + + print_header "Firmware Build Complete!" + print_success "Device: $FW_DESCRIPTION" + print_success "Location: $BUILD_DIR/firmware/$FW_DEVICE/" + + return 0 +} + # Show usage show_usage() { cat << EOF @@ -782,11 +1212,13 @@ COMMANDS: build Build all packages for x86_64 build Build single package build --arch Build for specific architecture + build-firmware Build full firmware image for device full Run validation then build clean Clean build directories + clean-all Clean all build directories including OpenWrt source help Show this help message -ARCHITECTURES: +ARCHITECTURES (for package building): x86-64 PC, VMs (default) aarch64-cortex-a53 ARM Cortex-A53 (ESPRESSObin) aarch64-cortex-a72 ARM Cortex-A72 (MOCHAbin, RPi4) @@ -794,6 +1226,13 @@ ARCHITECTURES: mips-24kc MIPS 24Kc (TP-Link) mipsel-24kc MIPS LE (Xiaomi, GL.iNet) +DEVICES (for firmware building): + espressobin-v7 ESPRESSObin V7 (1-2GB DDR4) + espressobin-ultra ESPRESSObin Ultra (PoE, WiFi) + sheeva64 Sheeva64 (Plug computer) + mochabin MOCHAbin (Quad-core A72, 10G) + x86-64 x86_64 Generic PC + EXAMPLES: # Validate all packages $0 validate @@ -807,17 +1246,27 @@ EXAMPLES: # Build for specific architecture $0 build --arch aarch64-cortex-a72 + # Build firmware image for MOCHAbin + $0 build-firmware mochabin + + # Build firmware image for ESPRESSObin V7 + $0 build-firmware espressobin-v7 + # Full validation and build $0 full # Clean build artifacts $0 clean + # Clean everything including OpenWrt source + $0 clean-all + ENVIRONMENT VARIABLES: OPENWRT_VERSION OpenWrt version (default: 23.05.5) SDK_DIR SDK directory (default: ./sdk) BUILD_DIR Build output directory (default: ./build) CACHE_DIR Download cache directory (default: ./cache) + OPENWRT_DIR OpenWrt source directory for firmware builds (default: ./openwrt) EOF } @@ -861,6 +1310,17 @@ main() { run_build "$single_package" ;; + build-firmware) + local device="$1" + if [[ -z "$device" ]]; then + print_error "Device not specified" + print_info "Usage: $0 build-firmware " + print_info "Available devices: ${!DEVICE_PROFILES[*]}" + exit 1 + fi + run_firmware_build "$device" + ;; + full) run_validation && run_build ;; @@ -871,6 +1331,12 @@ main() { print_success "Build directories cleaned" ;; + clean-all) + print_header "Cleaning All Build Directories" + rm -rf "$SDK_DIR" "$BUILD_DIR" "$OPENWRT_DIR" "$CACHE_DIR" + print_success "All build directories cleaned (SDK, build, OpenWrt source, cache)" + ;; + help|--help|-h) show_usage ;;