Updated for automatic pushing back - pulling already works via webhook

2026-01-13 20:03:36 +01:00
parent e5ba624aa9
commit 424c20923b
5 changed files with 404 additions and 125 deletions
--- a/README.md
+++ b/README.md
@@ -1,50 +1,84 @@
-# ESPHome Webhook Service
+# ESPHome Gitea Sync Service

-A Docker-based webhook service for automating ESPHome device compilation and deployment with file watching capabilities.
+A Docker-based synchronization service that bridges Gitea repositories with ESPHome device configurations. Automatically syncs YAML configuration files between your Gitea git repository and your ESPHome installation.

 ## Features

- **Webhook Endpoints**: Trigger ESPHome operations via HTTP POST requests
- **File Watching**: Automatically detect YAML changes and trigger compilation/upload
- **REST API**: List devices, check health, and manage operations
- **Docker Compose**: Easy deployment with ESPHome dashboard
- **Debouncing**: Prevent duplicate operations from rapid file changes
+- **Bidirectional Git Sync**: Sync configurations between Gitea and ESPHome
+- **File Watching**: Automatically detect YAML changes and push to Gitea
+- **Gitea Webhooks**: Receive push events and pull changes automatically
+- **REST API**: Manual sync endpoints and device listing
+- **Docker Polling**: Uses filesystem polling for reliable change detection on Docker bind mounts
 - **Thread-Safe**: Concurrent operation handling with locks
+- **Archive Support**: Detects when files are moved to archive folders
+
+## Architecture
+
+This service works alongside the official ESPHome container:
+- **ESPHome Container**: Handles all compilation and device deployment (immutable)
+- **Sync Service**: Manages git synchronization between Gitea and ESPHome (this service)
+
+The ESPHome container remains unchanged and simply consumes the YAML configs. This service only handles git operations.

 ## Quick Start

-### 1. Start the Services
+### 1. Configure Environment Variables
+
+Edit `docker-compose.yml` and set your Gitea details:
+
+```yaml
+environment:
+  - GITEA_URL=https://your-gitea-instance.com
+  - GITEA_REPO=username/esphome-configs
+  - GITEA_TOKEN=your_gitea_access_token
+  - GITEA_BRANCH=main
+  - AUTO_PUSH=false  # Set to true for automatic push on file changes
+```
+
+### 2. Start the Services

 ```bash
-cd esphome-webhook-service
 docker-compose up -d
 ```

 This starts two services:
- **ESPHome Dashboard**: `http://localhost:6052` (host network mode)
- **Webhook Service**: `http://localhost:5000`
+- **ESPHome Dashboard**: `http://localhost:6052` (compilation and deployment)
+- **Gitea Sync Service**: `http://localhost:5000` (git synchronization)

-### 2. Access the Services
+### 3. Access the Services

 - ESPHome Dashboard: http://localhost:6052
- Webhook API: http://localhost:5000/health
+- Sync API Health: http://localhost:5000/health
+- Device List: http://localhost:5000/devices

-### 3. Stop the Services
+### 4. Configure Gitea Webhook (Optional)

-```bash
-docker-compose down
-```
+To automatically pull changes when you push to Gitea:
+
+1. Go to your Gitea repository → Settings → Webhooks
+2. Add Webhook → Gitea
+3. Set Target URL: `http://your-server:5000/webhook/gitea`
+4. Content Type: `application/json`
+5. Trigger On: `Push Events`
+6. Click "Add Webhook"

 ## Environment Variables

-Configure the webhook service behavior in `docker-compose.yml`:
+Configure the sync service in `docker-compose.yml`:

 | Variable | Default | Description |
 |----------|---------|-------------|
 | `ESPHOME_CONFIG_DIR` | `/config` | Directory containing device configs |
-| `AUTO_COMPILE` | `true` | Auto-compile on YAML file changes |
-| `AUTO_UPLOAD` | `false` | Auto-upload on YAML file changes (⚠️ use with caution) |
-| `DEBOUNCE_SECONDS` | `5` | Delay before triggering operations after file change |
+| `DEBOUNCE_SECONDS` | `5` | Delay before triggering after file change |
+| `USE_POLLING` | `true` | Use polling for Docker compatibility (required) |
+| `POLLING_INTERVAL` | `1.0` | Seconds between filesystem polls |
+| `GITEA_URL` | - | URL of your Gitea instance |
+| `GITEA_REPO` | - | Repository path (username/repo) |
+| `GITEA_TOKEN` | - | Gitea access token (generate in user settings) |
+| `GITEA_BRANCH` | `main` | Git branch to sync |
+| `AUTO_PUSH` | `false` | Automatically push local changes to Gitea |
+| `GIT_USER_NAME` | `ESPHome Sync Service` | Git commit author name |
+| `GIT_USER_EMAIL` | `esphome-sync@localhost` | Git commit author email |

 ## API Endpoints

@@ -56,103 +90,116 @@ curl http://localhost:5000/health

 ### List Devices

+Lists all YAML files in the config directory:
+
 ```bash
 curl http://localhost:5000/devices
 ```

-### Validate Configuration
-
-```bash
-curl -X POST http://localhost:5000/webhook/validate/ades-office-control-panel
+Returns:
+```json
+{
+  "devices": [
+    {
+      "name": "bedroom-light",
+      "config_path": "/config/bedroom-light.yaml",
+      "last_modified": "2026-01-13T10:30:00"
+    }
+  ]
+}
 ```

-### Compile Device
+### Gitea Webhook
+
+Receives push events from Gitea and pulls changes:

 ```bash
-curl -X POST http://localhost:5000/webhook/compile/ades-office-control-panel
+curl -X POST http://localhost:5000/webhook/gitea \
+  -H "Content-Type: application/json" \
+  -H "X-Gitea-Event: push" \
+  -d '{"repository": {"full_name": "user/repo"}}'
 ```

-### Upload Device (OTA)
+### Manual Pull from Gitea
+
+Manually trigger a git pull:

 ```bash
-curl -X POST http://localhost:5000/webhook/upload/ades-office-control-panel
+curl -X POST http://localhost:5000/sync/pull
 ```

-### Compile and Upload
+### Manual Push to Gitea
+
+Manually trigger a git push:

 ```bash
-curl -X POST http://localhost:5000/webhook/run/ades-office-control-panel
+curl -X POST http://localhost:5000/sync/push \
+  -H "Content-Type: application/json" \
+  -d '{"message": "Manual sync from API"}'
 ```

 ## File Watching

-The service automatically watches all YAML files in the config directory. When a `main.yaml` file is modified:
+The service monitors all `.yaml` and `.yml` files in the root of the config directory.

-1. **Auto-Compile Enabled**: Automatically compiles the device configuration
-2. **Auto-Upload Enabled**: Automatically uploads firmware to the device (OTA)
+### Detected Events

-**⚠️ WARNING**: Only enable `AUTO_UPLOAD=true` if you're confident in your changes and have physical access to devices in case of failures.
+1. **File Modified** - Edit existing YAML file
+2. **File Created** - Add new YAML file
+3. **File Deleted** - Remove YAML file
+4. **File Renamed** - Rename YAML file
+5. **File Archived** - Move YAML file to subdirectory (e.g., `archive/`)
+6. **File Restored** - Move YAML file from subdirectory back to root

-## Integration Examples
+When `AUTO_PUSH=true`, all these events automatically trigger a git commit and push to Gitea.

-### GitHub Actions Webhook
+### Commit Messages

-Trigger compilation after pushing changes:
-
-```yaml
-name: ESPHome Compile
-
-on:
-  push:
-    paths:
-      - '**/*.yaml'
-
-jobs:
-  compile:
-    runs-on: ubuntu-latest
-    steps:
-      - name: Trigger Webhook
-        run: |
-          curl -X POST http://your-server:5000/webhook/compile/ades-office-control-panel
-```
-
-### Home Assistant Automation
-
-Create an automation to compile devices:
-
-```yaml
-automation:
-  - alias: "Compile ESPHome Device"
-    trigger:
-      platform: webhook
-      webhook_id: esphome_compile
-    action:
-      service: rest_command.compile_device
-      data:
-        device: "{{ trigger.data.device }}"
-
-rest_command:
-  compile_device:
-    url: "http://localhost:5000/webhook/compile/{{ device }}"
-    method: POST
-```
+- Modified/Created: `Auto-sync: device-name.yaml changed`
+- Deleted: `Auto-sync: device-name.yaml deleted`
+- Renamed: `Auto-sync: Renamed old-name.yaml to new-name.yaml`
+- Archived: `Auto-sync: device-name.yaml archived`
+- Restored: `Auto-sync: device-name.yaml restored from archive`

 ## Directory Structure

 ```
-esphome-webhook-service/
-├── app.py                  # Webhook service with file watching
-├── Dockerfile              # Container image definition
-├── requirements.txt        # Python dependencies
-├── docker-compose.yml      # Service orchestration
-├── README.md              # This file
-└── config/                 # Device configurations (mounted as /config)
-    ├── ades-office-control-panel/
-    │   └── main.yaml
-    ├── Old_Phone_Doorbell/
-    │   └── main.yaml
-    └── Oekoboiler/
-        └── main.yaml
+config/
+├── bedroom-light.yaml       # Device configs (flat structure)
+├── kitchen-sensor.yaml
+├── garage-door.yaml
+├── .gitignore              # Optional: ignore archive folder
+├── archive/                # Optional: archived configs (not monitored)
+│   └── old-device.yaml
+└── .git/                   # Git repository (managed by sync service)
+```
+
+## Workflows
+
+### Editing in ESPHome Dashboard
+
+1. Edit YAML files in ESPHome dashboard
+2. File watcher detects the change
+3. If `AUTO_PUSH=true`: Changes automatically committed and pushed to Gitea
+4. If `AUTO_PUSH=false`: Manually trigger push via API
+
+### Editing in Gitea (or git client)
+
+1. Commit and push changes to Gitea repository
+2. Gitea webhook triggers `/webhook/gitea` endpoint
+3. Service runs `git pull` to fetch changes
+4. ESPHome sees updated files and reloads dashboard
+
+### Manual Sync
+
+```bash
+# Pull latest changes from Gitea
+curl -X POST http://localhost:5000/sync/pull
+
+# Push local changes to Gitea
+curl -X POST http://localhost:5000/sync/push \
+  -H "Content-Type: application/json" \
+  -d '{"message": "My commit message"}'
 ```

 ## Troubleshooting
@@ -165,26 +212,40 @@ docker-compose logs webhook
 docker-compose logs esphome
 ```

-### Webhook Returns 404
+### File Watcher Not Detecting Changes

-Ensure the device name matches the directory name:
-```bash
-curl http://localhost:5000/devices
+**Cause**: inotify events don't propagate through Docker bind mounts on WSL2, Unraid, and macOS.
+
+**Solution**: The service uses polling mode by default (`USE_POLLING=true`). Ensure this is enabled in docker-compose.yml.
+
+Check logs for:
+```
+Using PollingObserver (interval: 1.0s) for Docker bind mount compatibility
 ```

-### OTA Upload Fails
+### Git Operations Failing

-1. Verify device is on the same network
-2. Check ESPHome logs: `docker-compose logs esphome`
-3. Ensure OTA password is correct in secrets.yaml
-4. Try using host network mode for webhook service (uncomment in docker-compose.yml)
+1. Verify Gitea token has correct permissions (read/write repository)
+2. Check `GITEA_URL`, `GITEA_REPO`, and `GITEA_TOKEN` are set correctly
+3. Check logs for git command errors:
+   ```bash
+   docker-compose logs webhook | grep -i "git command"
+   ```

-### File Watcher Not Triggering
+### Webhook Not Receiving Events

-1. Check that `AUTO_COMPILE` is set to `true`
-2. Verify the file path is correct
-3. Check webhook logs for file change events
-4. Ensure Docker has permission to watch the mounted volume
+1. Verify webhook URL is accessible from Gitea (firewall, network)
+2. Check Gitea webhook delivery logs (Repository → Settings → Webhooks → Recent Deliveries)
+3. Ensure `Content-Type` is `application/json` and trigger is `Push Events`
+
+### Changes Not Syncing to Gitea
+
+1. Check if `AUTO_PUSH=true` in docker-compose.yml
+2. Verify file is in the root of `/config/` (not in subdirectory)
+3. Check logs for file change detection:
+   ```bash
+   docker-compose logs webhook | grep -i "detected"
+   ```

 ## Development

@@ -193,12 +254,18 @@ curl http://localhost:5000/devices
 ```bash
 # Install dependencies
 pip install -r requirements.txt
-pip install esphome

 # Set environment variables
-export ESPHOME_CONFIG_DIR=/path/to/esphome2/config
-export AUTO_COMPILE=true
-export AUTO_UPLOAD=false
+export ESPHOME_CONFIG_DIR=/path/to/config
+export DEBOUNCE_SECONDS=5
+export USE_POLLING=true
+export GITEA_URL=https://gitea.example.com
+export GITEA_REPO=username/esphome-configs
+export GITEA_TOKEN=your_token
+export GITEA_BRANCH=main
+export AUTO_PUSH=false
+export GIT_USER_NAME="Your Name"
+export GIT_USER_EMAIL="your@email.com"

 # Run the service
 python app.py
@@ -207,17 +274,51 @@ python app.py
 ### Build Custom Image

 ```bash
-docker build -t esphome-webhook:custom .
+docker build -t esphome-gitea-sync:custom .
+```
+
+### Testing File Watcher
+
+Create, modify, delete, or move files in the config directory and watch the logs:
+
+```bash
+docker-compose logs -f webhook
+```
+
+You should see:
+```
+Detected change in /config/test.yaml
+Change detected in device: test
+AUTO_PUSH enabled, pushing changes to Gitea
 ```

 ## Security Considerations

- The webhook service has no authentication by default
+- The sync service has no authentication by default
+- Gitea token is stored in environment variables (keep docker-compose.yml secure)
 - Only expose port 5000 on trusted networks
 - Use a reverse proxy (nginx, Traefik) with authentication for external access
- Keep `AUTO_UPLOAD=false` unless absolutely necessary
- Review changes before enabling auto-upload
+- Keep `AUTO_PUSH=false` unless you trust all changes made in ESPHome dashboard
+- Review the `.gitignore` file to avoid committing sensitive data
+
+## Migration from Old Structure
+
+If migrating from `device-name/main.yaml` to flat `device-name.yaml` structure:
+
+```bash
+cd config
+for dir in */; do
+  if [ -f "${dir}main.yaml" ]; then
+    device_name="${dir%/}"
+    mv "${dir}main.yaml" "${device_name}.yaml"
+    rmdir "${dir}"
+  fi
+done
+git add -A
+git commit -m "Migrate to flat YAML structure"
+git push origin main
+```

 ## License

-Same as parent repository (see LICENSE file).
+MIT License - See LICENSE file for details.