adebaumann/ESP-Home-Git-Synchroniser

Fork 0

Files

Adrian A. Baumann 424c20923b Updated for automatic pushing back - pulling already works via webhook

2026-01-13 20:03:36 +01:00

13 KiB

Raw Permalink Blame History

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

This is an ESPHome Gitea Sync Service - a Flask-based synchronization tool that bridges Gitea repositories with ESPHome device configurations. The service:

Clones and syncs a Gitea repository containing ESPHome configs
Watches for local file changes and can auto-push to Gitea
Receives webhooks from Gitea to pull changes
Provides manual sync endpoints

The ESPHome container remains immutable and handles all compilation/deployment operations. This service only manages git synchronization.

Architecture

Core Components

Flask REST API (app.py)
- Health check and device listing endpoints
- Gitea webhook endpoint (/webhook/gitea) for receiving push events
- Manual sync endpoints (/sync/pull, /sync/push)
- Thread-safe operation management using locks
File Watcher System (app.py)
- Uses watchdog library to monitor YAML file changes
- Implements debouncing (default 5 seconds) to prevent duplicate triggers
- Tracks last modification time per file in ESPHomeFileHandler.last_modified
- Monitors all .yaml/.yml files directly in the config directory
- Optionally triggers git push when AUTO_PUSH=true
- In production (Gunicorn), started via post_fork hook in first worker only
Git Operations (app.py)
- git_clone() - Initial clone of Gitea repository on startup
- git_pull() - Pull changes from Gitea (triggered by webhook or manual)
- git_push() - Push local changes to Gitea (auto or manual)
- initialize_git_repo() - Called on startup to clone or sync repo
- Authentication via Gitea token injected into remote URL
Synchronization Flow
- Gitea → ESPHome: Gitea webhook triggers /webhook/gitea → git pull → ESPHome sees updated files
- ESPHome → Gitea: File change detected → debounced → optionally git push (if AUTO_PUSH enabled)
- Compilation/upload handled entirely by ESPHome container

Expected Directory Structure

config/
├── device-name-1.yaml
├── device-name-2.yaml
├── device-name-3.yaml
└── .git/

The service expects a flat structure with each device having its own .yaml file directly in the config directory. The device name is the filename without the extension.

Thread Safety

operation_lock (Lock) protects the pending_operations dictionary
Prevents concurrent operations on the same device
Operations run in separate threads spawned from file watcher or webhook handlers

Project Files

app.py - Main Flask application with file watcher, git operations, and API endpoints
gunicorn_config.py - Gunicorn configuration with hooks for file watcher initialization
requirements.txt - Python dependencies
Dockerfile - Container image definition
docker-compose.yml - Multi-container orchestration (webhook + esphome services)
CLAUDE.md - This documentation file

Development Commands

Local Development (Without Docker)

# Install dependencies
pip install -r requirements.txt

# Set environment variables
export ESPHOME_CONFIG_DIR=/path/to/config
export DEBOUNCE_SECONDS=5
export GITEA_URL=https://gitea.example.com
export GITEA_REPO=username/esphome-configs
export GITEA_TOKEN=your_gitea_token
export GITEA_BRANCH=main
export AUTO_PUSH=false
export GIT_USER_NAME="ESPHome Sync Service"
export GIT_USER_EMAIL="esphome-sync@localhost"

# Run the service directly
python app.py

Docker Development

# Start services
docker-compose up -d

# View logs
docker-compose logs webhook
docker-compose logs esphome

# Stop services
docker-compose down

# Rebuild webhook service after code changes
docker-compose build webhook
docker-compose up -d webhook

Building Custom Image

docker build -t esphome-webhook:custom .

API Endpoints

GET /health - Health check with configuration info
GET /devices - List all devices (scans for .yaml/.yml files in config directory)
- Returns device name (filename without extension), config path, and last modified time
- Example: bedroom-light.yaml → device name: bedroom-light
POST /webhook/gitea - Gitea webhook endpoint (handles push events, triggers git pull)
POST /sync/pull - Manually trigger git pull from Gitea
POST /sync/push - Manually trigger git push to Gitea (optional JSON body: {"message": "commit message"})

Configuration

Environment variables (set in docker-compose.yml or locally):

Core Settings:

ESPHOME_CONFIG_DIR - Path to device configurations (default: /config)
DEBOUNCE_SECONDS - Delay before triggering after file change (default: 5)
USE_POLLING - Use polling instead of inotify for file watching (default: true, required for Docker bind mounts)
POLLING_INTERVAL - Seconds between filesystem polls when using polling mode (default: 1.0)

Gitea Configuration:

GITEA_URL - URL of Gitea instance (e.g., https://gitea.example.com)
GITEA_REPO - Repository path (e.g., username/esphome-configs)
GITEA_TOKEN - Authentication token for Gitea API (generate in Gitea user settings)
GITEA_BRANCH - Git branch to use (default: main)
AUTO_PUSH - Auto-push local changes to Gitea (default: false)

Git User Configuration:

GIT_USER_NAME - Git commit author name (default: ESPHome Sync Service)
GIT_USER_EMAIL - Git commit author email (default: esphome-sync@localhost)

Setup Guide

Initial Setup

Create Gitea Repository
- Create a new repository in Gitea for your ESPHome configs
- Generate an access token in Gitea: Settings → Applications → Generate New Token
- Give token appropriate permissions (read/write repository)
Configure Environment Variables
- Update docker-compose.yml with your Gitea details:
  - GITEA_URL: Your Gitea instance URL
  - GITEA_REPO: Your repository path (username/repo)
  - GITEA_TOKEN: Your generated token
- Set AUTO_PUSH=true if you want local changes auto-pushed to Gitea
Start Services
```
docker-compose up -d
```
- On first startup, service will clone the Gitea repository to /config
- If /config already has files, manually initialize git first
Configure Gitea Webhook (Optional but recommended)
- Go to repository Settings → Webhooks → Add Webhook
- Set URL: http://your-server:5000/webhook/gitea
- Set Content Type: application/json
- Select events: Push events
- When you push to Gitea, service will automatically pull changes

Migration from Old Structure

If you have existing configs in the old device-name/main.yaml structure, you need to migrate to the flat structure:

# Move all device configs to the flat structure
cd config
for dir in */; do
  if [ -f "${dir}main.yaml" ]; then
    device_name="${dir%/}"
    mv "${dir}main.yaml" "${device_name}.yaml"
    rmdir "${dir}" 2>/dev/null || echo "Directory ${dir} not empty, manual cleanup needed"
  fi
done

# Commit the changes
git add -A
git commit -m "Migrate to flat YAML structure"
git push origin main

After migration, your structure should be:

config/
├── device-1.yaml  (was device-1/main.yaml)
├── device-2.yaml  (was device-2/main.yaml)
└── .git/

Workflows

Editing in ESPHome Dashboard:

Edit YAML files in ESPHome dashboard
If AUTO_PUSH=true, changes automatically pushed to Gitea
If AUTO_PUSH=false, manually push: curl -X POST http://localhost:5000/sync/push

Editing in Gitea (or git client):

Commit and push changes to Gitea repository
Gitea webhook triggers /webhook/gitea
Service runs git pull
ESPHome sees updated files

Manual Sync:

Pull from Gitea: curl -X POST http://localhost:5000/sync/pull
Push to Gitea: curl -X POST http://localhost:5000/sync/push -H "Content-Type: application/json" -d '{"message": "My commit message"}'

Key Implementation Details

Device Config Resolution

The find_device_config() function looks for device configs at:

{ESPHOME_CONFIG_DIR}/{device_name}.yaml

If the device name doesn't include an extension, .yaml is automatically appended.

File Watching Behavior

The file watcher monitors all .yaml and .yml files directly in the config directory:

Watched: Any *.yaml or *.yml file in the root of the config directory
Ignored: Subdirectories, non-YAML files, and hidden files (like .git)
Debouncing: Changes are debounced for DEBOUNCE_SECONDS (default 5) to prevent duplicate triggers
Auto-push: If AUTO_PUSH=true, any watched file change triggers a git commit and push
Polling Mode: Uses filesystem polling (default) instead of inotify to ensure compatibility with Docker bind mounts on WSL2, macOS, Unraid, and other environments where inotify events don't propagate correctly through volume mounts

Events Detected

File Modified (device.yaml edited)
- Logs: "Detected change in /config/device.yaml"
- Commit message: Auto-sync: device.yaml changed
File Created (new device.yaml added)
- Logs: "Detected change in /config/device.yaml"
- Commit message: Auto-sync: device.yaml changed
File Deleted (device.yaml removed)
- Logs: "Detected deletion of /config/device.yaml"
- Commit message: Auto-sync: device.yaml deleted
File Renamed (old-device.yaml → new-device.yaml)
- Logs: "Rename detected: old-device.yaml -> new-device.yaml"
- Commit message: Auto-sync: Renamed old-device.yaml to new-device.yaml
File Archived (device.yaml → archive/device.yaml)
- Logs: "Device moved to archive/subdirectory: device"
- Commit message: Auto-sync: device.yaml archived
- The file deletion from /config/ root is committed (archive folder typically in .gitignore)
File Restored (archive/device.yaml → device.yaml)
- Logs: "Device moved from subdirectory to config root: device"
- Commit message: Auto-sync: device.yaml restored from archive

Example watched files in /config:

bedroom-light.yaml ✓
kitchen-sensor.yaml ✓
garage-door.yml ✓
README.md ✗ (not YAML)
.gitignore ✗ (hidden file)
archive/old-config.yaml ✗ (in subdirectory, not watched but move events detected)

Git Authentication

Authentication is handled by injecting the Gitea token into the remote URL:

Input: GITEA_URL=https://gitea.com, GITEA_TOKEN=abc123, GITEA_REPO=user/repo
Remote URL: https://abc123@gitea.com/user/repo.git
Token is only stored in environment variables, never committed to git

Operation Flow

Startup: initialize_git_repo() → Clone if not exists OR Pull latest changes
File Change Detected → Debounced → Check if YAML file in config directory → Optionally git push (threaded)
Gitea Webhook → Parse payload → git pull → ESPHome sees updated files
Thread Safety → Operations use operation_lock to prevent concurrent access to pending_operations dict

Git Repository Initialization

On startup, the service:

Checks if /config/.git exists
If not, clones the repository to a temp directory, then moves contents to /config
If yes, configures git user and pulls latest changes
All git operations have 60-second timeout

Gunicorn Configuration

Configuration File: gunicorn_config.py

2 workers
600 second timeout
Binds to 0.0.0.0:5000

Hooks for File Watcher:

on_starting hook: Initializes git repository once before workers are forked
post_fork hook: Starts file watcher thread in the first worker only (worker.age == 0)
This ensures the file watcher runs correctly under Gunicorn without duplicates

Why hooks are needed: When running under Gunicorn, the if __name__ == '__main__': block in app.py is never executed. The hooks ensure that:

Git initialization happens once at startup
File watcher starts in exactly one worker process
Multiple workers don't create duplicate file watchers

Docker Compose Architecture

Two services work together:

esphome - Official ESPHome dashboard (host network mode for mDNS/OTA)
- Dashboard: http://localhost:6052
- Handles compilation and device deployment
- Shares ./config volume with sync service (read-only)
- Uses persistent esphome-cache volume
- Immutable - never modified, only reads configs
webhook (esphome-gitea-sync) - This service (bridged network)
- API: http://localhost:5000
- Synchronizes configurations between Gitea and ESPHome
- Read-write mount of ./config for git operations
- Has git installed and manages the git repository
- ESPHome container handles all compilation/upload operations

Volume Architecture

./config is shared between both containers
Sync service has read-write access (manages git repo)
ESPHome has read access (consumes configs)
On first startup, if ./config is empty, sync service clones Gitea repo

13 KiB Raw Permalink Blame History