ESP-Home-Git-Synchroniser/CLAUDE.md

# CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

## Project Overview

This is an ESPHome Gitea Sync Service - a Flask-based synchronization tool that bridges Gitea repositories with ESPHome device configurations. The service:
- Clones and syncs a Gitea repository containing ESPHome configs
- Watches for local file changes and can auto-push to Gitea
- Receives webhooks from Gitea to pull changes
- Provides manual sync endpoints

The ESPHome container remains immutable and handles all compilation/deployment operations. This service only manages git synchronization.

## Architecture

### Core Components

1. **Flask REST API** (app.py)
   - Health check and device listing endpoints
   - Gitea webhook endpoint (`/webhook/gitea`) for receiving push events
   - Manual sync endpoints (`/sync/pull`, `/sync/push`)
   - Thread-safe operation management using locks

2. **File Watcher System** (app.py)
   - Uses watchdog library to monitor YAML file changes
   - Implements debouncing (default 5 seconds) to prevent duplicate triggers
   - Tracks last modification time per file in `ESPHomeFileHandler.last_modified`
   - Monitors all `.yaml`/`.yml` files directly in the config directory
   - Optionally triggers `git push` when `AUTO_PUSH=true`
   - In production (Gunicorn), started via `post_fork` hook in first worker only

3. **Git Operations** (app.py)
   - `git_clone()` - Initial clone of Gitea repository on startup
   - `git_pull()` - Pull changes from Gitea (triggered by webhook or manual)
   - `git_push()` - Push local changes to Gitea (auto or manual)
   - `initialize_git_repo()` - Called on startup to clone or sync repo
   - Authentication via Gitea token injected into remote URL

4. **Synchronization Flow**
   - **Gitea → ESPHome**: Gitea webhook triggers `/webhook/gitea` → `git pull` → ESPHome sees updated files
   - **ESPHome → Gitea**: File change detected → debounced → optionally `git push` (if AUTO_PUSH enabled)
   - Compilation/upload handled entirely by ESPHome container

### Expected Directory Structure

```
config/
├── device-name-1.yaml
├── device-name-2.yaml
├── device-name-3.yaml
└── .git/
```

The service expects a flat structure with each device having its own `.yaml` file directly in the config directory. The device name is the filename without the extension.

### Thread Safety

- `operation_lock` (Lock) protects the `pending_operations` dictionary
- Prevents concurrent operations on the same device
- Operations run in separate threads spawned from file watcher or webhook handlers

## Project Files

- `app.py` - Main Flask application with file watcher, git operations, and API endpoints
- `gunicorn_config.py` - Gunicorn configuration with hooks for file watcher initialization
- `requirements.txt` - Python dependencies
- `Dockerfile` - Container image definition
- `docker-compose.yml` - Multi-container orchestration (webhook + esphome services)
- `CLAUDE.md` - This documentation file

## Development Commands

### Local Development (Without Docker)

```bash
# Install dependencies
pip install -r requirements.txt

# Set environment variables
export ESPHOME_CONFIG_DIR=/path/to/config
export DEBOUNCE_SECONDS=5
export GITEA_URL=https://gitea.example.com
export GITEA_REPO=username/esphome-configs
export GITEA_TOKEN=your_gitea_token
export GITEA_BRANCH=main
export AUTO_PUSH=false
export GIT_USER_NAME="ESPHome Sync Service"
export GIT_USER_EMAIL="esphome-sync@localhost"

# Run the service directly
python app.py
```

### Docker Development

```bash
# Start services
docker-compose up -d

# View logs
docker-compose logs webhook
docker-compose logs esphome

# Stop services
docker-compose down

# Rebuild webhook service after code changes
docker-compose build webhook
docker-compose up -d webhook
```

### Building Custom Image

```bash
docker build -t esphome-webhook:custom .
```

## API Endpoints

- `GET /health` - Health check with configuration info
- `GET /devices` - List all devices (scans for `.yaml`/`.yml` files in config directory)
  - Returns device name (filename without extension), config path, and last modified time
  - Example: `bedroom-light.yaml` → device name: `bedroom-light`
- `POST /webhook/gitea` - Gitea webhook endpoint (handles push events, triggers git pull)
- `POST /sync/pull` - Manually trigger git pull from Gitea
- `POST /sync/push` - Manually trigger git push to Gitea (optional JSON body: `{"message": "commit message"}`)

## Configuration

Environment variables (set in docker-compose.yml or locally):

**Core Settings:**
- `ESPHOME_CONFIG_DIR` - Path to device configurations (default: `/config`)
- `DEBOUNCE_SECONDS` - Delay before triggering after file change (default: `5`)
- `USE_POLLING` - Use polling instead of inotify for file watching (default: `true`, required for Docker bind mounts)
- `POLLING_INTERVAL` - Seconds between filesystem polls when using polling mode (default: `1.0`)

**Gitea Configuration:**
- `GITEA_URL` - URL of Gitea instance (e.g., `https://gitea.example.com`)
- `GITEA_REPO` - Repository path (e.g., `username/esphome-configs`)
- `GITEA_TOKEN` - Authentication token for Gitea API (generate in Gitea user settings)
- `GITEA_BRANCH` - Git branch to use (default: `main`)
- `AUTO_PUSH` - Auto-push local changes to Gitea (default: `false`)

**Git User Configuration:**
- `GIT_USER_NAME` - Git commit author name (default: `ESPHome Sync Service`)
- `GIT_USER_EMAIL` - Git commit author email (default: `esphome-sync@localhost`)

## Setup Guide

### Initial Setup

1. **Create Gitea Repository**
   - Create a new repository in Gitea for your ESPHome configs
   - Generate an access token in Gitea: Settings → Applications → Generate New Token
   - Give token appropriate permissions (read/write repository)

2. **Configure Environment Variables**
   - Update `docker-compose.yml` with your Gitea details:
     - `GITEA_URL`: Your Gitea instance URL
     - `GITEA_REPO`: Your repository path (username/repo)
     - `GITEA_TOKEN`: Your generated token
   - Set `AUTO_PUSH=true` if you want local changes auto-pushed to Gitea

3. **Start Services**
   ```bash
   docker-compose up -d
   ```
   - On first startup, service will clone the Gitea repository to `/config`
   - If `/config` already has files, manually initialize git first

4. **Configure Gitea Webhook** (Optional but recommended)
   - Go to repository Settings → Webhooks → Add Webhook
   - Set URL: `http://your-server:5000/webhook/gitea`
   - Set Content Type: `application/json`
   - Select events: `Push` events
   - When you push to Gitea, service will automatically pull changes

### Migration from Old Structure

If you have existing configs in the old `device-name/main.yaml` structure, you need to migrate to the flat structure:

```bash
# Move all device configs to the flat structure
cd config
for dir in */; do
  if [ -f "${dir}main.yaml" ]; then
    device_name="${dir%/}"
    mv "${dir}main.yaml" "${device_name}.yaml"
    rmdir "${dir}" 2>/dev/null || echo "Directory ${dir} not empty, manual cleanup needed"
  fi
done

# Commit the changes
git add -A
git commit -m "Migrate to flat YAML structure"
git push origin main
```

After migration, your structure should be:
```
config/
├── device-1.yaml  (was device-1/main.yaml)
├── device-2.yaml  (was device-2/main.yaml)
└── .git/
```

### Workflows

**Editing in ESPHome Dashboard:**
1. Edit YAML files in ESPHome dashboard
2. If `AUTO_PUSH=true`, changes automatically pushed to Gitea
3. If `AUTO_PUSH=false`, manually push: `curl -X POST http://localhost:5000/sync/push`

**Editing in Gitea (or git client):**
1. Commit and push changes to Gitea repository
2. Gitea webhook triggers `/webhook/gitea`
3. Service runs `git pull`
4. ESPHome sees updated files

**Manual Sync:**
- Pull from Gitea: `curl -X POST http://localhost:5000/sync/pull`
- Push to Gitea: `curl -X POST http://localhost:5000/sync/push -H "Content-Type: application/json" -d '{"message": "My commit message"}'`

## Key Implementation Details

### Device Config Resolution

The `find_device_config()` function looks for device configs at:
- `{ESPHOME_CONFIG_DIR}/{device_name}.yaml`

If the device name doesn't include an extension, `.yaml` is automatically appended.

### File Watching Behavior

The file watcher monitors all `.yaml` and `.yml` files directly in the config directory:
- **Watched**: Any `*.yaml` or `*.yml` file in the root of the config directory
- **Ignored**: Subdirectories, non-YAML files, and hidden files (like `.git`)
- **Debouncing**: Changes are debounced for `DEBOUNCE_SECONDS` (default 5) to prevent duplicate triggers
- **Auto-push**: If `AUTO_PUSH=true`, any watched file change triggers a git commit and push
- **Polling Mode**: Uses filesystem polling (default) instead of inotify to ensure compatibility with Docker bind mounts on WSL2, macOS, Unraid, and other environments where inotify events don't propagate correctly through volume mounts

#### Events Detected

1. **File Modified** (`device.yaml` edited)
   - Logs: "Detected change in /config/device.yaml"
   - Commit message: `Auto-sync: device.yaml changed`

2. **File Created** (new `device.yaml` added)
   - Logs: "Detected change in /config/device.yaml"
   - Commit message: `Auto-sync: device.yaml changed`

3. **File Deleted** (`device.yaml` removed)
   - Logs: "Detected deletion of /config/device.yaml"
   - Commit message: `Auto-sync: device.yaml deleted`

4. **File Renamed** (`old-device.yaml` → `new-device.yaml`)
   - Logs: "Rename detected: old-device.yaml -> new-device.yaml"
   - Commit message: `Auto-sync: Renamed old-device.yaml to new-device.yaml`

5. **File Archived** (`device.yaml` → `archive/device.yaml`)
   - Logs: "Device moved to archive/subdirectory: device"
   - Commit message: `Auto-sync: device.yaml archived`
   - The file deletion from `/config/` root is committed (archive folder typically in `.gitignore`)

6. **File Restored** (`archive/device.yaml` → `device.yaml`)
   - Logs: "Device moved from subdirectory to config root: device"
   - Commit message: `Auto-sync: device.yaml restored from archive`

Example watched files in `/config`:
- `bedroom-light.yaml` ✓
- `kitchen-sensor.yaml` ✓
- `garage-door.yml` ✓
- `README.md` ✗ (not YAML)
- `.gitignore` ✗ (hidden file)
- `archive/old-config.yaml` ✗ (in subdirectory, not watched but move events detected)

### Git Authentication

Authentication is handled by injecting the Gitea token into the remote URL:
- Input: `GITEA_URL=https://gitea.com`, `GITEA_TOKEN=abc123`, `GITEA_REPO=user/repo`
- Remote URL: `https://abc123@gitea.com/user/repo.git`
- Token is only stored in environment variables, never committed to git

### Operation Flow

1. **Startup**: `initialize_git_repo()` → Clone if not exists OR Pull latest changes
2. **File Change Detected** → Debounced → Check if YAML file in config directory → Optionally `git push` (threaded)
3. **Gitea Webhook** → Parse payload → `git pull` → ESPHome sees updated files
4. **Thread Safety** → Operations use `operation_lock` to prevent concurrent access to `pending_operations` dict

### Git Repository Initialization

On startup, the service:
1. Checks if `/config/.git` exists
2. If not, clones the repository to a temp directory, then moves contents to `/config`
3. If yes, configures git user and pulls latest changes
4. All git operations have 60-second timeout

### Gunicorn Configuration

**Configuration File:** `gunicorn_config.py`

- 2 workers
- 600 second timeout
- Binds to 0.0.0.0:5000

**Hooks for File Watcher:**
- `on_starting` hook: Initializes git repository once before workers are forked
- `post_fork` hook: Starts file watcher thread in the first worker only (worker.age == 0)
- This ensures the file watcher runs correctly under Gunicorn without duplicates

**Why hooks are needed:**
When running under Gunicorn, the `if __name__ == '__main__':` block in app.py is never executed. The hooks ensure that:
1. Git initialization happens once at startup
2. File watcher starts in exactly one worker process
3. Multiple workers don't create duplicate file watchers

## Docker Compose Architecture

Two services work together:

1. **esphome** - Official ESPHome dashboard (host network mode for mDNS/OTA)
   - Dashboard: http://localhost:6052
   - Handles compilation and device deployment
   - Shares `./config` volume with sync service (read-only)
   - Uses persistent `esphome-cache` volume
   - **Immutable** - never modified, only reads configs

2. **webhook** (esphome-gitea-sync) - This service (bridged network)
   - API: http://localhost:5000
   - Synchronizes configurations between Gitea and ESPHome
   - Read-write mount of `./config` for git operations
   - Has git installed and manages the git repository
   - ESPHome container handles all compilation/upload operations

### Volume Architecture

- `./config` is shared between both containers
- Sync service has read-write access (manages git repo)
- ESPHome has read access (consumes configs)
- On first startup, if `./config` is empty, sync service clones Gitea repo