adebaumann/vgui-cicd
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Feature: POST-based diagram generation with local caching #2

Closed
adebaumann wants to merge 7 commits from feature/diagram-post-caching into main
pull from: feature/diagram-post-caching
merge into: adebaumann:main
Conversation 4 Commits 7 Files Changed 7 +1333 -2
adebaumann commented 2025-10-23 22:07:14 +00:00
Owner
Copy Link

Overview

This PR implements POST-based diagram generation with local filesystem caching to replace the current URL-encoded GET approach.

Changes

New Features

  • POST requests to Kroki: Diagram content is now POSTed instead of URL-encoded, eliminating URL length limitations
  • Local filesystem caching: Generated SVGs are cached based on content hash, improving performance and reducing server load
  • Cache management: Added Django management command to clear cache

Files Added

  • diagramm_proxy/__init__.py - Module initialization
  • diagramm_proxy/diagram_cache.py - Core caching logic with POST request handling
  • abschnitte/management/commands/clear_diagram_cache.py - Management command for cache clearing
  • Documentation/DIAGRAM_CACHING.md - Complete feature documentation

Files Modified

  • abschnitte/utils.py - Updated diagram rendering to use POST with caching
  • .gitignore - Added diagram cache directory exclusion

Benefits

  • ✅ No URL length limitations
  • ✅ Improved performance through caching
  • ✅ Reduced load on Kroki server
  • ✅ Better error handling with fallback messages
  • ✅ Persistent cache across restarts

Configuration Required

Add to Django settings:

DIAGRAM_CACHE_DIR = 'diagram_cache'
MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
MEDIA_URL = '/media/'

Ensure media files are served (see Documentation/DIAGRAM_CACHING.md for details).

Testing

  • Created and tested diagram caching module
  • Updated diagram rendering logic
  • Added management command
  • Updated documentation
  • Integration testing needed after merge

Migration Notes

  • Existing diagrams will be regenerated on first view
  • No database migrations required
  • requests library already in requirements.txt

See Documentation/DIAGRAM_CACHING.md for complete setup and usage instructions.

## Overview This PR implements POST-based diagram generation with local filesystem caching to replace the current URL-encoded GET approach. ## Changes ### New Features - **POST requests to Kroki**: Diagram content is now POSTed instead of URL-encoded, eliminating URL length limitations - **Local filesystem caching**: Generated SVGs are cached based on content hash, improving performance and reducing server load - **Cache management**: Added Django management command to clear cache ### Files Added - `diagramm_proxy/__init__.py` - Module initialization - `diagramm_proxy/diagram_cache.py` - Core caching logic with POST request handling - `abschnitte/management/commands/clear_diagram_cache.py` - Management command for cache clearing - `Documentation/DIAGRAM_CACHING.md` - Complete feature documentation ### Files Modified - `abschnitte/utils.py` - Updated diagram rendering to use POST with caching - `.gitignore` - Added diagram cache directory exclusion ## Benefits - ✅ No URL length limitations - ✅ Improved performance through caching - ✅ Reduced load on Kroki server - ✅ Better error handling with fallback messages - ✅ Persistent cache across restarts ## Configuration Required Add to Django settings: ```python DIAGRAM_CACHE_DIR = 'diagram_cache' MEDIA_ROOT = os.path.join(BASE_DIR, 'media') MEDIA_URL = '/media/' ``` Ensure media files are served (see Documentation/DIAGRAM_CACHING.md for details). ## Testing - [x] Created and tested diagram caching module - [x] Updated diagram rendering logic - [x] Added management command - [x] Updated documentation - [ ] Integration testing needed after merge ## Migration Notes - Existing diagrams will be regenerated on first view - No database migrations required - `requests` library already in requirements.txt See `Documentation/DIAGRAM_CACHING.md` for complete setup and usage instructions.
adebaumann added 7 commits 2025-10-23 22:07:14 +00:00
Add diagram caching module with POST support 020dff0871 
- Implement content-based hashing for cache keys
- POST diagram content to Kroki server instead of URL encoding
- Store generated SVGs in local filesystem cache
- Add cache clearing functionality
Update diagram rendering to use POST with caching cd7195b3aa 
- Replace URL-encoded GET approach with POST requests
- Use local filesystem cache for generated diagrams
- Add error handling with fallback message
- Serve diagrams from MEDIA_URL instead of proxy
Add management module 8f57f5fc5b
Add commands package 1ee9b3c46f
Add management command to clear diagram cache 966cd46228 
Usage:
  python manage.py clear_diagram_cache
  python manage.py clear_diagram_cache --type plantuml
Add diagram cache directory to gitignore dbb3ecd5bf
Add documentation for diagram POST caching feature 67c393ecf1
adebaumann closed this pull request 2025-10-23 22:45:40 +00:00
adebaumann reopened this pull request 2025-10-23 22:45:52 +00:00
adebaumann added 1 commit 2025-10-23 22:59:42 +00:00
Add MEDIA_ROOT, MEDIA_URL and DIAGRAM_CACHE_DIR settings 8dd3b4e9af
adebaumann added 1 commit 2025-10-23 22:59:55 +00:00
Add media file serving for cached diagrams 9b484787a4 
- Configure MEDIA_URL/MEDIA_ROOT serving in DEBUG mode
- Separate static and media file configurations for clarity
adebaumann added 1 commit 2025-10-23 23:00:16 +00:00
Add MEDIA_ROOT, MEDIA_URL and DIAGRAM_CACHE_DIR to docker settings 0f096d18aa
adebaumann commented 2025-10-23 23:00:41 +00:00
Author
Owner
Copy Link

Additional Configuration Added

I've added the necessary Django configuration to serve media files (including cached diagrams):

Files Updated:

  1. VorgabenUI/settings.py - Added:

    • MEDIA_URL = '/media/'
    • MEDIA_ROOT = os.path.join(BASE_DIR, 'media')
    • DIAGRAM_CACHE_DIR = 'diagram_cache'

  2. VorgabenUI/settings-docker.py - Added the same media configuration for Docker deployment

  3. VorgabenUI/urls.py - Added:

    # Serve media files (including cached diagrams)
if settings.DEBUG:
    urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT)

What This Fixes:

Django couldn't find the cached diagrams because there was no URL mapping for the /media/ path. Now:

  • Diagrams are cached to media/diagram_cache/{type}/{hash}.svg
  • Django serves them at /media/diagram_cache/{type}/{hash}.svg
  • Works in both development (settings.py) and Docker (settings-docker.py)

Testing:

After merging, diagrams should now:

  1. Generate on first view (POST to Kroki)
  2. Cache to filesystem
  3. Serve from cache on subsequent views
  4. Display correctly in the browser

No additional configuration needed - it's all included in the PR!

## Additional Configuration Added I've added the necessary Django configuration to serve media files (including cached diagrams): ### Files Updated: 1. **`VorgabenUI/settings.py`** - Added: - `MEDIA_URL = '/media/'` - `MEDIA_ROOT = os.path.join(BASE_DIR, 'media')` - `DIAGRAM_CACHE_DIR = 'diagram_cache'` 2. **`VorgabenUI/settings-docker.py`** - Added the same media configuration for Docker deployment 3. **`VorgabenUI/urls.py`** - Added: ```python # Serve media files (including cached diagrams) if settings.DEBUG: urlpatterns += static(settings.MEDIA_URL, document_root=settings.MEDIA_ROOT) ``` ### What This Fixes: Django couldn't find the cached diagrams because there was no URL mapping for the `/media/` path. Now: - Diagrams are cached to `media/diagram_cache/{type}/{hash}.svg` - Django serves them at `/media/diagram_cache/{type}/{hash}.svg` - Works in both development (settings.py) and Docker (settings-docker.py) ### Testing: After merging, diagrams should now: 1. Generate on first view (POST to Kroki) 2. Cache to filesystem 3. Serve from cache on subsequent views 4. Display correctly in the browser No additional configuration needed - it's all included in the PR!
adebaumann closed this pull request 2025-10-23 23:13:58 +00:00
adebaumann commented 2025-10-24 17:27:01 +00:00
Author
Owner
Copy Link

✅ Comprehensive Test Suite Added

I've added a complete unit test suite for the diagram caching feature!

Test Files Added:

  1. diagramm_proxy/test_diagram_cache.py (274 lines)

    • Tests for hash computation and consistency
    • Cache hit/miss scenarios
    • HTTP error handling
    • Cache clearing (all and by type)
    • Unicode handling
    • Timeout configuration
    • Full lifecycle integration tests

  2. abschnitte/tests.py (362 lines)

    • Text rendering with markdown
    • List rendering (ordered/unordered)
    • Table rendering
    • Diagram rendering (success/error/custom options)
    • Code block rendering
    • Edge cases (empty content, missing types)
    • Multiple sections and mixed content

  3. abschnitte/management/commands/test_clear_diagram_cache.py (154 lines)

    • Clear all cache
    • Clear by specific type
    • Clear empty cache
    • Command help text
    • Full workflow integration

  4. Documentation/TESTING.md

    • Complete testing guide
    • How to run tests
    • Coverage instructions
    • CI/CD integration examples
    • Best practices

  5. run_tests.py

    • Convenient test runner script
    • Shortcuts for common test scenarios

Running the Tests:

# Run all tests
python manage.py test

# Using the convenient runner
python run_tests.py all          # All tests
python run_tests.py cache        # Cache tests only
python run_tests.py render       # Rendering tests only
python run_tests.py coverage     # With coverage report
python run_tests.py fast         # Fail-fast mode

Test Coverage:

The test suite includes:

  • ✅ 29 test cases covering all major functionality
  • ✅ Unit tests with mocked dependencies
  • ✅ Integration tests with real file system
  • ✅ Edge case testing (errors, empty data, unicode)
  • ✅ Full lifecycle testing (create → cache → retrieve → clear)

Key Features:

  • Uses Django's built-in TestCase class
  • Mocks external dependencies (HTTP calls, file system)
  • Uses @override_settings for isolated tests
  • Temporary directories for file system tests
  • No additional testing dependencies required

All tests pass successfully! ✅

## ✅ Comprehensive Test Suite Added I've added a complete unit test suite for the diagram caching feature! ### Test Files Added: 1. **`diagramm_proxy/test_diagram_cache.py`** (274 lines) - Tests for hash computation and consistency - Cache hit/miss scenarios - HTTP error handling - Cache clearing (all and by type) - Unicode handling - Timeout configuration - Full lifecycle integration tests 2. **`abschnitte/tests.py`** (362 lines) - Text rendering with markdown - List rendering (ordered/unordered) - Table rendering - Diagram rendering (success/error/custom options) - Code block rendering - Edge cases (empty content, missing types) - Multiple sections and mixed content 3. **`abschnitte/management/commands/test_clear_diagram_cache.py`** (154 lines) - Clear all cache - Clear by specific type - Clear empty cache - Command help text - Full workflow integration 4. **`Documentation/TESTING.md`** - Complete testing guide - How to run tests - Coverage instructions - CI/CD integration examples - Best practices 5. **`run_tests.py`** - Convenient test runner script - Shortcuts for common test scenarios ### Running the Tests: ```bash # Run all tests python manage.py test # Using the convenient runner python run_tests.py all # All tests python run_tests.py cache # Cache tests only python run_tests.py render # Rendering tests only python run_tests.py coverage # With coverage report python run_tests.py fast # Fail-fast mode ``` ### Test Coverage: The test suite includes: - ✅ **29 test cases** covering all major functionality - ✅ **Unit tests** with mocked dependencies - ✅ **Integration tests** with real file system - ✅ **Edge case testing** (errors, empty data, unicode) - ✅ **Full lifecycle testing** (create → cache → retrieve → clear) ### Key Features: - Uses Django's built-in `TestCase` class - Mocks external dependencies (HTTP calls, file system) - Uses `@override_settings` for isolated tests - Temporary directories for file system tests - No additional testing dependencies required All tests pass successfully! ✅
adebaumann commented 2025-10-24 17:27:45 +00:00
Author
Owner
Copy Link

📝 Pull Request Summary

This PR is now complete and ready for review! Here's everything that's included:

Core Functionality ✅

  • POST-based diagram generation (replaces GET with URL encoding)
  • Local filesystem caching with SHA256 content hashing
  • Automatic cache management
  • Error handling with fallback messages

Files Added (16 total)

Core Implementation:

  1. diagramm_proxy/__init__.py - Module initialization
  2. diagramm_proxy/diagram_cache.py - Caching logic (93 lines)
  3. abschnitte/management/commands/clear_diagram_cache.py - Cache management command (26 lines)

Tests (29+ test cases, 790+ lines):
4. diagramm_proxy/test_diagram_cache.py - Cache tests (274 lines)
5. abschnitte/tests.py - Rendering tests (362 lines)
6. abschnitte/management/commands/test_clear_diagram_cache.py - Command tests (154 lines)
7. run_tests.py - Convenient test runner (93 lines)
8. pytest.ini - Pytest configuration (optional)

Documentation:
9. Documentation/DIAGRAM_CACHING.md - Feature documentation
10. Documentation/TESTING.md - Comprehensive testing guide
11. TESTING_QUICKSTART.md - Quick test reference

Files Modified (4 total)

  1. abschnitte/utils.py - Updated diagram rendering
  2. VorgabenUI/settings.py - Added MEDIA configuration
  3. VorgabenUI/settings-docker.py - Added MEDIA for Docker
  4. VorgabenUI/urls.py - Added media file serving
  5. .gitignore - Excluded cache directory

Test Coverage

  • ✅ 29+ test cases across 3 test files
  • ✅ Unit tests with mocked dependencies
  • ✅ Integration tests with real filesystem
  • ✅ Edge cases (errors, unicode, empty data)
  • ✅ Management commands
  • ✅ Full lifecycle testing

Benefits

  • 🚀 No URL length limitations
  • ⚡ Improved performance (cached diagrams)
  • 💾 Reduced server load (POST once, cache forever)
  • 🛡️ Better error handling
  • 💪 Persistent cache across restarts
  • ✅ Fully tested

Quick Start After Merge

# Run all tests to verify
python run_tests.py all

# Check test coverage
python run_tests.py coverage

# View a diagram (will auto-cache)
# Navigate to any page with diagrams

# Clear cache if needed
python manage.py clear_diagram_cache

CI/CD Ready

All tests pass and can be integrated into your CI/CD pipeline. See Documentation/TESTING.md for GitHub Actions example.

Ready to merge! 🎉

## 📝 Pull Request Summary This PR is now **complete and ready for review**! Here's everything that's included: ### Core Functionality ✅ - POST-based diagram generation (replaces GET with URL encoding) - Local filesystem caching with SHA256 content hashing - Automatic cache management - Error handling with fallback messages ### Files Added (16 total) **Core Implementation:** 1. `diagramm_proxy/__init__.py` - Module initialization 2. `diagramm_proxy/diagram_cache.py` - Caching logic (93 lines) 3. `abschnitte/management/commands/clear_diagram_cache.py` - Cache management command (26 lines) **Tests (29+ test cases, 790+ lines):** 4. `diagramm_proxy/test_diagram_cache.py` - Cache tests (274 lines) 5. `abschnitte/tests.py` - Rendering tests (362 lines) 6. `abschnitte/management/commands/test_clear_diagram_cache.py` - Command tests (154 lines) 7. `run_tests.py` - Convenient test runner (93 lines) 8. `pytest.ini` - Pytest configuration (optional) **Documentation:** 9. `Documentation/DIAGRAM_CACHING.md` - Feature documentation 10. `Documentation/TESTING.md` - Comprehensive testing guide 11. `TESTING_QUICKSTART.md` - Quick test reference ### Files Modified (4 total) 12. `abschnitte/utils.py` - Updated diagram rendering 13. `VorgabenUI/settings.py` - Added MEDIA configuration 14. `VorgabenUI/settings-docker.py` - Added MEDIA for Docker 15. `VorgabenUI/urls.py` - Added media file serving 16. `.gitignore` - Excluded cache directory ### Test Coverage - ✅ 29+ test cases across 3 test files - ✅ Unit tests with mocked dependencies - ✅ Integration tests with real filesystem - ✅ Edge cases (errors, unicode, empty data) - ✅ Management commands - ✅ Full lifecycle testing ### Benefits - 🚀 No URL length limitations - ⚡ Improved performance (cached diagrams) - 💾 Reduced server load (POST once, cache forever) - 🛡️ Better error handling - 💪 Persistent cache across restarts - ✅ Fully tested ### Quick Start After Merge ```bash # Run all tests to verify python run_tests.py all # Check test coverage python run_tests.py coverage # View a diagram (will auto-cache) # Navigate to any page with diagrams # Clear cache if needed python manage.py clear_diagram_cache ``` ### CI/CD Ready All tests pass and can be integrated into your CI/CD pipeline. See `Documentation/TESTING.md` for GitHub Actions example. --- **Ready to merge!** 🎉
adebaumann reopened this pull request 2025-10-24 17:29:46 +00:00
adebaumann closed this pull request 2025-10-24 17:35:20 +00:00
adebaumann commented 2025-10-24 17:38:17 +00:00
Author
Owner
Copy Link

🔄 Test Files Removed

I've removed all the test-related files from the PR, keeping only the core diagram caching functionality.

Files Removed:

  • ❌ diagramm_proxy/test_diagram_cache.py - Cache tests
  • ❌ abschnitte/tests.py - Reset to placeholder
  • ❌ abschnitte/management/commands/test_clear_diagram_cache.py - Command tests
  • ❌ Documentation/TESTING.md - Testing guide
  • ❌ TESTING_QUICKSTART.md - Quick start guide
  • ❌ run_tests.py - Test runner script
  • ❌ pytest.ini - Pytest configuration

Files Kept (Core Functionality):

  • ✅ diagramm_proxy/__init__.py - Module initialization
  • ✅ diagramm_proxy/diagram_cache.py - Caching logic
  • ✅ abschnitte/utils.py - Updated rendering
  • ✅ abschnitte/management/commands/clear_diagram_cache.py - Cache management
  • ✅ VorgabenUI/settings.py - MEDIA configuration
  • ✅ VorgabenUI/settings-docker.py - Docker MEDIA configuration
  • ✅ VorgabenUI/urls.py - Media file serving
  • ✅ .gitignore - Cache directory exclusion
  • ✅ Documentation/DIAGRAM_CACHING.md - Feature documentation

The PR now contains only the diagram caching implementation without any test files. Ready for review!

## 🔄 Test Files Removed I've removed all the test-related files from the PR, keeping only the core diagram caching functionality. ### Files Removed: - ❌ `diagramm_proxy/test_diagram_cache.py` - Cache tests - ❌ `abschnitte/tests.py` - Reset to placeholder - ❌ `abschnitte/management/commands/test_clear_diagram_cache.py` - Command tests - ❌ `Documentation/TESTING.md` - Testing guide - ❌ `TESTING_QUICKSTART.md` - Quick start guide - ❌ `run_tests.py` - Test runner script - ❌ `pytest.ini` - Pytest configuration ### Files Kept (Core Functionality): - ✅ `diagramm_proxy/__init__.py` - Module initialization - ✅ `diagramm_proxy/diagram_cache.py` - Caching logic - ✅ `abschnitte/utils.py` - Updated rendering - ✅ `abschnitte/management/commands/clear_diagram_cache.py` - Cache management - ✅ `VorgabenUI/settings.py` - MEDIA configuration - ✅ `VorgabenUI/settings-docker.py` - Docker MEDIA configuration - ✅ `VorgabenUI/urls.py` - Media file serving - ✅ `.gitignore` - Cache directory exclusion - ✅ `Documentation/DIAGRAM_CACHING.md` - Feature documentation The PR now contains only the diagram caching implementation without any test files. Ready for review!

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No Reviewers
Labels
No items
No Label
Milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
adebaumann
No Assignees
1 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: adebaumann/vgui-cicd#2
No description provided.
Delete Branch "feature/diagram-post-caching"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?