Hub-and-Spoke Umbau: Multi-Tenant Zentrale + Satellite-Agent
- Backend: Customer/Satellite Models, customer_id auf Server/Job/Audit - Satellite-API: heartbeat, poll (atomares Claiming), logs, result, scan-result, health-report - Auth via X-Api-Key (SHA-256 gehasht) - Job-Queue: pending/claimed/running/success/failed + Stale-Janitor - Batch-Trigger: ein Job pro Server, Satellite arbeitet sequenziell ab - Credentials bleiben lokal: nur symbolische credential_ref zentral - Neues Paket satellite/: Pull-Loop, WinRM/SSH/CAU/Scanner, PyInstaller-tauglich - Frontend: Kunden-Switcher, Satelliten-View, Polling statt WebSocket - Entfernt: WebSocket/Socket.io, Redis, zentrale Credentials, JobRunner - Docs: README/AGENTS/PROMPT auf neue Architektur aktualisiert
This commit is contained in:
B0rbor4d
@@ -1,11 +1,13 @@
|
||||
# Insight Updater - Agent Orientation
|
||||
|
||||
## Project Purpose
|
||||
Self-hosted update orchestration for Windows (CAU/WSUS) and Linux servers via WinRM/SSH. Web UI to manage inventory, trigger updates, stream live progress via WebSocket.
|
||||
Zentrale, mandantenfaehige Update-Orchestrierung (Hub-and-Spoke). Zentrale (Docker) +
|
||||
Satelliten beim Kunden (Binary, kein Docker). Satelliten pollen Jobs, fuehren sie lokal
|
||||
im Kundennetz aus (WinRM/SSH/CAU/Scan) und melden Ergebnisse zurueck.
|
||||
|
||||
## Quick Start
|
||||
```bash
|
||||
# Local development
|
||||
# Zentrale lokal
|
||||
cd ~/projects/insight-updater
|
||||
docker compose up -d --build
|
||||
|
||||
@@ -14,137 +16,110 @@ cd backend && pip install -e . && uvicorn app.main:app --reload
|
||||
|
||||
# Frontend only
|
||||
cd frontend && npm install && npm run dev
|
||||
|
||||
# Satellite (Dev, gegen lokale Zentrale)
|
||||
cd satellite && pip install -e .
|
||||
cp config.example.yaml config.yaml # api_key eintragen
|
||||
insight-satellite
|
||||
```
|
||||
|
||||
## Architecture Overview
|
||||
|
||||
```
|
||||
┌─────────────┐ ┌─────────────┐ ┌─────────────┐
|
||||
│ Frontend │────▶│ Backend │────▶│ Database │
|
||||
│ (Vue 3) │ WS │ (FastAPI) │ │ (SQLite/ │
|
||||
│ Port 3000 │◀─── │ Port 8000 │ │ PostgreSQL)│
|
||||
└─────────────┘ └──────┬──────┘ └─────────────┘
|
||||
│
|
||||
┌────────────┼────────────┐
|
||||
▼ ▼ ▼
|
||||
┌─────────┐ ┌─────────┐ ┌─────────┐
|
||||
│ WinRM │ │ SSH │ │ CAU │
|
||||
│ Service │ │ Service │ │ Service │
|
||||
└─────────┘ └─────────┘ └─────────┘
|
||||
Kundennetz Zentrale
|
||||
┌────────────┐ outbound HTTPS ┌──────────────────┐
|
||||
│ Satellite │ ──────────────────▶ │ FastAPI Backend │
|
||||
│ (pollt │ X-Api-Key Auth │ /api/satellite │
|
||||
│ alle 30s) │ ◀────────────────── │ /api (Dashboard │
|
||||
└────────────┘ Jobs (claimed) │ JWT) + DB │
|
||||
│ WinRM/SSH/CAU lokal └──────────────────┘
|
||||
▼ ▲
|
||||
Server im Kundennetz Vue 3 Frontend (JWT, Kunden-Switcher)
|
||||
```
|
||||
|
||||
## Key Directories
|
||||
|
||||
| Path | Purpose |
|
||||
|------|---------|
|
||||
| `backend/app/api/` | FastAPI route definitions (REST + WS) |
|
||||
| `backend/app/core/` | Config, security, database, logging |
|
||||
| `backend/app/models/` | SQLAlchemy ORM models |
|
||||
| `backend/app/schemas/` | Pydantic request/response models |
|
||||
| `backend/app/services/` | Business logic: winrm, ssh, cau, audit |
|
||||
| `backend/app/websocket/` | Socket.io handlers for live updates |
|
||||
| `frontend/src/views/` | Page components (Dashboard, Servers, Updates, Audit) |
|
||||
| `frontend/src/components/` | Reusable UI components |
|
||||
| `frontend/src/stores/` | Pinia stores (auth, servers, updates) |
|
||||
| `frontend/src/api/` | Axios/Socket.io client setup |
|
||||
| Pfad | Zweck |
|
||||
|------|-------|
|
||||
| `backend/app/api/routes/satellite_api.py` | Agent-API: heartbeat, poll, logs, result, scan-result, health-report |
|
||||
| `backend/app/api/routes/` | Dashboard-REST: auth, customers, satellites, servers, updates, audit |
|
||||
| `backend/app/models/` | Customer, Satellite, Server, UpdateJob, UpdateLog, AuditLog, User |
|
||||
| `backend/app/services/janitor.py` | Markiert stale Jobs (Satellite meldet nicht mehr) als failed |
|
||||
| `satellite/satellite/runner.py` | Main-Loop: heartbeat, poll, execute, report |
|
||||
| `satellite/satellite/winrm_exec.py` | Windows Update via pywinrm |
|
||||
| `satellite/satellite/ssh_exec.py` | Linux Update via asyncssh |
|
||||
| `satellite/satellite/cau_exec.py` | Invoke-CauRun via WinRM |
|
||||
| `satellite/satellite/scanner.py` | Ping-Sweep + Port-Probe (5985/22) |
|
||||
| `frontend/src/stores/` | Pinia: auth, customers, satellites, servers, updates |
|
||||
|
||||
## Core Models
|
||||
|
||||
| Model | Description |
|
||||
|-------|-------------|
|
||||
| `Server` | Inventory item: Windows/WinRM, Linux/SSH, CAU-Cluster |
|
||||
| `Credential` | Encrypted credentials (WinRM user/pass, SSH key/pass) |
|
||||
| `UpdateJob` | One update execution: server, status, started_by, started_at, finished_at |
|
||||
| `UpdateLog` | Streamed log lines per job (WebSocket → DB) |
|
||||
| `AuditLog` | Immutable audit trail: user, action, target, result |
|
||||
| `User` | Local admin or LDAP-mapped user |
|
||||
| Model | Beschreibung |
|
||||
|-------|--------------|
|
||||
| `Customer` | Tenant: name, slug |
|
||||
| `Satellite` | Agent beim Kunden: api_key_hash, last_seen, version, hostname |
|
||||
| `Server` | Inventar pro Kunde: hostname, type, credential_ref (symbolisch!) |
|
||||
| `UpdateJob` | customer_id, server_id (null bei Scan), satellite_id, status, params (JSON) |
|
||||
| `UpdateLog` | Log-Zeilen pro Job (Batch-Upload vom Satellite) |
|
||||
| `AuditLog` | Wer, wann, was - mit customer_id |
|
||||
| `User` | Dashboard-User (lokal oder LDAP-Stub) |
|
||||
|
||||
## Key Services
|
||||
## Job Lifecycle (Pull-Modell)
|
||||
|
||||
| Service | File | Responsibility |
|
||||
|---------|------|----------------|
|
||||
| `WinRMService` | `services/winrm.py` | Connect, run PS commands, stream output |
|
||||
| `SSHService` | `services/ssh.py` | Connect, run commands, sudo handling |
|
||||
| `CAUService` | `services/cau.py` | `Invoke-CauRun`, cluster status, node phases |
|
||||
| `AuditService` | `services/audit.py` | Structured logging to DB + JSON file |
|
||||
| `EncryptionService` | `core/security.py` | Fernet encrypt/decrypt credentials |
|
||||
|
||||
## WebSocket Events
|
||||
|
||||
| Event | Direction | Payload |
|
||||
|-------|-----------|---------|
|
||||
| `job:start` | Server→Client | `{job_id, server_id, type}` |
|
||||
| `job:log` | Server→Client | `{job_id, line, timestamp, level}` |
|
||||
| `job:progress` | Server→Client | `{job_id, percent, phase, node?}` |
|
||||
| `job:complete` | Server→Client | `{job_id, status, duration}` |
|
||||
| `job:cancel` | Client→Server | `{job_id}` |
|
||||
|
||||
## API Endpoints (Key)
|
||||
|
||||
| Method | Path | Description |
|
||||
|--------|------|-------------|
|
||||
| GET | `/api/servers` | List all servers |
|
||||
| POST | `/api/servers` | Create server |
|
||||
| GET | `/api/servers/{id}/health` | Test WinRM/SSH connectivity |
|
||||
| POST | `/api/updates/trigger` | Start update job |
|
||||
| GET | `/api/updates/{job_id}/logs` | Get job logs (REST fallback) |
|
||||
| WS | `/ws/updates` | Socket.io connection |
|
||||
| GET | `/api/audit` | Paginated audit log |
|
||||
| POST | `/api/auth/login` | JWT login |
|
||||
| GET | `/health` | Health check |
|
||||
|
||||
## Environment Variables
|
||||
|
||||
See `.env.example` — key ones:
|
||||
- `DATABASE_URL` — SQLite (dev) or PostgreSQL (prod)
|
||||
- `SECRET_KEY` — JWT signing (32+ chars)
|
||||
- `ENCRYPTION_KEY` — Fernet key for credentials (32 bytes base64)
|
||||
- `WINRM_TRANSPORT` — `ntlm` \| `kerberos` \| `credssp`
|
||||
- `LDAP_ENABLED` — `true`/`false` (stub)
|
||||
|
||||
## Testing
|
||||
```bash
|
||||
# Backend
|
||||
cd backend && pytest -v
|
||||
|
||||
# Frontend
|
||||
cd frontend && npm run test
|
||||
|
||||
# E2E (Playwright)
|
||||
cd frontend && npm run test:e2e
|
||||
```
|
||||
pending -> claimed (beim Poll) -> running (erster Log-Push) -> success | failed
|
||||
| cancelled (nur aus pending)
|
||||
```
|
||||
|
||||
- Claiming passiert atomar im Poll (`with_for_update`), zwei Satelliten eines Kunden
|
||||
bekommen nie denselben Job
|
||||
- Janitor (`services/janitor.py`, alle 60s): claimed/running ohne Report seit
|
||||
`JOB_STALE_TIMEOUT` (default 3600s) -> failed
|
||||
- Cancel nur moeglich solange pending
|
||||
|
||||
## API Endpoints (Dashboard, JWT)
|
||||
|
||||
| Method | Path | Beschreibung |
|
||||
|--------|------|--------------|
|
||||
| POST | `/api/auth/login` | Login |
|
||||
| GET/POST/PATCH/DELETE | `/api/customers[/{id}]` | Kunden CRUD |
|
||||
| GET/POST/DELETE | `/api/satellites[/{id}]` | Satelliten CRUD |
|
||||
| POST | `/api/satellites/{id}/rotate-key` | Neuer API-Key |
|
||||
| GET/POST/PATCH/DELETE | `/api/servers[/{id}]` | Inventar (customer_id scoped) |
|
||||
| POST | `/api/updates/trigger` | Einzelner Job |
|
||||
| POST | `/api/updates/trigger-batch` | Ein Job pro Server |
|
||||
| GET | `/api/updates/{id}/logs` | Job-Logs |
|
||||
| GET | `/api/audit` | Audit (filterbar per customer_id) |
|
||||
|
||||
## API Endpoints (Satellite, X-Api-Key)
|
||||
|
||||
| Method | Path | Beschreibung |
|
||||
|--------|------|--------------|
|
||||
| POST | `/api/satellite/heartbeat` | Lebenszeichen + Version |
|
||||
| GET | `/api/satellite/poll` | Pending Jobs claimen + abholen |
|
||||
| POST | `/api/satellite/logs` | Log-Batch + Progress |
|
||||
| POST | `/api/satellite/result` | Abschluss success/failed |
|
||||
| POST | `/api/satellite/scan-result` | Gefundene Hosts (legt Server an) |
|
||||
| POST | `/api/satellite/health-report` | Health-Ergebnis pro Server |
|
||||
|
||||
## Satellite-Deployment beim Kunden
|
||||
|
||||
- Binary via PyInstaller: `pyinstaller --onefile satellite/runner.py`
|
||||
- `config.yaml`: central_url, api_key, poll_interval
|
||||
- `credentials.yaml`: WinRM/SSH Zugangsdaten (bleiben lokal!)
|
||||
- Start: Scheduled Task (Windows) oder systemd (Linux), siehe `satellite/README.md`
|
||||
- 1-2 Stueck pro Kunde reichen - steuern das ganze Netz
|
||||
|
||||
## Conventions
|
||||
- **Sprache**: Deutsch fuer User-facing Text, Englisch fuer Code/Kommentare
|
||||
- **Keine Umlaute in .ps1/Python-Dateien** (ae, oe, ue, ss)
|
||||
- **Keine Credentials zentral** - nur `credential_ref` Strings
|
||||
- **Async**: Backend vollstaendig async; Satellite async mit to_thread fuer pywinrm
|
||||
- **Types**: Strict mypy, Pydantic v2, SQLAlchemy 2.0
|
||||
|
||||
## Deployment
|
||||
|
||||
**Target**: `monitoring` (10.0.2.105)
|
||||
**User**: `b0rbor4d` (sudo via Vaultwarden)
|
||||
**Reverse Proxy**: Traefik (Docker labels)
|
||||
**Target Zentrale**: `monitoring` (10.0.2.105)
|
||||
**Reverse Proxy**: Traefik (Docker labels)
|
||||
**Git Remote**: `ssh://git@gitea.insight-it.de:2222/b0rbor4d/insight-updater.git`
|
||||
|
||||
```bash
|
||||
# On monitoring host
|
||||
git clone ssh://git@gitea.insight-it.de:2222/b0rbor4d/insight-updater.git
|
||||
cd insight-updater
|
||||
cp .env.example .env # fill secrets
|
||||
docker compose -f docker-compose.prod.yml up -d --build
|
||||
```
|
||||
|
||||
## Useful Commands
|
||||
```bash
|
||||
# DB migrations
|
||||
cd backend && alembic upgrade head
|
||||
|
||||
# Generate keys
|
||||
openssl genrsa -out keys/private.pem 2048
|
||||
openssl rsa -in keys/private.pem -pubout -out keys/public.pem
|
||||
|
||||
# Encrypt a test credential
|
||||
python -c "from app.core.security import encrypt; print(encrypt('secret'))"
|
||||
```
|
||||
|
||||
## Conventions
|
||||
- **Language**: German for user-facing text, English for code/comments
|
||||
- **Logging**: `structlog` JSON, level from `LOG_LEVEL` env
|
||||
- **Errors**: Custom exceptions in `core/exceptions.py`, mapped to HTTP in `main.py`
|
||||
- **Async**: All I/O async (asyncpg, asyncssh, httpx)
|
||||
- **Types**: Strict mypy, Pydantic v2, SQLAlchemy 2.0
|
||||
**Wichtig**: Zentrale muss von Kundenstandorten aus per HTTPS (443) erreichbar sein.
|
||||
|
||||
Reference in New Issue
Block a user