Five days after the first GPUShare post, I split the FastAPI app into a public middleware service and a backend for the hardware. The accounts and the GPU now had separate jobs, though the installer still pointed at the service I’d removed.
The backend split
Public middleware (port 8000) handles JWT and API-key authentication, users and invites, account aggregation, append-only ledger views, optional Stripe integration, MCP configuration, and short-lived in-process caches. This is the service the Cloudflare Tunnel targets.
Hardware backend (port 8080) handles Ollama and optional OpenRouter requests, Blender submission and rendering, R2 file operations, GPU and integration health checks, and MCP tool execution. Middleware calls it with an X-Internal-Token shared secret and an asserted user ID.
The backend is not a Cloudflare ingress target, but Docker Compose currently publishes port 8080 on the host. Its shared-secret check is therefore an application security boundary, not proof of complete network isolation. The backend also makes outbound calls when OpenRouter, R2, or other integrations are enabled.
Inference streams are proxied as Server-Sent Events. Model and health responses use short-lived process-local caches, and the account page uses one aggregate endpoint instead of a fan-out of browser requests.
OpenAI compatibility and routing
The OpenAI-compatible endpoints support streamed and non-streamed chat, model discovery, image-capable content shapes, and structured tool calls. Ollama executes locally through a FIFO queue. OpenRouter is an optional route for selected cloud models, while auto chooses from configured local or cloud options using token thresholds and user preferences.
If a local request fails, it won’t retry through OpenRouter. A cloud model has to be selected, so its availability, cost and privacy terms need to be considered separately.
Usage accounting
The account totals now sum explicit ledger types for local inference, cloud inference, and rendering instead of deriving chart segments from limited recent logs or unrelated balance movements. The ledger is append-only and balances are calculated from its rows.
Local charges are still estimates. They combine configured or detected wattage, the electricity tariff, estimated token counts and throughput, or measured render duration. NVIDIA detection reads the configured power limit as an input assumption; it does not measure each request’s wall-power draw. Optional Tapo monitoring reports whole-device telemetry, not request-level attribution.
Blender rendering and its trust model
The render path validates .blend uploads, removes Blender text blocks, stores files in optional R2, queues a separate worker, records frame progress, packages output, and writes a render-usage ledger entry. Text-block removal reduces one risk but is not a full sandbox, so this remains a trusted-user workflow.
Setup automation, and what I broke
The macOS, Linux and Windows scripts still do the hardware detection and configuration, including model sizes, wattage estimates, secrets, container builds and health checks. Though I left the database migration and troubleshooting commands pointing at the old fastapi Compose service. The services are now middleware, backend and render-worker, so those commands need fixing before the installer works all the way through.
Current status
The accounts, API keys, streaming and tool calls are implemented, along with local and cloud routing, MCP server settings, queued Blender jobs and usage accounting. I’m still building this for a trusted group, and it depends on the host being available.
The code is open source at github.com/Slaymish/GPUShare.