Screenmind

The problem I wanted to solve

I wanted something like Rewind.ai but fully open source and completely local. No cloud uploads, no subscriptions, no trusting a company with screenshots of my entire digital life. Just my screen, my GPU, my data.

So I built ScreenMind.

What it does

ScreenMind runs in the background, captures your screen when content actually changes (not on a dumb timer), and feeds each frame through a local Gemma 4 model running on your GPU via llama.cpp. It figures out what app you're using, what you're doing, categorizes it, and writes a rich description — all stored in a local SQLite database.

The result is a fully searchable timeline of your entire workday. You can full-text search it, semantic search it with embeddings, or just ask it questions in natural language like "what was I reading about around 3pm?" and it pulls up the exact screenshots with context.

How it works under the hood

The architecture is pretty straightforward:

• Capture loop — mss grabs screenshots, imagehash (pHash) deduplicates them, and a11y APIs grab text from the active window at capture time (not analysis time — that was a fun bug to fix)

• Analysis pipeline — OCR (EasyOCR) extracts text, Gemma 4 analyzes the screenshot + text context, and a layout analyzer organizes OCR bounding boxes into spatial sections. There's a per-app pHash cache so identical screens don't burn another 12-76 seconds of GPU time

• Three-signal app reconciliation — Gemma sometimes hallucinates app names, window titles can be misleading, and process names are often cryptic. So I cross-reference all three to get the actual app right

• Storage — SQLite with WAL mode, FTS5 for full-text search, 384-dim MiniLM embeddings packed as BLOBs for semantic search

• API — FastAPI serving a web dashboard on localhost:7777. SSE streaming for chat responses

Everything talks to llama-server over HTTP. Single inference slot — analysis, chat, and audio transcription take turns. Chat pre-empts running analysis (cancels the HTTP request, re-queues at front) so it feels responsive even on a single GPU.

Things I'm proud of

• Privacy is real, not a marketing checkbox. Screenshots are AES-encrypted at rest with keys in the OS keyring. Credit cards, API keys, SSNs, and passwords are regex-redacted before they ever touch the LLM. Blocked apps never get captured.

• Meeting transcription actually works. It auto-detects meeting apps, probes for voice activity (so a notification sound doesn't trigger a fake meeting), captures dual-channel audio (system + mic), and transcribes through Gemma 4's native audio encoder. When the meeting ends, it generates a structured summary.

• The agent system. You can drop a markdown file in ~/.screenmind/agents/ with a prompt and a schedule, and it'll run on autopilot — analyzing your activity data and outputting reports to local files, your Obsidian vault, or a Slack webhook. Python plugins get a full SDK with persistent state and LLM access.

• Developer context. If you're coding, it detects your git repo, branch, changed files, and insertions/deletions — then bakes that into the activity record. Auto-generated standup notes that actually reflect what you worked on.

• Cross-platform. Windows, macOS, Linux (X11 + Wayland). Each platform has its own adapter for window detection, process names, and accessibility text extraction.

What you need

• A GPU with 4GB+ VRAM (for Gemma 4 E2B). 6-10GB gets you the bigger models.

• Python 3.10+

• That's it. pip install screenmind && screenmind

It downloads the model and llama-server binary on first run, starts the server, and opens the dashboard.

Tech stack

Python · FastAPI · SQLite · llama.cpp · Gemma 4 (GGUF) · EasyOCR · sentence-transformers (MiniLM) · mss · imagehash · Pydantic · sounddevice

Contributing

It's fully open source. The codebase has 29 test files, type hints everywhere, and every module has docstrings explaining the "why" not just the "what." PRs welcome.