If you’re handling sensitive documents—client proposals, internal strategies, financial records—you probably don’t want them floating around OpenAI’s servers. Yet traditional ChatGPT remains the easiest way to ask questions about your documents. AnythingLLM and Ollama change that equation. Together, they let you chat with your documents using a large language model that runs entirely on your own machine, with zero cost beyond your hardware. This post walks you through setting both up.
What is AnythingLLM?
AnythingLLM is an open-source application that acts as your personal document assistant. Feed it PDFs, Word docs, text files, websites—anything with information you want to query—and it indexes them into a searchable format. Then you ask questions in plain English, and AnythingLLM retrieves relevant passages from your documents and sends them to a language model for answer generation.
Think of it as giving ChatGPT perfect memory of your documents. The difference is: AnythingLLM can use a local model like Ollama instead of sending data to the cloud, keeping everything private on your network.
AnythingLLM runs as a desktop application (Windows, Mac, Linux) or as a Docker container if you prefer server-based deployment. The interface is clean—drag documents into a sidebar, create workspaces to organise different projects, and start asking questions. No coding required.
Why Ollama + Privacy + No Monthly Bills
Ollama is a lightweight framework for running large language models locally. Instead of paying £15/month for ChatGPT Plus or subscribing to Claude, you download open-source models—Llama 2, Mistral, Neural Chat—and run them on your own hardware. The first time you run a model, it downloads (a few gigabytes typically); after that, it’s stored locally and runs instantly.
For organisations handling regulated data—healthcare, finance, legal, government—local inference is essential. Your documents never leave your building. No API keys to rotate, no SaaS vendor to depend on, no surprise rate limits. You own the whole pipeline.
The trade-off is modest: you need a decent machine. An Intel i7 or AMD Ryzen 7 with 16GB RAM handles models fine; 32GB is ideal for larger models. GPU acceleration (NVIDIA CUDA) makes it faster, but not mandatory. Even a MacBook Pro runs Ollama smoothly.
Installation and Setup
Step 1: Install Ollama. Head to ollama.ai, download the installer for your OS, and run it. On Linux and Mac, it’s straightforward. Windows users should grab the installer as well—recent versions work reliably. Once installed, Ollama runs as a background service.
Step 2: Pull a language model. Open a terminal and run:
ollama pull mistral— excellent general-purpose 7B model, lean and responsiveollama pull llama2— popular, good at reasoning, slightly largerollama pull neural-chat— lightweight, good for older machines
The command downloads the model and saves it locally. This takes 5–15 minutes depending on your internet speed and model size.
Step 3: Install AnythingLLM. Download from anythingllm.com. Windows and Mac installers are available; Linux users can use Docker or build from source. Once installed, launch the app.
Step 4: Connect AnythingLLM to Ollama. Open AnythingLLM settings. Look for “LLM Configuration” or “Model Settings.” Select Ollama as your provider. It auto-detects your local Ollama instance running on localhost:11434. Choose the model you pulled (e.g., Mistral). Save.
Step 5: Create a workspace and add documents. Start a new workspace—name it something meaningful, like “Q1 Strategy” or “Client Contracts.” Drag and drop your PDFs and documents into the sidebar. AnythingLLM automatically splits them into chunks, embeds them using a local embedding model, and stores the vectors in its built-in database. No external vector DB needed.
Step 6: Start querying. Type a question in the chat box. AnythingLLM retrieves relevant document passages, feeds them to Ollama along with your question, and returns an answer with source citations. First query may take 10–30 seconds as the model processes; subsequent queries are faster.
Practical Tips and Next Steps
A few things that help in practice: keep your embedding model lightweight—AnythingLLM defaults to a small, fast model that works well locally. If response speed matters, choose a leaner LLM like Mistral over larger ones. For document-heavy workflows, consider allocating 32GB RAM and a GPU; the speed difference is noticeable.
Use separate workspaces for different projects. This keeps context clean and lets you manage document access if you run a shared AnythingLLM instance for a team.
Finally, remember these are general-purpose models, not specialists. They’re excellent for summarising documents, answering straightforward questions, and extracting key points. They can hallucinate—occasionally inventing facts—so always verify critical answers against your source documents. AnythingLLM shows you which passages informed each answer, making verification straightforward.
If you’re running a UK-based firm with sensitive data, this setup eliminates vendor lock-in and keeps compliance simpler. You control the infrastructure, the data never leaves your network, and you own nothing to no monthly bill. Start small: install Ollama and AnythingLLM on a test machine, upload a sample document, and see how it feels. Most users find the experience immediately useful.
