In this guide · 8 sections
Short answer
A local model is worth it for narrow, private tasks such as summarising or redacting sensitive documents offline, on a PC with 16 GB or more of GPU or unified memory. For general work, open models that fit on one machine score well below frontier cloud models.
What running a model locally means
Ollama is a free app for Windows, macOS and Linux that downloads open-weight models and runs them on your own hardware. You chat with the model in a terminal or a simple app, or connect other tools through its local API.
Ollama's FAQ says that when you run models locally, Ollama does not see your prompts or data, and that by default it binds to 127.0.0.1 on port 11434, which means only that computer can reach it. Two points matter for a business:
- Ollama also offers cloud-hosted models. Some models in its library carry a cloud tag. Ollama says it processes but does not store or train on that content. If your reason for going local is that data must not leave the machine, follow the FAQ's steps to disable cloud features.
- Do not expose it to the internet. Changing OLLAMA_HOST to share the model on your network is possible, but an unprotected AI server on a public IP is an open door. Keep it on the local machine or behind authentication.
Hardware you need, by model size
The rule of thumb: the model's download size must fit in your GPU memory (or a Mac's unified memory) with a few GB to spare. If it only fits in normal RAM, it will run, but slowly. Sizes below are Ollama's default downloads as of October 2026.
| Model | Download size | Typical hardware | Good for |
|---|---|---|---|
| Qwen3.5 4B, Gemma 4 E4B | 3.3 GB, 6.6 GB | Recent laptop, 16 GB RAM | Short summaries, simple drafting, tagging |
| Qwen3.5 9B, Gemma 4 12B | 6.6 GB, 7.7 GB | GPU with 12 GB, or Mac with 16 GB | Document summaries, email drafts, extraction |
| gpt-oss 20B | 14 GB | 16 GB of GPU or unified memory | Reasoning-style tasks offline |
| Gemma 4 31B, Qwen3.8 27B | 19 GB, 18 GB | GPU with 24 GB, or Mac with 32 GB | The strongest single-machine options |
| gpt-oss 120B | 65 GB | Workstation GPU with 80 GB | Rarely practical for an SME |
Ollama's page for gpt-oss notes the 20B model can run on systems with as little as 16 GB of memory, and the 120B model fits on a single 80 GB GPU. Expect slower responses than cloud services, especially for long documents.
How good are local models? Published benchmarks
We have not run our own tests. The figures below are from the Artificial Analysis leaderboard, checked on 18 October 2026, which combines many evaluations into one Intelligence Index. Higher is better.
| Model | Type | Intelligence Index |
|---|---|---|
| Claude Opus 5.5 (max) | Cloud, top of leaderboard | 58 |
| GPT-6.1 Sol (max) | Cloud | 52 |
| Qwen3.8 27B (xhigh) | Open, runs on one 24 GB GPU | 34 |
| Claude Haiku 5.5 (medium) | Cloud, small and cheap | 34 |
| Gemma 4 26B A4B | Open, runs locally | 17 |
| Gemma 4 31B | Open, runs locally | 15 |
| Qwen3.5 9B | Open, runs on a laptop | 11 |
| gpt-oss-20b (high) | Open, runs locally | 9 |
The pattern is clear: the best open model that fits on one machine is roughly level with a small, low-cost cloud model, and far behind the frontier. For a quick primer on what these indices measure, see our LLM benchmarks guide.
Licences: can you use open models commercially?
Open weights do not always mean free for any use, so check the licence on the model card. As of October 2026, the Hugging Face model cards for Gemma 4, Qwen3.5, Qwen3.8 and OpenAI's gpt-oss list the Apache 2.0 licence, which permits commercial use. Meta's Llama models use Meta's own community licence, which adds conditions, so read it before relying on one.
Where a model was made matters less when it runs entirely on your own machine, because no data goes back to the developer. It still matters for bias, accuracy on local topics and what the model refuses to discuss. Our guide to DeepSeek, Qwen and other open models covers those questions, and SEA-LION is worth a look for Southeast Asian languages.
Realistic uses for a small business
A local model is a specialist tool, not a replacement for a business AI subscription. It fits best where privacy or offline access matters more than top quality:
Good fit for a local model
- Summarising or redacting sensitive documents that must not leave the office
- Extracting fields from batches of internal files
- Working offline, such as on site visits
- Experimenting with AI without sending client data anywhere
Better in the cloud
- Complex drafting, analysis or coding where quality matters
- Staff-wide everyday use with admin controls
- Long documents where speed matters
- Anything needing current web information
If cost is the reason, compare first: low-cost cloud models are often cheaper than buying a 24 GB GPU once you count staff time. Our LLM API pricing comparison shows per-token prices.
How to try it in 30 minutes
You can test a local model on an existing computer before buying anything.
Install Ollama
Download it from ollama.com for your operating system and install it like any other app.
Pick a model that fits your machine
On a 16 GB laptop, start with a 4B or 9B model. On a 24 GB GPU or 32 GB Mac, try a 27B-class model.
Turn off cloud features if privacy is the goal
Follow Ollama's FAQ to set local-only mode, so nothing can route to cloud models by accident.
Run five real tasks
Use non-sensitive copies of real work: summarise a contract, extract invoice fields, draft a reply. Note the time taken.
Compare with your cloud tool
Run the same five tasks in your business AI plan and compare quality and speed side by side before deciding.
Do it yourself or get help?
Trying Ollama on one computer is a simple do-it-yourself job. It becomes a project when you want several staff to use a local model, connect it to your documents, or run it on a shared server, because then you need access control, logging and updates. If you want a model to answer from your own files, our fine-tuning vs RAG guide explains the right approach.
We scope private, self-hosted AI set-ups, including a local model with document search and staff logins, as a Custom Tool or Dashboard project (from S$1,990), with a written quote before any hardware is bought.
Frequently asked questions
Run locally, Ollama says it does not see your prompts or data, and by default it only accepts connections from the same computer. Two cautions: Ollama also offers cloud-hosted models, so disable cloud features if data must stay local, and never expose the Ollama server to the internet without authentication.
Small models of around 4 billion parameters run on a recent laptop with 16 GB of RAM. The stronger 27B to 31B models are 18 to 19 GB downloads and need a graphics card with about 24 GB of memory, or a Mac with 32 GB of unified memory, to run at a usable speed.
Not the ones that fit on a single PC. On the Artificial Analysis Intelligence Index in October 2026, the best single-GPU open model, Qwen3.8 27B, scored 34, similar to small low-cost cloud models, while the top cloud model scored 58. Smaller laptop-sized models scored around 9 to 15.
Many allow it. As of October 2026, Gemma 4, Qwen3.5, Qwen3.8 and OpenAI's gpt-oss models list the Apache 2.0 licence on their Hugging Face model cards, which permits commercial use. Some families, such as Meta's Llama, use their own licences with extra conditions, so always check the model card.
Often not for a small team. A capable local set-up may need a graphics card or Mac with 24 to 32 GB of memory, and staff time to maintain it, while business AI plans and low-cost API models are inexpensive per user. Local models make most sense when privacy or offline use is the main requirement.