Tools
Sign InSign Up
Notifications 0
Every week, there's a new AI model.
Every month, your API bill looks a little more "enterprise."
If all you want is to chat with an LLM, write code, summarize documents, or build an AI side project, you don't need to send every prompt halfway across the planet.
You can run surprisingly capable LLMs locally with Ollama. It's free, takes minutes to set up, and won't judge you for asking the same question 17 times.
Let's get started.
What is Ollama?
Ollama is a lightweight runtime that makes running open-source LLMs as simple as installing Docker.
Think of it as:
Docker, but for AI models.
Instead of pulling containers, you pull models.
ollama pull qwen3
Done.
No CUDA configuration. No Python virtual environments with 147 dependencies. No mysterious ImportError from a package you never installed.
Step 1: Install Ollama
macOS
brew install ollama
Or download it directly from the official website.
Linux
curl -fsSL https://ollama.com/install.sh | sh
Windows
Download the installer and follow the wizard.
That's it.
Step 2: Start Ollama
ollama serve
Ollama starts a local server on
http://localhost:11434
Every application on your machine can now talk to your local AI.
Step 3: Download a Model
This is where the fun begins.
Some popular models:
Qwen 3
ollama pull qwen3
Excellent all-round model.
Llama 3.3
ollama pull llama3.3
Great reasoning and general conversations.
Gemma
ollama pull gemma3
Smaller and faster.
DeepSeek R1
ollama pull deepseek-r1
Strong reasoning capabilities.
Want to see what's installed?
ollama list
Step 4: Start Chatting
Run:
ollama run qwen3
Example:
>>> Explain Kubernetes like I'm five. Imagine you have many toy boxes...
Congratulations.
You now own an AI that works even when your Wi-Fi decides it's on vacation.
Step 5: Use the REST API
Every model is exposed through HTTP.
curl http://localhost:11434/api/generate \
-d '{
"model":"qwen3",
"prompt":"Explain Docker",
"stream":false
}'
Or in JavaScript:
const response = await fetch("http://localhost:11434/api/generate", {
method: "POST",
headers: {
"Content-Type": "application/json"
},
body: JSON.stringify({
model: "qwen3",
prompt: "Write a haiku about Linux",
stream: false
})
});
const data = await response.json();
console.log(data.response);
That's literally enough to build your own AI application.
Streaming Responses
For ChatGPT-like typing effects:
{
"stream": true
}
Ollama streams tokens as they're generated.
No extra libraries required.
Useful Commands
Download a model
ollama pull qwen3
Run a model
ollama run qwen3
List installed models
ollama list
Delete a model
ollama rm qwen3
Show model information
ollama show qwen3
Which model should you use?
Which Model Should You Use?
PurposeModelGeneral ChatQwen 3CodingQwen 3, DeepSeek R1ReasoningDeepSeek R1Small LaptopGemma 3Larger GPULlama 3.3
If you're unsure, start with Qwen 3. It's a solid default.
How Much RAM Do You Need?
Rule of thumb:
- 2B model → ~2 GB RAM
- 7B model → ~8 GB RAM
- 14B model → ~16 GB RAM
- 32B+ → Bring snacks. It'll take a while.
GPU acceleration helps, but modern CPUs handle smaller models surprisingly well.
Why Run Models Locally?
Running locally gives you:
- No API costs
- Better privacy
- Lower latency
- Offline usage
- Complete control over your models
If you're building AI tools, coding assistants, document search, or internal company applications, local inference is often all you need.
Final Thoughts
Five years ago, running an LLM locally required reading research papers, compiling CUDA kernels, and sacrificing at least one weekend.
Today it's:
brew install ollama ollama pull qwen3 ollama run qwen3
That's it.
The hardest part is deciding which model to download first.
(And explaining to your SSD why it suddenly lost another 8 GB.)
If you've never tried running AI locally, Ollama is the easiest place to start. You'll spend less time configuring software and more time building things that actually matter.