How to Run Local LLMs with Ollama: The Complete Tutorial

Running large language models (LLMs) locally has shifted from an experimental hobby to a critical enterprise capability. Whether you want absolute data privacy, zero recurring subscription costs, or the ability to code without an internet connection, deploying models on your own hardware is the definitive solution. Enter Ollama: an open-source framework designed to get large language models up and running locally with a single command.

This comprehensive guide covers everything you need to know about setting up, configuring, and leveraging Ollama for fully private artificial intelligence.

What is Ollama and Why Use It?

Ollama acts as a lightweight, extensible wrapper around the core inference engines (such as llama.cpp), packaging model weights, configurations, and datasets into a single bundle. Traditionally, setting up an LLM required compiling complex C++ libraries, managing Python virtual environments, and manually converting tensor weights. Ollama eliminates this friction entirely.

Key advantages of using Ollama include:

• Complete Privacy: Your prompts and data never leave your local machine, making it ideal for processing sensitive source code, medical data, or proprietary business documents.
• Zero Latency and Offline Access: Operate without network dependencies or API rate limits.
• Hardware Optimization: Automatically offloads computations to Apple Silicon GPUs (M1/M2/M3/M4), NVIDIA GPUs via CUDA, or falls back efficiently to standard multi-core CPUs.

Step 1: Installing Ollama on Your System

Ollama supports macOS, Linux, and Windows (via WSL2 or native preview installers). To get started, navigate to the official download source or use your operating system’s package manager.

For macOS and Windows, download the official installer directly from the Ollama website and run the setup wizard. For Linux users, installation is streamlined via a shell script:

curl -fsSL https://ollama.com/install.sh | sh

Once the installation completes, verify that the service is running correctly by checking the version in your terminal:

ollama --version

Step 2: Downloading and Running Your First Model

Ollama hosts a robust library of open-weights models, ranging from general-purpose assistants like Meta’s Llama 3 to specialized coding models like DeepSeek-Coder and lightweight models like Microsoft’s Phi-3.

To pull and immediately run a model, execute the run command. If the model is not present locally, Ollama automatically downloads it:

ollama run llama3

After the download completes, an interactive terminal prompt appears. You can begin conversing with the model directly:

>>> Write a bash script to back up a PostgreSQL database daily.

To exit the interactive session at any time, type /exit or press Ctrl+D.

Step 3: Managing Local Models and Storage

Because LLM model files are large (ranging from 2GB to over 20GB depending on quantization and parameter counts), efficient model management is essential.

To view all models currently stored on your local machine, run:

ollama list

If you need to free up disk space by removing a model you no longer use, use the rm command:

ollama rm llama3

Step 4: Integrating Ollama with Applications via API

One of Ollama’s most powerful features is its built-in REST API. By default, Ollama exposes an endpoint at http://localhost:11434 that is fully compatible with the OpenAI API specification. This means you can effortlessly swap out cloud-based APIs in your custom applications for your local Ollama instance.

Here is a quick example of how to query your local model using curl:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3",
  "prompt": "Explain quantum computing in one sentence.",
  "stream": false
}'

This API compatibility allows you to connect Ollama to popular desktop clients like AnythingLLM, Open WebUI, or standard developer environments like VS Code extensions for offline pair programming.

Frequently Asked Questions

How much RAM or VRAM do I need to run local LLMs?

Hardware requirements depend entirely on the size and quantization level of the model. For smaller 3B to 8B parameter models (such as Llama 3 8B quantized to 4-bit), you need at least 8GB to 12GB of system RAM or dedicated VRAM. For larger 70B models, enterprise-grade hardware with 48GB+ of VRAM or multiple unified memory Macs is typically required.

Can I run Ollama without a dedicated GPU?

Yes. Ollama runs seamlessly on standard multi-core CPUs. While inference speed (tokens per second) will be slower compared to running on a dedicated NVIDIA or Apple Silicon GPU, it is entirely functional for testing, writing scripts, and light usage.

How do I make Ollama accessible across my local network?

By default, Ollama binds exclusively to 127.0.0.1 for security. To expose it to your local network, you must configure the OLLAMA_HOST environment variable to listen on all interfaces (0.0.0.0:11434). On Linux systems managed via systemd, edit the service configuration using sudo systemctl edit ollama.service and add the environment variable under the [Service] block.

Conclusion

Running local LLMs with Ollama transforms your workstation into a powerful, private AI sandbox. By bypassing cloud APIs, you retain absolute ownership of your data while unlocking limitless experimentation. Ready to take your local AI workflow further? Explore the Ollama model library today, pull your first model, and build your next secure application offline.

Disclosure: BriteHacks is reader-supported. When you buy through links on our site or deploy infrastructure, we may earn an affiliate commission at no extra cost to you. As an Amazon Associate, we earn from qualifying purchases.