How to Run AI Locally on Your PC: Ollama, LM Studio, and What Your Hardware Can Really Handle

Yes, you can run a real AI model on your own PC today. It is free, it is private, and it does not take a computer science degree. There are exactly three main doors into local AI: a friendly desktop app called LM Studio, a behind-the-scenes engine called Ollama, and a do-it-yourself toolkit called llama.cpp. All three run the same kind of model files (GGUF files, the standard format you download from Hugging Face), so you are not picking sides in a war. You are picking the door that matches how you like to use software.

There is a second question, though, and it matters just as much: what can your hardware actually hold? A model that is too big for your machine will crawl or refuse to load. A model that fits will feel snappy and wonderful. The good news is that one simple rule of thumb - model file size is roughly parameters times bits-per-weight divided by 8 - tells you almost everything you need to know.

By the end of this guide, you will know which door is yours, what your hardware can run, and you will have a model chatting with you in about ten minutes.

In short

• LM Studio is the easiest start: point, click, chat. Ollama is best for tools and scripts: one command, and other software can talk to it. llama.cpp gives the most control: build it, tune everything.
• New in 2026: typing ollama on its own now launches an interactive agent, LM Studio has a sibling app called Bionic that delegates coding and document work to local models, and llama.cpp finally has an official one-line installer at llama.app.
• Rule of thumb: file size is roughly parameters x bits-per-weight / 8. An 8B model at Q4_K_M is about 4.6 GiB; the same model at F16 is about 15 GiB.
• 16 GB of RAM or VRAM comfortably runs 7-8B models at 4-bit quantization. A 70B model needs around 40 GiB or more.
• Local AI is free after the hardware and never sends your text to a server. Ollama says it plainly: "Nothing you run locally ever leaves your machine."
• The tradeoff: the biggest local models still trail frontier cloud models in raw quality. Pick local for privacy, cost, and control - not for maximum IQ.

Three doors into local AI

Here is the part nobody tells you up front: every local AI setup funnels into the same family of runtimes. All three doors consume GGUF files, and LM Studio literally runs llama.cpp under the hood (plus Apple MLX on Apple Silicon Macs). You are not choosing between different universes of AI. You are choosing an interface to the same engine room.

So what actually differs? Three things:

• Ease - how fast you get from download to first chat.
• Power - how easily other software can plug into your local AI.
• Control - how deep you can dig into settings and tuning.

LM Studio scores highest on ease. Ollama scores highest on power, because it is built to serve other tools. llama.cpp scores highest on control, because you can touch every knob that exists. Let's meet each door.

LM Studio: the easy chair

LM Studio is the easiest way to run AI locally: a point-and-click desktop app for Mac, Windows, and Linux where you search for, download, and chat with local AI models. The classic app is alive and actively updated (version 0.4.23 arrived in August 2026), and in July 2026 the team shipped something new beside it: LM Studio Bionic, a separate agent app that can inspect your code, edit documents, build slides and spreadsheets, and transcribe your voice offline with a local model called Voxtral. Bionic is its own download, not a mode inside LM Studio - but the classic app remains the place to simply browse models and chat.

What makes the classic app the easy chair? Everything is built in. There is a chat window. There is a model downloader that pulls any supported model straight from Hugging Face - search by keyword or paste a Hugging Face link. There is chat-with-documents (RAG) that works entirely offline, an OpenAI-compatible REST endpoint for scripts (plus an Anthropic-compatible one, so tools like Claude Code can use your local models), MCP support, a headless mode, and the lms command line tool. The 2026 releases added more reasons to stay: LM Link lets you reach your models from other machines with no waitlist, a companion mobile app called Locally puts your largest models on your iPhone or iPad, and the default context length was raised to 8k tokens. You get a surprising amount of power without ever opening a terminal.

The GUI also surfaces the quantization choices (names like Q3_K_S, Q8_0) with simple guidance: choose a 4-bit option or higher if your machine is capable enough.

The official system requirements: on Mac, Apple Silicon M1-M4 with macOS 14 or newer (16 GB RAM recommended, though 8 GB works with smaller models and modest context). On Windows, x64 machines with AVX2 support - and ARM machines like the Snapdragon X Elite are supported now too - with 16 GB of RAM recommended plus at least 4 GB of dedicated VRAM. On Linux, an AppImage for x64 and ARM64 (Ubuntu 20.04 or newer).

Ease: the highest of the three. Control: GUI-managed, so the lowest of the three. If you want to point and click your way into local AI, this is your door.

Ollama: the engine room

Ollama is the best local LLM when you want to run AI on your computer and have other tools talk to it. It is not an app with windows and buttons. It is a service that lives in your terminal - an engine that other software connects to. Its homepage pitch is privacy first: "Nothing you run locally ever leaves your machine," and only you decide whether to touch the optional cloud extras.

Install it and type ollama, and here is the 2026 surprise: you no longer get a boring menu. Since version 0.32 (July 2026), the bare command drops you into an interactive agent that can chat across turns, read and edit files in your working directory, write code, and search the web when you ask it to. If you just want plain chat, ollama run gemma4 still does exactly that: the model downloads and the chat starts. Type /bye to leave. The official quickstart is three short steps.

Where Ollama shines is making local AI available to other tools. The officially documented integrations now include Claude Code, Codex, OpenCode, Claude Desktop, and even ChatGPT Desktop - run ollama launch to see the current list. It speaks a native API plus OpenAI- and Anthropic-compatible endpoints, and ships Python and JavaScript libraries. If you have ever been curious about the MCP protocol that lets AI tools talk to each other, Ollama fits neatly into that world.

The model library at ollama.com spans everything from 0.5B to 671B parameter classes, with pull counts so you can see what is popular. The current star family is Gemma 4: from a 7.2 GB edge model (e2b) to a 20 GB 31B flagship, all one ollama run away. On a Mac, Ollama's MLX engine makes Gemma 4 generate tokens nearly 90% faster on Apple Silicon, per the official release notes. There are even :cloud tags for models bigger than your hardware - clearly flagged as cloud, and entirely optional.

Best party trick: ollama ps shows you exactly where the model loaded - 100% GPU, 100% CPU, or a split like 48%/52% CPU/GPU. When something feels slow, that one command usually explains why.

Ease: middle. Power: the highest of the three. If you want your editor, your agents, and your scripts to share one local AI brain, this is your door.

llama.cpp: the machine shop

llama.cpp describes itself as "LLM inference in C/C++" - a lean engine built on the ggml library, with no dependencies and an MIT license. It is the layer the other two doors are built on or bundle. LM Studio literally ships it inside. When you learn llama.cpp, you are learning what is under everyone's hood.

Getting started got much easier in 2026: the project now has an official home at llama.app, a one-line installer for macOS and Linux (curl -LsSf https://llama.app/install.sh | sh), winget on Windows, Docker, pre-built binaries - or build it from source. The installer drops a single llama command, a Git-style multiplexer where llama serve is the server and llama cli is the chat interface. One line pulls a model straight from Hugging Face and starts a server:

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

That is an OpenAI-compatible server with a built-in web UI, downloaded and running. For plain terminal chat instead, llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF does the same without the server.

Why pick the hard door? Control. Every quantization class from 1.5-bit to 8-bit. GPU layer splits. Server flags for everything. And the widest hardware net in the business: CUDA for NVIDIA, Metal for Apple Silicon, HIP for AMD, Vulkan, SYCL for Intel, plus CPU paths like AVX2, AVX512, and AMX. It even supports CPU+GPU hybrid inference, so you can partially run models bigger than your total VRAM.

Ease: the lowest of the three. Control: the highest. And once you have picked which model family you want to feed it, you can tune every last detail of how it runs.

What your hardware can actually run

Time for the one piece of math this whole hobby runs on. Local LLM hardware requirements come down to one rule of thumb (an approximation, but a very good one):

File size is roughly parameters x bits-per-weight / 8.

Why does this work? Quantization squishes model weights from 32-bit floats down to smaller, lower-precision numbers - 4-bit, 8-bit, whatever you choose. Fewer bits per weight means a smaller file and faster inference. The tradeoff, per llama.cpp's official docs, is that it "may introduce some accuracy loss" - usually measured as perplexity (ppl) or Kullback-Leibler divergence (kld), and minimizable with a suitable imatrix file.

You can see the whole story in the official llama.cpp table for Llama-3.1-8B:

Read that table again, because it is the hardware decision in disguise: the same model, ten times smaller depending on the format you pick. At Q4_K_M, an 8B model is about 4.6 GiB. At F16, it is about 15 GiB.

Now the corollary: at Q4_K_M (about 4.9 bits per weight), 16 GB of memory holds roughly 26B parameters of weights. So an 8B model at Q4 fits in 16 GB with plenty of headroom left for context. An 8B at F16 does not even fit, never mind headroom.

A few practical notes:

• VRAM vs RAM: VRAM is the fast lane - the model lives on your GPU and flies. System RAM works too, but slower. Check where your model landed with ollama ps and its Processor column: 100% GPU is the dream; a split like 48%/52% CPU/GPU means it did not all fit.
• Context is the hidden cost: the KV cache grows as your conversation gets longer, which eats memory beyond the weights. Ollama now picks your default context window from detected VRAM: roughly 4,096 tokens below 24 GB, 32,768 from 24-48 GB, and 262,144 at 48 GB and up (adjustable via OLLAMA_CONTEXT_LENGTH or num_ctx). Treat that default as a boot-safely baseline, not a promise - Ollama's own docs recommend at least 64K tokens for agent and coding workloads. If you want to go deeper on taming context, context engineering is the skill to learn.
• GPU support reality: per Ollama's official hardware page, NVIDIA GPUs need compute capability 5.0 or higher with driver 550 or newer (and CC 5.0 through 6.2 cards need driver 570 or newer). Apple Silicon works via Metal. AMD and Intel are covered by llama.cpp's backends.

The honest tradeoffs

Local AI is wonderful, but it is not magic. Here is the honest ledger.

Privacy: this is the headline win. Local inference is the default mode - nothing you type is shipped anywhere for processing. Ollama states it flatly: "Nothing you run locally ever leaves your machine." LM Studio says it can "operate entirely offline" - just download your model files first. (Both vendors also offer optional cloud services, like Ollama's :cloud tags and Bionic's cloud models; those are opt-in extras, and their marketing claims - like LM Studio's zero-data-retention statement about its cloud - are the vendors' own words, not something to take on faith.)

Cost: after the hardware and the electricity, inference is free. No per-token billing. No rate limits. No monthly subscription just to chat.

Speed and quality: here quantization is actually your friend. On llama.cpp's official reference machine, an 8B model at Q4_K_M generated around 72 tokens per second versus about 29 at F16 - quantization buys you speed as well as memory. Those numbers are reference-only and depend on your hardware, but the pattern holds: smaller quants run faster.

The ceiling: frontier cloud models are still larger and more finely tuned than what a consumer PC comfortably runs. The biggest local library entries (the 671B class) need server-grade hardware. So pick local for privacy, cost, offline use, and control - not for maximum model quality. If you are used to cloud chats that keep asking you questions to pin down what you want, expect a bit less polish from a small local model, and love it for what it is.

Pick your door

Here is the decision path, distilled:

• You want a GUI and zero terminal time - LM Studio.
• You want other tools (editors, agents, scripts) to talk to your local AI - Ollama.
• You want exact quants, GPU layer control, exotic hardware, custom builds - llama.cpp.

And here is the best part: whichever door you pick, your knowledge transfers. All three doors consume GGUF files, and LM Studio literally bundles llama.cpp. Learn quantization once, use it everywhere.

Quick start-by-hardware guide, with current model families:

Your first model in 10 minutes

The fastest path to a running model and a usable API is Ollama:

1. Download the installer for macOS, Windows, or Linux from ollama.com/download.
2. Run ollama in your terminal and follow the setup prompts - pick a local model (sign in only if you want the cloud ones).
3. Run ollama run gemma4 - the model downloads (about 9.6 GB for the default build, so give it a few minutes) and the chat starts.
4. Chat away. Type /bye when you are done.
5. Run ollama ps and check the Processor column to confirm the model loaded onto your GPU (100% GPU or a favorable split).

Prefer to click? The LM Studio path, in three steps: download from lmstudio.ai, open the Discover tab (Command/Ctrl+2), search a model, pick a 4-bit-or-higher option, and chat. And if what you really want is an agent that edits code and builds documents for you, that is what the separate Bionic app is for.

And for the curious, the llama.cpp one-liner: llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF pulls a small model straight from Hugging Face and starts a local server.

That is it. You are running a real AI model, on your own machine, for free.

Mistakes that ruin first runs

Almost every bad first local-AI experience traces back to one of these five:

1. Wrong quant for your memory. The official 8B table spans 1.87 GiB (IQ1_S) to 14.96 GiB (F16) - a 10x swing for the same model. Do not grab F16 on an 8 GB machine; check what fits first.
2. Expecting cloud quality. Quantization "may introduce some accuracy loss" (official docs), and the smallest quants degrade the most. A 4-bit-or-higher quant is the sane default.
3. Ignoring the context window. On most consumer GPUs Ollama's default window is still just 4,096 tokens, and when your prompt is longer than the window, Ollama does not error - it silently drops the front of the conversation, which is exactly where your instructions live. That is the classic "why did it forget my document?" moment. Raising num_ctx or OLLAMA_CONTEXT_LENGTH without the RAM or VRAM to back it fails in the other direction: layers get shoved onto the CPU and everything crawls.
4. Never checking where the model loaded. A silent 100% CPU load is the usual answer to "why is it slow?" One ollama ps settles it.
5. Requantizing from a quant. llama.cpp's own docs warn that requantizing already-quantized tensors "can severely reduce quality." Always quantize from 16-bit or 32-bit sources.

The door is chosen, the model is running, and the next step is yours: bigger quants, tool integrations, or a whole home lab. Welcome to local AI.

Local AI FAQs

What do GGUF names like Q4_K_M, Q8_0, and IQ2 mean - which should I download?

They are quantization formats: the same model, stored at lower precision so the file is smaller. Q4_K_M is the popular 4-bit sweet spot; Q8_0 is a high-fidelity 8-bit copy; IQ-quants are the smallest and most degraded. LM Studio's official guidance says it plainly: choose a 4-bit option or higher if your machine is capable enough.

Is local AI actually private - does anything leave my machine?

Local inference stays on your device - that is the default mode, per both Ollama ("Nothing you run locally ever leaves your machine") and LM Studio (which can operate entirely offline once models are downloaded). Both vendors do offer optional cloud modes, but they are opt-in, clearly flagged extras.

Why is my local model slower and shorter-memory than ChatGPT?

First check where it loaded: ollama ps shows the CPU/GPU split, and a 100% CPU load is the usual culprit. Second, check the context window - Ollama now defaults to 4K, 32K, or 256K tokens depending on detected VRAM, and a small window silently trims the front of long prompts. Quantization actually speeds things up (the official table shows Q4_K_M far faster than F16), but hardware always decides your ceiling.

Can I run a 70B model on my gaming PC?

Probably not comfortably. At Q4_K_M (about 4.9 bits per weight), 70B weighs in around 43 GiB - beyond a typical 8-12 GB gaming GPU. You would need CPU+GPU hybrid inference and lots of system RAM, or you can accept a smaller model: 8B to 14B at Q4 is the sweet spot for that class of card.