How to Run Local AI Models on Your PC: A Beginner's Guide to Free, Private LLM Tools

I ran local AI models on my PC without knowing what the acronyms mean, and they save me hundreds on cloud subscriptions

Abhinav pivoted from a career in banking to pursue his first love in writing. Even while working full-time, he continued contributing as an editor-at-large, a role he has held for more than 7 years. A lifelong tech enthusiast who has built three gaming and productivity powerhouse PCs since 2018, his passion for technology keeps him closely following the semiconductor industry, from NVIDIA and AMD to ARM. His MSc dissertation explored how artificial intelligence will reshape the future of work, reflecting his curiosity about the wider social impact of emerging technologies. Local AI inference is hugely popular at the moment, and open models have found a home in the workflow of most developers, analysts, creative workers, and home lab hobbyists alike. After all, who doesn't want an LLM that's limitless, free to use, can be run privately on your own hardware, and potentially save you hundreds of dollars on cloud subscription fees? Unfortunately, though, most of the discourse surrounding local AI models on community forums and subreddits tends to push more casual users away than to draw them in, while beginner guides simply assume a level of knowledge most users don't have regarding a technology that's still rather nascent. I too, found the terms intimidating before deciding to dive head-first into local AI inference, and most of what I know comes from my usage, experience, and tinkering alone. If you're in the same boat as I was, here's a way to get started. The first step is knowing your hardware The only acronym you need to know to get started is what "VRAM" stands for If you were to go through a local AI model hub like Hugging Face, you'd notice strings like Q4_K_M, IQ4_XS, GGUF or MTP quite often. More often than not, they become part of the reason why most beginners give up before picking a model. The great news about this is the fact that you can completely ignore most of that at step one. The first thing you need to familiarize yourself is your own hardware, wherever you're planning on running the models on. The primary component for all local AI inference is your GPU, and the part of it that matters the most is your VRAM. The VRAM is the memory that sits on your GPU, and it has to "hold" the model along with the working memory that's built as it runs. If the size of your model exceeds your VRAM, your GPU will offload the overflow to your system RAM, at which point, the speed of inference (measured by tokens per second) drops sharply. In that way, VRAM sets the ceiling on what you can run well. For example, my RTX 4070 Ti Super with 16GB of GDDR6X memory can comfortably run 13B-class models (models with 13 billion parameters). On the other hand, my Lenovo Legion 5 Pro on a mobile RTX 3070 with 8GB of VRAM is much better suited to the 4B-7B tier, which leaves room for the model to work. There are other variables and adjustments that can make running larger models run comfortably well on a smaller VRAM budget, but at this step, as a beginner, it's best to stick with what your card permits before you get the hang of it. The next step is choosing a "harness" Which is how you'll interact with your models Before you get anxious, a "harness" is simply the software you choose to run and interact with a local model on. It's the software you install, open, use to load a model, and then use it as you would. Two of the easiest options are Ollama and LM Studio. I'm personally fond of Ollama, because it provides just the right mix of command-line approach to interaction and also allows similar controls in its GUI, while taking away most of the burden that is associated with setting things up. LM Studio is easier still, with a more navigable GUI and a built-in model browser that lets you find, download, load, and chat with models. There is, however, something that even beginners should take note of. Ollama and LM Studio both make a lot of decisions on your behalf regarding inference, and while they are sensible and conservative for your hardware tier most of the time, they aren't universally optimal for every model or PC. Newer optimizations can also take time to arrive, meaning a convenient front-end has a convenience cost of leaving some performance on the table. But that's not a reason to avoid these tools together. On the contrary, I would recommend them to beginners who are just starting out, as I got started with Ollama myself. Once you start running into their limits, you can move on to llama.cpp or vLLM to get some more control over how your models run. Finally, you pick a model that complements your workflow Would you like a coder? A general assistant? Or something to handle your notes? Close When you know what models your hardware can handle, it's time to pick one. When it comes to local models, bigger isn't always better, and a model's capability often matters more. For example, a smaller model that's trained for coding can outperform a larger general-purpose model when you're writing Python scripts, and only be a worse choice for everyday conversations. As such, it's prudent to start by understanding what your workflow really needs. For coding, I am particularly fond of the Qwen family of models developed by Alibaba Cloud, and I can comfortably run Qwen3.5-9B at 87.76 tokens per second, which is fast enough to write Python scripts within seconds on my 4070 Ti Super, but this, of course, depends on the complexity of the prompt. For general purpose assistance, I like to look at Gemma 4 variants, which come in various sizes and can be run on a wide variety of hardware. Many Gemma 4 models also come with native vision and agentic capabilities, but you'll have to check if your specific model supports them via the model hub. Besides that, you'd also want to have a look and check the model's page to see if they support "reasoning" (sometimes referred to as "thinking"), and that means the model works through a problem before answering, which helps multi-step tasks at the cost of speed and tokens. Some models also support tool calling, which means you can rely on them for web search or code execution if your harness supports them. At this point though, it would be massively helpful to know a few basic facts about model quantization. Quantization is how the same model gets shrunk to fit different VRAM sizes. Whereas a model's full-precision weights can be quite large, quantized versions compress them at a small quality cost, and the tag on the downloads (Q4_K_M, Q8_0, IQ4_XS) tell you the kind of compression that was used. Q4_K_M means 4-bit quantization, and it roughly halves the size when compared with Q8 with minimal impact on quality for many models. Context length also matters when you're dealing with smaller VRAM sizes, and it also affects how much VRAM the model needs beyond its weights alone. The harness defaults, such as Ollama's, are usually a good starting point for most beginners. Hosting your own local models can save you money If you rely extensively on AI tools for your workflow, and you've got the hardware to host your local models, there's practically nothing to lose from running a model. For my coding workflow, I would frequently go over my Claude Code usage limits, but introducing Qwen 3-Coder 30B at Q4_K_M quantization and 16K context length in my workflow (averaging 56 tokens per second) allowed me to reduce my dependence on the cloud AI provider and saved me a lot of $5 top-ups every time I went over the limit so that I'd not have to wait for five hours to continue working.This is exactly the kind of advantage you unlock with running LLMs locally, and very little of the process requires you to obsess over the technical bits until you're ready to delve into it yourself.

4 more excellent local LLM projects you can run for free on a slow laptop

Nick Lewis is an editor at How-To Geek. He has been using computers for 20 years --- tinkering with everything from the UI to the Windows registry to device firmware. Before How-To Geek, he used Python and C++ as a freelance programmer. In college, Nick made extensive use of Fortran while pursuing a physics degree. Nick's love of tinkering with computers extends beyond work. He has been running video game servers from home for more than 10 years using Windows, Ubuntu, or Raspberry Pi OS. He also uses Proxmox to self-host a variety of services, including a Jellyfin Media Server, an Airsonic music server, a handful of game servers, NextCloud, and two Windows virtual machines. He enjoys DIY projects, especially if they involve technology. He regularly repairs and repurposes old computers and hardware for whatever new project is at hand. He has designed crossovers for homemade speakers all the way from the basic design to the PCB. Nick enjoys the outdoors. When he isn't working on a computer or DIY project, he is most likely to be found camping, backpacking, or canoeing. An old laptop isn't the kind of device you think of when someone says "local AI," but as long as you pick the right AI, they can be pretty useful. Whatever device you use, it needs 8GB of RAM at a minimum, though 16GB is better. Luckily, even laptops from five years ago frequently had that much memory available. Cherry Studio is a local-first AI studio Don't upload sensitive documents to the cloud Close If you've ever wanted to use an AI to analyze a bank statement, medical records, or a tax document, but you don't want to upload it to the internet, you should try Cherry Studio instead. Cherry Studio is a free, open-source desktop app that acts as a front end for local AI models running with Ollama. It supports built-in retrieval-augmented generation (RAG), so you can provide it with information to work with. You can load in PDFs, documents, spreadsheets, and then talk to an AI running on your local PC about the data. Some small AI models (4-12 billion parameters) will run reasonably well on a CPU, and if you have a GPU, you can run even faster models. It also includes a whole range of built-in tools and MCP, which allows you to connect it to most anything you want. Remember that RAG is only fully private if you use a local model. If you use a cloud embedding service, your documents still leave your PC. Jan is a polished chatbot It is similar to ChatGPT in many ways Close If you're not a fan of the command line, Jan is a pretty good option. It is a fully open-source desktop app that combines a model downloader, an inference engine, and a chat interface into one package. That means you don't need to work with the terminal to pull models with Ollama or anything like that. It is designed for convenience, and you don't get the granular customization with Jan that you do using llama.cpp directly, but that is a small price to pay for the simplicity. You also need to try and keep your model sizes and expectations in check. 8GB of RAM (or even 16GB) can't match the reasoning capabilities of a frontier, cloud-based model. Still, it can be useful for extracting data or discussing the contents of a folder. Transcribe audio for free with whisper.cpp Fast speech-to-text on a CPU If you ever need to transcribe audio to text, Whisper.cpp is a great option. It is a CPU-optimized variant of OpenAI's whisper models. On even a moderately capable CPU, you can get real-time or close to real-time audio transcription. That makes it suitable for simple voice recognition tasks for a voice assistant. If you have a low-power CPU, Whisper is still great for batch processing audio transcripts that you don't need done in real time. Your only costs are the laptop itself (which is often something you already own) and the electricity, The small or base models are accurate enough for most applications and will happily run in 8GB of RAM. Depending on your workflow, you may also need FFmpeg to handle file conversion first. However, with a little bit of setup, you get automated AI transcription that is completely private. Add a coding autocomplete to VS Code with Continue A small local coder model can do small, simple tasks well If you're a programmer working with an NDA, or just someone that values their privacy, you could try out Continue. Continue is an extension for VS Code that allows you to connect your IDE to a local AI server to provide inline autocomplete. You can also chat with the model about your code in a menu on the side. Typically, when you think of vibe coding you think of large, expensive frontier models, but they're not really necessary for simple things like generating regex and simple lines of code. You can find models in the 1.5B to 4B parameter range that are actually fairly capable. The only significant trade-off is latency. Completion on an older laptop CPU is not instantaneous like it is with a cloud service, and the larger your model, the worse that gets. I'd recommend using the smallest possible model for tab-autocomplete and only using a larger model for the chat panel to get the best performance-to-utility ratio. Sometimes a bit slow is completely fine Local AI on an old laptop when you can keep your projects focused, or you don't need it to run in real time. So long as you're careful about your application, it is an easy win for privacy and cost. If you aren't sure where to start, try Jan. It is very easy to get running and is useful "out of the box." Whisper.cpp is another favorite. It runs on almost anything, including a Raspberry Pi 4 or 5. That makes it suitable for a DIY home voice assistant that is completely private.