Developers ditch ChatGPT for local AI coding agents, saving $20+ monthly with powerful local LLM

How to roll your own local AI coding agents

Take those token limits and shove them by vibe coding with a local LLM With model devs pushing more aggressive rate limits, raising prices, or even abandoning subscriptions for usage-based pricing, that vibe-coded hobby project is about to get a whole lot more expensive. Fortunately, you're not without cost-saving options. Over the past few weeks, we've seen Anthropic toy with dropping Claude Code from its most affordable plans while Microsoft has skipped testing the waters and moved GitHub Copilot to a purely usage-based model. The whole debacle got us thinking. Do we even need Anthropic or OpenAI's top models, or can we get away with a smaller local model? Sure, it might be slower, less capable, and a little more frustrating to work with, but you can't beat the price of free... Well, assuming you've already got the hardware that is. It just so happens that Alibaba recently dropped Qwen3.6-27B, which the cloud and e-commerce giant boasts packs "flagship coding power" into a package small enough to run on a 32 GB M-series Mac or 24 GB GPU. This isn't the first time we've looked at local code assistants. Previously we explored using Continue's VS Code extension for tasks such as code completion and generation. At the time, the models and software stack were quite immature, making them useful tools, but not necessarily good enough to compete with larger frontier models. Since then, model architectures and agent harnesses have improved dramatically. "Reasoning" capabilities allow small models to make up for their size by "thinking" for longer, mixture-of-experts models mean you don't need terabytes a second of memory bandwidth for an interactive experience, and vastly improved function and tool calling capabilities mean that these models can actually interact with code bases, shell environments, and the web. In this hands on, we'll be looking at how to deploy and configure local models like Qwen3.6-27B, for coding on your computer, and explore some of the agent frameworks you can use with them. What you'll need: Note: Older M-series Macs may struggle with the large context lengths required for agentic coding. You may have better luck with an inference engine like oMLX, which can take better advantage of Apple's hardware accelerators, but your mileage may vary. Running LLMs locally is a dead simple process these days. Install your favorite inference engine. Download the model, and connect your app via the API. However, for code assistants in particular, there are a couple of parameters we need to dial in, otherwise the model is apt to churn out garbage and broken code. Some models require specific hyper-parameters to function properly in different applications, and Qwen3.6-27B is no exception. When using Qwen3.6-27B for vibe coding, Alibaba recommends setting the following parameters: We also need to set the model's context window as large as we can fit in memory. If you're not familiar, a model's context window defines how many tokens the model can keep track of for any given request. When working with large code bases containing thousands of lines of code, this adds up quickly. What's more, the system prompts used by many agent frameworks can be quite large, so we want to set our context window as high as possible. Qwen3.6-27B supports a 262,144 token context window, but unless you have a high-end Mac or a workstation GPU, you probably don't have enough memory to take advantage of all of that, at least not at 16-bit precision. The good news is that we don't need to store the key-value caches, which track the model state, at 16-bits. We can get away with lower precisions without too much performance and quality degradation. To maximize our context window, we'll be compressing the key value pairs to 8-bits. Finally, we'll want to make sure prefix caching is turned on. For workloads where large sections of the prompt are going to be reprocessed over and over again, like a system prompt or code base, this will speed up inference by ensuring only new tokens are processed. In newer builds of Llama.cpp this should be enabled by default, but we'll call those flags just in case. With all that out of the way, here's the launch command we're using for a 24GB Nvidia RTX 3090 TI, but the same code command should work just fine if you're using an AMD or Intel GPU or are running Llama.cpp on a Mac. If you're running this on a machine with more memory, try bumping up the context window to 131,072 or 262,144. If you're planning on running Llama.cpp and accessing it on another machine, you'll also want to add --host 0.0.0.0 to the command, which will expose it to your local area network. If Llama.cpp is running in a VPC, you'll want to configure your firewall rules before passing this flag for the sake of security. Now that our model is up and running, we need to connect it to an agentic coding harness. On their own, models can generate code, but they have no way to implement, test, or debug it without an active development environment. Part of what has helped vibe coding take off where other AI ventures have struggled, is that code is verifiable. It either runs or compiles, or it doesn't. To keep things simple we'll be looking at three popular options: Claude Code, Pi Coding Agent, and Cline. We'll kick things off with Claude Code. Despite what you might think, you don't have to use Claude Code with Anthropic's models. The framework works just fine with local models, assuming you've got enough resources to run them. Install Claude Code as you normally would. You can find Anthropic's one-liner here. Next, we'll need to tell Claude Code we want to use the model running locally on our machine rather than a Claude account or Anthropic's API services. This is done by setting a few shell variables before launching Claude Code. These will need to be run each time you launch Claude from a new session. Now when you start Claude, it'll connect directly to your local model. Claude Code itself continues to function as it normally would. Let's say you not only want to use your own local models, but would prefer an open source harness as well. If you like Claude Code, you'll probably like the Pi Coding Agent. And just like Claude Code, it's not picky about what model you use with it. One of the main attractions of Pi Coding Agent is how lightweight it is. Long input sequences can be extremely taxing on lower end or older GPUs or accelerators. Claude Code and Cline both have system prompts that can bring less capable hardware to a crawl. By comparison, Pi Coding Agent's default system prompt is short enough to keep things snappy, especially with prompt-caching enabled. However, that speed comes at the expense of many of the guardrails and safety features we see on other coding agents. This is one you'll probably want to spin up in a virtual machine, container, or even a Raspberry Pi. Much like Claude, the Pi Coding Agent can be installed using the appropriate one liner for your system. After that, all that's required is a little bit of JSON telling the agent harness where to find your model. If you've been following along, the setup is fairly simple. Using your preferred text editor, create the following file: Next, paste in the following template. If you've set an API key, replace no_API_key_required with your key. The rest of these will depend on what model and port you're using. You'll also want to adjust the contextWindowSize to match what you set in Llama.cpp. With that out of the way, we can navigate to our working directory, launch Pi Coding Agent, and get to work vibe coding our next hobby project. Claude Code integrates directly with popular integrated development environments (IDEs) like VS Code, but if you're going this route, we also recommend checking out another open source app called Cline. Installing Cline is as simple as finding it in VS Code's -- or a supported IDE's -- extension manager and adding it to your library. Next, we'll point Cline at our Llama.cpp server and adjust a few hyperparameters like temperature and context size: Once it is configured, you can interact with Cline through its chat interface. Any files or edits will appear in VS Code as they're generated. One of Cline's more useful features is the ability to switch between a pure planning mode and an action mode. If you've ever gotten frustrated because Claude interpreted a question as a call to action when what you really want to do is workshop a problem, this is a huge help. So can Qwen3.6-27B replace Opus 4.7 or GPT-5.5? Not exactly. As you probably guessed, a 27B LLM isn't a replacement for a multi-trillion parameter frontier model. However, you might be surprised with just how far you can get with local models these days. In our testing, Qwen3.6-27B easily one shot an interactive solar system web app and was able to accurately identify and patch bugs in an existing code base. Admittedly, these are fairly trivial projects. To get a better sense of how well the model performs, I handed it over to fellow vulture Thomas Claburn to see how it compares to his recent experience with Claude Code. He writes: I've only recently started playing around with local models, but Tobias's experience seems similar to my own. I've been using the pi coding agent, with OMLX as the model server, and while the token rate is a lot slower, I'm satisfied with Qwen so far, at least for small scripts. For example, I asked the model to write a Python script for resizing images to a specified width and it did so - after about five minutes with a few manual approvals. Claude Code's assessment of the Qwen model's work is more positive than I expected - "Overall: Strong, production-quality script." Claude had some improvements to suggest, but none of them were necessary. For example: silently treats all non-PNG as JPEG A .webp file in the directory would be filtered out by , but if that set ever grows, the fallthrough to JPEG would be a silent misbehavior. An explicit elif or a lookup dict would be safer. Given the time required to generate that code, I can see using local agents for focused, discrete code changes, scripts, and minimal web projects. With a more substantial project, I expect there would be too many things that need correction. But a lot is going to depend on the skills and tools available to the local model. The best way to figure out if local models are plausible is to give them a try - they might work for your purposes. Make sure you have memory-heavy hardware - and make sure you have your data backed up. With all the hullabaloo over the security nightmare known as OpenClaw, it's a good question. Thankfully, most of the frameworks we've discussed here are fairly limited in their autonomy. By default, Claude Code, and Cline rely on having a human-in-the-loop to approve code changes and execute shell commands. Unless you've whitelisted a set of commands or are spamming the enter key before reading without taking the time to understand what it is that the agent is trying to do, the blast radius should be manageable. We emphasize "should be" because a basic understanding of the programming language and common CLI commands goes a long way here. If the model starts asking to run rm -rf on files or folders outside your working directory, something probably has gone wrong. This isn't the case with Pi Coding Agent, which operates in YOLO mode out of the box, which gives it free rein to read and modify anything it has access to. In a dedicated development environment like a virtual machine or Raspberry Pi, this might be an acceptable risk, but if it's not, you may want to consider running the agent in a proper sandbox. Containerization offers an easy avenue for this. It's fairly simple to spin up a Docker container and pass your working directory through to it. Docker is a whole can of worms on its own, but the following run command should give you a reasonable starting point for a sandboxed environment. You can find instructions on installing Docker on your preferred OS here. This will spin up a new Ubuntu docker container and pass through our working directory to the container. Any changes will be limited to that folder or the container. If you'd like to see a comprehensive guide on building agent sandboxes, let us know in the comments section. ®

I replaced ChatGPT and Claude with this powerful local LLM and saved over $20 a month while gaining full control

Ayush Pande is a PC hardware and gaming writer. When he's not working on a new article, you can find him with his head stuck inside a PC or tinkering with a server operating system. Besides computing, his interests include spending hours in long RPGs, yelling at his friends in co-op games, and practicing guitar. As a home labber, I've always been an advocate for running services on my local hardware instead of relying on cloud platforms. And my stance on FOSS tools hasn't changed even after I started running large language models. If anything, I've grown to love the 9B and 12B LLMs I've deployed on my server nodes, as I don't have to worry about external firms gaining access to all the documents and log files I upload to my models. But no matter how much I adore my local models, they can't hold a candle to the hundreds of billions of parameters Claude, Perplexity, ChatGPT, and other cloud-based models can crunch, especially for demanding coding tasks requiring massive context windows. Or at least, that's what I thought until I began using Qwen3.6-35B-A3B on my gaming PC. And let me tell you, this powerhouse of an LLM can walk toe-to-toe with pricey cloud models for development workloads - all while running on my weak hardware. Getting Qwen3.6-35B-A3B to run on my outdated GPU was a bit of a challenge But with a few tweaks, I got this beast of a model to generate 20+ tokens/s Before I talk about my experience with Qwen3.6, let me go over the hardware aspect of my LLM-hosting setup. As a broke home labber, the RTX 3080 Ti is the fastest GPU in my arsenal, and it has held up pretty decently for running 12B models, and even GPT-OSS-20B with tweaked parameters. However, it only has 12GB of VRAM and, to be honest, it's terribly outdated for LLM-powered tasks in 2026. In theory, it would be difficult to run a 27B model on this card, let alone something as bulky as a 35B LLM. But as it turns out, it's not only possible to load Qwen3.6-35B-A3B on my old system, but it can even drive this LLM at a respectable token generation rate of 24t/s. Since it's a mixture of experts model, I can use the --n-cpu-moe flag to offload some expert weights on the CPU instead of forcing them on my graphics card, while -ngl 999 ensures my GPU gets utilized for the KV cache and attention layers. Increasing the CPU threads via -t increases its computational prowess, but since I wanted enough context size for my coding tasks, I set -c to 65536. After asking our resident LLM maestro, Adam Conway, for some tips, I used the following command to get my llama-server instance running with Qwen3.6: llama-server.exe -m "C:\Users\Ayush\.lmstudio\models\lmstudio-community\Qwen3.6-35B-A3B-GGUF\Qwen3.6-35B-A3B-Q4_K_M.gguf" -c 65536 -ngl 999 --n-cpu-moe 30 -fa on -t 20 -b 2048 -ub 2048 --no-mmap --jinja Just to test things out, I logged into the web UI created by llama-server and ran a quick query about XDA-Developers. To my surprise, the LLM was able to hit above 20 tokens/s, so I switched to llama-bench to see how far I could go. The max my system could handle was a 16K prompt length with -ctk set to q8_0‑style quantization (though setting -ctv to q8_0‑style would cause it to crash). llama-bench.exe -m "C:\Users\Ayush\Downloads\LLMs\Qwen3.6-35B-A3B-Q4_K_M.gguf" -p 16000 -n 256 -ngl 999 --n-cpu-moe 32 -fa on -t 20 -b 2096 -ub 2096 -ctk q8_0 In fact, one look at the system resources confirmed that the 32GB RAM on my PC was the bottleneck with these commands. So, if I could track two more 16GB sticks (which may be plausible once the RAM apocalypse ends), I could push the context window and quantization settings even further. But since I wanted to check whether Qwen3.6 could replace its cloud counterparts for my specific tasks, it was time to pair it with my code editors, agentic tools, and typical FOSS apps. Qwen3.6-35B-A3B is a beast of an LLM It's a rock-solid coding companion Besides my productivity tasks (which I'll get to in a moment), I extensively use LLMs as my coding companions. And no, I'm not talking about vibe-coding tasks, either. Rather than letting clankers create apps, I use them to troubleshoot tasks, rearrange syntax, and autocomplete my code. Up until now, I'd cycle between Qwen2.5-Coder and DeepSeek R1 for my coding tasks, and while they were decent companions, I'd often have to run them a couple of times and specify the context in great detail before they'd start dishing out helpful suggestions. Subscribe for hands-on guides to running Qwen3.6 locally Discover why subscribing to this newsletter pays off: detailed coverage of running Qwen3.6-35B-A3B locally, step-by-step hardware tweaks, exact llama-server and bench commands, quantization and context tips, and practical FOSS integrations. Get Updates By subscribing, you agree to receive newsletter and marketing emails, and accept our Terms of Use and Privacy Policy. You can unsubscribe anytime. Qwen3.6, on the other hand, gives solid troubleshooting tips from the very first prompt - to the point where it correctly identifies where my Home Assistant automation was malfunctioning after I paired it with Claude Code and uploaded the trigger-action YAML file and HASS logs. Heck, I've even run it through a couple of other terminal logs, and aside from two configs (which, in all fairness, didn't have enough details about the errors), it pinpointed the faulty packages and configs accurately. Likewise, I paired it with the Continue.Dev extension on VS Code, and it's great at autofilling my variable names and modifying syntax to fit different languages. I've also paired it with a handful of self-hosted FOSS tools If you've read my articles at XDA, you're probably aware that I use LLMs with certain LXCs and Docker containers. Well, Qwen3.6 is just as effective for my productivity-centric toolkit. Thanks to llama-server's OpenAPI-compliant nature, I can pair this LLM with everything from my Paperless-ngx companion apps to good ol' Open WebUI. In fact, I've paired it with OpenNotebook, and it's significantly better than every other local model I've used for aggregating my research notes. Likewise, I've also added it to Blinko, where it goes through my notes and answers all my queries. The best part? Not only do I get to keep my notes off the prying eyes of cloud-based AI models, but I also don't have to spend extra bucks on subscription fees. For example, I've been feeding Claude Code error logs every time my home lab breaks, and without Qwen3.6-35B-A3B, I'd have to pay for API usage. And since this 35B behemoth is good enough for research, I can just pair it with Open WebUI + SearXNG instead of relying on ChatGPT. llama.cpp Llama.cpp is an open-source framework that runs large language models locally on your computer. See at Official Website Expand Collapse