Local LLMs are moving fast. Within six months or so, I was running them on my computerwhat actually exists has changed a lot, and not always for the greater good. The people who build these models understand who actually runs them, which is not labs or server farms, but mostly normal people with consumer hardware.
So a lot has been done lately smaller models that hold up well rather than chasing parameter counts on conventional GPUs, no one can run without it. Google new Gemma 4 12B is a pretty clean example of this direction. It’s built with laptops and dedicated GPUs in mind, things that regular people actually own, and Google is pushing the architecture as the main reason for this.
Want to stay up-to-date on the latest developments in artificial intelligence? The XDA AI Insider newsletter drops every week with deep dives, tool tips, and hands-on coverage you won’t find anywhere else on the site. subscribe by changing your newsletter preferences!
Google’s new mid-sized outdoor model
It’s not built for labs, it’s built for regular hardware
Gemma 4 12B launched in June 2026. It sits between the smaller E4B and the heavier 26B MoE in Google’s lineup, so it’s a mid-range option, which I think is helpful because it’s a size that fits the GPUs people actually have at home.
What Google talks about the most is architecture. Most multimodal models use separate encoders that convert images and audio into a format that the language model can read. Gemma 4 12B skipped this step entirely; image and audio are fed directly into the LLM via light projection, without a separate encoder doing the work at first. The result is lower latency because LLM doesn’t wait for the encoder to finish before starting to run.
However, the lag win is mostly a multi-modal thing and you only really feel it when you’re feeding them images or audio. Refactoring the architecture for text usage doesn’t change much because text is always entered straight. But the Gemma 4 12B uses the same decoder structure as the larger Gemma 4 31B Dense, so on the text side you’re actually getting the same basic reasoning engine as the larger model offloading to a normal GPU.
It’s also the first mid-range Gemma to handle native audio input. And the context window now expands to 256K. Google says it runs on less than half the memory near the 26B, though no one has done a comparison that I’ve seen.
My gaming PC has become a decent LLM machine…finally
It took a few tries at first
My computer is a gaming rig that someone else put together for me. To be honest, I just wanted something that could run Rocket League smoothly, but was also good for local LLMs. My first model was a gpt-oss 20B, which I upgraded to an 8GB GPU with some system RAM loading. Since then I’ve moved to smaller, more optimized models that strangely get more hardware than the 20B.
Loading the Gemma 4 12B was a task in itself. LM Studio has 356 versions of it, all of which look pretty much the same, so choosing one was a headache above all else. I went with the QAT Q4_0 version because Google trained it with quantization built into the training itself, so the quality is kept at a smaller size.
My first download attempt failed. I had all 48 layers going to the GPU, context at the default 30K, and KV cache quantization enabled, one of the experimental settings. The estimated memory was 11.35 GB, which was obviously too much for my hardware. So I dropped the GPU load to 32 and dropped the context to 4K, which came out to about 7.3GB. Still failed. At this point I updated LM Studio because the runtime I have is slightly earlier than Gemma 4. Tried again and still failed.
What actually worked was to disable quantization of the experimental KV cache and further reduce the GPU load to 28. The UI itself warns that the KV cache can cause problems on some models, and the Gemma 4 turns out to be one of them. Not sure if the LM Studio update did anything in the end, the KV cache might have been the only thing that mattered the whole time. I ended up running it with over 20K contexts.
Another thing to note is that the “failed to load model” popup never says why. The only useful signal the entire time was the memory estimate at the top of the boot settings, which updates live as you change settings. If you’re running this on hardware not built for native LLMs, this estimate is really something to watch.
Gemma 4 12B is launched
What a few tests revealed
Gemma 4 12B is also good for coding, but I don’t code. So for me, the test was about how he handled the thought. I started with a logic puzzle about the seating arrangement. Thoughtfully, he ran for ages and never found an answer; but the same thing happened with Thinking off. Finally, I ran the same puzzle in the cloud model, the puzzle told me that the puzzle itself was unsolvable, so it was on me. But what bothers me is that Gemma never realized that it couldn’t be solved, which I think shows how the on-premises models are still behind the cloud models.
I gave him a solvable puzzle and he found it either way in seconds. The mode of thinking seems a bit strange to me. Reasoning with it takes place in the hidden block, and the final answer remains short. When you turn it off, the model prompts louder and the response is longer because it ends up on the page instead of sitting in the thinking block. I wrote before how Gemma 4 E4B bleeds its justification To the answer that seems to be a bug in LM Studio, so it might be the same here.
After that, a JSON assignment. Something I really want for a front item that creates a batch with a stack of schemas and constraints in my obsidian notes. And I kept thinking about it. The structure was perfect as expected, every field conformed to the schema, and it didn’t wrap the JSON in the markup fences that many models do. However, several limits are still exceeded and some values are doubled.
In addition to these tests, there was some back-and-forth for a document on the design, and it followed a better structure (without my prompting) than my other Gemma models, and somehow gave more comprehensive answers while getting to the point faster (everything had the same settings).
It’s worth it, with caveats
The Gemma 4 12B is a solid pickup, even freewheeling if you have the equipment to run it. It fell into the cloud puzzle immediately, but held its own for the rest and was a better choice for my paperwork. Sure, I probably won’t be able to use it to its full potential with the GPU, and I haven’t tested the sound properly yet, but it holds up really well so I don’t feel the need to trade it for something smaller.
