★ open weights insideshippers operate / open source ai
themodels areloose.
run them in the crib.
open weights, local inference, scavenged hardware. this is the field guide: what people are actually building, what it runs on, and the trail through all of it. every card cracks open.
i
the moves
nine plays, each one a real thing people run right now. every card names the actual tool, and cracks open for the deeper cut.
01 · exo
the e-waste supercomputer
the actual "grab your old console" move. exo finds every device on your wifi, old macbooks, iphones, ipads, raspberry pis, and shards one big model across the pile. four forgotten laptops become one machine that runs what none of them could alone.
learn thismodels can be split layer by layer across devices. memory stacks across the mesh; the wifi is the motherboard.
deeper cut: exo-labs/exo on github. auto-discovery means zero config: same network, same cluster. the famous demos chain macbooks and ipads into one model. reality check: wifi latency makes it a party trick until you wire thunderbolt between them.
02 · llama.cpp + gguf
it runs on anything with a pulse
the whole local scene stands on one c++ repo. quantization crushes a model to a quarter of its size for barely any quality tax; that's how an 8B fits on a raspberry pi and a 70B fits in a gaming tower. people run this on 2012 thinkpads and android phones.
learn thisQ4 means 4-bit weights. smaller number format, same brain, a fraction of the memory.
deeper cut: the format to know is GGUF; hugging face is full of pre-quantized models (search any model name + "GGUF"). Q4_K_M is the community's sweet spot. and r/LocalLLaMA is the scene's front page: benchmarks, leaks and breakthroughs, daily.
03 · ollama / lm studio
your mac is already a rig
apple silicon's unified memory is the quiet cheat code: the gpu can borrow ALL the ram, so an m-series mac out-runs most gaming pcs at this. one pull and you've got a free offline model that works on a plane. lm studio if you want a face on it.
learn thisfor local inference, memory beats flops. unified memory is why the mac punches up.
deeper cut: rough maths: a Q4 model needs about half its parameter count in GB, so an 8B wants ~5GB and fits a 16GB mac with room to breathe. qwen and gemma small models are the current pound-for-pound kings. the same mac will run whisper and image gen at the same time.
04 · home assistant + whisper + piper
kill your alexa
the most-built local ai project going: whisper for the ears, piper for the voice, a local model for the brain, home assistant as the body. lights, heating, timers, questions, and not one syllable of audio ever leaves the house.
learn thispipelines beat monoliths. ears, brain and voice are three swappable models, not one product.
deeper cut: home assistant sells ready-made voice hardware now (voice preview edition) if you don't fancy soldering. wake words run on openWakeWord, and the whole pipeline is configurable in the UI these days: no yaml martyrdom required.
05 · open webui / anythingllm
a second brain that stays home
point a local model at your own pdfs, notes and exports. it indexes the lot and answers with citations: chat ui, self-hosted, zero upload. your archive becomes searchable intelligence without a byte leaving the machine.
learn thisRAG means the model reads your files at answer time. it never memorises them, so swapping models keeps your library.
deeper cut: the real unlock is exports: whatsapp archives, kindle highlights, bank statements, old notes apps. anything that becomes text becomes queryable. embeddings run local too (nomic-embed), so even the indexing stays in the house.
06 · whisper / whisper.cpp
transcription is free now
local speech-to-text good enough to retire the paid apis. voice notes, meetings, calls, all transcribed on your own laptop with no meter running. the cpp port even runs on phones.
learn thissmall task-specific models beat giant general ones at their own game. transcription is a solved local problem.
deeper cut: whisper-large-v3 is still the open benchmark; faster-whisper runs the same model several times quicker. macwhisper if you want zero terminal. pair it with a local model to summarise what it hears: fully offline meeting notes.
07 · used 3090 / ex-datacenter cards
the used-gpu come-up
the canonical rig move: a used rtx 3090 for its 24GB of vram, or retired datacenter cards for pennies. yesterday's mining rigs and gaming towers are this year's inference servers.
learn thisvram decides what fits; speed only decides how fast. buy memory, not marketing.
deeper cut: 24GB runs the 30B class comfortably at Q4; two 3090s bridge to 48GB and open the 70B door (it's the last consumer card with nvlink). check fan bearings on ex-mining stock. amd is getting usable, but nvidia is still the path of least pain.
08 · lora / unsloth
teach a small model your moves
fine-tuning stopped needing a datacenter. lora trains a thin adapter on top of a frozen model, so a 7B learns your tone, your formats, your domain, overnight on one consumer gpu.
learn thisyou're not retraining the model, you're teaching it an accent. adapters are megabytes, not gigabytes.
deeper cut: unsloth roughly halves the memory needed, so a 7B adapter trains on an 8GB card. the data matters more than the compute: 500 clean examples of YOUR pattern beat 50k scraped ones. graduate to axolotl when notebooks feel small.
09 · stable diffusion / flux + comfyui
images without the meter
image gen on the gaming gpu you already own. comfyui gives you the node graph: models, loras and upscalers wired together like modular synth patches. no credits, no queue, weird experiments encouraged.
learn thisopen image models are moddable. communities ship whole art styles as tiny lora files you stack.
deeper cut: civitai is the lora bazaar: thousands of styles as small files you drop in. flux for realism, sdxl still king of the fine-tune scene. comfy workflows are shareable json; steal one, mutate it. that IS the culture.
10 · kimi k3 / deepseek / qwen / llama
the frontier went open, and the trickle-down is the point
the biggest models on earth ship their weights now; k3's 2.8 trillion parameters land in the open on july 27. you will not run that in the crib, and it does not matter: open frontier weights get distilled into the small models you WILL run. every big open drop upgrades the whole food chain a few months later, which is why the 8B on your mac keeps getting scary good.
learn thisdistillation means the big model teaches the small one. bet on the food chain, not any single model.
deeper cut: the houses to watch: deepseek (shipped open reasoning first), qwen (best small models pound-for-pound), llama (the ecosystem standard everything tools against). one catch: "open weights" is not always "do anything you like". licences differ, so read one before you build a business on it.
ii
the trail
no clock on this. eight territories, wander in any order. the map's yours.
ifirst contact
ollama, pull a small model, talk to it offline. feel how close free gets.
iigive it a face
lm studio or open webui: a proper chat home for every model you collect.
iiifeed it your files
private RAG over your pdfs, notes and exports. the archive wakes up.
ivgive it ears
whisper everywhere: voice notes, meetings, calls. free text from the air.
vwire the house
piper + home assistant: the fully local voice pipeline. alexa never hears it.
vigive it hands
open interpreter / goose / aider: local agents that run your terminal.
viibuild the rig
a used 24GB gpu. bigger brains at home, and a lora trained on your own writing.
viiiraise the mesh
exo across every device you were about to sell. the e-waste supercomputer.