Running a local LLM for fiction writing

Why local models suit fiction unusually well

Editorial work — critique, consistency checks, tightening a paragraph — doesn't need a frontier model. A well-chosen 4–12B parameter model running on consumer hardware handles "where does this chapter lose momentum?" surprisingly well, because the hard part (the manuscript, the characters, the intent) is supplied by you, as context. You trade a little polish for total privacy and zero per-token cost.

Option 1 — LM Studio or Ollama on your computer

The mainstream route: install LM Studio (point-and-click) or Ollama (command line), download a general instruct model in the 4–12B range, and start the built-in local server. Both expose an OpenAI-compatible endpoint — usually http://localhost:1234/v1 or http://localhost:11434/v1 — which is the only thing your writing tool needs to know. 8 GB of RAM runs small models; 16 GB or a modest GPU runs good ones.

Option 2 — a spare Android phone with AuctorServ

The route almost nobody offers: modern phones are capable AI hardware. AuctorServ is a free Android app that loads a language model onto the phone and serves the same OpenAI-compatible API over your Wi-Fi. Your old phone becomes a dedicated, silent AI appliance: no fan noise, negligible power draw, and your manuscript never touches the internet at all. Install it, pick a model, and it shows you the address to plug into your writing tool.

It's a genuinely pleasant setup for writers with a laptop that can't spare the RAM — the phone does the thinking, the laptop does the writing.

Getting useful feedback from a small model

• Ask narrow questions. "Critique the pacing of this chapter" beats "is this good?" every time — small models excel at focused tasks.
• Give it the context. Feedback improves sharply when the model can see who a character is, not just the scene text. Tools that pass your story world along with the passage get better answers from the same model.
• Use it for revision, not drafting. Local models are at their weakest generating long prose and at their strongest reacting to yours — which happens to be the ethical sweet spot too.