Running a language model inside your own environment, rather than calling a hosted provider, is sometimes necessary and frequently proposed for reasons that do not survive examination. Knowing which situation you are in saves a large amount of money.
The reasons that actually hold
A contract forbids the transfer. Client agreements, particularly in professional services and manufacturing, sometimes prohibit sending the client material to a third party processor without consent. This is the most common genuine driver, and it is decided by a document rather than by a preference.
A regulator or a customer security review requires it. Some procurement processes will not approve a supplier whose architecture sends data outside a defined boundary. Arguing with that is usually slower than building for it.
The data cannot leave a network at all. Air gapped or tightly restricted environments exist, and in those the question is not whether to run privately but which hardware.
Latency to an external provider is the constraint. Rare, but real where a feature runs inside a tight loop or in a location with poor connectivity.
The reasons that usually do not
General discomfort about data. Hosted providers offer contractual terms on processing and retention. If the concern is specific, read the terms and check whether they address it. If the concern is that a third party is involved at all, note that this is already true of your hosting, your email and your payment processing.
Assumed cost savings. A private deployment has hardware, operations, monitoring, updates and staffing costs. Compare those with the actual hosted workload, including utilisation and availability requirements. Neither private nor hosted inference is automatically the cheaper choice.
Wanting a fine tuned model. Some hosted providers support fine tuning. Retrieval and fine tuning solve different problems: access to changing documents is not the same as adapting task behaviour. Evaluate which capability the feature needs before deciding where it must run.
What changes when you go private
The index moves. Retrieval has to run inside the boundary too, otherwise you have kept the model private and sent the document passages out. That means the vector store, the search infrastructure and the embedding generation all live inside.
The latency budget changes. A hosted inference provider optimised for speed will often be faster than a self hosted model on modest hardware. If your feature was designed around a fast first token, verify that the private deployment still meets it before committing.
Model choice narrows. You are limited to open weight models you can run on hardware you have. These are capable, and for classification, extraction and grounded answering they are frequently sufficient. For the hardest reasoning tasks the gap is real and should be measured rather than assumed.
Someone operates it. Capacity planning, monitoring, updates, security patching and a plan for what happens when the machine fails. That is an ongoing responsibility rather than a setup task.
On device is a different question
Running a model on the end user machine rather than a server inside your network removes the transfer entirely and gives you a different set of constraints: whatever hardware the user happens to have, a download for the model weights, and no central visibility into quality.
Narrow tasks such as classification, redaction and simple extraction are useful starting points for evaluation. A broader assistant needs device specific tests for quality, memory, battery use and response time. Capability depends on the model and hardware, so do not dismiss or promise it from deployment location alone.
Keep the decision reversible
Whatever you choose, isolate provider calls behind an application interface. Similar chat APIs can still differ in tool calling, supported parameters, context limits and output behaviour. A provider change needs compatibility tests and a fresh task evaluation, not only a configuration edit.
This separation can reduce the work of a later change. It does not make model licences, procurement approval or data residency interchangeable.
Decide it early
This is the part that costs money when it goes wrong. Private deployment affects the index location, the retrieval infrastructure, the latency budget and the hardware plan. Adding it to a system built against a hosted provider is not a switch, it is a re architecture of the data layer.
Ask the question during planning: is there any client, contract or regulator who would require this within the life of the system? If the answer is possibly, record the boundary and the cost of preserving that option.
Draw the boundary around every supporting service
A private model is only one component. Inventory document parsing, embedding generation, search, logging, crash reporting and update delivery. Each can move information outside the boundary even when inference stays inside it. A diagram should mark which connections are permitted and what data each carries.
For an on device feature, separate model downloads from document upload. Downloading weights does not inherently require sending the user's documents to the download server. Conversely, an error reporter that captures the prompt can undo the privacy benefit. Inspect outbound requests during normal operation and during a failure, then compare them with the declared design.
Offline operation needs an explicit user experience. If the feature cannot load its model, say so without silently switching to hosted inference. If a hosted fallback is available, explain the data transfer and obtain the appropriate permission before using it. Retain the user's unfinished task so they can choose another route.
Finally plan updates as changes to behaviour, not just files. Keep a supported model version, a way to verify downloaded artefacts, and a rollback path if the replacement fails the task evaluation. Test upgrades on the least capable supported device. Otherwise a successful test on an engineer's workstation says little about the people expected to use the feature.