Public API, private cloud or on-premises: where should sensitive data be processed?
The right answer is not determined by the model location alone. Map data, logs, keys and access paths, then choose the level of control your risk and operating capacity actually require.
Short answer
Use a public API when data minimisation, a clear provider contract and strong access controls are enough. Choose a private cloud when the organisation needs a controlled network perimeter and ownership of identities, keys and logs. Reserve on-premises processing for real constraints on residency, connectivity or local operations. In all three cases, the decision covers the complete data path, not only the place where the model generates a response.
This is an architecture rule, not a universal ranking. Capabilities vary by provider, model and enabled feature. OpenAI, AWS, Google Cloud and Microsoft each describe different conditions for retention, residency and access. Verify the exact service before signing a contract or moving to production (OpenAI endpoint data controls, AWS Bedrock data protection, Google Vertex AI zero data retention, Microsoft Foundry data privacy).
Nexxom diagram: data classification and policy determine which architecture to examine. It illustrates a decision logic, not a regulatory requirement.
Location is not the only control
Geographic residence matters, but it does not answer four operational questions: which data leaves the source system, who can read it, how long it is retained, and how it can be deleted or audited. A workload described as private can still expose prompts in application logs, tool traces or backups. A public API can be suitable when the request is minimised, retention is documented contractually and business identifiers are kept away from the payload.
Before comparing options, describe the complete route: business source, proxy or gateway, model provider, application logs, tools called by an agent, output storage, backups and the human systems that can view traces. This map exposes copies that teams often overlook. It complements an AI inventory and a risk analysis based on the NIST AI Risk Management Framework and its generative AI profile, NIST AI 600-1.
The zero trust approach in NIST SP 800-207 is useful here: trust no network or service by default, and explicitly verify the identity, context and authorisation of every call. A selected location does not remove the need for identity, segmentation or logging controls.
Public API: speed with contractual controls
A public API is a managed service reached over HTTPS. It fits a pilot, a low-volume workload or a case where the organisation prefers the provider to operate the infrastructure. The main risk moves to endpoint selection, contract terms, configuration and the application gateway.
OpenAI states that data sent to its API is not used to train models by default. The same documentation says abuse-monitoring logs may be retained for up to 30 days by default and that eligibility for zero data retention depends on the endpoint. A retention policy must therefore be checked for the endpoint actually called, rather than inferred from the word “API” (OpenAI endpoint documentation).
An API should never receive more data than the task needs. The gateway should remove direct identifiers, restrict fields, enforce size limits and reject unapproved attachments. It should separate technical logs, prompts, responses and security events. API keys belong in a secrets manager, never in a browser or source repository. Access permissions and audit logs should follow least privilege, as described in AWS IAM for Bedrock.
The weakness of a public API is not automatically a lack of security. It is the inability to control some provider-operated layers. Obtain those guarantees from documentation, contract language, the data-processing addendum and account settings. AWS shared responsibility guidance explains that AWS protects the service while the customer remains responsible for data, identities and configuration.
Private cloud: network perimeter and managed services
Private cloud means, in this article, a deployment in an organisation-controlled account or project with private networking, IAM, customer-managed keys and centralised logs. The model may be a managed service reachable inside that perimeter or a component deployed on dedicated resources. Private does not mean that no third party is involved: the cloud provider, model provider, support staff and platform operators each have a role to verify.
Google Cloud publishes a control matrix by model and feature. Data residency, customer-managed encryption keys, VPC Service Controls and Access Transparency are not identical for every service. The practical conclusion is to read the row for the selected model, region and feature, then retain it in the architecture record (Vertex AI security controls). Google also explains that some data can be retained to provide a feature, prevent abuse or support operations, with different rules by scenario (Vertex AI zero data retention).
Private cloud often gives better control over network paths, segmentation and keys. It also adds configuration work: egress rules, certificates, secret rotation, failover routes, observability, restore tests and operator access reviews. Real security is what the team can maintain after six months, not what appears in the first architecture diagram.
On-premises: local control and local responsibility
On-premises processing makes sense when data cannot leave a site or administrative zone, when external connectivity is intermittent, or when the organisation must control every inference component. It can also address a strict physical-latency requirement. It does not make sensitive data safe by itself: storage, GPUs, backups, administrator accounts, software dependencies and logs still need protection.
Local deployment requires version inventory, patching, model lifecycle management, regression tests and a hardware replacement plan. Define who responds to an alert, who can extract traces and how long outputs remain available. The NIST AI RMF Playbook and NIST work on AI security and resilience provide a structure for these responsibilities without prescribing a topology.
The hidden cost is operational. Someone must maintain security updates for the operating system, inference runtime, drivers, network layer and monitoring tools. If those tasks are not funded, the theoretical control of on-premises infrastructure becomes an availability and vulnerability risk.
Comparing the three options
The table separates observable properties from context-dependent recommendations. “High” does not mean “better”: more control usually means more work and responsibility.
| Criterion | Public API | Private cloud | On-premises |
|---|---|---|---|
| Network-path control | Limited to the gateway and provider guarantees | High when private networking and egress rules are actually configured | Very high inside the local perimeter |
| Operating effort | Low to medium | Medium to high | High |
| Residency and connectivity | Depends on endpoint and region | Depends on service, region and controls | Local control with evidence to maintain |
| Model operations | Provider-operated | Shared or managed | Fully owned by the team |
| Log control | Provider settings and contract | Can be centralised in the account | Fully internal, including backups |
| Suitable case | Minimised data, experimentation, short lead time | Sensitive data with advanced network and IAM needs | Strict egress, autonomy or connectivity constraints |
Static Plotly matrix produced by Nexxom. Values 1, 2 and 3 are an explicit editorial heuristic. They help read the trade-offs in the table; they are not a provider measurement or benchmark.
A reproducible decision grid
To avoid an abstract “cloud versus local” debate, score each constraint with a documented answer. The team can keep the rationale in the processing and architecture record.
| Decision question | If yes | Likely consequence |
|---|---|---|
| Must raw data remain in a specific zone? | Strict residency | Exclude endpoints that cannot prove residency; examine private cloud or on-premises |
| Can a provider retain operational logs? | External retention is acceptable and documented | Public API may work with gateway, filtering and contract; otherwise reduce data or change topology |
| Must the system run without internet egress? | External connectivity is forbidden or unstable | On-premises, or private cloud with a dedicated path if policy permits |
| Can the team maintain models and patches? | Local operating capacity exists | On-premises is viable; otherwise prefer a controlled managed service |
| Can data be pseudonymised before inference? | Minimisation is feasible | Public API or private cloud often suffice for the reduced scope |
This grid is not legal advice. The NIST AI RMF is voluntary and does not establish regulatory compliance by itself. Contract terms, data classification, sector obligations and the data-protection officer remain decisive.
Cross-cutting controls to require
Whatever the architecture, require a minimum sequence:
- Classify inputs, outputs, attachments, embeddings and logs.
- Minimise before the call: remove unnecessary identifiers, truncate fields and separate join keys.
- Authenticate every service with distinct identities, rotated secrets and limited permissions.
- Encrypt in transit and at rest, then decide who owns keys and who can revoke them.
- Log decisions and errors without automatically copying sensitive content into a general log.
- Test deletion, restoration, model changes and incident response.
- Review provider documentation whenever an endpoint, model or region changes.
Provider capabilities are not interchangeable. Microsoft notes that controls and retention vary by deployment type and Azure feature (Azure OpenAI and data privacy). AWS documents TLS, IAM, CloudTrail and encryption controls, but activation and configuration remain customer responsibilities (Bedrock data protection).
Three selection scenarios
Internal document assistant with pseudonymised data. A public API can be reasonable when the gateway removes identifiers, endpoint retention is documented and responses are not injected into a decision system without review. A private cloud becomes preferable when the organisation must centralise logs or block general egress.
Regulated case analysis with no site exit. On-premises is the natural candidate, but verify that the team can patch and monitor the platform. If that capacity is absent, a private cloud with proven region, keys and segmentation may reduce operational risk, provided policy allows it.
Prototype where real data is unnecessary. Do not move sensitive data merely to test an interface. Use synthetic or strongly pseudonymised records, measure outputs and document limitations. The architecture choice then remains a choice about speed and reversibility, not a permanent exception to governance.
Limits and watch points
A cloud region does not guarantee that every processing step, support activity, metadata record or backup stays in that region. A customer-managed key does not prove that an application never stores a clear copy. A local model does not prevent leakage through prompts, logs or called tools. Every statement must be tied to a component, a configuration and evidence.
Do not send real sensitive data to a provider test environment to validate a privacy promise. Use synthetic data, a contract review procedure and a controlled deletion test. When the question concerns a legal obligation, ask the legal function or DPO to validate the interpretation.
Conclusion
The best architecture is the one that makes the data path observable and maintainable. Start with classification and minimisation. Choose a public API for low-exposure cases where speed matters, a private cloud for stronger network and IAM control, and on-premises for a real egress or connectivity constraint. In every case, retain evidence for retention, residency, access, deletion and responsibility before declaring the processing compliant.
For the next step, connect this decision to an AI inventory, your governance controls and your LLM application security strategy. Nexxom can help turn the grid into a testable architecture and operating procedures.
Primary sources verified on 19 August 2026
- OpenAI, endpoint data controls
- AWS, Amazon Bedrock data protection
- AWS, IAM security guide for Bedrock
- Google Cloud, Vertex AI security controls
- Google Cloud, Vertex AI zero data retention
- Microsoft, Azure OpenAI and Foundry data privacy
- NIST, AI Risk Management Framework
- NIST, Generative AI profile AI 600-1
- NIST, AI RMF Playbook
- NIST, Zero Trust Architecture SP 800-207
- NIST, AI security and resilience
