Skip to content
Book a consult

Insights  /  Platforms

Evaluating AI vendors: questions executives should insist on

Security, privacy, model behavior, IP, and exit strategy — in plain language.

Sopra Labs AI  ·   ·  8 min read
SopraLabs Hero share

Key takeaways

  • A good demo shows what a product can do. The contract decides what happens to your data, who answers for the output, and how you leave.
  • Vendors are now a main route to a breach. Verizon’s 2026 Data Breach Investigations Report found third-party involvement in 48% of breaches, up 60% from the prior year’s dataset (Verizon).
  • Whether your data trains a model depends on the plan you buy. Published business terms from OpenAI and Anthropic say it does not by default, and Anthropic’s consumer plans run on separate terms, so get the answer in writing for your plan.
  • Five heads cover the ground: security, privacy, model behavior, intellectual property and exit. Each answer should arrive as a document or a contract clause, not as a reassurance on a call.

The demo went well. The contract is forty pages.

The demo is on Tuesday and it goes well. The tool reads a stack of customer contracts and produces a clean summary of renewal terms in under a minute. The head of operations wants it live by the end of the quarter.

By Friday the order form and a forty-page master agreement are in the CFO’s inbox. She is the one who signs. She has a price, a discount that expires at month end, and a note from the account executive calling the security questionnaire “standard.”

She reads the agreement twice. She can tell what it costs. She cannot tell where the contracts go once they are uploaded, whether anyone will learn from them, or what she gets back if this vendor is bought next year.

She asks the head of operations what else they should be asking. He says the demo answered his questions.

A demo answers one question. The contract answers the rest.

Nobody here is careless. A demo is built to show whether the product works on a good day. It cannot show where data is stored, what the vendor may do with it, or what happens when something changes. Those answers live in documents that nobody puts on a screen.

The exposure is real. Verizon’s 2026 Data Breach Investigations Report covers more than 22,000 confirmed breaches and found that “breaches with third-party involvement have increased by 60% from last year’s dataset, reaching 48% of total breaches” (Verizon). Every vendor that holds your data is part of your security perimeter, whether or not your security team has met them.

AI adds a layer. An AI application is often built on a model, the engine that generates the output, which the vendor rents from another company. When that is the case, your data passes through at least two companies and two sets of terms.

Those terms vary by plan. OpenAI’s enterprise privacy page, updated January 8, 2026, says: “By default, we do not use your business data for training our models” (OpenAI). Anthropic’s commercial terms, effective June 17, 2025, say “Anthropic may not train models on Customer Content from Services” (Anthropic). Yet when Anthropic changed its consumer terms in August 2025 to let users choose whether their data is used for training, it noted that the updates “do not apply to services under our Commercial Terms” (Anthropic). Same company, different plan, different rules.

So a trial run on someone’s personal account says little about the agreement on the table. The only terms that count are the ones attached to the plan being bought. Every vendor quotation here is the published wording as read on October 3, 2026. Terms change.

NIST’s AI Risk Management Framework treats all of this as ordinary management work. It expects that “policies and procedures are in place that address AI risks associated with third-party entities, including risks of infringement of a third-party’s intellectual property or other rights” and that “contingency processes are in place to handle failures or incidents in third-party data or AI systems deemed to be high-risk” (NIST AI RMF, GOVERN 6.1 and 6.2).

The CFO’s instinct is sound. Price is simply the one term the vendor has made easy to read.

Ask five sets of questions before you sign

Each answer should arrive as a document or a contract clause, not as a reassurance on a call.

Follow the data. Ask where your data is stored and processed, which of the vendor’s staff can see it, and who is told if there is a breach. Then ask for the independent security audit report, such as a SOC 2 Type 2, and read what the auditor flagged. A badge on a website is not the report.

Get the training answer in writing. Ask whether your inputs and outputs are used to train or improve any model, the vendor’s or its provider’s. Ask how long data is kept after you delete it. Ask for the list of sub-processors, the other companies that handle your data on the vendor’s behalf. The words “by default” matter, so ask what changes the default.

Ask what sits underneath. Which model does the product use, and who provides it? How was accuracy tested on work like yours, and can you run your own documents before signing? Ask what notice you get when the model changes. OpenAI’s policy lists “at least 6 months” before retiring generally available models, absent safety or compliance concerns, but preview models “may be retired with much shorter notice, such as 2 weeks” (OpenAI). Your vendor inherits that clock. Ask how errors are reported and corrected.

Read the indemnity, then its exclusions. Confirm that you own what the tool produces. Then ask who defends you if an output is alleged to infringe a copyright. Providers publish such commitments with conditions. Microsoft’s applies “as long as the customer used the guardrails and content filters we have built into our products” (Microsoft, 2023). Google describes separate indemnities for training data and generated output (Google, 2023). Anthropic’s excludes claims arising from “the combination of the Services or Outputs with technology or content not provided by Anthropic” (Anthropic). Ask whether any of this reaches you through your vendor’s contract.

Plan the exit before the entrance. Ask in what format you can export your data, prompts and configurations, and for how long after termination. Ask how much notice the vendor must give before ending the service. Anthropic’s standard commercial terms, for example, specify “30 days prior Notice.” Ask what happens to your data and your contract if the vendor is acquired or shuts down.

The usual objection is that a mid-market buyer has no leverage, so asking is pointless. The questions still work, because a vendor ready for business customers can produce an audit report, a sub-processor list and a written position on training within days. One that cannot has told you something. Record the answers in the decision log and the vendor evaluation rubric, and revisit them at renewal. NIST expects that “AI risks and benefits from third-party resources are regularly monitored” (NIST AI RMF, MANAGE 3.1).

What she signs

On Monday the CFO sends five short sets of questions. The vendor answers four within the week and attaches its audit report. The fifth, on exit, takes a call and ends in a one-page amendment on data export.

She signs ten days later than the discount wanted. The discount, it turns out, could wait. The price is the same and so is the product. The difference is that she can now say what she agreed to.

The opening scenario is a composite of common patterns, not a client engagement. SopraLabs helps leadership teams evaluate vendors and architectures against their requirements and constraints in the Platform Selection module of our services.

Frequently asked questions

What questions should we ask an AI vendor before signing a contract?

Ask about five things: security, privacy, model behavior, intellectual property and exit. In practice that means where your data goes and who can see it, whether it trains any model, which model sits underneath and what happens when it changes, who owns and defends the output, and how you get your data back. Ask for each answer as a document or a contract clause.

Do AI vendors use our company data to train their models?

It depends on the plan and the contract, so get the answer in writing. OpenAI’s enterprise privacy page says “By default, we do not use your business data for training our models” (OpenAI, updated January 8, 2026), and Anthropic’s commercial terms say it “may not train models on Customer Content from Services” (Anthropic, effective June 17, 2025). Consumer plans can run on different terms, and an application vendor’s terms are separate from those of the model provider behind it.

Who owns the output of an AI tool, and who is liable if it infringes copyright?

Under OpenAI’s and Anthropic’s published business terms, the customer owns the output. OpenAI says “You own your inputs and outputs (where allowed by law)” (OpenAI), and Anthropic’s commercial terms say the customer “owns its Outputs” (Anthropic). Liability is a separate matter covered by the indemnity clause, which carries conditions and exclusions worth reading before you sign.

What happens if an AI vendor changes or retires the model behind its product?

The product’s behavior can change, and the notice you get depends on your contract. OpenAI’s published policy lists “at least 6 months” of notice before it retires generally available models, unless safety or compliance concerns require a faster timeline, with far shorter notice possible for preview models (OpenAI, read October 3, 2026). Ask your vendor which model it uses, how it retests accuracy after a change, and how much warning you will receive.

Sources

Keep reading

Browse all insights