Twenty-Five Questions to Ask an AI Development Company Before You Sign
An AI development company checklist for vendor calls and security reviews, helping you finalize your shortlist and choose the right partner.

The questions to ask an AI development company fall into five groups: who will do the work, how they will prove the AI works, what happens to your data, what you will own, and how you leave. Ask all twenty-five below, listen for specific answers with names, numbers and documents, and treat vagueness on security or IP as a no.
This is a working list for the last few weeks before signature, when you are on calls with two or three finalists and your security and legal teams are reviewing paperwork. If you need a method for building and scoring the shortlist itself, use our ten-point framework on how to choose an AI development partner. If the vendor will embed engineers inside your team, add the nine questions on choosing a forward deployed engineering company. Both sit alongside this guide in our AI Strategy: Pilot to Production series.
How should you use these questions?
Use them as a script, not a survey. Each question below comes with two things: what a good answer sounds like (paraphrased, in the vendor's voice) and the red flag that should make you slow down. You are not scoring style. You are listening for whether the answer is specific, testable and something the vendor will put in writing.
Not every question belongs in the same meeting:
| Group | Questions | Best asked | Who should ask on your side |
|---|---|---|---|
| Delivery and team | 1 to 5 | Live, with the proposed delivery lead present | Business owner, engineering lead |
| Technical and evals | 6 to 10 | Live technical deep dive with the vendor's engineers | Engineering lead, data owner |
| Security and data | 11 to 15 | In writing first, as part of your security questionnaire, then a follow-up call | Security, privacy |
| IP and contracts | 16 to 20 | In writing, against the vendor's draft MSA and SOW | Legal, procurement |
| Commercial and exit | 21 to 25 | Proposal stage, once scope is concrete | Procurement, business owner |
Why the rigor? Gartner predicted in June 2025 that over 40% of agentic AI projects will be canceled by the end of 2027, citing escalating costs, unclear business value and inadequate risk controls. Most of the questions below are aimed at exactly those three causes.
Delivery and team: who will actually do the work?
AI projects are judgment-heavy. The same brief handed to two teams can produce very different systems, so the people matter more than the logo.
| # | Question | What a good answer sounds like | Red flag |
|---|---|---|---|
| 1 | Who exactly will work on our project, and can we meet them before we sign? | "Here are the named engineers and their recent AI work. Book a technical call with any of them this week." | "We'll staff the right resources after kickoff," or only the sales architect is available to talk. |
| 2 | What happens if one of those people leaves or is moved to another client? | "You approve any replacement, we overlap the handover, and we don't bill for the ramp-up." | No written substitution terms, or replacements announced rather than agreed. |
| 3 | Who on your side is accountable for the outcome, and what will they report each week? | "One delivery lead owns it. Each week you get eval scores, open risks and next week's plan, not just hours spent." | Accountability spread across an account manager and "the team," with status reported as tasks completed. |
| 4 | What would make you tell us to stop or change direction? | "If the baseline eval in the first weeks shows the data can't support the target, we'll say so and propose a smaller scope." | "We've never had a project that didn't work." Every AI team has, and the honest ones can tell you when. |
| 5 | What will you need from our team, and how much of our time? | "A product owner for a few hours a week, a data owner for access decisions, and 20 minutes from end users for each review." | "Nothing, we'll handle everything." AI systems built without your subject-matter experts rarely survive real users. |
Push further: ask question 4 again at the end of the call, phrased as "Tell me about a project you advised a client to stop." A vendor who can describe one is showing you how it will behave when your project hits trouble.
Technical and evals: how will they prove it works?
This group separates teams that build AI systems from teams that build demos. An evaluation suite (evals) is a set of test cases, graded automatically where possible, that tells you whether the system is getting better or worse. Anthropic's guidance on building evaluations recommends designing them to "mirror your real-world task distribution," including edge cases, and automating grading wherever possible.
| # | Question | What a good answer sounds like | Red flag |
|---|---|---|---|
| 6 | In the work you showed us, which parts are actually AI, and which are rules, scripts or people? | "The triage step is an LLM with retrieval; routing is plain rules; a human reviews low-confidence cases. Here's the split." | Everything is "AI-powered," and nobody can say where the model ends. |
| 7 | How will we measure whether the system works, and who builds the test set? | "We build a test set from your real cases with your experts, agree the metrics and thresholds before we build, and you keep it." | "We'll test it manually," or success defined as "the users like it." |
| 8 | How do you catch regressions when a prompt, model version or retrieval setting changes? | "The eval suite runs on every change in CI. A drop past an agreed threshold blocks the release." | No answer to "how would you know if a model upgrade made things worse?" |
| 9 | Which models would you start with, and what would it take to switch? | "We'd start with one provider for reasons X and Y, but the model sits behind an interface, and the evals let us compare a replacement in days." | One model hard-wired everywhere, or a recommendation that tracks the vendor's partnership rather than your use case. |
| 10 | How will you track latency and running cost per task, and what stops it running away? | "Every request is traced with tokens, latency and tool calls. We set budgets and rate limits, and alert before you hit them." | Running cost is "your cloud bill." OWASP lists Unbounded Consumption in its Top 10 for LLM Applications (LLM06:2026). |
Question 6 deserves extra weight in agentic work. Gartner has warned about "agent washing," the rebranding of assistants, RPA tools and chatbots as agents, and estimates that only about 130 of the thousands of agentic AI vendors are real. The FTC's Operation AI Comply, announced in September 2024, is a reminder that regulators treat AI claims like any other marketing claim; then-Chair Lina Khan said there is "no AI exemption from the laws on the books." For what production-grade evals and guardrails look like in practice, see our guide on how to build a production AI agent.
Security and data: what goes into your AI vendor security questionnaire?
Send these five in writing before the security call, attached to whatever questionnaire you already use. Standard questionnaires help, but many were written before LLMs and miss AI-specific risks. The Cloud Security Alliance's AI Controls Matrix (v1.1, July 2026) covers AI risks across 18 domains and comes with a companion questionnaire, the AI-CAIQ, which can save you writing your own. Either way, add the questions below, because they are the ones generic forms most often skip. Our internal AI vendor security questionnaire page is planned as a starting template.
| # | Question | What a good answer sounds like | Red flag |
|---|---|---|---|
| 11 | Trace our data end to end. Where does it go, including every model provider, and what are their retention and training terms? | "Inputs go to provider A under its API terms, configured with these retention settings. Here is the data-flow diagram. Nothing lands on our laptops." | "The model provider is secure," with no account-level settings named, or copies of your data on vendor machines. |
| 12 | Who else touches our data: subcontractors, subprocessors, third-party tools? How will you tell us about changes? | "Here's the full subprocessor list. We notify you in advance of any change, and you can object." | Subcontractors revealed late. Under GDPR Article 28(2), a processor may not engage another processor without the controller's prior written authorization. |
| 13 | How do you test for prompt injection, sensitive data leakage and other LLM-specific attacks? | "We maintain an adversarial test set mapped to the OWASP Top 10 for LLM Applications, run it before each release, and share results." | "Our prompts tell the model not to do that." Instructions in a prompt are not a security control. |
| 14 | What can the system do without a human approving it? | "It reads freely, drafts freely, and anything that writes, sends money or contacts a customer needs approval. Each tool has its own scoped credentials." | Agents running on a shared admin credential, or no clear list of actions. OWASP names this risk Excessive Agency (LLM03:2026). |
| 15 | Can you back your questionnaire answers with evidence, and how fast will you tell us about an incident? | "Here are the audit reports under NDA, our access-review logs, and an incident notification commitment in hours, written into the contract." | A certification badge on the website but no report available, or incident notice "as soon as practicable." |
The written answers to 11 to 15 should become contract schedules, not just procurement paperwork. For the controls that should sit behind those answers, see data security in offshore AI development.
IP and contracts: what will you actually own?
AI projects create more kinds of IP than ordinary software: prompts, retrieval configurations, eval datasets, fine-tuned weights, agent tool definitions. Ask about each by name.
| # | Question | What a good answer sounds like | Red flag |
|---|---|---|---|
| 16 | What exactly will we own at the end? | "All project deliverables, listed: code, prompts, configs, eval sets, fine-tuned weights, documentation. It's in clause X of our draft." | "You own the code," with silence on prompts, data and weights. |
| 17 | Which pre-existing tools or accelerators will you use, and what license do we get to them? | "These three internal libraries. You get a perpetual, royalty-free license to use them in this system, even after we part ways." | A broad "background IP" clause that could swallow most of the system, or an accelerator you can only run under an ongoing contract. |
| 18 | Do your engineers use AI coding assistants on our code, and under what terms? | "Yes, only approved enterprise tools with training disabled on your code, and engineers review and edit all output." | No policy at all. The U.S. Copyright Office concluded in January 2025 that prompts alone do not make a user the author of AI output, so heavy unreviewed generation raises ownership questions as well as confidentiality ones. |
| 19 | How does IP pass from your engineers and subcontractors to you, and then to us? | "Every engineer and subcontractor signs an assignment to us. We'll show you the template." | Reluctance to show employee-side or subcontractor terms. Assignment to you only works if the vendor owns the work first. |
| 20 | What do you warrant, and what are you excluding from liability? | "We warrant non-infringement of what we write and indemnify you for it. Model outputs aren't guaranteed accurate, which is why the evals are a deliverable." | A liability cap so low it is meaningless, or a blanket exclusion for anything "AI-generated." |
Cross-border projects add jurisdiction-specific rules on who owns work by default; our plain-English guide to IP ownership in offshore AI development covers the US, UK and India positions. This is general information, not legal advice; involve your counsel before you sign.
Commercial and exit: what happens when plans change?
Scope will change. Model prices, data quality and user feedback all move during an AI build, so the commercial terms need to handle change without a dispute.
| # | Question | What a good answer sounds like | Red flag |
|---|---|---|---|
| 21 | What happens when scope changes, as it will? | "Changes go through a short written change note that shows the effect on timeline and outcome. Small swaps within the same goal don't need a new SOW." | Every change becomes a priced change request, or no change process at all. |
| 22 | What is the smallest paid first step, and what will we have at the end of it? | "A short pilot on one real workflow, with success criteria written up front. You keep the code, evals and findings whether or not you continue." | A long minimum commitment before you have seen any output. |
| 23 | Where will everything live from day one: repos, cloud accounts, dashboards, eval results? | "In your accounts. We work under your identity provider and you can see everything we can see." | Core components hosted in the vendor's accounts "for convenience." |
| 24 | What does handover look like, and how will we know our team can run it? | "Handover criteria are agreed at kickoff. For example, one of your engineers ships a change and passes the eval suite without our help." | Handover as a final-week slide deck. |
| 25 | If we end the contract early, what happens over the following month? | "Access revoked on a defined timeline, a transition period with named people, documentation complete, and your data returned or deleted with written confirmation." | No exit clause, or transition help available only at "then-current rates" with no commitment. GDPR Article 28(3)(g) already requires processors to delete or return personal data at the end of services. |
The engagement model shapes most of these answers. Our guide to AI development engagement models explains who carries the risk under each option.
How do you read the answers?
A few patterns matter more than any single answer:
- Specific beats confident. "We notify you within a fixed number of hours, here's the clause" is a better answer than "security is our top priority," however warmly it is delivered.
- "It depends" is fine if it's followed by what. Good vendors say what it depends on and how they would find out. Weak ones stop at "it depends."
- Watch who answers. If the salesperson answers every technical question in group two, you have learned who you will be working with.
- Count red flags by group, not in total. One red flag on delivery is a negotiation point. One on security (11 to 15) or IP (16 to 20) should pause the deal until it's fixed in writing, because those are the failures you cannot repair mid-project.
- Ask for the paper. Any good answer you want to rely on should appear in the MSA, SOW or a security schedule. If the vendor resists writing it down, the answer was not real.
If you are earlier in the process and still deciding which kind of partner fits, weigh an AI development agency against a freelancer or a dedicated pod before you start asking these questions.
How fwdpod approaches this
Fwdpod builds dedicated AI engineering pods for startups and enterprises, covering LLM development, RAG systems, AI agents and product delivery, with engineering delivery from India for buyers in the US, UK, EU and Gulf. Each pod is 4 to 6 experts working around a single measurable outcome, which is the kind of answer question 3 is looking for. We would rather be asked all twenty-five of these than none. Substitution terms, subprocessors, incident notification, IP assignment scope and exit terms are all things the team can walk you through on a first call.
To get something concrete to test these questions against, configure your AI engineering pod: answer five questions and receive a custom pod proposal within 48 hours, based on templates for Voice AI, Agentic Operations, RAG and Knowledge, Compliance AI or a custom build, ranging from 8 to 16 weeks. For pricing or written answers to any of the twenty-five, talk to the team.
Frequently Asked Questions
1. How many questions should I ask an AI development company on the first call?
Pick five to eight, weighted toward the people and the evidence: who will do the work, how success will be measured, and where your data goes. Send the security and IP questions in writing beforehand so the call is spent on follow-ups rather than first answers.
2. How do I evaluate an AI development vendor if I am not technical?
Bring one technical person to the deep-dive call, even an outside advisor, and focus your own questions on outcomes and accountability. Ask the vendor to explain how they will prove the system works in terms your business users would accept. A vendor that cannot explain its evaluation plan in plain language is unlikely to report honestly on it later.
3. What is an AI vendor security questionnaire?
It is a written set of security and data-protection questions a buyer sends to a supplier before signing. A standard IT questionnaire covers access control, encryption and incident response; an AI-specific one adds model provider data terms, prompt injection testing, agent permissions and handling of prompts and logs. The Cloud Security Alliance's AI-CAIQ is one published starting point.
4. Should these questions go into the RFP?
Some should. The security, IP and exit questions work well in an RFP or RFI because written answers can be compared side by side and carried into the contract. The delivery and technical questions work better live, where you can hear who answers and ask follow-ups.
5. What if two vendors give equally good answers?
Run a short paid pilot with both, or with the stronger of the two, on a real but contained problem with success criteria agreed in advance. How each team behaves when the first results disappoint will tell you more than any answer on a call.
6. Do I need different questions for an offshore AI development company?
The questions stay the same, but some answers need more detail: which laws govern the contract, where engineers sit, how personal data crosses borders, and how many working hours overlap with your team. Ask for those specifics in writing.
7. How long should a vendor take to answer a security questionnaire?
There is no fixed standard, but an established vendor with documented controls can usually answer a focused AI questionnaire within a week or two. Long delays, or answers that arrive without any supporting evidence, tell you something about how mature those controls are.


