TL;DR. Every vendor in 2026 says "AI teammate." Most of them ship an assistant. These eight questions, plus the answer patterns under each, tell you which one you are actually buying before you sign an annual contract.
The AI employee category exploded in 2026. Viktor's Series A, Pancake's launch, Adapt's growth, Dust's Series B, plus a dozen new entrants every YC batch. Every product page uses variations of the same words: "AI teammate," "AI employee," "AI coworker," "always-on," "hire not tool."
Most of them are one of the four categories from our previous piece: tools, chatbots, assistants, or actual AI hires. The buyer's job is to tell them apart before signing. Below each question is what a real AI hire will answer versus what a marketing-only AI hire will dodge.
Use it during sales calls, product research, or reading landing pages. If a vendor cannot answer clearly, you have your answer.
The eight questions at a glance
| # | Question | What it tests |
|---|---|---|
| 1 | What is the smallest task the AI can own end-to-end? | Ownership |
| 2 | What memory does it have, and where does the memory live? | Memory |
| 3 | Which tools does the AI act on, not just read? | Actions |
| 4 | How does the AI hire fail, and what happens when it does? | Failure modes |
| 5 | How do I create an AI employee for a role I do not see in your product? | Custom roles |
| 6 | How is pricing structured as we grow? | Pricing |
| 7 | What security posture does the vendor actually have? | Security |
| 8 | Show me a real customer running this in production. Can I talk to them? | References |
1. What is the smallest task the AI can own end-to-end?
The word to listen for is "own." Not "assist with." Not "help draft." Own.
Good answer pattern: "It can own a support ticket end-to-end. Read the incoming message, look up the customer's history, draft the reply, send it, and mark the ticket resolved. You review the responses that mattered."
Bad answer pattern: "It helps your team by suggesting responses that a human then reviews and sends."
The bad answer describes an assistant. It is a useful product. It is not an AI hire.
2. What memory does it have, and where does the memory live?
Memory is what separates an AI hire from a chatbot. Every vendor should have a clear answer to what their memory model is.
Good answer pattern: "The AI hire reads Slack, Gmail, Drive, and your tools continuously. Everything it learns goes into a shared memory that the hire and any other agent can query. Memory is scoped by permission. You can inspect what it knows about any topic."
Bad answer pattern: "It uses your data to give better answers." Vague. No commitment to persistence, structure, or inspectability.
Follow-up: ask to see the memory. A real vendor can show you. A vendor that cannot is either hiding a weak implementation or the memory does not exist as a structured store. More on the distinction in company memory vs company brain.
3. Which tools does the AI act on, not just read?
Every vendor lists integrations. Read the list carefully. Distinguish between "reads from" and "acts on."
Good answer pattern: "It reads from and acts on Slack, Teams, Gmail, GitHub, Sentry, Linear, HubSpot, WhatsApp, and Notion. It also reads from Amplitude, PostHog, PagerDuty, Datadog, and any MCP-compatible tool."
Bad answer pattern: a wall of integration logos with no distinction between passive and active.
The distinction matters because "acts on" is where the AI hire actually saves you time. Reading is table stakes. Acting is the product.
4. How does the AI hire fail, and what happens when it does?
An AI hire that never fails is either not being used enough to have data, or the vendor is not telling you.
Good answer pattern: "It fails in three main ways: it drafts a wrong reply and you catch it in review; it flags something as an anomaly that is normal; it asks for approval on an action that could have been autonomous. Failures are logged and the failure rate is visible per hire. High-leverage actions are always human-in-the-loop; low-risk actions run autonomously with retroactive review."
Bad answer pattern: "It very rarely fails" or "our accuracy is 99%."
You want a vendor that owns failure modes and lets you calibrate the autonomy line, not one that pretends failure does not exist.
5. How do I create an AI employee for a role I need but do not see?
This is the custom role test. Some vendors give you a fixed menu (Viktor is the clearest example: analyst, ops lead, engineer). Some let you configure your own but require code, like most agent platforms. Some let you write a job description in plain English and get an AI hire out, which is how Qualia works.
- Fixed-menu vendor: "Our menu covers 80% of asks. If your role is not in it, we may add it to the roadmap. For now, adapt the closest role."
- Custom-code vendor: "Our platform lets your engineering team build any agent. Here is the SDK documentation."
- User-defined vendor: "Write the role in a sentence. The product parses it into a spec, sets up the agent, and deploys it within minutes."
All three answers can be right for the right buyer. The wrong answer is a vendor that dodges the question or promises yes to anything without explaining how. Compare the approaches in Qualia vs Viktor.
6. How is pricing structured as we grow?
Pricing surprises are the top reason AI hire deployments die in month three. Ask now.
| Model | Vendor examples | Best for | Watch out for |
|---|---|---|---|
| Per-user + credits | Viktor, Lindy Pro | Individuals, small teams | Credit burn on heavy use |
| Per-agent | Some platforms | Teams with 1-2 agents | Cost stacks with more hires |
| Portfolio-based | Qualia | Teams with multiple hires | Requires up-front commitment |
| Enterprise-only, quoted | Glean, Sana, Adapt for large deals | Large orgs | No transparent pricing |
| Freemium + Pro | Notion AI, Granola | Individuals, evaluation | Pro caps that squeeze at scale |
The model that fits you depends on your team shape. What you want is transparency: a vendor who tells you exactly how the bill grows over the next 12 months as usage grows.
7. What security posture does the vendor actually have?
Do not accept "we are enterprise-ready" as an answer. Get specific. Ask for:
- SOC 2 status: Type I, Type II, in progress, or none
- Data encryption at rest and in transit
- Access controls and the permission model
- Data retention policy
- Where data is hosted: US, EU, self-hosted option
- Compliance certifications: ISO 27001, HIPAA, GDPR handling
Not every buyer needs all of these. Regulated industries need most. Small teams may need only encryption and reasonable access controls. Know what you need before the call so you can filter the answers. Ours is documented on our security page.
8. Show me a real customer running this in production
The last test. Every AI hire vendor should have at least three customers using the product in production at scale, and should be able to introduce you to one.
Good answer pattern: "Yes. Here are three customers with your team shape. I will make the intro this week."
Bad answer pattern: "We have many customers but they prefer to stay anonymous."
Reference customers are the single most valuable input in evaluating an AI hire. Talk to them for 30 minutes. Ask what worked, what failed, how long setup took, whether they would buy again.
Bonus: the tie-breaker question
If you are between two products after these eight questions, ask each vendor: "Can I write a job description for a role I actually need and see what your product does with it, live, on this call?"
The vendor that says yes, does it in front of you, and produces something you would trust to run tomorrow is the vendor to buy. The vendor that says "let me get back to you with a customized demo" is the vendor whose product does not yet exist the way their landing page describes it.
The bottom line
The AI employee market in 2026 is loud. Every vendor claims the same properties. The eight questions above are the filter that separates the actual AI hires from the assistants, tools, and chatbots wearing AI teammate branding.
Use them. If you are on the vendor side reading this, notice which questions your product answers cleanly and which are dodges. Fix the dodges.
FAQ
How long should the evaluation process take?
Two to four weeks from first sales call to signed contract for most teams. Longer for enterprise procurement. Shorter for solo operators buying a personal hire.
Is it OK to run parallel trials of multiple AI hires?
Yes, and it is often the right move. Give each vendor the same job description or task. See which one produces the output you would trust.
How much of the evaluation should be technical vs product?
For an AI hire, roughly 30% technical (integrations, security, memory) and 70% product: does it actually own the outcome, is the customer happy, is the failure rate acceptable.
Should I evaluate with real data or dummy data?
Real data. Dummy data hides the memory quality issues that only show up when the AI hire has to reason across your actual history.
What if the vendor requires an NDA to see the product?
For most AI hires today, a full product demo should not require an NDA. If a vendor requires one before showing you the basics, they are usually optimizing for enterprise-only sales and may not be a fit for smaller teams.
Can I switch AI hire vendors after a few months?
Yes, but not without cost. Memory does not transfer between vendors, and custom-configured agents do not transfer. Budget two to four weeks for re-onboarding when switching.
What is the biggest mistake buyers make?
Focusing on features instead of ownership. Every vendor has a great feature list. The question that matters is: what does this AI hire actually own, end-to-end, that I no longer have to do?
Run the tie-breaker on us
Write the job description. Qualia becomes that AI hire, working from one shared company brain across every tool your team already uses. Book a demo and test it on your own data.
By Doğan Turan, Co-founder, Qualia ·