A buyer's checklist for AI agent platforms: twelve questions, and which answers matter
Twelve questions for any AI agent platform, what a good answer to each looks like, and an honest scoring of UtilitySpaces including the three it answers weakly.
Last updated
9 minute read · Written by the team that builds UtilitySpaces. More about UtilitySpaces.
A buyer’s checklist for an AI agent platform is twelve questions about credentials, approvals, the audit log, write permissions, cost visibility, data residency, exit, pricing shape, integrations, MCP, what happens when the agent is wrong, and support. For each, this piece says what a good answer looks like, and then answers it for UtilitySpaces — including the three where the answer is weak, named as such rather than skipped.
The list is ordered by how much a bad answer costs you later, not by how often vendors talk about it. Model choice is not on it, because every platform lets you choose a model and none of them will be wrong about it for long. Permissions and records are where the durable mistakes live.
Questions one to three: who holds the keys, and who says yes?
- How does the agent authenticate to my applications? A good answer is a per-application, scoped credential — an OAuth grant or a key you minted for this purpose — stored encrypted and bound to your tenant, never your own password and never a parked browser session. Ask whether the credential is revocable on its own and what is logged when it is created, changed and removed. The four options and their blast radius are ranked in the piece on giving an agent app access safely.
- What stops for a human, by default, on day one? A good answer names a default: every write, or every write above some stake, stops for a person before it takes effect, and the vendor can tell you who is allowed to change that and whether the change is logged. “You can add an approval step” is a legitimate answer for a workflow builder and a weak one for an agent platform, because it puts the safety property on whoever forgot to add it. The three models are laid out in what human in the loop actually means.
- What does the audit log prove, and can it be edited? A good answer is an append-only record of every action and every approval — the tool, the arguments, the actor, the time — that distinguishes a person from an agent and that cannot be silently rewritten. Ask on which plan it is available; a log that appears only on the top tier is a log most of your workspaces will not have.
Questions four to six: what can it change, what does it cost, where does it live?
- Is there a write allowlist, and does it fail closed? A good answer is that writes are permitted only for connectors somebody explicitly cleared, that anything the policy does not recognise is treated as a write and refused, and that destructive operations are off unless deliberately enabled. “The agent can do whatever the token allows” means the vendor’s scope is your only control.
- Can I see what every run cost, and can I cap it? A good answer attributes cost per run, per agent and per model at stated rates, lets you set a budget that actually stops work rather than sending an email, and tells you how complete its own numbers are. Ask whether you can bring your own provider key so the bill is under your own account.
- Where is my data processed, and can I choose? A good answer names the hosting region, lists the sub-processors — including which model providers see your prompts — and offers a choice of region if that is a contractual requirement for you. Ask separately about the model provider, because a platform hosted in one place may route prompts somewhere else.
Questions seven to nine: can I leave, what am I paying for, does it reach my apps?
- How do I get my data out, and what is deleted when I go? A good answer is an export you can request or run, a deletion path that tells you exactly what it removes, and a stated exception for the records a log must keep. Ask what happens to data when an invoice is late: retained for you to export, or gone.
- What is the unit of pricing? A good answer is one you can predict in advance. Per-seat pricing penalises adding people; credit metering penalises using the product; per-workspace pricing with a stated limit on agents and schedules is the easiest to forecast. None is wrong, but you should know which you are buying and what it rewards. Never accept a number from a blog post — read the vendor’s pricing page, ours included: pricing.
- Does it connect to the applications I actually use? A good answer is a named list you can read before buying, with the tools and the scopes each connection asks for printed, plus a generic escape hatch — HTTP or MCP — for anything unlisted. A large catalogue genuinely matters if your stack is unusual; a small one with printed scopes genuinely matters if you care what each connection can reach. Decide which you are.
Questions ten to twelve: MCP, the bad day, and the human at the other end
- Does it speak MCP, and what happens to tools it discovers? A good answer is that you can connect any Model Context Protocol server, that its tools are discovered at connect time, and — this is the part to listen for — that discovered tools are treated as writes and quarantined until somebody clears them, because the platform cannot know what an unfamiliar tool does.
- What happens when the agent is wrong? A good answer describes a concrete path: the bad write was stopped at an approval, or it was made and the log shows exactly what was sent to which system with which arguments, and here is how you reverse it. Ask also what the platform does about prompt injection, since everything an agent reads is untrusted input; the credible answer is that permissions are narrow enough that an injection cannot reach anything expensive, not that the prompt is clever.
- Who answers when I write in, and how fast? A good answer is a named channel, stated hours and a stated reply commitment. Ask whether a person replies. Ask what is not covered. A vendor that states a modest commitment and keeps it is a better bet than one that states none.
How does UtilitySpaces answer the twelve?
Scored honestly. Three of these are weak and say so; the table is the short version and the notes below it are the long one.
| Question | UtilitySpaces today | Strength |
|---|---|---|
| 1. Credentials | Per-connection OAuth or key, encrypted and bound to the workspace; never your password | Strong |
| 2. Approvals by default | Every new connection holds connector writes for a person in Approvals; an admin can set a connection to run without asking, and that change is logged; deletes always wait | Good; later flow steps do not resume on their own |
| 3. Audit log | Append-only, hash-chained, on every workspace from day one | Strong |
| 4. Write allowlist | Fails closed; unrecognised tools count as writes; a policy per connection, set by an admin; DELETE always waits for a person | Strong |
| 5. Cost visibility | Per run, agent and model at a dated rate card; budgets that pause work; bring your own key | Good; no per-person attribution |
| 6. Data residency | Sub-processors listed; no choice of region | Weak |
| 7. Exit and export | Export and deletion on request, answered by a person; data retained if an invoice is late | Good; not self-serve |
| 8. Pricing shape | Per workspace, no per-seat charge, limits stated on the pricing page | Predictable |
| 9. Integrations | Seventy-odd definitions with tools and scopes printed, plus HTTP and MCP connectors | Weak on count, strong on disclosure |
| 10. MCP | Any MCP server; discovered tools write-quarantined by default | Strong |
| 11. When the agent is wrong | Held for a person or refused by the write policy, or logged with tool and arguments; narrow permissions as the injection defence | Strong |
| 12. Support | Email, stated hours, reply within one business day, a person answers | Modest and kept |
Where the answers are weak, and what that means for you
No third-party security certification. It is not on the list above because it is not one of the twelve, but it is the question many procurement teams ask first, so: none has been done and none is claimed. If an audited certification is a hard requirement, that is a disqualifier today and you should not spend an hour on a demo finding out. Two-factor authentication (TOTP) and single sign-on over OpenID Connect are available, and a workspace can require them; there is no SAML or SCIM.
No choice of data region. The sub-processor list names who sees what, including that prompts go to the model provider you choose and only to that provider if you bring your own key. But if your contract requires processing in a named jurisdiction, UtilitySpaces cannot promise it.
Integration count. Seventy-odd application definitions is a fraction of what the large catalogues carry. The generic HTTP and MCP connectors reach most authenticated APIs, and each listed integration prints its tools and scopes on its own page before you connect. If your stack is unusual and you do not want to configure a generic connector, a bigger catalogue is the right buy.
The strengths are three of the four questions about control: credentials, the log and the write policy. The approval default is good rather than strong, because an admin can set a connection to run without asking, and a flow that stops at an approval does not resume its later steps on its own. All four are covered on the connected apps page, the approvals page and the cost tracking page, each of which ends with a section on what it deliberately does not do.
How to use the list
Send the twelve questions to every vendor on your shortlist and ask for answers in writing. Score each as strong, partial, weak or absent. Then decide which absences are disqualifying for you — a certification, a region, a connector — and strike those vendors before you open a single demo, because the demo will be persuasive and the absence will still be there. What remains is a comparison on the control questions, which is where the platforms actually differ.
If UtilitySpaces survives your strikes, open a demo workspace: the full product for about an hour, no card, no connected apps, enough to see the gate and the log rather than read about them. Plans are on the pricing page, and support@utilityspaces.com replies within one business day, including to the twelve questions above if you would rather have them answered in writing first.