The buying decision
If exact model identity is essential, require documented pinning and routing behavior. If you accept a router, compare the quality and budget of the complete task instead of assuming it matches a named direct model.
Separate routing from model identity
A router may intentionally select among several models. A fixed-model endpoint makes a different promise. Ask what happens on failure, whether fallbacks can change the model, and what the response or dashboard reports. The model string in an API response is a provider assertion, not independent proof of which weights ran. Standard Compute's standardcompute identifier is a router; use the dashboard and current model documentation to understand your selection.
Start with a real tool loop
Use a disposable copy of a small repository. Ask the agent to read one file, make one bounded change and run an acceptance check. Confirm the tool arguments parse, the tool result returns to the model and the final diff matches the request. Roo's current native-tool requirement and Continue's model-capability settings show why generic chat compatibility alone is insufficient.
Run a comparison you can reproduce
Choose the tasks before seeing either result. Use the same starting commit, task prompt, available tools and stopping conditions for each provider.
- Task A: fix a bug with an existing failing test.
- Task B: make a bounded refactor while keeping the tests passing.
- Task C: add a feature with an explicit acceptance check.
- For every attempt record acceptance, repair turns, elapsed time, all model requests and allowance or cash consumed.
- Repeat the sample before making a broad quality claim. Keep failed attempts in the report.
Price useful work, not a perfect demonstration
Divide the cost of all attempts by accepted outcomes. Keep monthly fees, unused allowance and cash overages visible. A gateway can be a good fit even if it is not identical to a direct provider, provided its actual results meet your requirements. Equally, a low bill does not compensate for changes you cannot trust. The downloadable worksheet records evidence; it contains no prefilled performance claims.
Run your own comparison
Use the same task, starting revision and acceptance check. Record every retry and failed attempt so the result reflects the work you actually paid for.