When agents do a lot of your work, it helps to know how much they use and which one does best. Analytics answers both from data already on your Mac.

Cost and tokens

  • Cost: what the work would cost at public API list prices, broken down by type and speed. Subscriptions bill differently, so read it as a comparison, not a bill.
  • Tokens: how many tokens each tool and model processed, hourly or daily.
  • Limits: where you stand against your allowances.

The numbers come from each tool’s own logs.

How each agent does

The Harnesses scorecard compares the chats you run in DevMode, for each tool and model:

  • time per turn, typical and slow;
  • how turns ended, including failed and interrupted;
  • tokens per turn and how much came from cache;
  • friction: how often your own messages push back, ask it to stop or repeat themselves.

Only counts are kept for friction, never your messages.

Questions

Is the cost what I pay?

No. It is what the tokens would cost at public API list prices. Subscriptions bill differently, so it is a way to compare, not a bill.

Where do the numbers come from?

From each tool’s own logs on your Mac.

What is friction?

A count of your messages that push back, ask to stop, repeat the last one or are written in capitals. Only the counts are kept.

Try DevMode on your Mac

Free for Apple silicon Macs running macOS 15 or later.

Download for Mac