84,000 tokens at full price, on every step.
At 100,000 repeat planner steps a month, that is about $19,700 a month.
PostHog's AI insight generation sends an ~84,000-token static prompt to the model on every planner step. Prompt caching should make the repeat sends nearly free, but a timestamp accurate to the second sat ahead of the static prefix, so no two requests ever matched, and every call paid full input price.
The fix pins that timestamp to the start of the conversation. The prefix becomes stable, the cache starts hitting, and repeat sends are billed at the cached rate. Nothing changes for users.
prompt = construct_messages( utc_now=timezone.now(),+ conversation_start_dt=state.start_dt, )The same thing in your product
Anything that changes with each request, such as a timestamp, an ID or a user name, stops caching for everything that comes after it in the prompt. It is the most common leak we find: we found it in Chatwoot and Metabase too. If your prompts are long and mostly fixed, it is the first thing we check.