I've mentioned this before, but I'd be curious about a BOTEC of the cost of using reasoning models and things like Perplexity that do a ton of web search and inference.
No time like the present, I guess!
*Crude guessing:*
- a Perplexity search typically runs 3-5 searches, turns up 20-100 links, does some amount of inference with each in context, then does a final bout of inference to summarize results. Crudely this implies it's something like 20-100x as costly as a single model query.
- I assume if you use their default proprietary model, it's smaller & more efficient than ChatGPT etc. But I often use Claude via Perplexity instead, bc I assume it has better reasoning quality and I want that for analyzing text results.
*Does that match up with the financial data?*
Perplexity subscriptions cost about the same as ChatGPT/Claude, but I don't seem to run into usage limits like I do with Claude. From the scant info online, their economics are similar to e.g. OpenAI, which iirc loses about $1-2 for each dollar of revenue. (In 2024, Perplexity had a burn rate of $65m/year on $35m of revenue. Source: https://www.theinformation.com/articles/google-challenger-perplexity-growth-comes-high-cost )
That's a little surprising, if each Perplexity query is so much more costly.
The answer might be that most people aren't crazy power users. If you make 1-2 Perplexity queries per day, that's similar to an extended conversation with ChatGPT. And despite having a subscription, and some days making a bunch of Perplexity searches for research, I probably average less than that.
*OK, so does this change the argument that all this is fine?*
- Well, it saves me a bunch of time and thinking. I think it's good value for money, and insofar as what I do is valuable for society that swamps the carbon cost. (Consequentialism, icky.)
- Your little red bar for 50k ChatGPT queries roughly corresponds to "ask Perplexity 500-2500 fewer questions", which might correspond to 1-5 years of usage for me. Seems decently smaller than other lifestyle choices but no longer just hilariously out of scope.
- Another way to see this is, we're still just running an LLM for a share of the day, maybe as much as a few LLM-hours on heavy usage days if we account for parallel instances. It looks like GPUs use between a few hundred watts and a few kilowatts. So with aggressive assumptions this is a few kilowatt-hours = maybe a dollar of electricity a day. On average, maybe a tenth of that? Similar to e.g. running the dishwasher every day (~1 KWh). That's a nontrivial part of my electric bill but not a majority of it.
I ate so much of the Trader Joe's chili that my roommates incorporated it into a forfeit in one of our drinking games.
Andy, fantastic! Long Live TJ's Veggie Chili!
On the very off chance that any of your readers don't already, everyone should read Hannah Ritchie. She has another great one out overnight.
Also, https://mattball.substack.com/p/carbon-cult-cruelty
What is the carbon footprint of self-promotion? ;-)
I've mentioned this before, but I'd be curious about a BOTEC of the cost of using reasoning models and things like Perplexity that do a ton of web search and inference.
No time like the present, I guess!
*Crude guessing:*
- a Perplexity search typically runs 3-5 searches, turns up 20-100 links, does some amount of inference with each in context, then does a final bout of inference to summarize results. Crudely this implies it's something like 20-100x as costly as a single model query.
- I assume if you use their default proprietary model, it's smaller & more efficient than ChatGPT etc. But I often use Claude via Perplexity instead, bc I assume it has better reasoning quality and I want that for analyzing text results.
*Does that match up with the financial data?*
Perplexity subscriptions cost about the same as ChatGPT/Claude, but I don't seem to run into usage limits like I do with Claude. From the scant info online, their economics are similar to e.g. OpenAI, which iirc loses about $1-2 for each dollar of revenue. (In 2024, Perplexity had a burn rate of $65m/year on $35m of revenue. Source: https://www.theinformation.com/articles/google-challenger-perplexity-growth-comes-high-cost )
That's a little surprising, if each Perplexity query is so much more costly.
The answer might be that most people aren't crazy power users. If you make 1-2 Perplexity queries per day, that's similar to an extended conversation with ChatGPT. And despite having a subscription, and some days making a bunch of Perplexity searches for research, I probably average less than that.
*OK, so does this change the argument that all this is fine?*
- Well, it saves me a bunch of time and thinking. I think it's good value for money, and insofar as what I do is valuable for society that swamps the carbon cost. (Consequentialism, icky.)
- Your little red bar for 50k ChatGPT queries roughly corresponds to "ask Perplexity 500-2500 fewer questions", which might correspond to 1-5 years of usage for me. Seems decently smaller than other lifestyle choices but no longer just hilariously out of scope.
- Another way to see this is, we're still just running an LLM for a share of the day, maybe as much as a few LLM-hours on heavy usage days if we account for parallel instances. It looks like GPUs use between a few hundred watts and a few kilowatts. So with aggressive assumptions this is a few kilowatt-hours = maybe a dollar of electricity a day. On average, maybe a tenth of that? Similar to e.g. running the dishwasher every day (~1 KWh). That's a nontrivial part of my electric bill but not a majority of it.
GPOY: https://chatgpt.com/s/m_688733263bd08191a9e7e339cdb66fbe