yeah thats is my biggest issue - im okay with paying 20-30% more but what is the ROI? i dont see an equivalent improvement in performance. Anthropic hasnt published any data around what these improvements are - just some vague “better instruction following"
Its enshittificating real fast. They'll just keep releasing model after model, more expensive than the last, marginal gains, but touted as "the next thing". Evangelists will say that they're afraid, it's the future, in 6 months it's all over. Anthropic will keep astroturfing on Reddit. CEOs will make even more outlandish claims.
You raised a good point, what's a good metric for LLM performance? There's surely all the benchmarks out there, but aren't they one and done? Usually at release? What keeps checking the performance of those models. At this point it's just by feel. People say models have been dumbed down, and that's it.
I think the actual future is open source models. Problem is, they don't have the huge marketing budget Anthropic or OpenAI does.
This is most likely trajectory I fear. It reminds me a lot of Oracle, where they rebrand and reskin products just to change pricing/marketing without adding anything.
My only problem lumping them in is a huge part of those products are the UI and UX. It’s what you interact with as a user. So, a reskin with no new features is a valid version in my opinion.
It’s not always for the best mind you, but I do think major changes to the GUI or UX do deserve a version indicator given they are relatively infrequent. Without some requirement of course.
Nobody buys Oracle because of the UI or upgrades because of a new UI. It’s the underlying tool that matters and if my ugly version is working there is no incentive to upgrade to a prettier version.
The other thing is most people don't really care about price per token or whatever but how much it will cost to execute (successfully) a task they want.
It doesn't matter if a model is e.g. 30% cheaper to use than another (token-wise) but I need to burn 2x more tokens to get the same acceptable result.
Anthropic intruduces fake tool calls to prevent distillation of their models. Others still distill. Anthropic distils third party models. Claude now hallucinates tools.
choice is often a great dark-pattern (lack of choice is too but...). Choices generally grow cost to discover optimality in an np way. This means if the entity giving choice has more ability to compute the value prop than the entity deciding the choice you can easily create an exploitive system. Just create a bunch of choices, some actually do save money with enough thought but most don't, and you will gain:
People that think they got what they wanted, the feature is there!, so they can't complain but...
People that end up essentially randomly picking so the average value of the choices made by customers is suboptimal.
Once you've seen a few results of an LLM given too much sway over product decisions, 5 effort modes expressed as various english adjectives is pretty much par for the course
I find 5 thinking levels to be super confusing - I dont really get why they went from 3 -> 5