Average cost per attempt, without → with RTK:
Claude/Fable: $1.72 → $1.64 (~5% cheaper)
DeepSeek: $0.115 → $0.121 (~5% more expensive)
Almost all Claude savings came from a single task.
Excluding it, savings were under 1%.
It took me a few rereads to parse out the top-line.
This article really buries the lede.
Cool, there is no check on the direction of the camera. Various camera's I checked do not seem to point in the direction of the sun during totality. It does filter candidate cameras down though.
This video benchmarks slop-style indicators with different skill/prompt solutions including the STE skill vs. George Orwell's six rules of writing prompt: Orwell came out on top overall.
Additional bonus: it doesn't add much more tokens to input context. I have compared prose prompts with these rules and without and I am liking the results.
1. Never use a metaphor, simile, or other figure of speech which you are used to seeing in print.
2. Never use a long word where a short one will do.
3. If it is possible to cut a word out, always cut it out.
4. Never use the passive where you can use the active.
5. Never use a foreign phrase, a scientific word, or a jargon word if you can think of an everyday English equivalent.
6. Break any of these rules sooner than say anything outright barbarous.
I don't know if I have been damaged by reading so much AI, but I swear the narrator of that video had the script for it written by Claude without any of those rules applied. There are so many platitudes and AI-isms in the way he speaks:
"now the honesty - those translation gains were real but small"
"and here is the honest part"
"same disease but with different symptom"
"so that brutal 3% number was a Claude quirk, not a law of nature"
Same person that was mocking the hands in image generation in 2023, is the same person that was saying 'hands are fixed but it can't generate "the red dog jumps over the jump rope held by the blue pelican while juggling 5 balls"' in 2024, is the same person that posted this.
In ML, you want to test general capability of a model (generalizability), because you want it to perform well on unseen tasks. In that benchmark, the literal reference is leaking through web search, the agent can see the matching real codebase online and the commits so that's test set leakage. I know no programmer that was ever paid to rewind an existing codebase to a previous commit and implement a feature/fix a bug that exists in the next commits.
reply