The mistake there is to point it at code to figure out performance optimizations.
The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.
The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.
> The place to find them would be performance profiles, query plans, telemetry.
Tbh, once this information is available (which is the tricky part), in 99% of cases there's really no AI needed to analyze the data, since the 'low hanging fruits' will usually stand out anyway. And once you get into the area where optimization hotspots are no longer obvious, you're already deep in the diminishing returns area and optimizations for one use case or hardware configuration may degrade performance on others. That's my experience anyway.
> there's really no AI needed once this information is available
Most people are not arguing that problems are too tricky for a human to solve once presented with it, but AI can look at 1000 things at the same time across a whole code base and then just come back with the results a human can review.
True, but at least IME, once you get beyond fixing the 'obvious bugs' (how many there are depends on the initial quality of the code base - but I agree that AI is really useful to find those), you'll get into a murky grey area of "maybe false positives" which require a lot of time to validate. As a result you sink a lot of time creating and tweaking code-base-specific rules to try filtering out false positives, and IMHO this is exactly the tipping point where the whole thing becomes pointless because it becomes a bureaucratic monster.
E.g. a good code analysis tool needs to work predictably at button press on any code base.
Still better than nothing of course, e.g. I actually think bug scanning / code analsysis is indeed the one area where LLMs are actually useful, but it suffers from the same 'diminishing returns' problem as traditional approaches, if not worse (e.g. still no silver bullet, but a mostly useful additional tool in the toolbox).
>in 99% of cases there's really no AI needed to analyze the data
As I've seen it, there's a large chunk of those getting the most out of AI doing so because they weren't aware of or couldn't be bothered by already existing, more reliable alternatives.
It really depends on the context. On a pure computational code, it can work really well.
I recently optimized some code asking Claude to "make it faster" 2 different programs. For each, it wrote a benchmark, gathered initial data. Emitted some hypothesis and measured data around them with profilers, then did some changes (behind feature flags), checked the output was identical in either side and through profiling that the right code path was taken, and then benchmarked both.
And it did that for multiple successive changes in both codebases. Some changes were purely algorithmic (better complexity), some were related to tradeoffs between memory and computation (caching intermediate results better), some were about creating less intermediate objects to relieve the memory pressure, some where about a better memory layout to improve data locality.
The annoying part is that some of the code that was optimized was generated by Claude in the first place, so it's a bit frustrating that it didn't do those in the first place. But I guess, my iterative approach to making those tools didn't work on big enough data sets at first where it would have been an issue.
Not really a rebuttal. There's a lot of low hanging fruit in code bases were you can read code and see what's making things slow. Like the parent comment stated, you can visually tell if there's a lot of allocations in a certain part, and easily see if those allocs are present in a loop and often reason simply how many times that loop can be called.
I can see in the code when data layouts aren't optimal, and fix that.
There's a lot of optimizations that need more of a deep dive, but you can get a lot of gains by just reading and reasoning about your code/data.
EDIT: To add, there are cases were you specifically can't read code and understand performance issue, but you should first ask, is that because you just don't understand the APIs/Libs/tools you're using, or is it fundamentally difficult. For example, often at work I see people complain about their torch code being slow and needing to bust out a profiler, but often those people just don't understand how tensor operations work internally so of course they can't reason about the code and see the way they're using the lib is suboptimal.
That these performance hogs were in production reinforces my point that you can't spot slow code just by looking at it, unless you already know that it's slow.
I even have an example myself, but I'm not sure I can share it in detail. It was a common idiom that was used throughout the code. After tracing a performance problem to a specific function, it was immediately obvious to me that the common idiom was the problem in this function, and how to fix it very easily. But only because I traced a performance problem to this function. Otherwise I would have skimmed over it.
Are you serious? Of course you can. Some hash algorithms are slower than others, you can read that and know that. Allocating in a hot loop is probably slower than allocating outside of it. Duh? I can't believe this even has to be justified, of course you can look at code and get an idea of its performance properties lol
Obviously you want empirical evidence to justify changes, but like... duh, you can read code and understand how it executes.
Sometimes when you look at code you know is slow, you can immediately see why it's slow. Obviously you didn't see it when you wrote it, which disproves your hypothesis.
What? Sorry, I genuinely have no idea what you're talking about. Is it possible to look at code and intuit, just by reading, how to speed it up? Duh, yes. If you disagree, lol.
Keep in mind that I've never stated that all performance issues are findable through code review. Just that there are many that are.
> You can very easily spot performance issues through code.
If you think that this is false, there's literally nothing you can contribute.
I can't take this seriously. The claim "you can look at SOME code and see how to optimize it" is trivially true, and saying "you can't look at ALL code and see how to optimize it" changes nothing.
Exactly, you have to find the slow code first, with profiling, and then sometimes it is obvious why that code is slow. It is unlikely you can just look at a project and find the slow code in it.
You absolutely can. I've given enough examples that are obviously true. If you can't read code and reason about its performance, that is a personal issue.
The place to find them would be performance profiles, query plans, telemetry. The guidance for perf still applies, measure before and after change.
The issue is that the code often does not contain the information to do a perf optimization. Eg. you can't tell your cache size, the volumes of data in your DB or the latency of your network through just the text. Should you provide this context, you can get better results.