At my company there's a lot of discussion about AI and complaints that people run out of tokens within a day, but zero results are shown. No measurable (or measured) gains. Or nothing that people are willing to talk about, in any case.

I’ve been asking since months on HN for proofs that companies using AI see positive ROI from it. So far I didn’t get a single concrete example

I hope to see more of this across the industry.

I definitely make use of AI but in my experience I almost always could have done it better myself, the places where I threw AI at the problem I didn't care about the results being good, only good enough.

When we see memory and compute requirements for version x+1 of software decrease instead of increase I will happily say AI is the oracle people proclaim it to be.

In $COMPANY, for the mid-yearly review, the employees were asked whether their AI usage was 1/ efficient, 2/ adoptive (integrated in the way they work) or 3/ transformative.

Saying “never used it” or “I tried and it was useless” was literally not possible.

You'd have to be a pretty stubborn software engineer to not find AI useful for anything at this point, unless they're making you use Github Copilot or something far behind the frontier. You don't have repetitive tests to write? You never need to write a script to run something?

And you'd have to be a pretty pig-headed exec to assume that the only experience an employee could possibly have with LLMs is positive.

I don’t use AI for ethical reasons. I don’t want to support companies that are destroying the planet and inflating hardware prices.

Currently the only thing I personally find LLMs useful for are code reviews.

"It can't be that stupid—you must be prompting it wrong." - Ed Zitron

> At Amazon we have many forums [...], but none to share AI failures or disappoinments.

I come to HN to read about AI failures and disappointments.

This is kind of a weird take.

If you were the CEO of Amazon, would you be setting up channels for people to talk about their AI failures? The general way technology is deployed is that we try to find ways to make it work, because those are the most interesting. We're not as interested in all the ways it doesn't work.

From my perspective, some people are trying to use AI in the same way somebody might use a laptop to paddle a canoe. Sure, you can do it, but it's not a good idea. The fact that it doesn't work well is not particularly interesting.

If I steelman your position, I guess the ideal repository would be a set of cases where it's known to work well and a set of cases where it's known to not work well. A little bit like ProtonDB or SteamDB, perhaps.

>The general way technology is deployed is that we try to find ways to make it work, because those are the most interesting.

That's not the way AI is being deployed. At all.

Sure it is. It's just like the .com bust: people didn't know what the internet was good for, so they did a lot of stupid stuff with it before settling on the most interesting uses.

No, AI is more like: people do know what it is good for, but that's immaterial because hundreds of billions of dollars of debt were poured into it, such that the only glimmer of hope of getting an ROI out of it is force-feeding AI everywhere, good or bad, and normal people are sick of that.

That's fair. I still think there big similarities (people with money seeing it as the future and investing heavily, even if in reckless ways), but I agree there's a forcefulness to it this time that wasn't present before, for the reasons you outline.

> If you were the CEO of Amazon, would you be setting up channels for people to talk about their AI failures?

I would set up channels to talk about AI, encouraging both successes and failures, with proofs required for both, and punishing people who intentionally misreport on either.

Wow. Setting up a forum for discussion where people could potentially be punished for what they say is not going to encourage discussion.

That's fine, lies are more harmful than anecdotes are beneficial. There's no forum to discuss the best IDE and people still manage, in reality there's no need for a forum around AI.

Well, you're at Amazon. Is the Dev Improvement discussion group still around? I feel like that was a pretty good place to learn about neat stuff (in my case, Emacs-related). I enjoyed the discussion.