It really depends on the task. Like Sabine, I’m operating on the very frontier of a scientific domain that is extremely niche. Every single LLM out there is worse than useless in this domain. It spits out incomprehensible garbage.
But ask it to solve some leet code and it’s brilliant.
The question I ask afterwords then is: is solving some leet code brilliant? Is designing a simple inventory system brilliant if they've all been accomplished already? My answer tends towards no, since they still make mistakes in the process, and it harms newer developers from learning.
It is a matter of speech. That said I have seen LLMs do brilliant work. There are just some things like the hard sciences where its understanding is only surface deep.
I should start collecting examples, if only for threads like this. Recently I tried to llm a tsserver plugin that treats lines ending with "//del" as empty. You can only imagine all the sneaky failures in the chat and the total uselessness of these results.
Anything that is not literally millions (billions?) of times in the training set is doomed to be fantasized about by an LLM. In various ways, tones, etc. After many such threads I came to conclusion that people who find it mostly useful are simply treading water as they probably have done most of their career. Their average product is a react form with a crud endpoint and excitement about it. I can't explain their success reports otherwise, cause it rarely works on anything beyond that.
Welcome to the new digital divide people, and the start of a new level of "inequality" in this world. This thread is proof that we've diverged and there is a huge subset of people that will not have their minds changed easily.
Hallucinating incorrect information is worse than useless. It is actively harmful.
I wonder how much this affects our fundraising, for example. No VC understands the science here, so they turn to advisors (which is great!) or to LLMs… which has us starting off on the wrong foot.
I work in a field that is not even close to a scientific nishe - software reverse engineering - and LLM will happily lie to me all the time, for every question I have. I find out useful to generate some initial boilerplate but... that's it. AI autocompletion saved me an order of magnitude more time, and nobody is hyped about it.
Sabine is lex Friedman for women. Stay in your lane about quantum physics and stop trying to opine on LLMs. I’m tired of seeing the huge amount of FUD from her.
But ask it to solve some leet code and it’s brilliant.