Technology

Hallucinating is the term for when ai generate incorrect information.

This is the best article I've seen yet on the topic. It does mention the "how" in brief, but this analogy really explains the "why" Gonna bookmark this in case I ever need to try to save another friend or family member from drinking the Flavor-Aid

That's exactly the issue here. ChatGPT's current training set ends right around the time the Meta Quest 3 came out. It's not going to have any discussions in there of No Man's Sky with tech that wasn't out yet.

It’s trained on internet discussions and people on the internet rarely say, “I don’t know”.

It all depends on the training data and preprompt. With the right combination of those, it will admit when it doesn't know an answer most of the time.

I've had a similar experience. Except in my case I used lyrics for a really obscure song where I knew the writer. I asked Chat GPT, and it gave me completely the wrong artist. When I corrected it, it apologized profusely and agreed with exactly what I had said. Of course, it didn't remember that correct answer, because it can't add to it update its data source.

LLMs don’t “feel”, “know”, or “understand” anything. They spit out statistically most significant answer from it’s data-set, that is all they do.

they have to provide an answer

Indeed. That's the G in chatGPT. It stands for generative. It looks at all the previous words and "predicts" the most likely next word. You could see this very clearly with chatGPT-2. It just generated good looking nonsense based on a few words.

Then you have the P in chatGPT, pre-trained. If it happens to have received training data on what you're asking, that data is shown. It it's not trained on that data, it just uses what is more likely to appear and generates something that looks good enough for the prompt. It appears to hallucinate, lie, make stuff up.

It's just how the thing works. There is serious research to fix this and a recent paper claimed to have a solution so the LLM knows it doesn't know.

The issue is: What is right and what is wrong?

"mondegreen"s are so ubiquitous that there are multiple websites dedicated to it. Is it "wrong" to tell someone that the song where Jimi Hendrix talked about kissing a guy is Purple Haze? And even pointing out where in the song that happens has value.

In general, I would prefer it if all AI Search Engines provided references. Even a top two or three pages. But that gets messy when said reference is telling someone they misunderstood a movie plot or whatever. "The movie where Anthony Hopkins pays Brad Pitt for eternal life using his daughter is Meet Joe Black. Also you completely missed the point of that movie" is a surefired way to make customers incredibly angry because we live in bubbles where everything we do or say (or what influencers do or say and we pretend we agree with...) is reinforced, truth or not.

And while it deeply annoys me when I am trying to figure out how to do something in Gitlab CI or whatever and get complete nonsense based on a single feature proposal from five years ago? That... isn't much better than asking for help in a message board where people are going to just ignore the prompt and say whatever they Believe.

In a lot of ways, the backlash against the LLMs reminds me a lot of when people get angry at self checkout lines. People have this memory of a time that never was where cashiers were amazingly quick baggers and NEVER had to ask for help to figure out if something was an Anaheim or Poblano pepper or have trouble scanning something or so forth. Same with this idea of when search (for anything non-trivial) was super duper easy and perfect and how everyone always got exactly the answer they wanted when they posted on a message board rather than complete nonsense (if they weren't outright berated for not searching for a post from ten years ago that is irrelevant).

Not all AIs, since many AIs (maybe even most) are not LLMs. But for LLMs, you're right. Minor nitpick.

It’s literally just Markov chains with extra steps

Yes!!! It doesn’t know Trump has been convicted and told me that even when I give it sources, it won’t upload to a central database for privacy reasons. 🤷‍♀️

It does remind me of that recent Joe Scott video about the split brain. One part of the brain would do something and the other part of the brain that didn't get the info because of the split just makes up some semi-plausible answer. It's like one part of the brain does work at least partially like an LLM.

It's more like our brain is like a corporation, with a spokesperson, a president and vice president and a number of departments that with semi-independently. Having an LLM is like having only the spokesperson and not the rest of the work force in that building that makes up an AGI.

LLMs are basically just really fancy search engines. The reason the initial code is garbage is that it's cut and pasted together from random crap the LLM found on the net under various keywords. It gets more performant when you ask because then the LLM is running a different search. The first search was "assemble some pieces of code to accomplish X", while the second search was "given this sample of code find parts of it that could be optimized", two completely different queries.

As noted in another comment the true fatal flaw of LLMs is that they don't really have a threshold for just saying " I don't know that" as they are inherently probabilistic in nature. When asked something they can't find an answer for they assemble a lexically probable response from similar search results even in cases where it's wildly wrong. The more uncommon and niche your search is the more likely this is to happen. In other words they work well for finding very common information, and increasingly worse the less common that information is.

If you already know what you want but want to save time writing it

IME, going to ChatGPT for code usually meant losing time, cause I'd go back and forth trying to get a usable snippet and it would just keep refactoring the same slop that didn't work in its first attempt

They're pretty reasonable for consensus-based programming prompts as well like "Compare and contrast popular libraries for {use case} in {language}" or "I want to achieve {goal/feature} in {summary of project technologies}, what are some ways I could structure this?"

Of course you still shouldn't treat any of the output as factual without verifying it. But at least in the former case, I've found it more useful than traditional search engines to generate leads to look into, even if I discard some or all of the specific information it asserts

Edit: Which is largely due to traditional search engines getting worse and worse in recent years, sadly

Best use I've had for them (data engineer here) is things that don't have a specific answer. Need a cover letter? Perfect. Script for a presentation? Gets 95% of the work done. I never ask for information since it has no capability to retain a fact.

The copilot app doesn't seem to be any better.