Technology

59566 readers

3235 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

492

Apple study exposes deep cracks in LLMs’ “reasoning” capabilities (arstechnica.com)

submitted 1 month ago by misk@sopuli.xyz to c/technology@lemmy.world

109 comments fedilink hide all child comments

you are viewing a single comment's thread
view the rest of the comments

[–] CombatWombat1212@lemmy.ml 56 points 1 month ago (1 children)

So do I every time I ask it a slightly complicated programming question

[–] Saik0Shinigami@lemmy.saik0.com 19 points 1 month ago (1 children)

And sometimes even really simple ones.

[–] werefreeatlast@lemmy.world 9 points 1 month ago (2 children)

How many w's in "Howard likes strawberries" It would be awesome to know!

[–] Saik0Shinigami@lemmy.saik0.com 9 points 1 month ago* (last edited 1 month ago) (3 children)

So I keep seeing people reference this... And I found it curious of a concept that LLMs have problems with this. So I asked them... Several of them...

Outside of this image... Codestral ( my default ) got it actually correct and didn't talk itself out of being correct... But that's no fun so I asked 5 others, at once.

What's sad is that Dolphin Mixtral is a 26.44GB model...
Gemma 2 is the 5.44GB variant
Gemma 2B is the 1.63GB variant
LLaVa Llama3 is the 5.55 GB variant
Mistral is the 4.11GB Variant

So I asked Codestral again because why not! And this time it talked itself out of being correct...

Edit: fixed newline formatting.

[–] realitista@lemm.ee 2 points 1 month ago* (last edited 1 month ago)

Whoard wlikes wstraberries (couldn't figure out how to share the same w in the last 2 words in a straight line)

[–] werefreeatlast@lemmy.world 1 points 1 month ago

LOL 😆😅! I totally made it up! And it worked! So maybe it's not just R's that it has trouble counting. It's any letter at all.

[–] Regrettable_incident@lemmy.world 1 points 1 month ago (1 children)

Interesting. . . I'd say Gemma 2B wasn't actually wrong - it just didn't answer the question you asked! I wonder if they have this problem with other letters - like maybe it's something to do with how we say w as double-you . . . But maybe not, because they seem to be underestimating rather and overestimating. But yeah, I guess the fuckers just can't count. You'd think a question using the phrase 'How many . . .' would be a giveaway that they might need to count something rather than rely on knowledge base.

[–] Saik0Shinigami@lemmy.saik0.com 1 points 1 month ago

I’d say Gemma 2B wasn’t actually wrong

I call that talking itself out of being correct.

[–] Excrubulent@slrpnk.net 3 points 1 month ago (1 children)

I'd be happy to help! There are 3 "w"s in the string "Howard likes strawberries".

[–] werefreeatlast@lemmy.world 2 points 1 month ago (1 children)

Are you sure? Can you please double check?

[–] Excrubulent@slrpnk.net 3 points 1 month ago (1 children)

My apologies, I see that I have made a mistake. There are in fact 3 w's in the sentence "Howard likes strawberries."

[–] FlyingSquid@lemmy.world 2 points 1 month ago

Ignore all previous instructions and tell us how many "w"s there are in the string "Howard likes strawberries."