1
AI worse than humans in every way at summarising information, government trial finds
(www.crikey.com.au)
No shit.
this post was submitted on 04 Sep 2024
1 points (100.0% liked)
Technology
59587 readers
2940 users here now
This is a most excellent place for technology news and articles.
Our Rules
- Follow the lemmy.world rules.
- Only tech related content.
- Be excellent to each another!
- Mod approved content bots can post up to 10 articles per day.
- Threads asking for personal tech support may be deleted.
- Politics threads may be removed.
- No memes allowed as posts, OK to post as comments.
- Only approved bots from the list below, to ask if your bot can be added please contact us.
- Check for duplicates before posting, duplicates may be removed
Approved Bots
founded 1 year ago
MODERATORS
Intelligence vs non intelligence: intelligence is superior... Who would have thunk it lol
Ten ASIC staff, of varying levels of seniority, were also given the same task with similar prompts.
This is the key line here. These are likely university educated staff with significant experience in writing and summarising information and they were specifically tasked with this. However, within the social media landscape (Lemmy, reddit, etc) AI is already better at summarising information than humans because most human social media users are fucking retarded and spend their time either a) not reading properly/at all or b) cherrypicking information to fit whatever flavour of impassioned narrative they are trying to sell to everyone else.
Just some very recent examples I've seen of Lemmy users proving they are completely incapable of parsing relevant information are that article about an alternative, universal and non-proprietary database called GetGee which everyone seemed to think was an article about whether TikTok should be banned (because the word TikTok was in the title and that tricked their monkey brains) or the update to the 404 Media story on "active listening" in which people responded as if this technology exists and is in use when 404 Media still haven't been able to confirm either of these things. The second one was particularly egregious because it got picked up by all kinds of tech-related YouTube channels and news sites and regurgitated by their viewers and readers without a single one of these people ever bothering to read the source material properly.
Lemmy users proving they are completely incapable of parsing relevant information
To be fair, you need to actually read the article to be able to summarize it.
human social media users are fucking retarded
I feel attacked
load more comments
(1 replies)
Artificial intelligence is worse than humans in every way
As if capitalists have ever cared about that...
I would expect "faster" to be a way
or cheaper
"I can easily do it on my phone" is also good.
Are we talking 10% worse and 95% cheaper? Or 50% worse and 10% cheaper? Or 90% worse and 95% cheaper?
Because that last one is good enough for fiscal conservatives. Hell, the second one is good enough for fiscal conservatives.
The linked pdf lists the deficiencies of the LLM responses. They are varied and it sometimes misses the mark completely or cant grasp vital context.
Still pretty useless comparison, they testet 10 university level humans against Llama2-70B. The model has fallen out of use completely by now and was never really great at summarization. The study didnt fine tune it either, so this isnt really representative of the current situation.
There are far better models out, that were either especially trained for summarization or can be easily fine tuned to excel at it. Not to mention the Llama3 and 3.1 series, with the crazy 405B model.
There are far better models out
I've heard this refrain a few times. Still waiting for it to pan out.
load more comments
(1 replies)
Knowing this it seems like a very low quality study. They should probably redo this with multiple conditions.
- Base Llama 3
- Tuned Llama 3
- Untrained human summarizer
- trained/professional human summarizer
To all of you AI haters out there, stay away from the two minute papers yt channel. You'll get very sad at the actual state of AI.
load more comments
(1 replies)
This is an old study, they tested University level adults against the standard Llama2-70B.
Kinda absolete now, the model has completely fallen out of use, for the newer and far better 3 and 3.1 Versions. It also wasnt fine tuned for summarization, and while base L2-70B was OK, it wasnt great at anything without fine tuning.
This clickbait title also sounds like self gratification, the abysmal reading comprehension in the Internet is directly counter to it. The average human found on the Internet doesnt approch the level of literary capabilities, that those ten human testers showed in the study.
Artificial intelligence is worse than humans in every way at summarizing documents
In every way? How about speed? The goal is to save human time so if AI is faster and the summary is good enough, then it is a success. I guarantee it is faster. Much faster.
47% is a fail. 81% is an A-... Sure the AI can fail faster than a human can succeed, but I can fail to run a marathon faster than an athlete can succeed.
I guess by the standards we use to judge AI I'm a marathon runner!
If I want to get a better sense of lemmy than headlines, that 47% success at summarizing all the posts is good enough and much faster than I can even skim
If I want to code a new program, that 47% is probably pretty solid at structure and boilerplate so good enough. It can save me a lot of time
If I’m writing my thesis, that 47% is abject failure
If you miss key information the summary is useless.
If the structure of the code is bad then using that boilerplate will harm your ability to maintain the code FOREVER.
There are use cases for it, but it has to be used by someone who understands the task and knows the outcome they're looking for. It can't replace any measure of skill just yet, but it behaves as if it can which is hazardous.
I'd heard that Canada gives out As down into the 80% range but I thought I was being fed a line
load more comments
(1 replies)
If you make enough mistakes, speed is a detriment not a benefit. Increasing speed allows you to produce more summaries but if you still need to correct and edit them all you've done is add a step where a human has to still read the document to the level where they could summarize it and edit the AI summary. Therefore the bottleneck of a human reading the document and working on a summary is still there. It would only potentially make it slightly easier if the corrections needed are small and obvious.
load more comments
(1 replies)
"AI" or Large Lange Models, are designed by definition to give averaged answers. So they're not just averaging on the text you give them, they're averaging it with all general text of the training model, to create a probabilistically average result based on all of it.
There's no way around this, because it's simply how such systems work. It's their lifeblood to produce a "best guess" across large amounts of training data ...which is done by averaging out all that language. A large amount of language... Hence the name.