this post was submitted on 07 Oct 2024

453 points (98.7% liked)

Technology

58550 readers

5733 users here now

This is a most excellent place for technology news and articles.

Our Rules

Follow the lemmy.world rules.
Only tech related content.
Be excellent to each another!
Mod approved content bots can post up to 10 articles per day.
Threads asking for personal tech support may be deleted.
Politics threads may be removed.
No memes allowed as posts, OK to post as comments.
Only approved bots from the list below, to ask if your bot can be added please contact us.
Check for duplicates before posting, duplicates may be removed

Approved Bots

founded 1 year ago

MODERATORS

453

TikTok’s parent launched a web scraper that’s gobbling up the world’s online data 25-times faster than OpenAI (fortune.com)

submitted 1 day ago by LuuTuyen@lemmy.world to c/technology@lemmy.world

86 comments fedilink hide all child comments

top 50 comments

sorted by: hot top controversial new old

[–] GnuLinuxDude@lemmy.ml 1 points 33 minutes ago* (last edited 32 minutes ago)

As for what ByteDance plans to do with a new LLM, a person familiar with the company’s ambitions said one goal has to do with the search function for TikTok.

Last week, TikTok released an update to its current search function focused on [keywords for ads], basically allowing advertisers to search in real time for words that are trending on TikTok. It allows marketers to build an ad with relevant keywords that would ostensibly help the ad show up on the screens of more users.

…

“Given the audience and the amount of use, TikTok with a search environment that is a completely biddable space with keywords and topics, that would be very interesting to a lot of people spending a ton of money with Google right now,” the person said.

A dark vision just flashed in my mind. And I am certain this is what will happen. AI-generated ads done in real time based on the latest “trending” thing. Presented to users basically as soon as the topic has the slightest amount of “trend”.

Just emitting untold amounts of CO2 to show you generated ads in near real time.

[–] Brown_dude69@lemmy.world 4 points 4 hours ago (1 children)

Every major ai company did this let them do that what is to loose here?

[–] echodot@feddit.uk 2 points 2 hours ago (1 children)

People like to act as if archiving has never been a thing until about a year ago at which point it was suddenly invented and is now a threat in some nebulous way.

[–] hamsterkill@lemmy.sdf.org 5 points 40 minutes ago

It's not that it's a threat, it's that there's a difference between archiving for preservation and crawling other people's content for the purpose of making money off it (in a way that does not benefit the content creator).

[–] Roflmasterbigpimp@lemmy.world 18 points 12 hours ago (2 children)

I can not contribute to anything here, I just came to say I really really like the phrase "gobbling something up" :D

[–] GreenKnight23@lemmy.world 8 points 4 hours ago

better than "slammed"

Screenshot_20241008-091015_Firefox

[–] Iheartcheese@lemmy.world 3 points 11 hours ago

[–] zod000@lemmy.ml 80 points 22 hours ago (1 children)

We've had this thing hammering our servers. The scraper uses randomized user-agents browser/OS combinations and comes from a number of distinct IP ranges in different datacenters around the world, but all the IPs track back to Bytedance.

[–] UnderpantsWeevil@lemmy.world 31 points 20 hours ago (1 children)

Wouldn't be surprised if they're just cashing out while TikTok is still public in the US. One last desperate grab at value-add for the parent company before the shut down.

Also a great way to burn the infrastructure for subsequent use. After this, you can guarantee every data security company is going to add the TikTok servers to their firewalls and blacklists. So the American company that tries to harvest the property is going to be tripping over these legacy bullwarks for years after.

[–] Maggoty@lemmy.world 7 points 16 hours ago

This has nothing to do with Tik Tok other than ByteDance being a shareholder in Tik Tok

[–] dinckelman@lemmy.world 249 points 1 day ago (11 children)

It's illegal when a regular person steals something, but it's innovation and courage, when a huge corporation steals something. Interesting how that works

[–] Blackmist@feddit.uk 5 points 49 minutes ago

Not that there's anything right about anything right now, but a web crawler crawling the web hardly seems newsworthy. It's not like everyone else's crawlers haven't been feeding data into giant AI mulchers for years now.

This is just "you know that thing everyone else does? Now the Chinese do it too! Boooo!"

[–] b3an@lemmy.world 86 points 1 day ago (8 children)

Honestly it’s fucking angering. So much regulation and geo-restrictions and licensing schemes… but it’s cool that there are data brokers, and shit like this. On top of it all Chrome screwing us with manifest v3 and killing ad blocking on chrome. It’s already in canary build.

WHAT THE FUCK IS WRONG WITH THIS SPECIES?!

[–] ZombieMantis@lemmy.world 11 points 6 hours ago

WHAT THE FUCK IS WRONG WITH THIS SPECIES?!

Capitalism.

[–] militaryintelligence@lemmy.world 5 points 11 hours ago (1 children)

But muh stocks and bonds

[–] Chouxfleur@lemmy.world 1 points 2 hours ago

The line must go UP.

[–] undefined@links.hackliberty.org 3 points 12 hours ago

I get it that everyone wants ad blockers in their browser, but it doesn’t solve the problem of resources loading outside the browser.

I think DNS or IP filtering is much more effective. I only bring it up because everyone uses apps all the time and I’m constantly seeing apps trying to connect to tracking domains.

load more comments (5 replies)

[–] Chozo@fedia.io 49 points 1 day ago (6 children)

They're not stealing your data, they're pirating it.

load more comments (6 replies)

[–] alphapuggle@programming.dev 32 points 1 day ago (3 children)

Aaron Schwartz killed himself over punishments for less

[–] UnderpantsWeevil@lemmy.world 8 points 20 hours ago

Should have waited until he was in the billionaire class before breaking the law.

load more comments (2 replies)

load more comments (7 replies)

[–] affiliate@lemmy.world 19 points 19 hours ago (2 children)

from the article:

Robots.txt is a line of code that publishers can put into a website that, while not legally binding in any way, is supposed to signal to scraper bots that they cannot take that website’s data.

i do understand that robots.txt is a very minor part of the article, but i think that’s a pretty rough explanation of robots.txt

[–] echodot@feddit.uk 3 points 2 hours ago

It's literally a text document it's not even "a line of code".

[–] Corkyskog@sh.itjust.works 10 points 19 hours ago (3 children)

Out of curiosity, how would you word it?

[–] affiliate@lemmy.world 12 points 11 hours ago

i would probably word it as something like:

Robots.txt is a document that specifies which parts of a website bots are and are not allowed to visit. While it’s not a legally binding document, it has long been common practice for bots to obey the rules listed in robots.txt.

in that description, i’m trying to keep the accessible tone that they were going for in the article (so i wrote “document” instead of file format/IETF standard), while still trying to focus on the following points:

robots.txt is fundamentally a list of rules, not a single line of code
robots.txt can allow bots to access certain parts of a website, it doesn’t have to ban bots entirely
it’s not legally binding, but it is still customary for bots to follow it

i did also neglect to mention that robots.txt allows you to specify different rules for different bots, but that didn’t seem particularly relevant here.

load more comments (2 replies)

[–] BlackEco@lemmy.blackeco.com 65 points 1 day ago (3 children)

Also it doesn't respect robots.txt (the file that tells bots whether or not a given page can be accessed) unlike most AI scrapping bots.

[–] kboy101222@sh.itjust.works 37 points 23 hours ago (2 children)

My personal website that primarily functions as a front end to my home server has been getting BEAT by these stupid web scrapers. Every couple of days the server is unusable because some web scraper demanded every single possible page and crashed the damn thing

[–] echodot@feddit.uk 1 points 2 hours ago (1 children)

Can't you just disallow all external requests other than your own IP? If it's a personal website that's just for you then it really doesn't need to be accessible by anyone else and if anyone comes along that needs access you can just manually add their IP.

It's a minor pain to have to implement it, but it's an easy solution

[–] kboy101222@sh.itjust.works 1 points 1 hour ago

I have family and friends that also access the sites contents, so that's sadly not feasible without getting the IPs from dozens of different devices

[–] assaultpotato@sh.itjust.works 12 points 20 hours ago (1 children)

I do the same thing, and I've noticed my modem has been absolutely bricked probably 3-4 times this month. I wonder if this is why.

load more comments (1 replies)

load more comments (2 replies)

[–] Dindonmasker@sh.itjust.works 45 points 1 day ago (8 children)

Not surprising that Bytedance would want to gobble up every bit of data they can as fast as possible.

load more comments (8 replies)

load more comments