this post was submitted on 24 Aug 2024
38 points (95.2% liked)
Linux
47561 readers
676 users here now
From Wikipedia, the free encyclopedia
Linux is a family of open source Unix-like operating systems based on the Linux kernel, an operating system kernel first released on September 17, 1991 by Linus Torvalds. Linux is typically packaged in a Linux distribution (or distro for short).
Distributions include the Linux kernel and supporting system software and libraries, many of which are provided by the GNU Project. Many Linux distributions use the word "Linux" in their name, but the Free Software Foundation uses the name GNU/Linux to emphasize the importance of GNU software, causing some controversy.
Rules
- Posts must be relevant to operating systems running the Linux kernel. GNU/Linux or otherwise.
- No misinformation
- No NSFW content
- No hate speech, bigotry, etc
Related Communities
Community icon by Alpár-Etele Méder, licensed under CC BY 3.0
founded 5 years ago
MODERATORS
you are viewing a single comment's thread
view the rest of the comments
view the rest of the comments
I tried to ingest a four terabyte epub library once. Even getting the data ingested with the author and title in the right spot was almost impossible. If a duplicates weren't just slightly wrong would be a different story but the duplicates are often misspells or different spellings.
Realistically the best thing you can do is get an output of file name, title, author and hand dedupe, but even then you're going to have to be careful about quality and language and all kinds of other strange issues you run into with large libraries.
In the end I gave up and only stored what I really wanted and would realistically ever need and that was small enough to hand cull.
When I had to match against misspells I found Levenshtein distance to be most useful.
https://en.wikipedia.org/wiki/Levenshtein_distance?wprov=sfla1
Yeah, I wrote some python once to give me voice control over a plex server. Distance algorithms did okay I had a lot better results out of fuzzy wuzzy though
Kind of a dicey prospect with risk when you're doing duplication though.
Then you run into problems with things like Harry Potter and the philosopher's Stone versus Harry Potter and the sorcerer's Stone. Depending on how badly your database is degenerated you can even end up with words out of order and all kinds of crazy crap. If the database is truly large just truing the database up can be unreasonably time-consuming.
I was pretty amazed at all the different versions of string search and comparison algorithms.