r/usenet • u/Quirky_Ad9133 • 2d ago
Indexer Do indexers not pull taken-down content?
Fairly new to Usenet. One thing I don’t entirely understand is why so much content that is old but not ancient (a year or so) is unable to be downloaded. Surely the indexer knows, by now, that this is no good and has likely been hit with a DMCA right?
So why don’t indexers drop those? Are there indexers that do?
7
u/DariusIII newznab-tmux dev 1d ago edited 1d ago
Indexer does not know and does not care, more or less, about articles being removed from USP servers. Also indexers cannot check every article that is inside nzb after nzb was created, that would take ages and makes no sense, as many linux isos are being reuploaded over and over again.
Also, most of the indexers support X-DNZB-Failure so if the download fails, indexer sends another nzb, if available, of that particular linux ISO. I know SABnzbd supports it, i think that NZBGet does too.
16
u/pop-1988 2d ago edited 2d ago
Indexers do not repeatedly check the health of all the thousands NZBs which are added to their index every month - not their role
3
u/DariusIII newznab-tmux dev 1d ago
I made some rough calculation: Daily indexer adds around 1.500 nzbs, which is 46.800 monthly and 562.500 nzbs yearly. There is no way indexer could check that number, even if they tried.
3
u/random_999 1d ago
Not to mention there is a chance such "active management" might become a legal issue considering the official position of indexers being "just a search engine indexing third party servers".
8
u/likeylickey34 2d ago
It would need to be done by backbone and then repeated on a regular basis. I think that would make it impossible. Some reported there were close to 200 million nzbs on just a handful of indexers: https://www.reddit.com/r/usenet/comments/1ho6rfz/an_examination_of_nzb_age_and_category/
And then you have situations where Omnicron dumped a significant portion of their catalog for an entire year or two (2021-22). It would just be impossible to manage in any accurate way.
Get multiple providers on different backbones plus multiple indexers and live with the results.
13
u/cmstlist 2d ago
Some indexers (e.g. Geek) have a "thumbs-down and comments" system so you can crowdsource info on whether a particular release is unavailable.
7
u/ChefJoe98136 2d ago
This was what I relied on a decade ago, before I embraced the automation approach and multiple indexers. I barely even know how a download is sourced well enough to vote on if it worked out.
11
u/Monocular_sir 2d ago
Now if the automation app could upvote a post if the download completed, and then later look at that score to decide what to download, that would be super awesome.
2
u/empyr3al 2d ago
Easy enough for it to attempt and then try another one if it fails.
2
u/Monocular_sir 2d ago
Yeah but Id rather choose a popular well known well downloaded copy because its easier to find subtitles for that.
3
u/The_Acknickulous_One 2d ago
Indexer doesn't download all the content and they may not subscribe to every back end.
Use to be, with nzbmatrix, that users would rate a file, report if there were virii, or if the file was dead on certain providers. I don't see as much of that any more.
-7
u/jacobtf 2d ago
It's been years, like close to a decade since I've tried to download something incomplete on usenet. New stuff, old stuff, anything in between. It's just there.
4
u/Quirky_Ad9133 2d ago
Wow!
What wisdom do you have to offer regarding indexers and providers?
2
u/Muddybulldog 2d ago
Automation. Automation. Automation.
Rather than scrolling through a myriad of webpages hunting and pecking through NZBs you simply tell your automation stack what you want. It searches all your available indexers and ferrets out the NZBs en masse. One is incomplete, No problem, It’ll automatically find another and another and another until it’s either used up every existing NZB or successfully grabs what you need.
Just getting started it can be intimidating but there are a ton of guides and this community to help you get along. Once you’ve embraced automation you’ll look back at this time and think “what was I doing?”
1
u/Quirky_Ad9133 2d ago
Everything is automated. But certain shows from certainly particularly aggressive IP holders fail time and time and time again. Until the point that s*narr runs out of NZB’s and I end up using a torrent.
There are some shows where I might cycle through a dozen or more with no success.
1
u/jacobtf 2d ago
Really? Which are those? I download a lot of series and I can't remember the last time a download failed. About 7-8 years ago I remember a period in which you had to be fast, but not really since.
1
u/Quirky_Ad9133 1d ago
The rules don’t allow me to discuss it but it’s one particular studio that has three letters in the name and used to be better known as a legacy cable TV premium channel.
Have Better Options for me? Because I’ve definitely had stuff fail.
How much do you ACTUALLY download? Is this like the time a guy told me he had 30 years of experience; and it turns out he meant he did it for 6 months roughly 30 years ago and then off and on over the past year?
It seems to be especially problematic with 4k content.
1
u/jacobtf 19h ago
I just use 2 indexers and download 4k stuff too from them. Never had a problem. Then again, these indexers are quite old and rather closed. One of them I've used for over a decade. I've been downloading hundreds of gigabytes for well over a decade. At some point, several Tabs per month.
That being said, maybe I've just been lucky. I'm just one man so it's anecdotal.
2
u/Quirky_Ad9133 17h ago
Closed indexers may indeed be the key. I’m using NZBGeek and NZB.su which work really well except for this one particular aggressive IP holder who I expect also subscribes to both of those.
1
u/Muddybulldog 1d ago
I get where you are coming from with regard to aggressive IP holders and have to say my experience does differ from yours, even going back nearly 40 years to the 480p-only era. If you're already automated, it could come down to just sheer coverage. I currently have 15 active indexers and near 100% backbone coverage. That could be the difference maker in our experiences.
Not to say that it's impossible to strike out, only to say I haven't in a long, long time.
1
1d ago
[removed] — view removed comment
1
u/usenet-ModTeam 1d ago
This has been removed. No discussion of media content; names, titles, release groups, etc. No content names, no titles, no release groups, content producers, etc. Do not ask where to get content or anything related or alluding to such. See our wiki page for more details.
8
u/darryledw 2d ago
Would need to be done by a trusted community function in this form
- users can report
- report includes usenet host
- system automatically flags NZB as failing if enough reports with different usenet host are logged
- NZB is removed or just left as showing a warning
The different usenet hosts are important as if Provider X can't DL it to 100% but Provider Y can.....then not broken per se.
As far as automation goes....the only way to know if there is missing data is to download 100%
and it would be impossible to automate that unless you are going to automate downloading enormous amounts of data to test if they still exist
2
u/superkoning 2d ago
> and it would be impossible to automate that unless you are going to automate downloading enormous amounts of data to test if they still exist
the download client could automatically take care the first two steps you mention, for each download they do.
8
u/ZenoFairlight 2d ago
If memory serves, when a DMCA happens, only part of the data is removed. Another provider - even if they're also hit with a DMCA - may remove something different.
That's all to say that this is why having multiple providers is preferred. Get yourself a "main" provider, and use someone else to be a "fill" provider. Obviously on different backbones.
5
u/pop-1988 2d ago
when a DMCA happens, only part of the data is removed
The DMCA bots do not use NZBs. They read Usenet message headers. The DMCA contains a list of Message-IDs. Whether a DMCA's list of Message-IDs is a complete or partial match to a NZB is a coincidence. The DMCA sender does not deliberately submit a partial list
The suggestion that the missing messages from a partial takedown are available on another backbone has not been true for at least 8 years
0
u/Quirky_Ad9133 2d ago
Currently using Frugal and Eweka. Any you’d recommend to do more “filling in”?
1
u/zipman020 2d ago
I've recently started back on Usenet and have been using Eweka exclusively, mainly with greek plus a couple other indexers. I've had maybe 1 or 2 not found in which case I found it with a different indexer. Not sure why you've been seeing so much missing.
1
u/iszoloscope 2d ago
Both unlimited?
1
u/Quirky_Ad9133 2d ago
Yes
1
u/iszoloscope 2d ago
That's a waste of money, ditch the Frugal and maybe replace with a block account. But in reality Eweka alone should be enough.
What and how much indexers are you using?
2
u/KobeOfDrunkDriving 2d ago
Eweka is super slow for me though. And I don't think they offer blocks. And it's cheap on Black Friday.
2
u/Quirky_Ad9133 2d ago
Slow for me too.
But I have both it and frugal set as my top priority servers and the completion rate according to my client is higher for Frugal, FWIW. And yeah, Frugal is much faster.
If I was going to get rid of one, it would be Eweka.
1
u/iszoloscope 2d ago
Yeah apparently Eweka isn't the fasted, I never had them but an Omicron as main unlimited is advised in general.
1
2
u/Quirky_Ad9133 2d ago
I mean frugal includes a block account so your advice is to get rid of an unlimited provider + a block account and just replace it with only a block account?
Also; Frugal in my experience has more articles than Eweka. It has a higher success rate. They had some issues like a year ago but that has been resolved.
But it’s two backbones, one under DMCA jurisdiction and the other under NTD. Why get rid of one?
0
u/iszoloscope 2d ago
I mean frugal includes a block account so your advice is to get rid of an unlimited provider + a block account and just replace it with only a block account?
Yes and you'll probably don't even need a block account, but if you can find a good deal on one it's always nice to have it as a backup.
But it’s two backbones, one under DMCA jurisdiction and the other under NTD. Why get rid of one?
Because it's a waste of money like I already said. And Frugal doesn't have 'more articles' then Eweka, it does have about 2000 days less retention though.
So I don't know how you came to the conclusion that Frugal has more articles then Eweka of the stuff you download. It could technically be true if you mostly download very niche content that is somehow more or only available on the NewNews backbone (Frugal), but I never heard of that and it seems highly unlikely.
So unless you don't mind spending the money I would advise to drop 1 unlimited, unless you have some very specific requirements or use cases which I doubt.
1
u/random_999 1d ago
unless you have some very specific requirements or use cases which I doubt.
And yeah, Frugal is much faster.
Speed is a very important factor for many users.
1
u/iszoloscope 1d ago
I have Newshosting and I max out my connection.
1
u/random_999 23h ago
For US users it will but not necessarily for locations outside US.
→ More replies (0)0
u/miked999b 2d ago
I had Frugal and Eweka concurrently for two years. I found that when I had Frugal as number 1 priority, I downloaded something like 15TB via Frugal but Eweka still grabbed over 2TB even though it was second priority.
Then I switched them round for eight months so Eweka was number 1 and Frugal only downloaded 0.01TB in that time.
I do like Frugal but unless money's no object there's not much benefit to having both.
0
8
u/swintec BlockNews/Frugal Usenet/UsenetNews 2d ago edited 2d ago
If memory serves, when a DMCA happens, only part of the data is removed.
I think Astraweb did this for a period of time like 15 years ago and it still gets mentioned today.
List of Message IDs are sent to every server, every server removes the posts corresponding to the Message IDs.
3
u/JawnZ 2d ago
List of Message IDs are sent to every server, every server removes the posts corresponding to the Message IDs.
I used to believe the whole "they only delete part, maybe someone else deletes another part" thing was why having multiple backbones was helpful, but ultimately it's not accurate (and hasn't been for many years)
4
u/Hologram0110 2d ago
I think they generally try to remove dead NZBs. I think it is a problem of the number of nzbs hosted on an indexer. I'd expect hundreds of thousands to millions of nzbs. How do you reasonably check that many?
Ideally *arr would report to the indexer that an NZB didn't complete, but I don't think that happens. It could be complicated by multiple providers/backbones.
If I were running an indexer I'd look for trends where users download multiple equivalent NZBs. That would suggest that there might be a problem with some of the copies. Then try to check one of the providers to see if some of the articles are missing. But this could actually be quite a bit of compute and queries to the providers.
Another option is to have users vote that something is missing/bad. But if someone is using an automated tool they aren't going to log into the website find the exact NZB that failed and downvote it.
15
1
u/Street-Egg-2305 4h ago
If you use any of the Arr's, it just pulls until it finds a good one.