Fun with adblocker filter lists
-
Several times over the past few years I've played around with various filter lists, hoping to find a group with the least amount of lines that provided the most protection from ads and trackers. What is the definition of "lines" in this context? Total row count in the filter file itself. My reasoning is the fewer the rules an algorithm has to parse, the faster the performance will be. I don't want to sacrifice results for a lower line count though, when a compromise had to be made I always chose function over total rows. Along the way I have come up with several combinations that have worked quite well, the smallest of which was about 162,000 rows contained in 6 lists. That sounds like a big number, but when you consider how many filter lists most adblockers use it's actually a small number relative to them.
In the past I've been able to keep the line count down, but I still needed to use something like Privacy Badger to handle the bulk of trackers. I'd rather use a single solution for both ads and trackers - my preference being the one built into Vivaldi - but in order to achieve passing scores on the objective test sites I use to measure, along with blocking ads and trackers without making a website look odd by leaving things like gaps and white space all over the place, I could either use several lists and swell the line count to a large number or keep the line count down and use Privacy Badger. Having more than one bit of software doing essentially the same thing made me wonder if that could led to conflicts between them, and possibly impart a performance penalty, two things I want to avoid if possible. Before I go further, here are the objective test sites I use...
All along my goal was 150,000 lines or less, something I never quite achieved. With the filter list group that had 162k lines I could ditch Privacy Badger and get excellent scores on all three test sites. It fulfilled 2 of my 3 target objectives; using one ad/tracker blocker, and pass all the test sites (the missed objective was total line count). I figured that was as close as I would ever get, until about a week ago.
While browsing around the internet I stumbled upon a list I had never heard of before, Combined Privacy Block List (CPBL). At 151k lines it was effectively my 150k target so I gave it a shot (I don't imagine the comment lines in a list are loaded into memory, only the rules themselves, but correct me if I'm wrong). Unfortunately it didn't do so well on 2 of the 3 test sites, it only passed CoverYourTracks. Cosmetics were lacking too, leaving white space where most ads used to be. They also have a "mini" version which is about 60% of the full versions size, around 90k lines. The size reduction is because the curator of the list removes rules he's determined are no longer relevant, or are needed so infrequently as to be expendable. The mini version scored the same as the full one, leading me to believe that if I added an additional list that focused primarily on ads the two might supplement each other well. I decided to try the granddaddy of them all, EasyList.
When combined EasyList and the full CPBL list are about 223k lines though, well above my target of 150k. I ran them against the 3 test sites and they passed, that combo meant no tracker-specific extension required. But could I do better? I swapped the full CPBL list for the mini version and also switched to the "optimized" EasyList. Most people aren't aware there is a version of EasyList that strips a bunch of the little or not used rules out, dropping it from roughly 72k lines to 50k. Between the 90k lines in the mini CPBL and the 50k from the optimized EasyList that's only 140k in total. Actually below my target, but what would the results be? Turns out the numbers achieved on the 3 test sites were identical to the full version of each. Apparently the only thing I lost was about 83k lines, there was no loss of capability.
Using only those 2 lists I spent the better part of a week surfing around the internet like I normally would, much to my surprise I saw no difference in how webpages looked or acted relative to my 162k line filter list setup. Everything I would expect to see as far as content - and not see with regards to ads - was exactly as it should be. I checked the shields icon on many occasions and the number of trackers and ads were as high as I had become accustom to in the past (since the CPBL list seemed better at trackers I used that as a tracker source, EasyList optimized was an ad source). After a few days I thought maybe I had finally hit all 3 of my goals, but then I ran into a problem.
I was viewing an article on yahoo.com and decided to leave a comment, as soon as I clicked the Show Comments link at the end of the article that section of the webpage went blank. Where there should have been a dialog box to enter text, and a Submit button to post it, there was neither. It was just white space, like someone had erased the content. After a bit of troubleshooting I was able to determine it was the CPBL list causing the problem. I loaded DevTool and in the console saw dozens of blocked elements but I have no idea which of them is crippling the comments function, so that leads me to a question for those of you who made it this far; how do I interpret the data?
I dumped the debug info to a log file but can't see a way to attach it this post, is there a method to do that? I tried looking it over myself but to no avail, I don't know how to decipher the information. This is the only issue I've found with this filter list combination so far, if I can get past that problem I might be able to use them going forward.
Whether or not you're able to assist with that, maybe there's another way you can contribute to this thread. How about posting your filter lists and the results from the 3 test sites for your particular configuration. I'm curious to see what other configs are being used and how they fare objectively. No other extensions should be running when you test though, just shields and your lists. Maybe you have something that's even better than what I came up with.
-
@theJman, yes, it's somewhat puzzeling to find a good combination of filters, but I think that muy current combination is pretty good, 100% on all tests, except in Cover your Tracks, which say "You have a strong protection but with an unique Fingerprint", but this can't be avoided with the Vivaldi blockerlists.
The adblock filters I use:
https://downloads.vivaldi.com/lists/abp/abp-filters-anti-cv-current.txt https://downloads.vivaldi.com/lists/abp/antiadblockfilters-current.txt https://downloads.vivaldi.com/easylist/easylist-current.txt https://secure.fanboy.co.nz/fanboy-cookiemonster.txt https://www.i-dont-care-about-cookies.eu/abp/ https://gitlab.com/curben/urlhaus-filter/raw/master/urlhaus-filter-agh.txt https://raw.githubusercontent.com/uBlockOrigin/uAssets/master/filters/filters.txtWith these I don't noticed any slowdown of Vivaldi in my humble laptop.
-
The most important thing about adblocker lists is how well-maintained and updated they are. Not line count.
Performance-wise, I'd say the difference would be negligible. Unless we're talking about those "mega-giga-super" lists some muppets create by cat'ing all the popular lists together into one giant list, put online and then forget about ever maintaining.
These are the line counts for the default lists in Vivaldi, including a couple extra I usually enable:
1939 abp-filters-anti-cv-current.txt 2150 antiadblockfilters-current.txt 72030 easylist-current.txt 51296 easyprivacy-current.txt 49464 fanboy-annoyance.txt 26418 fanboy-cookiemonster.txt 24338 i-dont-care-about-cookies.eu.txt 3560 peter-lowe.txt 231195 totalRemoving the optional Fanboy and Cookie popup lists, ends at ~130k.
The DDG Tracker Radar list is not a regular blocklist but a JSON file, so counting the lines does not count as the # of rules.The lists used by Vivaldi are the most common ones, well maintained and updated regularly, often daily, with a large community behind them, not a single-person effort which is doomed to death once the maintainer loses interest.
I was viewing an article on yahoo.com and decided to leave a comment, as soon as I clicked the Show Comments link at the end of the article that section of the webpage went blank.
When using adblockers you have to accept some site breakage.
Worst case if you're really desperate to comment, disable the adblocker and do what you need, then enable it.Usually on a lot of news sites, these comment fields are powered by Disqus, blocked by several lists as tracker. Whether or not that's fair to Disqus I don't know.
As for troubleshooting site breakage with blocklists, it's usually a process of elimination, as Vivaldi lacks a logger like uBO has, giving what requests are blocked and what rule applied.
And when you have only one lists - well you already know what list is causing it. Then, trying to find what rule in that list would apply to the site. Looking at the Network log and making some guesses what blocked requests are related to what's missing. This requires time, patience and experience. There is no silver-bullet fix for this.
The comments field on Yahoo seems to load fine with the default blocklists in Vivaldi. Whether or not it actually works I don't know as I don't have a Yahoo account.
🎻Volunteer helper · Forum moderator · Sopranos tester 🛠️Troubleshooting 🐛Report a bug 📜Markdown help
🦆"With a rubber duck, one's never alone" -Douglas Adams🦆 -
@Pathduck, until now all sites work fine with my blocklists, only exception is YT which detect the Vivaldi blocker sometimes, in this case I use the Embed Redirect script.
-
@Catweazle said in Fun with adblocker filter lists:
@Pathduck, until now all sites work fine with my blocklists, only exception is YT which detect the Vivaldi blocker sometimes, in this case I use the Embed Redirect script.
It's not only YouTube.
My issues are on other sites too. For example Outlook.
Microsoft adds an ad promo email at top of your emails. This needs scriptlets to be removed.
Vivaldi supports basically ABP filters, so this kind of ads cannot be not removed.
I am not visiting shady piracy sites, but many people do. All these sites are adding popups everywhere, at every click you make.
This kind of sites.. with the current ABP lists and without UBO lists many people will be very annoyed when they stop using uBO.
uBO Lite works on the above, but... the latest update of it that was published today in chrome store.
It took 4 days to be reviewed for simple list updates.
All these sites will throw a party when uBO is out of the picture.
They will break lists hours after uBO Lite get updated on purpose.
They be having a party for many days until the next update is published in chrome's store. -
Said:
They will break lists hours after uBO Lite get updated on purpose.
They be having a party for many days until the next update is published in chrome's store.
Adguard Version history:
"Quick fixes. AdGuard Ad Blocker MV3 introduces the AdGuard Quick Fixes filter. Why do you need it?
With Manifest V3, we can’t update filter rules dynamically because the rules are now built into the extension and are only updated with the extension itself. This is not very convenient.
We found a way around this: the new filter uses dynamic rules and provides the most reliable operation of our filters. AdGuard Quick Fixes uses dynamic rules to make adjustments in real time without significantly affecting the user’s dynamic rules limit. It’s fast, efficient, and perfect for seamless filtering in the demanding MV3 environment."
-
@barbudo2005 said in Fun with adblocker filter lists:
"Quick fixes. AdGuard Ad Blocker MV3 introduces the AdGuard Quick Fixes filter. Why do you need it?
With Manifest V3, we can’t update filter rules dynamically because the rules are now built into the extension and are only updated with the extension itself. This is not very convenient.
We found a way around this: the new filter uses dynamic rules and provides the most reliable operation of our filters. AdGuard Quick Fixes uses dynamic rules to make adjustments in real time without significantly affecting the user’s dynamic rules limit. It’s fast, efficient, and perfect for seamless filtering in the demanding MV3 environment."Nice to see that.
I personally have Adguard Home anyway.
What I don't like about Adguard extension is that it uses workarounds unlike uBO Lite for making custom lists to work.
So it doesn't remove the ads immediately like uBO Lite which follows MV3 standards completely.
My dream scenario is if they split the addon.
1 for the normal lists that follows the MV3 standard and it doesn't "wake up" the service worker, so the ads will be removed immediately.
And 1 helper addon which works with workarounds for MV3 and has only the quick fixes, custom lists and the element picker.
Or.. gorhill to make a uBO extra addon for quick fixes, custom lists and an element picker.
Or even better, the native adblocker to be improved and match Brave's adblocker.
Watching ads in my Outlook email in not an option:) -
@electryon, well, I don't use Outlook or any other thing from M$, apart of the guuted W10 I use. In all other sites I don't see any ads or pop-ups and I visit a lot of sites every day (> 1000 every month), naturally also no shady ones, which is always risky, also with adblocker. The user itself is the biggest security hole, that can't be avoided by the best browser protections.
-
Said:
What I don't like about Adguard extension is that it uses workarounds unlike uBO Lite for making custom lists to work.
What workarounds?
So it doesn't remove the ads immediately like uBO Lite which follows MV3 standards completely.
I don't see ads with Adguard ‘instantly’.
How can you not follow the MV3 standards in a MV3 extension?
-
@barbudo2005 said in Fun with adblocker filter lists:
What workarounds?
How can you not follow the MV3 standards in a MV3 extension?
Adguard in order to be able to import filter lists or create custom filters uses a trick.
Its service worker is force-restarted every 30 seconds when the browser is idle.
But all event handlers are non-blocking in MV3, so uBO Lite is declarative to avoid that.
Look at the following videos, you will understand what I mean. YouTube, a heavy site is used in order to make the difference more noticable.
If gorhill or Adguard Team splitted their ablockers in order to have a declarative for normal lists and a non declarative for custom lists, then we would have the best possible MV3 solution.
But they both won't:)
Vivaldi will have to save us, I am still hopeful they will eventually catch up with Brave's adblocker.Adguard MV3 when launching Chromium and restoring the last youtube page
uBO Lite when launching Chromium and restoring the last youtube page
-
Please post the link of the video in YT.
-
@barbudo2005 said in Fun with adblocker filter lists:
Please post the link of the video in YT.
I don't have a Google account: Any other site?
-
I'm referring to the music video on YT where you showed the proof.
-
@barbudo2005 said in Fun with adblocker filter lists:
I'm referring to the music video on YT where you showed the proof.
-
PPathduck moved this topic from Vivaldi for Windows on -
No ad in Chrome with Adguard with all the default filters and this custom:
PS:
-
@barbudo2005 said in Fun with adblocker filter lists:
No ad in Chrome with Adguard with all the default filters and this custom:
PS:
In your video you haven't done what i did.
You just showed the video playing.
Go to the video.
Clear YT cookies, YT may not show the ad for various reasons if it has already shown the ad in the video many times.
Close Chrome without closing the tab.
Launch Chrome....Then do the same with uBO Lite...
-
Clear YT cookies.
Close Chrome without closing the tab.
Launch Chrome.No ad with Adguard.
-
That's a solid group of lists. ABP's anti-cv and anti-adblock are 2 I'm going to add back, they're both around 2k lines each so not a big deal at all. uAssets is another favorite of mine, that does a lot for a filter list with less than 15k lines. I did see one I'm unfamiliar with though, what is urlhaus-filter-agh.txt for?
-
The most important thing about adblocker lists is how well-maintained and updated they are. Not line count.
The line count goal was just to keep the rules down, a personal crusade to make it as efficient as possible. I do agree that a carefully maintained list is critically important. The two I'm using seem to fall into that category, but only time will tell.
The DDG Tracker Radar list is not a regular blocklist but a JSON file
When I downloaded it I did see the format was decidedly different. I don't use it however because, rather ironically, it also prevents commenting on Yahoo articles. I've tested at least 20 filter lists while I've been messing around and DDG Tracker and CPBL are the only ones with the issue. That suggests to me the tracker behind the comments functionality isn't deemed egregious by most of the list maintainers.
As for troubleshooting site breakage with blocklists, it's usually a process of elimination
That's what I was trying to avoid, blindly poking around and hoping I accidentally find it. I've looked through the log file but I can't make much sense out of it, I'll see if the maintainers Git page has a way to report an issue.
-
@barbudo2005 said in Fun with adblocker filter lists:
Clear YT cookies.
Close Chrome without closing the tab.
Launch Chrome.No ad with Adguard.
barbudo, just upload a video where you open Vivaldi and the first page it loads is the YT video.
And with only Adguard MV3 installed, no enabled native adblocker, no uBO, no UBO Lite, no sofware like Adguard Home runnning in background, nothing, just Adguard MV3.
