Generates a host list of first-party trackers for ad-blocking.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
 
 
 
Geoffrey Frogeye e882e09b37
Added outdated documentation warning in README
2 years ago
dist Added possibility to add personal sources 2 years ago
rules Tracker: intendmedia? 2 years ago
rules_adblock Improved rules handling 2 years ago
rules_asn Workflow: Automatically import IP ranges from ASN 2 years ago
rules_hosts Improved rules handling 2 years ago
rules_ip Workflow: Automatically import IP ranges from ASN 2 years ago
subdomains Added possibility to add personal sources 2 years ago
temp Separated DNS resolution from filtering 2 years ago
tests Tracker: intendmedia? 2 years ago
websites Added RED by SFR website 2 years ago
.gitignore Workflow: SQL -> Tree 2 years ago
README.md Added outdated documentation warning in README 2 years ago
adblock_to_domain_list.py Removed third-parties from easyprivacy 2 years ago
collect_subdomains.py Added some delay for websites subdomains collecting 2 years ago
collect_subdomains.sh Fix log in scripts 2 years ago
database.py Save dupplicates too 2 years ago
db.py Forgot to push this little guy 2 years ago
eulaurarien.sh Improved rules handling 2 years ago
export.py Save dupplicates too 2 years ago
export_lists.sh Save dupplicates too 2 years ago
feed_asn.py Save dupplicates too 2 years ago
feed_dns.py Save dupplicates too 2 years ago
feed_rules.py Added first_party tracking 2 years ago
fetch_resources.sh Typo in source 2 years ago
filter_subdomains.py Optimized IP matching 2 years ago
import_rules.sh Added first_party tracking 2 years ago
new_workflow.sh Workflow: Can now import DnsMass output 2 years ago
resolve_subdomains.sh Added intermediate representation for DNS datasets 2 years ago

README.md

eulaurarien

Generates a host list of first-party trackers for ad-blocking.

The latest list is available here: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt

DISCLAIMER: I'm by no way an expert on this subject so my vocabulary or other stuff might be wrong. Use at your own risk.

What's a first-party tracker?

Traditionally, websites load trackers scripts directly. For example, website1.com and website2.com both load https://trackercompany.com/trackerscript.js to track their users. In order to block those, one can simply block the host trackercompany.com.

However, to circumvent this easy block, tracker companies made the website using them load trackers from somethingirelevant.website1.com. The latter being a DNS redirection to website1.trackercompany.com, directly pointing to a server serving the tracking script. Those are the first-party trackers.

Blocking trackercompany.com doesn't work any more, and blocking *.trackercompany.com isn't really possible since:

  1. Most ad-blocker don't support wildcards
  2. It's a DNS redirection, meaning that most ad-blockers will only see somethingirelevant.website1.com

So the only solution is to block every somethingirelevant.website1.com-like subdomains known, which is a lot. That's where this scripts comes in, to generate a list of such subdomains.

How does this script work

Notice: This section is a tad outdated. I'm still experimenting to make the generation process better. I'll update this once I'm done with this.

It takes an input a list of websites with trackers included. So far, this list is manually-generated from the list of clients of such first-party trackers (latter we should use a general list of websites to be more exhaustive). It open each ones of those websites (just the homepage) in a web browser, and record the domains of the network requests the page makes.

Additionaly, or alternatively, you can feed the script some browsing history and get domains from there.

It then find the DNS redirections of those domains, and compare with regexes of known tracking domains. It finally outputs the matching ones.

Requirements

Notice: This section is a tad outdated. I'm still experimenting to make the generation process better. I'll update this once I'm done with this.

Just to build the list, you can find an already-built list in the releases.

(if you don't want to collect the subdomains, you can skip the following)

  • Firefox
  • Selenium
  • seleniumwire

Usage

Notice: This section is a tad outdated. I'm still experimenting to make the generation process better. I'll update this once I'm done with this.

This is only if you want to build the list yourself. If you just want to use the list, the latest build is available here: https://hostfiles.frogeye.fr/firstparty-trackers-hosts.txt It was build using additional sources not included in this repository for privacy reasons.

Add personal sources

The list of websites provided in this script is by no mean exhaustive, so adding your own browsing history will help create a better list. Here's reference command for possible sources:

  • Pi-hole: sqlite3 /etc/pihole-FTL.db "select distinct domain from queries" > /path/to/eulaurarien/subdomains/my-pihole.custom.list
  • Firefox: cp ~/.mozilla/firefox/<your_profile>.default/places.sqlite temp; sqlite3 temp "select distinct rev_host from moz_places" | rev | sed 's|^\.||' > /path/to/eulaurarien/subdomains/my-firefox.custom.list; rm temp

Collect subdomains from websites

Just run collect_subdomain.sh. This is a long step, and might be memory-intensive from time to time.

This step is optional if you already added personal sources. Alternatively, you can get just download the list of subdomains used to generate the official block list here: https://hostfiles.frogeye.fr/from_websites.cache.list (put it in the subdomains folder).

Extract tracking domains

Make sure your system is configured with a DNS server without limitation. Then, run filter_subdomain.sh. The files you need will be in the folder dist.

Contributing

Adding websites

Just add the URL to the relevant list: websites/<source>.list.

Adding first-party trackers regex

Just add them to regexes.py.