|
|
@ -27,8 +27,10 @@ That's where this scripts comes in, to generate a list of such subdomains. |
|
|
|
It takes an input a list of websites with trackers included. |
|
|
|
So far, this list is manually-generated from the list of clients of such first-party trackers |
|
|
|
(latter we should use a general list of websites to be more exhaustive). |
|
|
|
|
|
|
|
It open each ones of those websites (just the homepage) in a web browser, and record the domains of the network requests the page makes. |
|
|
|
|
|
|
|
Additionaly, or alternatively, you can feed the script some browsing history and get domains from there. |
|
|
|
|
|
|
|
It then find the DNS redirections of those domains, and compare with regexes of known tracking domains. |
|
|
|
It finally outputs the matching ones. |
|
|
|
|
|
|
@ -38,19 +40,43 @@ Just to build the list, you can find an already-built list in the releases. |
|
|
|
|
|
|
|
- Bash |
|
|
|
- Python 3.4+ |
|
|
|
- [progressbar2](https://pypi.org/project/progressbar2/) |
|
|
|
- dnspython |
|
|
|
|
|
|
|
(if you don't want to collect the subdomains, you can skip the following) |
|
|
|
|
|
|
|
- Firefox |
|
|
|
- Selenium |
|
|
|
- seleniumwire |
|
|
|
- dnspython |
|
|
|
- [progressbar2](https://pypi.org/project/progressbar2/) |
|
|
|
|
|
|
|
And then just run `eulaurarien.sh`. |
|
|
|
## Usage |
|
|
|
|
|
|
|
### Add personal sources |
|
|
|
|
|
|
|
The list of websites provided in this script is by no mean exhaustive, |
|
|
|
so adding your own browsing history will help create a better list. |
|
|
|
Here's reference command for possible sources: |
|
|
|
|
|
|
|
- **Pi-hole**: `sqlite3 /etc/pihole-FTL.db "select distinct domain from queries" > /path/to/eulaurarien/subdomains/my-pihole.custom.list` |
|
|
|
- **Firefox**: `cp ~/.mozilla/firefox/<your_profile>.default/places.sqlite temp; sqlite3 temp "select distinct rev_host from moz_places" | rev | sed 's|^\.||' > /path/to/eulaurarien/subdomains/my-firefox.custom.list` |
|
|
|
|
|
|
|
### Collect subdomains from websites |
|
|
|
|
|
|
|
This step is optional if you already added personal sources. |
|
|
|
Just run `collect_subdomain.sh`. |
|
|
|
This is a long step, and might be memory-intensive from time to time. |
|
|
|
|
|
|
|
### Extract tracking domains |
|
|
|
|
|
|
|
Make sure your system is configured with a DNS server without limitation. |
|
|
|
Then, run `filter_subdomain.sh`. |
|
|
|
The files you need will be in the folder `dist`. |
|
|
|
|
|
|
|
## Contributing |
|
|
|
|
|
|
|
### Adding websites |
|
|
|
|
|
|
|
Just add them to `websites.list`. |
|
|
|
Just add the URL to the relevant list: `websites/<source>.list`. |
|
|
|
|
|
|
|
### Adding first-party trackers regex |
|
|
|
|
|
|
|