Generates a host list of first-party trackers for ad-blocking.
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.

179 lines
8.4 KiB

1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
1 year ago
  1. # eulaurarien
  2. This program is able to generate a list of every hostnames being a DNS redirection to a list of DNS zones and IP networks.
  3. It is primarilyy used to generate [Geoffrey Frogeye's block list of first-party trackers](https://hostfiles.frogeye.fr) (learn about first-party trackers by following this link).
  4. If you want to contribute but don't want to create an account on this forge, contact me the way you like: <https://geoffrey.frogeye.fr>
  5. ## How does this work
  6. This program takes as input:
  7. - Lists of hostnames to match
  8. - Lists of DNS zone to match (a domain and their subdomains)
  9. - Lists of IP address / IP networks to match
  10. - Lists of Autonomous System numbers to match
  11. - An enormous quantity of DNS records
  12. It will be able to output hostnames being a DNS redirection to any item in the lists provided.
  13. DNS records can either come from [Rapid7 Open Data Sets](https://opendata.rapid7.com/sonar.fdns_v2/) or can be locally resolved from a list of subdomains using [MassDNS](https://github.com/blechschmidt/massdns).
  14. Those subdomains can either be provided as is, come from [Cisco Umbrella Popularity List](http://s3-us-west-1.amazonaws.com/umbrella-static/index.html), from your browsing history, or from analyzing the traffic a web browser makes when opening an URL (the program provides utility to do all that).
  15. ## Usage
  16. Remember you can get an already generated and up-to-date list of first-party trackers from [here](https://hostfiles.frogeye.fr).
  17. The following is for the people wanting to build their own list.
  18. ### Requirements
  19. Depending on the sources you'll be using to generate the list, you'll need to install some of the following:
  20. - [Bash](https://www.gnu.org/software/bash/bash.html)
  21. - [Coreutils](https://www.gnu.org/software/coreutils/)
  22. - [Gawk](https://www.gnu.org/software/gawk/)
  23. - [curl](https://curl.haxx.se)
  24. - [pv](http://www.ivarch.com/programs/pv.shtml)
  25. - [Python 3.4+](https://www.python.org/)
  26. - [coloredlogs](https://pypi.org/project/coloredlogs/) (sorry I can't help myself)
  27. - [numpy](https://www.numpy.org/)
  28. - [python-abp](https://pypi.org/project/python-abp/) (only if you intend to use AdBlock rules as a rule source)
  29. - [jq](http://stedolan.github.io/jq/) (only if you have a Rapid7 API key)
  30. - [massdns](https://github.com/blechschmidt/massdns) in your `$PATH` (only if you have subdomains as a source)
  31. - [Firefox](https://www.mozilla.org/firefox/) (only if you have websites as a source)
  32. - [selenium (Python bindings)](https://pypi.python.org/pypi/selenium) (only if you have websites as a source)
  33. - [selenium-wire](https://pypi.org/project/selenium-wire/) (only if you have websites as a source)
  34. - [markdown2](https://pypi.org/project/markdown2/) (only if you intend to generate the index webpage)
  35. ### Create a new database
  36. The so-called database (in the form of `blocking.p`) is a file storing all the matching entities (ASN, IPs, hostnames, zones…) and every entity leading to it.
  37. It exists because the list cannot be generated in one pass, as DNS redirections chain links do not have to be inputed in order.
  38. You can purge of old records the database by running `./prune.sh`.
  39. When you remove a source of data, remove its corresponding file in `last_updates` to fix the pruning process.
  40. ### Gather external sources
  41. External sources are not stored in this repository.
  42. You'll need to fetch them by running `./fetch_resources.sh`.
  43. Those include:
  44. - Third-party trackers lists
  45. - TLD lists (used to test the validity of hostnames)
  46. - List of public DNS resolvers (for DNS resolving from subdomains)
  47. - Top 1M subdomains
  48. ### Import rules into the database
  49. You need to put the lists of rules for matching in the different subfolders:
  50. - `rules`: Lists of DNS zones
  51. - `rules_ip`: Lists of IP networks (for IP addresses append `/32`)
  52. - `rules_asn`: Lists of Autonomous Systems numbers (IP ranges will be deducted from them)
  53. - `rules_adblock`: Lists of DNS zones, but in the form of AdBlock lists (only the ones concerning domains will be extracted)
  54. - `rules_hosts`: Lists of DNS zones, but in the form of hosts lists
  55. See the provided examples for syntax.
  56. In each folder:
  57. - `first-party.ext` will be the only files considered for the first-party variant of the list
  58. - `*.cache.ext` are from external sources, and thus might be deleted / overwrote
  59. - `*.custom.ext` are for sources that you don't want commited
  60. Then, run `./import_rules.sh`.
  61. If you removed rules and you want to remove every record depending on those rules immediately,
  62. run the following command:
  63. ```
  64. ./db.py --prune --prune-before "$(cat "last_updates/rules.txt")" --prune-base
  65. ```
  66. ### Add subdomains
  67. If you plan to resolve DNS records yourself (as the DNS records datasets are not exhaustive),
  68. the top 1M subdomains provided might not be enough.
  69. You can add them into the `subdomains` folder.
  70. It follows the same specificities as the rules folder for `*.cache.ext` and `*.custom.ext` files.
  71. #### Add personal sources
  72. Adding your own browsing history will help create a more suited subdomains list.
  73. Here's reference command for possible sources:
  74. - **Pi-hole**: `sqlite3 /etc/pihole-FTL.db "select distinct domain from queries" > /path/to/eulaurarien/subdomains/my-pihole.custom.list`
  75. - **Firefox**: `cp ~/.mozilla/firefox/<your_profile>.default/places.sqlite temp; sqlite3 temp "select distinct rev_host from moz_places" | rev | sed 's|^\.||' > /path/to/eulaurarien/subdomains/my-firefox.custom.list; rm temp`
  76. #### Collect subdomains from websites
  77. You can add the websites URLs into the `websites` folder.
  78. It follows the same specificities as the rules folder for `*.cache.ext` and `*.custom.ext` files.
  79. Then, run `collect_subdomain.sh`.
  80. This is a long step, and might be memory-intensive from time to time.
  81. > **Note:** For first-party tracking, a list of subdomains issued from the websites in the repository is avaliable here: <https://hostfiles.frogeye.fr/from_websites.cache.list>
  82. ### Resolve DNS records
  83. Once you've added subdomains, you'll need to resolve them to get their DNS records.
  84. The program will use a list of public nameservers to do that, but you can add your own in the `nameservers` directory.
  85. Then, run `./resolve_subdomains.sh`.
  86. Note that this is a network intensive process, not in term of bandwith, but in terms of packet number.
  87. > **Note:** Some VPS providers might detect this as a DDoS attack and cut the network access.
  88. > Some Wi-Fi connections can be rendered unusable for other uses, some routers might cease to work.
  89. > Since massdns does not support yet rate limiting, my best bet was a Raspberry Pi with a slow ethernet link (Raspberry Pi < 4).
  90. The DNS records will automatically be imported into the database.
  91. If you want to re-import the records without re-doing the resolving, just run the last line of the `./resolve_subdomains.sh` script.
  92. ### Import DNS records from Rapid7
  93. If you have a Rapid7 Organization API key, make sure to append to `.env`:
  94. ```
  95. RAPID7_API_KEY=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
  96. ```
  97. Then, run `./import_rapid7.sh`.
  98. This will download about 35 GiB of data the first time, but only the matching records will be stored (about a few MiB for the tracking rules).
  99. Note the download speed will most likely be limited by the database operation thoughput (a quick RAM will help).
  100. The script remembers which were the last sets downloaded, and will only newer sets.
  101. If the first-party rules changed, the corresponding sets will be re-imported anyway.
  102. If you want to force re-importing, run `rm last_updates/rapid7_*.txt`.
  103. ### Export the lists
  104. For the tracking list, use `./export_lists.sh`, the output will be in the `dist` folder (please change the links before distributing them).
  105. For other purposes, tinker with the `./export.py` program.
  106. #### Explanations
  107. Note that if you created an `explanations` folder at the root of the project, a file with a timestamp will be created in it.
  108. It contains every rule in the database and the reason of their presence (i.e. their dependency).
  109. This might be useful to track changes between runs.
  110. Every rule has an associated tag with four components:
  111. 1. A number: the level of the rule (1 if it is a rule present in the `rules*` folders)
  112. 2. A letter: `F` if first-party, `M` if multi-party.
  113. 3. A letter: `D` if a dupplicate (e.g. `foo.bar.com` if `*.bar.com` is already a rule), `_` if not.
  114. 4. A number: the number of rules relying on this one
  115. ### Generate the index webpage
  116. This is the one served on <https://hostfiles.frogeye.fr>.
  117. Just run `./generate_index.py`.
  118. ### Everything
  119. Once you've made sure every step runs fine, you can use `./eulaurarien.sh` to run every step consecutively.