geoffrey
/
eulaurarien
1
0
Fork 0
Code Issues 6 Pull Requests Releases 1 Wiki Activity
eulaurarien/fetch_resources.sh

55 lines
2.7 KiB
Bash
Raw Normal View History

Automatically get top 1M subdomains
2019-11-14 10:23:59 +00:00
#!/usr/bin/env bash
Fix log in scripts Closes #8
2019-12-07 17:45:48 +00:00
function log() {
echo -e "\033[33m$@\033[0m"
}
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem).
2019-12-03 07:48:12 +00:00
function dl() {
Fix log in scripts Closes #8
2019-12-07 17:45:48 +00:00
echo "Downloading $1 to $2…"
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem).
2019-12-03 07:48:12 +00:00
curl --silent "$1" > "$2"
if [ $? -ne 0 ]
then
echo "Failed!"
fi
}
Fix log in scripts Closes #8
2019-12-07 17:45:48 +00:00
log "Retrieving rules…"
Removed false-positive source Also had edgekey.net for blocking. Thanks @TorchedPoseidon for the report!
2019-12-03 20:25:20 +00:00
rm -f rules*/*.cache.*
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem).
2019-12-03 07:48:12 +00:00
dl https://easylist.to/easylist/easyprivacy.txt rules_adblock/easyprivacy.cache.txt
Added more rule sources
2019-12-03 16:33:46 +00:00
# From firebog.net Tracking & Telemetry Lists
Workflow: Some modifications
2019-12-14 15:04:19 +00:00
# dl https://v.firebog.net/hosts/Prigent-Ads.txt rules/prigent-ads.cache.list
Removed another source of false-positives
2019-12-04 23:50:32 +00:00
# dl https://gitlab.com/quidsup/notrack-blocklists/raw/master/notrack-blocklist.txt rules/notrack-blocklist.cache.list
# False positives: https://github.com/WaLLy3K/wally3k.github.io/issues/73 -> 69.media.tumblr.com chicdn.net
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem).
2019-12-03 07:48:12 +00:00
dl https://raw.githubusercontent.com/StevenBlack/hosts/master/data/add.2o7Net/hosts rules_hosts/add2o7.cache.txt
Added more rule sources
2019-12-03 16:33:46 +00:00
dl https://raw.githubusercontent.com/crazy-max/WindowsSpyBlocker/master/data/hosts/spy.txt rules_hosts/spy.cache.txt
Removed another source of false-positives
2019-12-04 23:50:32 +00:00
# dl https://raw.githubusercontent.com/Kees1958/WS3_annual_most_used_survey_blocklist/master/w3tech_hostfile.txt rules/w3tech.cache.list
# False positives: agreements.apple.com -> edgekey.net
Removed false-positive source Also had edgekey.net for blocking. Thanks @TorchedPoseidon for the report!
2019-12-03 20:25:20 +00:00
# dl https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt rules_hosts/ads-and-tracking-extended.cache.txt # Lots of false-positives
Added more rule sources
2019-12-03 16:33:46 +00:00
# dl https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/android-tracking.txt rules_hosts/android-tracking.cache.txt
# dl https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/SmartTV.txt rules_hosts/smart-tv.cache.txt
# dl https://raw.githubusercontent.com/Perflyst/PiHoleBlocklist/master/AmazonFireTV.txt rules_hosts/amazon-fire-tv.cache.txt
Can now use AdBlock lists for tracking matching It's not very performant by itself, especially since pyre2 isn't maintained nor really compilableinstallable anymore. The performance seems to have decreased from 200 req/s to 0.2 req/s when using 512 threads, and to 80 req/s using 64 req/s. This might or might not be related,as the CPU doesn't seem to be the bottleneck. I will probably add support for host-based rules, matching the subdomains of such hosts (as for now there doesn't seem to be any other pattern for first-party trackers than subdomains, and this would be a very broad performace / compatibility with existing lists improvement), and convert the AdBlock lists to this format, only keeping domains-only rules.
2019-11-15 07:57:31 +00:00
Fix log in scripts Closes #8
2019-12-07 17:45:48 +00:00
log "Retrieving nameservers…"
Automatically get top 1M subdomains
2019-11-14 10:23:59 +00:00
rm -f nameservers
touch nameservers
[ -f nameservers.head ] && cat nameservers.head >> nameservers
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem).
2019-12-03 07:48:12 +00:00
dl https://public-dns.info/nameservers.txt nameservers.temp
sort -R nameservers.temp >> nameservers
rm nameservers.temp
Automatically get top 1M subdomains
2019-11-14 10:23:59 +00:00
Fix log in scripts Closes #8
2019-12-07 17:45:48 +00:00
log "Retrieving top subdomains…"
Append top 1M subdomains rather than replacing it
2019-12-03 08:02:59 +00:00
dl http://s3-us-west-1.amazonaws.com/umbrella-static/top-1m.csv.zip top-1m.csv.zip
Automatically get top 1M subdomains
2019-11-14 10:23:59 +00:00
unzip top-1m.csv.zip
Append top 1M subdomains rather than replacing it
2019-12-03 08:02:59 +00:00
sed 's|^[0-9]\+,||' top-1m.csv > temp/cisco-umbrella_popularity.fresh.list
Automatically get top 1M subdomains
2019-11-14 10:23:59 +00:00
rm top-1m.csv top-1m.csv.zip
Added more rule sources
2019-12-03 16:33:46 +00:00
if [ -f subdomains/cisco-umbrella_popularity.cache.list ]
then
cp subdomains/cisco-umbrella_popularity.cache.list temp/cisco-umbrella_popularity.old.list
pv temp/cisco-umbrella_popularity.old.list temp/cisco-umbrella_popularity.fresh.list | sort -u > subdomains/cisco-umbrella_popularity.cache.list
Nitpicking
2019-12-05 18:38:26 +00:00
rm temp/cisco-umbrella_popularity.old.list temp/cisco-umbrella_popularity.fresh.list
Added more rule sources
2019-12-03 16:33:46 +00:00
else
mv temp/cisco-umbrella_popularity.fresh.list subdomains/cisco-umbrella_popularity.cache.list
fi
Typo in source
2019-12-15 00:52:45 +00:00
dl https://www.orwell1984.today/cname/eulerian.net.txt subdomains/orwell-eulerian-cname-list.cache.list