eulaurarien/resolve_subdomains.sh

#!/usr/bin/env bash

# Resolve the CNAME chain of all the known subdomains for later analysis
echo "Compiling subdomain lists..." > /dev/stderr
cat subdomains/*.list | sort -u > temp/all_subdomains.list
# Sort by last character to utilize the DNS server caching mechanism
rev temp/all_subdomains.list | sort | rev > temp/all_subdomains_reversort.list
./resolve_subdomains.py --input temp/all_subdomains_reversort.list --output temp/all_resolved.csv
sort -u temp/all_resolved.csv > temp/all_resolved_sorted.csv
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem). 2019-12-03 08:48:12 +01:00			`#!/usr/bin/env bash`

			`# Resolve the CNAME chain of all the known subdomains for later analysis`
Improved DNS resolving performances Also various fixes. Also some debug stuff, make sure to remove that later. 2019-12-03 15:35:21 +01:00			`echo "Compiling subdomain lists..." > /dev/stderr`
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem). 2019-12-03 08:48:12 +01:00			`cat subdomains/*.list \| sort -u > temp/all_subdomains.list`
Improved DNS resolving performances Also various fixes. Also some debug stuff, make sure to remove that later. 2019-12-03 15:35:21 +01:00			`# Sort by last character to utilize the DNS server caching mechanism`
			`rev temp/all_subdomains.list \| sort \| rev > temp/all_subdomains_reversort.list`
			`./resolve_subdomains.py --input temp/all_subdomains_reversort.list --output temp/all_resolved.csv`
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem). 2019-12-03 08:48:12 +01:00			`sort -u temp/all_resolved.csv > temp/all_resolved_sorted.csv`