eulaurarien/resolve_subdomains.sh

#!/usr/bin/env bash

function log() {
    echo -e "\033[33m$@\033[0m"
}

log "Compiling locally known subdomain…"
# Sort by last character to utilize the DNS server caching mechanism
pv subdomains/*.list | sed 's/\r$//' | rev | sort -u | rev > temp/all_subdomains.list
log "Resolving locally known subdomain…"
pv temp/all_subdomains.list | ./resolve_subdomains.py --output temp/all_resolved.csv
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem). 2019-12-03 08:48:12 +01:00			`#!/usr/bin/env bash`

Fix log in scripts Closes #8 2019-12-07 18:45:48 +01:00			`function log() {`
			`echo -e "\033[33m$@\033[0m"`
			`}`

Integrated DNS resolving to workflow Since the bigger datasets are only updated once a month, this might help for quick updates. 2019-12-13 13:38:23 +01:00			`log "Compiling locally known subdomain…"`
Improved DNS resolving performances Also various fixes. Also some debug stuff, make sure to remove that later. 2019-12-03 15:35:21 +01:00			`# Sort by last character to utilize the DNS server caching mechanism`
Added intermediate representation for DNS datasets It's just CSV. The DNS from the datasets are not ordered consistently, so we need to parse it completly. It seems that converting to an IR before sending data to ./feed_dns.py through a pipe is faster than decoding the JSON in ./feed_dns.py. This will also reduce the storage of the resolved subdomains by about 15% (compressed). 2019-12-13 21:59:35 +01:00			`pv subdomains/*.list \| sed 's/\r$//' \| rev \| sort -u \| rev > temp/all_subdomains.list`
Integrated DNS resolving to workflow Since the bigger datasets are only updated once a month, this might help for quick updates. 2019-12-13 13:38:23 +01:00			`log "Resolving locally known subdomain…"`
Added intermediate representation for DNS datasets It's just CSV. The DNS from the datasets are not ordered consistently, so we need to parse it completly. It seems that converting to an IR before sending data to ./feed_dns.py through a pipe is faster than decoding the JSON in ./feed_dns.py. This will also reduce the storage of the resolved subdomains by about 15% (compressed). 2019-12-13 21:59:35 +01:00			`pv temp/all_subdomains.list \| ./resolve_subdomains.py --output temp/all_resolved.csv`
Improved rules handling Rules can now come in 3 different formats: - AdBlock rules - Host lists - Domains lists All will be converted into domain lists and aggregated (only AdBlock rules matching a whole domain will be kept). Subdomains will now be matched if it is a subdomain of any domain of the rule. It is way faster (seconds rather than hours!) but less flexible (although it shouldn't be a problem). 2019-12-03 08:48:12 +01:00