geoffrey/eulaurarien
1
0
Fork 0
Code Issues 7 Pull requests Releases 1 Wiki Activity

Integrated DNS resolving to workflow

Browse source 
Since the bigger datasets are only updated once a month,
this might help for quick updates.
This commit is contained in:
Geoffrey Frogeye 2019-12-13 13:38:23 +01:00
parent 9050a84670
commit 8d94b80fd0
Signed by: geoffrey
GPG key ID: D8A7ECA00A8CD3DD
2 changed files with 34 additions and 57 deletions
Download patch file Download diff file Expand all files Collapse all files

10
resolve_subdomains.sh
View file

@ -4,11 +4,9 @@ function log() {
echo -e "\033[33m$@\033[0m"
}
# Resolve the CNAME chain of all the known subdomains for later analysis
log "Compiling subdomain lists..."
pv subdomains/*.list | sort -u > temp/all_subdomains.list
log "Compiling locally known subdomain…"
# Sort by last character to utilize the DNS server caching mechanism
pv temp/all_subdomains.list | rev | sort | rev > temp/all_subdomains_reversort.list
./resolve_subdomains.py --input temp/all_subdomains_reversort.list --output temp/all_resolved.csv
sort -u temp/all_resolved.csv > temp/all_resolved_sorted.csv
pv subdomains/*.list | rev | sort -u | rev > temp/all_subdomains.list
log "Resolving locally known subdomain…"
pv temp/all_subdomains.list | ./resolve_subdomains.py --output temp/all_resolved.json