Browse Source

Added some delay for websites subdomains collecting

Some websites load their trackers after the page is done loading.
newworkflow_parseropti
Geoffrey Frogeye 3 years ago
parent
commit
00a0020914
  1. 2
      collect_subdomains.py

2
collect_subdomains.py

@ -9,6 +9,7 @@ accessed by the websites.
import sys
import typing
import urllib.parse
import time
import progressbar
import selenium.webdriver.firefox.options
@ -34,6 +35,7 @@ def collect_subdomains(url: str) -> typing.Iterable[str]:
executable_path='geckodriver', options=options)
driver.get(url)
time.sleep(10)
for request in driver.requests:
if request.response:
yield subdomain_from_url(request.path)

Loading…
Cancel
Save