Browse Source

Added some delay for websites subdomains collecting

Some websites load their trackers after the page is done loading.
newworkflow_parseropti
Geoffrey Frogeye 2 years ago
parent
commit
00a0020914
1 changed files with 2 additions and 0 deletions
  1. +2
    -0
      collect_subdomains.py

+ 2
- 0
collect_subdomains.py View File

@ -9,6 +9,7 @@ accessed by the websites.
import sys
import typing
import urllib.parse
import time
import progressbar
import selenium.webdriver.firefox.options
@ -34,6 +35,7 @@ def collect_subdomains(url: str) -> typing.Iterable[str]:
executable_path='geckodriver', options=options)
driver.get(url)
time.sleep(10)
for request in driver.requests:
if request.response:
yield subdomain_from_url(request.path)


Loading…
Cancel
Save