Geoffrey Frogeye
269b8278b5
Worflow: Fixed rules counts
2019-12-13 18:36:08 +01:00
Geoffrey Frogeye
ab7ef609dd
Workflow: Various optimisations and fixes
...
I forgot to close this one earlier, so:
Closes #7
2019-12-13 18:08:22 +01:00
Geoffrey Frogeye
f3eedcba22
Updated now based on timestamp
...
Did I forget to add feed_asn.py a few commits ago?
Oh well...
2019-12-13 13:54:00 +01:00
Geoffrey Frogeye
8d94b80fd0
Integrated DNS resolving to workflow
...
Since the bigger datasets are only updated once a month,
this might help for quick updates.
2019-12-13 13:38:23 +01:00
Geoffrey Frogeye
231bb83667
Threaded feed_dns
...
Largely disapointing
2019-12-13 12:36:11 +01:00
Geoffrey Frogeye
9050a84670
Read-only mode
2019-12-13 12:35:05 +01:00
Geoffrey Frogeye
e19f666331
Workflow: Automatically import IP ranges from ASN
...
Closes #9
2019-12-13 08:23:38 +01:00
Geoffrey Frogeye
57416b6e2c
Workflow: POO and individual tables per types
...
Mostly for performances reasons.
First one to implement threading later.
Second one to speed up the dichotomy,
but it doesn't seem that much better so far.
2019-12-13 00:11:21 +01:00
Geoffrey Frogeye
b076fa6c34
Typo in new source URL
2019-12-12 23:28:00 +01:00
Geoffrey Frogeye
12dcafe606
Added alternate source of Eulerian CNAMES
...
It was requested so.
It should be temporary, once I have a bigger subdomain list
that shouldn't be required.
2019-12-12 19:13:54 +01:00
Geoffrey Frogeye
1484733a90
Workflow: Small tweaks
2019-12-09 18:21:08 +01:00
Geoffrey Frogeye
55877be891
IP parsing C accelerated, use bytes everywhere
2019-12-09 09:47:48 +01:00
Geoffrey Frogeye
7937496882
Workflow: Base for new one
...
While I'm automating this you'll need to download the A set from
https://opendata.rapid7.com/sonar.fdns_v2/ to the file a.json.gz.
2019-12-09 08:12:48 +01:00
Geoffrey Frogeye
62e6c9005b
Tracker: intendmedia?
2019-12-08 01:32:49 +01:00
Geoffrey Frogeye
dc44dea505
Optimized IP matching
2019-12-08 01:23:36 +01:00
Geoffrey Frogeye
b634ae5bbd
Updated IP ranges for Criteo
2019-12-07 23:23:39 +01:00
Geoffrey Frogeye
16f8bed887
Tracker: Otto Group
2019-12-07 21:30:15 +01:00
Geoffrey Frogeye
d6df0fd4f9
Tracker: Webtrekk
2019-12-07 21:21:33 +01:00
Geoffrey Frogeye
4dd3d4a64b
Preliminary structure for testing
...
In preparation of #4
2019-12-07 19:19:37 +01:00
Geoffrey Frogeye
ae71d6b204
Tracker: 2o7
2019-12-07 19:17:18 +01:00
Geoffrey Frogeye
2b0a723c30
Fix log in scripts
...
Closes #8
2019-12-07 18:45:48 +01:00
Geoffrey Frogeye
0b2eb000c3
FP: ThreatMetrix
2019-12-07 18:23:11 +01:00
Geoffrey Frogeye
cbb0cc6f3b
Rules lists are optional
2019-12-07 18:22:20 +01:00
Geoffrey Frogeye
a5e768fe00
Filtering by IP range
...
Closes #5
2019-12-07 13:56:04 +01:00
Geoffrey Frogeye
28e33dcc7a
Fixed description generation
2019-12-05 20:51:53 +01:00
Geoffrey Frogeye
95d4535abd
Nitpicking
2019-12-05 19:38:26 +01:00
Geoffrey Frogeye
025370bbbe
Splitted list with curated and not curated
...
Closes #2
2019-12-05 19:15:24 +01:00
Geoffrey Frogeye
1c20963ffd
Removed third-parties from easyprivacy
2019-12-05 01:19:10 +01:00
Geoffrey Frogeye
188a8f7455
Removed another source of false-positives
2019-12-05 00:50:32 +01:00
Geoffrey Frogeye
f2bab3ca3f
Added contact information
2019-12-03 21:45:29 +01:00
Geoffrey Frogeye
08f25e26ba
Removed false-positive source
...
Also had edgekey.net for blocking.
Thanks @TorchedPoseidon for the report!
2019-12-03 21:27:37 +01:00
Geoffrey Frogeye
8c744d621e
Removed too restrictive source
...
Was blocking ssl.ovh.net and akaimi.net
2019-12-03 18:43:23 +01:00
Geoffrey Frogeye
fe5f0c6c05
Added more rule sources
2019-12-03 17:33:46 +01:00
Geoffrey Frogeye
0159c6037c
Improved DNS resolving performances
...
Also various fixes.
Also some debug stuff, make sure to remove that later.
2019-12-03 15:35:21 +01:00
Geoffrey Frogeye
c609b90390
Append top 1M subdomains rather than replacing it
2019-12-03 09:04:19 +01:00
Geoffrey Frogeye
69b82d29fd
Improved rules handling
...
Rules can now come in 3 different formats:
- AdBlock rules
- Host lists
- Domains lists
All will be converted into domain lists and aggregated
(only AdBlock rules matching a whole domain will be kept).
Subdomains will now be matched if it is a subdomain of any domain of the
rule.
It is way faster (seconds rather than hours!) but less flexible
(although it shouldn't be a problem).
2019-12-03 08:48:12 +01:00
Geoffrey Frogeye
c23004fbff
Separated DNS resolution from filtering
...
This effectively removes the parallelism of filtering,
which doubles the processing time (5->8 hours),
but this allows me to toy around with the performances of this step,
which I aim to improve drastically.
2019-12-02 19:03:08 +01:00
Geoffrey Frogeye
7d01d016a5
Can now use AdBlock lists for tracking matching
...
It's not very performant by itself, especially since pyre2 isn't
maintained nor really compilableinstallable anymore.
The performance seems to have decreased from 200 req/s to 0.2 req/s when
using 512 threads, and to 80 req/s using 64 req/s.
This might or might not be related,as the CPU doesn't seem to be the
bottleneck.
I will probably add support for host-based rules, matching the
subdomains of such hosts (as for now there doesn't seem to be any other
pattern for first-party trackers than subdomains, and this would be a
very broad performace / compatibility with existing lists improvement),
and convert the AdBlock lists to this format, only keeping domains-only
rules.
2019-11-15 08:57:31 +01:00
Geoffrey Frogeye
87bb24c511
Shell typo
2019-11-14 15:40:25 +01:00
Geoffrey Frogeye
300fe8e15e
Added real argument parser
...
Just so we can have color output when running the script :)
2019-11-14 15:37:32 +01:00
Geoffrey Frogeye
88f0bcc648
Refactored for correct retry logic
2019-11-14 15:03:20 +01:00
Geoffrey Frogeye
b343893c72
Merge branch 'master' of git.frogeye.fr:geoffrey/eulaurarien
2019-11-14 13:45:42 +01:00
Geoffrey Frogeye
ae93593930
Statistics about explicit first-parties
2019-11-14 13:31:39 +01:00
Geoffrey Frogeye
bdc691e647
Upped timeout
2019-11-14 13:10:14 +01:00
Geoffrey Frogeye
08a8eaaada
Use threads not subprocesses
...
You dumbo
2019-11-14 12:57:06 +01:00
Geoffrey Frogeye
32377229db
Retry failed requests
2019-11-14 11:35:05 +01:00
Geoffrey Frogeye
04fe454d99
Automatically get top 1M subdomains
2019-11-14 11:23:59 +01:00
Geoffrey Frogeye
7df00fc859
Automatically download nameserver list
2019-11-14 10:56:53 +01:00
Geoffrey Frogeye
1bbc17a8ec
Greatly optimized subdomain filtering
2019-11-14 10:45:06 +01:00
Geoffrey Frogeye
00a0020914
Added some delay for websites subdomains collecting
...
Some websites load their trackers after the page is done loading.
2019-11-14 06:29:24 +01:00
Geoffrey Frogeye