geoffrey/eulaurarien
1
0
Fork 0
Code Issues 6 Pull requests Releases 1 Wiki Activity
68 commits 5 branches 12 tags 269 KiB
Commit graph

10 commits

Author SHA1 Message Date
Geoffrey Frogeye 189deeb559
Workflow: Multiprocess 
Still trying.
It's better than multithread though.

Merge branch 'newworkflow' into newworkflow_threaded
 2019-12-14 17:27:46 +01:00
Geoffrey Frogeye d7c239a6f6 Workflow: Some modifications 2019-12-14 16:04:19 +01:00
Geoffrey Frogeye 5023b85d7c
Added intermediate representation for DNS datasets 
It's just CSV.
The DNS from the datasets are not ordered consistently,
so we need to parse it completly.
It seems that converting to an IR before sending data to ./feed_dns.py
through a pipe is faster than decoding the JSON in ./feed_dns.py.
This will also reduce the storage of the resolved subdomains by
about 15% (compressed).
 2019-12-13 21:59:35 +01:00
Geoffrey Frogeye ab7ef609dd
Workflow: Various optimisations and fixes 
I forgot to close this one earlier, so:
Closes #7
 2019-12-13 18:08:22 +01:00
Geoffrey Frogeye f3eedcba22
Updated now based on timestamp 
Did I forget to add feed_asn.py a few commits ago?
Oh well...
 2019-12-13 13:54:00 +01:00
Geoffrey Frogeye 231bb83667
Threaded feed_dns 
Largely disapointing
 2019-12-13 12:36:11 +01:00
Geoffrey Frogeye 9050a84670
Read-only mode 2019-12-13 12:35:05 +01:00
Geoffrey Frogeye 57416b6e2c
Workflow: POO and individual tables per types 
Mostly for performances reasons.
First one to implement threading later.
Second one to speed up the dichotomy,
but it doesn't seem that much better so far.
 2019-12-13 00:11:21 +01:00
Geoffrey Frogeye 55877be891
IP parsing C accelerated, use bytes everywhere 2019-12-09 09:47:48 +01:00
Geoffrey Frogeye 7937496882
Workflow: Base for new one 
While I'm automating this you'll need to download the A set from
https://opendata.rapid7.com/sonar.fdns_v2/ to the file a.json.gz.
 2019-12-09 08:12:48 +01:00