Geoffrey Frogeye
2bcf6cbbf7
Added SINGLE_PROCESS environment variable
3 years ago
Geoffrey Frogeye
b310ca2fc2
Clever pruning mechanism
3 years ago
Geoffrey Frogeye
bb9e6de62f
Profiling is now optional
3 years ago
Geoffrey Frogeye
c543e0eab6
Make multi-processing optional for feed_dns
3 years ago
Geoffrey Frogeye
195f41bd9f
Use smaller cache if it cannot allocate
3 years ago
Geoffrey Frogeye
0e7479e23e
Added handling for IPs too big
3 years ago
Geoffrey Frogeye
9f343ed296
Removed debug print
3 years ago
Geoffrey Frogeye
c65ae94892
Added ability to use Rapid7 API
Closes #11
3 years ago
Geoffrey Frogeye
7d1c1a1d54
Implement pruning
3 years ago
Geoffrey Frogeye
1a6e64da3d
Forgot numpy dependency
3 years ago
Geoffrey Frogeye
d66040a7b6
Added some litterature
Well not really litterature in the scientific term but still something
to read
3 years ago
Geoffrey Frogeye
57e2919f25
Added information about CORS security issue
3 years ago
Geoffrey Frogeye
94acd106da
Acknwoledgments
Gesundheit
3 years ago
Geoffrey Frogeye
885d92dd77
Added LICENSE
3 years ago
Geoffrey Frogeye
8b7e538677
Updated links
(could not bother guessing them)
3 years ago
Geoffrey Frogeye
cd46b39756
Merge branch 'newworkflow'
3 years ago
Geoffrey Frogeye
38cf532854
Updated README
Split in two actually (program and list).
Closes #3
Also,
Closes #1
Because I forgot to do it earlier.
3 years ago
Geoffrey Frogeye
53b14c6ffa
Removed TODO placeholders in commands description
It's better than nothing but not by that much
4 years ago
Geoffrey Frogeye
c81be4825c
Automated tests
Very rudimentary but should do the trick
Closes #4
4 years ago
Geoffrey Frogeye
4a22054796
Added optional cache for faster IP matching
4 years ago
Geoffrey Frogeye
06b745890c
Added other first-party trackers
4 years ago
Geoffrey Frogeye
aca5023c3f
Fixed scripting around
4 years ago
Geoffrey Frogeye
dce35cb299
Harder verficiation before adding entries to DB
4 years ago
Geoffrey Frogeye
747fe46ad0
Script to automatically download from Rapid7 datasets
4 years ago
Geoffrey Frogeye
b43cb1725c
Autosave
Not needed but since the import may take multiple hour I get frustrated
if this gets interrupted for some reason.
4 years ago
Geoffrey Frogeye
f5c60c482a
Merge branch 'master' of git.frogeye.fr:geoffrey/eulaurarien
4 years ago
Geoffrey Frogeye
12ecfa1a5d
Added outdated documentation warning in README
4 years ago
Geoffrey Frogeye
e882e09b37
Added outdated documentation warning in README
4 years ago
Geoffrey Frogeye
d65107f849
Save dupplicates too
Maybe I won't publish them but this will help me for tracking trackers.
4 years ago
Geoffrey Frogeye
ea0855bd00
Forgot to push this little guy
Good thing I cleaned up my working directory.
It only exists because pickles created from database.py itself
won't be openable from a file simply importing databse.py.
So we create it when in 'imported state'.
4 years ago
Geoffrey Frogeye
7851b038f5
Reworked rule export
4 years ago
Geoffrey Frogeye
8f6e01c857
Added first_party tracking
Well, tracking if a rule is from a first or a multi rule...
Hope I did not do any mistake
4 years ago
Geoffrey Frogeye
c3bf102289
Made references work
4 years ago
Geoffrey Frogeye
03a4042238
Added level
Also fixed IP logic because this was real messed up
4 years ago
Geoffrey Frogeye
3197fa1663
Remove list usage for IpTreeNode
4 years ago
Geoffrey Frogeye
a0e68f0848
Reworked match and node system
For level, and first_party later
Next: add get_match to retrieve level of source and have correct levels
... am I going somewhere with all this?
4 years ago
Geoffrey Frogeye
aec8d3f8de
Reworked how paths work
Get those tuples out of my eyes
4 years ago
Geoffrey Frogeye
7af2074c7a
Small optimisation of feed_switch
4 years ago
Geoffrey Frogeye
45325782d2
Multi-processed parser
4 years ago
Geoffrey Frogeye
ce52897d30
Smol fixes
4 years ago
Geoffrey Frogeye
954b33b2a6
Slightly better Rapid7 parser
4 years ago
Geoffrey Frogeye
d976752797
Store Ip4Path as int instead of List[int]
4 years ago
Geoffrey Frogeye
4d966371b2
Workflow: SQL -> Tree
Welp. All that for this.
4 years ago
Geoffrey Frogeye
040ce4c14e
Typo in source
4 years ago
Geoffrey Frogeye
b50c01f740
Merge branch 'master' into newworkflow
4 years ago
Geoffrey Frogeye
ddceed3d25
Workflow: Can now import DnsMass output
Well, in a specific format but DnsMass nonetheless
4 years ago
Geoffrey Frogeye
189deeb559
Workflow: Multiprocess
Still trying.
It's better than multithread though.
Merge branch 'newworkflow' into newworkflow_threaded
4 years ago
Geoffrey Frogeye
d7c239a6f6
Workflow: Some modifications
4 years ago
Geoffrey Frogeye
5023b85d7c
Added intermediate representation for DNS datasets
It's just CSV.
The DNS from the datasets are not ordered consistently,
so we need to parse it completly.
It seems that converting to an IR before sending data to ./feed_dns.py
through a pipe is faster than decoding the JSON in ./feed_dns.py.
This will also reduce the storage of the resolved subdomains by
about 15% (compressed).
4 years ago
Geoffrey Frogeye
269b8278b5
Worflow: Fixed rules counts
4 years ago
