Synthetic data generation

[iamgroot42]

Hi.

Fascinating work! Nice to finally see a usable collection of network-based datasets for GNN driven. Could you please add the code used for synthetic graph generation? I'm looking to generate more graphs with some additional constraints, so I'd appreciate even any useful links or pointers to generation code used by the authors 😸

[jzhou316]

Hi @iamgroot42 thanks for your interest! We are working to sort some code for the graph generation part. Will point to you asap!

[iamgroot42]

@jzhou316 thanks! Looking forward to it :D

[jzhou316]

Hi @iamgroot42 we have added the code for generating botnets for detection (overlaying synthetic botnet topologies on real network traces) here. Hope that can give you some ideas!

[iamgroot42]

Thanks a lot, @jzhou316 ! Really appreciate the effort :)

[iamgroot42]

@jzhou316 I had a follow-up question on the exact process used to generate data used in the paper (and available via download links).

• What timestamp (I can see a particular one is used as the default value in botnetGenerator.py) files were used?
• What time ranges are used?

[jzhou316]

looping in @xuzhiying9510 for better explanation

[xuzhiying9510]

@iamgroot42 Thanks for the question!

• We generate one graph per minute from the CAIDA Anonymized Internet Traces Dataset.
• We consider all available datasets in 2018 from the IP backbone equinix-nyc, where an one-hour trace are collected each month.

[iamgroot42]

@xuzhiying9510, thanks for the clarification!

• Each month's directory has 2 files per minute: dirA and dirB. Do you combine both their pcap files to generate the graph for that particular minute?
• I was looking at the dataset, and in total, there are more than 1200 graphs in total for 2018, whereas the current dataset has less than 1000. Did you perform some form of thresholding to not consider some graphs (I saw some of the pcap files were only a few MBs, while others were in GBs)?

[iamgroot42]

Bumping this up again @xuzhiying9510 @jzhou316 :) I would greatly appreciate it if you could help with these clarifications!

[iamgroot42]

@jzhou316 would it be possible to get the underlying .pcap files (the ones before botnets are overlayed on top of them)? I've been trying to recreate the graphs based on instructions and the provided code for generation, but the generated graphs are FAR from the current graphs - they have around 272K nodes and 949K edges, and are much more sparse than the graphs provided as part of the dataset.

[jzhou316]

Hi @iamgroot42 I see. There might be some processing there. We'll check about it and get back to you.

[iamgroot42]

Clarification (fix) added via #14 Thanks a lot for looking into it!