files aren't downloaded #182
Comments
|
I can confirm.. nothing is being downloaded. |
|
Thanks for the follow up. If you have any suggestions as to how to get the script working again, please let me know. I've found this script to be extremely useful and appreciate you making it available. |
|
Add sudo before 'python maltrieve.py' or change python to python3 |
|
Sorry, I gave you an wrong answer just now. But I got it now. to: def process_urlquery(response): |
|
Hello,
Thanks for your response. I made the suggested changes and the script still isn’t downloading files. Please let me know if you have any additional suggestions. Thanks.
def process_urlquery(response):
soup = BeautifulSoup(response, "html.parser")
urls = set()
for t in soup.find_all("table", class_="test"):
for a in t.find_all("a"):
urls.add('http://' + re.sub('&', '&', a.text))
return urls
root@ubuntu:~/scripts/maltrieve-master# python maltrieve.py
Processing source URLs
Completed source processing
Downloading samples, check log for details
Completed downloads
From: hi-T0day [mailto:[email protected]]
Sent: Saturday, March 04, 2017 7:31 AM
To: krmaxwell/maltrieve <[email protected]>
Cc: Lindsay Hunt <[email protected]>; Author <[email protected]>
Subject: Re: [krmaxwell/maltrieve] files aren't downloaded (#182)
Sorry, I gave you an wrong answer just now. But I got it now.
You can change in "maltrieve.py"
def process_urlquery(response):
soup = BeautifulSoup(response)
urls = set()
for t in soup.find_all("table", class_="test"):
for a in t.find_all("a"):
urls.add('http://' + re.sub('&', '&', a.text))
return urls
to:
def process_urlquery(response):
soup = BeautifulSoup(response, "html.parser")
urls = set()
for t in soup.find_all("table", class_="test"):
for a in t.find_all("a"):
urls.add('http://' + re.sub('&', '&', a.text))
return urls
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_krmaxwell_maltrieve_issues_182-23issuecomment-2D284148635&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=F3HbLa-PYZ_cfGJw-BWkR8CsJX-ZYnlKAn5rGrHLKdo&m=q5gVyZzvxIN7Ph17gPhHTO7Q4aRkyOZ3mFKqvntA0Is&s=Bo83Xt6s_y-i4zVfjz2RploQJkZU9XGrLykpI64rA1I&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AYWdcofBGCSbHHGMCdu44BJXxVTK8oyxks5riVl5gaJpZM4L3jQB&d=DwMFaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=F3HbLa-PYZ_cfGJw-BWkR8CsJX-ZYnlKAn5rGrHLKdo&m=q5gVyZzvxIN7Ph17gPhHTO7Q4aRkyOZ3mFKqvntA0Is&s=bYq5lVPDDGgRSi42bxp-2wHPiMbGqrvR1YoaWdCdWz4&e=>.
|
|
Having the same issue. attrs==15.2.0 user1@ubuntu-template:~/maltrieve/maltrieve-0.7/files$ lsb_release -a user1@ubuntu-template:~/maltrieve/maltrieve-0.7$ sudo ./maltrieve.py user1@ubuntu-template:~/maltrieve/maltrieve-0.7$ more urls.json user1@ubuntu-template: |
|
I use another branch:https://github.com/HarryR/maltrieve. Now it works. I believe that you can success too. |
|
Hello,
Please see below and let me know if you have any suggestions.
python maltrieve.py -d /home/lhunt/malware/030817/
Traceback (most recent call last):
File "maltrieve.py", line 580, in <module>
main()
File "maltrieve.py", line 520, in main
cfg = config(args, 'maltrieve.cfg')
File "maltrieve.py", line 131, in __init__
self.cuckoo_dist = self.configp.get('Maltrieve', 'cuckoo_dist')
File "/usr/lib/python2.7/ConfigParser.py", line 623, in get
return self._interpolate(section, option, value, d)
File "/usr/lib/python2.7/ConfigParser.py", line 669, in _interpolate
option, section, rawval, e.args[0])
ConfigParser.InterpolationMissingOptionError: Bad value substitution:
section: [Maltrieve]
option : cuckoo_dist
key : dist_port_9003_tcp_addr
rawval : http://%(DIST_PORT_9003_TCP_ADDR)s:9003<http://%25(DIST_PORT_9003_TCP_ADDR)s:9003>
sudo python maltrieve.py
Traceback (most recent call last):
File "maltrieve.py", line 580, in <module>
main()
File "maltrieve.py", line 520, in main
cfg = config(args, 'maltrieve.cfg')
File "maltrieve.py", line 131, in __init__
self.cuckoo_dist = self.configp.get('Maltrieve', 'cuckoo_dist')
File "/usr/lib/python2.7/ConfigParser.py", line 623, in get
return self._interpolate(section, option, value, d)
File "/usr/lib/python2.7/ConfigParser.py", line 669, in _interpolate
option, section, rawval, e.args[0])
ConfigParser.InterpolationMissingOptionError: Bad value substitution:
section: [Maltrieve]
option : cuckoo_dist
key : dist_port_9003_tcp_addr
rawval : http://%(DIST_PORT_9003_TCP_ADDR)s:9003<http://%25(DIST_PORT_9003_TCP_ADDR)s:9003>
python3 maltrieve.py
File "maltrieve.py", line 125
self.priority = args.priority
^
TabError: inconsistent use of tabs and spaces in indentation
From: hi-T0day [mailto:[email protected]]
Sent: Wednesday, March 08, 2017 3:20 AM
To: krmaxwell/maltrieve <[email protected]>
Cc: Lindsay Hunt <[email protected]>; Author <[email protected]>
Subject: Re: [krmaxwell/maltrieve] files aren't downloaded (#182)
I use another branch:https://github.com/HarryR/maltrieve<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_HarryR_maltrieve&d=DwMCaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=F3HbLa-PYZ_cfGJw-BWkR8CsJX-ZYnlKAn5rGrHLKdo&m=hnR4CUG_5RW7St8kny3Zj2jYyESlnu1fnxyBNkp7e_w&s=ImMWXvy9JguyGyD18hgz8h_EksXC54OyxhexyLaAmVc&e=>. Now it works. I believe that you can success too.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_krmaxwell_maltrieve_issues_182-23issuecomment-2D284976575&d=DwMCaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=F3HbLa-PYZ_cfGJw-BWkR8CsJX-ZYnlKAn5rGrHLKdo&m=hnR4CUG_5RW7St8kny3Zj2jYyESlnu1fnxyBNkp7e_w&s=06gLrt0YoyxjvFtkm4a7GAvquZxAmJ7e6NTS2c3pEmc&e=>, or mute the thread<https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AYWdcmZoQZP1fBLF6IFcFh5qZLgSybn7ks5rjmSngaJpZM4L3jQB&d=DwMCaQ&c=V9IgWpI5PvzTw83UyHGVSoW3Uc1MFWe5J8PTfkrzVSo&r=F3HbLa-PYZ_cfGJw-BWkR8CsJX-ZYnlKAn5rGrHLKdo&m=hnR4CUG_5RW7St8kny3Zj2jYyESlnu1fnxyBNkp7e_w&s=7jmQV-TS7HXIQUgqqzHYvJ4krDZGvRk2JCBgST_Vbdk&e=>.
|
|
IF you add "#" before line8,9 in file "maltrieve.cfg" can maltrieve work? |
|
upon review, they were already commented out. [Maltrieve] #viper = http://127.0.0.1:8080 Filter Lists are based on mime type NO SPACE BETWEEN ,#black_list = text/html,text/plain |
|
utilized the other branch as mentioned by hi-T0day but still no luck user1@ubuntu-template: hashesh.json/urls.json files are empty 2017-03-13 16:13:32 140601425241856 Loaded urls from urls.json user1@ubuntu-template: |
|
are people still able to get help on issues w/ maltrieve? |
|
bump |
|
confirming that nothing is being downloaded. |
|
Same problem, nothing is downloaded. |
|
I realize questions are 2 years old but i have the same case of this issue, did anybody came up with a solution or cause ? |
Hello,
I'm running Maltrieve on Ubuntu 16.0.4. I installed the dependencies as described in the installation instructions. When I 'python maltrieve.py', the script doesn't download any files. Please see the output below and let me know if you have any suggestions.
python maltrieve.py -d /home/acme/malware/020517
Processing source URLs
Completed source processing
/usr/local/lib/python2.7/dist-packages/bs4/init.py:181: UserWarning: No parser was explicitly specified, so I'm using the best available HTML parser for this system ("lxml"). This usually isn't a problem, but if you run this code on another system, or in a different virtual environment, it may use a different parser and behave differently.
The code that caused this warning is on line 514 of the file maltrieve.py. To get rid of this warning, change code that looks like this:
BeautifulSoup([your markup])
to this:
BeautifulSoup([your markup], "lxml")
markup_type=markup_type))
Downloading samples, check log for details
Completed downloads
tail maltrieve.log
2017-02-05 11:49:24 140020353632000 Starting new HTTP connection (1): malc0de.com
2017-02-05 11:49:29 140020353632000 http://www.malwaredomainlist.com:80 "GET /hostslist/mdl.xml HTTP/1.1" 200 4938
2017-02-05 11:49:29 140020353632000 http://malc0de.com:80 "GET /rss/ HTTP/1.1" 200 None
2017-02-05 11:49:30 140020353632000 http://malwareurls.joxeankoret.com:80 "GET /normal.txt HTTP/1.1" 200 11192
2017-02-05 11:49:30 140020353632000 http://support.clean-mx.de:80 "GET /clean-mx/rss?scope=viruses&limit=0%2C64 HTTP/1.1" 200 918
2017-02-05 11:49:30 140020353632000 http://vxvault.net:80 "GET /URL_List.php HTTP/1.1" 200 None
2017-02-05 11:49:30 140020353632000 https://zeustracker.abuse.ch:443 "GET /monitor.php?urlfeed=binaries HTTP/1.1" 200 3869
2017-02-05 11:49:32 140020353632000 http://urlquery.net:80 "GET / HTTP/1.1" 200 4766
2017-02-05 11:49:33 140020353632000 Dumping past URLs to urls.json
2017-02-05 11:49:33 140020353632000 Dumping hashes to hashes.json
The text was updated successfully, but these errors were encountered: