Skip to content

Commit

Permalink
Feature: CSV export supports detailed analysis
Browse files Browse the repository at this point in the history
rfc-st committed Dec 28, 2024
1 parent 5a42376 commit 8327a17
Showing 8 changed files with 1,300 additions and 1,255 deletions.
10 changes: 5 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
<a target="_blank" href="https://www.python.org/downloads/" title="Minimum Python version required to run this tool"><img src="https://img.shields.io/badge/Python-%3E%3D3.8-blue?labelColor=343b41"></a>
<a target="_blank" href="LICENSE" title="License of this tool"><img src="https://img.shields.io/badge/License-MIT-blue.svg?labelColor=343b41"></a>
<a target="_blank" href="https://github.com/rfc-st/humble/releases" title="Latest release of this tool"><img src="https://img.shields.io/github/v/release/rfc-st/humble?display_name=release&label=Latest%20Release&labelColor=343b41"></a>
<a target="_blank" href="https://github.com/rfc-st/humble/commits/master" title="Latest commit of this tool"><img src="https://img.shields.io/badge/Latest_Commit-2024--12--24-blue.svg?labelColor=343b41"></a>
<a target="_blank" href="https://github.com/rfc-st/humble/commits/master" title="Latest commit of this tool"><img src="https://img.shields.io/badge/Latest_Commit-2024--12--28-blue.svg?labelColor=343b41"></a>
<a target="_blank" href="https://github.com/rfc-st/humble/actions?query=workflow%3ACodeQL" title="Results of the last analysis of this tool with CodeQL"><img src="https://github.com/rfc-st/humble/workflows/CodeQL/badge.svg"></a>
<a target="_blank" href="https://pkg.kali.org/pkg/humble" title="Official tool in Kali Linux"><img src="https://img.shields.io/badge/Kali%20Linux-Tool-blue?labelColor=343b41"></a>
<br />
@@ -132,7 +132,7 @@ Options used: -f -g -p -U -s --hints
<img src="https://github.com/rfc-st/humble/blob/master/screenshots/humble_fng.jpg" alt="(Linux) - List of HTTP fingerprint headers based on a specific term" width=70% height=70%>
</p>
<br />
.: (Linux) - Brief analysis saved as CSV. <a href="https://github.com/rfc-st/humble/raw/master/samples/humble_https_facebook_com_20241121_174721_en.csv">Example.</a><br />
.: (Linux) - Detailed analysis saved as CSV. <a href="https://github.com/rfc-st/humble/raw/master/samples/humble_https_facebook_com_20241228_202046_en.csv">Example.</a><br />
<p></p>
<p align="center">
<img src="https://github.com/rfc-st/humble/blob/master/screenshots/humble_csv_s.PNG" alt="(Linux) - Brief analysis saved as CSV" width=70% height=70%>
@@ -264,7 +264,7 @@ $ docker rmi humble:1.42
usage: humble.py [-h] [-a] [-b] [-c] [-df] [-e [TESTSSL_PATH]] [-f [FINGERPRINT_TERM]] [-g] [-grd] [-if INPUT_FILE] [-l {es}] [-lic] [-o {csv,html,json,pdf,txt,xml}]
[-of OUTPUT_FILE] [-op OUTPUT_PATH] [-r] [-s [SKIP_HEADERS ...]] [-u URL] [-ua USER_AGENT] [-v]

'humble' (HTTP Headers Analyzer) | https://github.com/rfc-st/humble | v.2024-12-20
'humble' (HTTP Headers Analyzer) | https://github.com/rfc-st/humble | v.2024-12-28

options:
-h, --help show this help message and exit
@@ -276,10 +276,10 @@ options:
-f [FINGERPRINT_TERM] Shows fingerprint statistics; if 'FINGERPRINT_TERM' (e.g., 'Google') is omitted the top 20 results will be shown
-g Shows guidelines for enabling security HTTP response headers on popular frameworks, servers and services
-grd Shows the checks to grade an analysis, along with advice for improvement
-if INPUT_FILE Analyzes 'INPUT_FILE': must contain HTTP response headers and values separated by ': '; E.g. 'server: nginx'.
-if INPUT_FILE Analyzes 'INPUT_FILE': must contain HTTP response headers and values separated by ': '; E.g. 'server: nginx'
-l {es} Defines the language for displaying analysis, errors and messages; if omitted, will be shown in English
-lic Shows the license for 'humble', along with permissions, limitations and conditions.
-o {csv,html,json,pdf,txt,xml} Exports analysis to 'humble_scheme_URL_port_yyyymmdd_hhmmss_language.ext' file; csv/json/xml will have a brief analysis
-o {csv,html,json,pdf,txt,xml} Exports analysis to 'humble_scheme_URL_port_yyyymmdd_hhmmss_language.ext' file; json/xml will have a brief analysis
-of OUTPUT_FILE Exports analysis to 'OUTPUT_FILE'; if omitted the default filename of the parameter '-o' will be used
-op OUTPUT_PATH Exports analysis to 'OUTPUT_PATH'; must be absolute. If omitted the PATH of 'humble.py' will be used
-r Shows HTTP response headers and a detailed analysis; '-b' parameter will take priority
2,372 changes: 1,186 additions & 1,186 deletions additional/fingerprint.txt

Large diffs are not rendered by default.

14 changes: 7 additions & 7 deletions humble.py
Original file line number Diff line number Diff line change
@@ -133,7 +133,7 @@
URL_STRING = ('rfc-st', ' URL : ', 'caniuse')

current_time = datetime.now().strftime("%Y/%m/%d - %H:%M:%S")
local_version = datetime.strptime('2024-12-24', '%Y-%m-%d').date()
local_version = datetime.strptime('2024-12-28', '%Y-%m-%d').date()


class SSLContextAdapter(requests.adapters.HTTPAdapter):
@@ -766,7 +766,7 @@ def print_basic_info(export_filename):
def print_extended_info(args, reliable, status_code):
if args.skip_headers:
print_skipped_headers(args)
if args.output in ('csv', 'json', 'xml'):
if args.output in ('json', 'xml'):
print(get_detail('[limited_analysis_note]', replace=True))
if (status_code is not None and 400 <= status_code <= 451) or reliable or \
args.redirects or args.skip_headers:
@@ -857,7 +857,7 @@ def get_fingerprint_headers():
encoding='utf8') as fng_source:
l_fng_ex = [line.strip() for line in
islice(fng_source, SLICE_INT[0], None) if line.strip()]
l_fng = [line.split(' [')[0].strip() for line in l_fng_ex]
l_fng = [line.split(' (')[0].strip() for line in l_fng_ex]
titled_fng = [item.title() for item in l_fng]
return l_fng_ex, l_fng, titled_fng

@@ -1358,7 +1358,7 @@ def format_html_fingerprint(args, ln, sub_d, l_fng):
for i in l_fng:
if (ln and i in ln and not args.brief):
try:
idx = ln.index(' [')
idx = ln.index(' (')
except ValueError:
return ln
if 'class="ko"' not in ln:
@@ -1700,7 +1700,7 @@ def custom_help_formatter(prog):
the checks to grade an analysis, along with advice for improvement")
parser.add_argument("-if", dest='input_file', type=str, help="Analyzes \
'INPUT_FILE': must contain HTTP response headers and values separated by ': ';\
E.g. 'server: nginx'.")
E.g. 'server: nginx'")
parser.add_argument("-l", dest='lang', choices=['es'], help="Defines the \
language for displaying analysis, errors and messages; if omitted, will be \
shown in English")
@@ -1709,7 +1709,7 @@ def custom_help_formatter(prog):
parser.add_argument("-o", dest='output', choices=['csv', 'html', 'json', 'pdf',
'txt', 'xml'], help="Exports\
analysis to 'humble_scheme_URL_port_yyyymmdd_hhmmss_language.ext' file; \
csv/json/xml will have a brief analysis")
json/xml will have a brief analysis")
parser.add_argument("-of", dest='output_file', type=str, help="Exports \
analysis to 'OUTPUT_FILE'; if omitted the default filename of the parameter \
'-o' will be used")
@@ -1795,7 +1795,7 @@ def custom_help_formatter(prog):
args.URL_A is None):
print_error_detail('[args_several]')

if args.output in ['csv', 'json', 'xml'] and not args.brief:
if args.output in ['json', 'xml'] and not args.brief:
print_error_detail('[args_brief_filetype]')

skip_list, unsupported_headers = [], []
4 changes: 2 additions & 2 deletions l10n/details.txt
Original file line number Diff line number Diff line change
@@ -872,7 +872,7 @@ HTTP Response Headers
Note : The analysis may not be reliable because of the time it took for the URL to respond.

[limited_analysis_note]
Note : Exporting to CSV/JSON/XML is currently limited to a brief analysis
Note : Exporting to JSON/XML is currently limited to a brief analysis

[analysis_redirects_note]
Note : The exact URL will be analyzed, without following redirects.
@@ -1558,7 +1558,7 @@ Error: The parameters '-b', '-c', '-df', '-'o', '-r' and '-s' require the parame
Error: The parameter '-of' requires the parameters '-u' and '-o'.

[args_brief_filetype]
Error: The parameters '-o csv', '-o json' and '-o xml' require the parameter '-b'.
Error: The parameters '-o json' and '-o xml' require the parameter '-b'.

[notestssl_file]
Error: 'testssl.sh' is not found in that PATH.
4 changes: 2 additions & 2 deletions l10n/details_es.txt
Original file line number Diff line number Diff line change
@@ -874,7 +874,7 @@ Cabeceras de respuesta HTTP
Nota : El análisis puede no ser fiable por el tiempo que tardó en responder la URL.

[limited_analysis_note]
Nota: Exportar a CSV/JSON/XML limita actualmente a un análisis breve
Nota: Exportar a JSON/XML limita actualmente a un análisis breve

[analysis_redirects_note]
Nota : Se analizará la URL exacta, sin seguir redireccionamientos.
@@ -1563,7 +1563,7 @@ Error: Los parámetros '-b', 'c', '-df', -'o', '-r' y '-s' requieren el parámet
Error: El parámetro -of' requiere los parámetros '-u' y '-o'.

[args_brief_filetype]
Error: Los parámetros '-o csv', '-o json' y '-o xml' requieren el parámetro '-b'.
Error: Los parámetros '-o json' y '-o xml' requieren el parámetro '-b'.

[notestssl_file]
Error: 'testssl.sh' no existe en la ruta indicada.
53 changes: 0 additions & 53 deletions samples/humble_https_facebook_com_20241121_174721_en.csv

This file was deleted.

98 changes: 98 additions & 0 deletions samples/humble_https_facebook_com_20241228_202046_en.csv
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
"Section","Values"
"[0. Info]","Date: 2024/12/28 - 20:20:46"
"[0. Info]","URL: https://facebook.com"
"[0. Info]","File: humble_https_facebook_com_20241228_202046_en.csv"
"[0. Info]","Note: The URL returns an error (HTTP code 400, 'Bad Request')"
"[0. Info]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/400"
"[1. Enabled HTTP Security Headers]","Cache-Control: private, no-cache, no-store, must-revalidate"
"[1. Enabled HTTP Security Headers]","Content-Security-Policy: default-src data: blob: 'self' https://*.fbsbx.com *.facebook.com *.fbcdn.net;script-src *.facebook.com *.fbcdn.net 'unsafe-inline' blob: data: 'self' 'wasm-unsafe-eval';style-src *.fbcdn.net data: *.facebook.com 'unsafe-inline';connect-src *.facebook.com facebook.com *.fbcdn.net wss://*.facebook.com:* wss://*.fbcdn.net attachment.fbsbx.com blob: *.cdninstagram.com 'self' http://localhost:3103 wss://gateway.facebook.com wss://edge-chat.facebook.com wss://snaptu-d.facebook.com wss://kaios-d.facebook.com/ *.fbsbx.com;font-src data: *.facebook.com *.fbcdn.net *.fbsbx.com;img-src *.fbcdn.net *.facebook.com data: https://*.fbsbx.com facebook.com *.cdninstagram.com fbsbx.com fbcdn.net blob: android-webview-video-poster: *.oculuscdn.com https://paywithmybank.com/ https://*.paywithmybank.com/;media-src *.cdninstagram.com blob: *.fbcdn.net *.fbsbx.com www.facebook.com *.facebook.com data:;frame-src *.facebook.com *.fbsbx.com fbsbx.com data: *.fbcdn.net https://paywithmybank.com/ https://*.paywithmybank.com/;worker-src blob: *.facebook.com data:;block-all-mixed-content;upgrade-insecure-requests;report-uri https://www.facebook.com/csp/reporting/?m=c&minimize=0;"
"[1. Enabled HTTP Security Headers]","Content-Type: text/html; charset=""utf-8"""
"[1. Enabled HTTP Security Headers]","Cross-Origin-Opener-Policy: same-origin-allow-popups;report-to=""coop_report"""
"[1. Enabled HTTP Security Headers]","(*) Origin-Agent-Cluster: ?1"
"[1. Enabled HTTP Security Headers]","Pragma: no-cache"
"[1. Enabled HTTP Security Headers]","Report-To: {""max_age"":2592000,""endpoints"":"
"[2. Missing HTTP Security Headers]","Clear-Site-Data"
"[2. Missing HTTP Security Headers]","Clears browsing data (cookies, storage, cache) associated with the requesting website."
"[2. Missing HTTP Security Headers]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Clear-Site-Data"
"[2. Missing HTTP Security Headers]","Cross-Origin-Embedder-Policy"
"[2. Missing HTTP Security Headers]","Prevents documents and workers from loading non-same-origin requests unless allowed."
"[2. Missing HTTP Security Headers]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Cross-Origin-Embedder-Policy"
"[2. Missing HTTP Security Headers]","Cross-Origin-Resource-Policy"
"[2. Missing HTTP Security Headers]","Protect servers against certain cross-origin or cross-site embedding of the returned source."
"[2. Missing HTTP Security Headers]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Cross-Origin_Resource_Policy_(CORP)"
"[2. Missing HTTP Security Headers]","(*) NEL"
"[2. Missing HTTP Security Headers]","Enables web applications to declare a reporting policy to report errors."
"[2. Missing HTTP Security Headers]","Ref: https://scotthelme.co.uk/network-error-logging-deep-dive/"
"[2. Missing HTTP Security Headers]","(*) Permissions-Policy"
"[2. Missing HTTP Security Headers]","Previously called ""Feature-Policy"", allow and deny the use of browser features."
"[2. Missing HTTP Security Headers]","Ref: https://scotthelme.co.uk/goodbye-feature-policy-and-hello-permissions-policy/"
"[2. Missing HTTP Security Headers]","Referrer-Policy"
"[2. Missing HTTP Security Headers]","Controls how much referrer information should be included with requests."
"[2. Missing HTTP Security Headers]","Ref: https://scotthelme.co.uk/a-new-security-header-referrer-policy/"
"[2. Missing HTTP Security Headers]","X-Permitted-Cross-Domain-Policies"
"[2. Missing HTTP Security Headers]","Limit which data external resources (e.g. Adobe Flash/PDF documents), can access on the domain."
"[2. Missing HTTP Security Headers]","Ref: https://owasp.org/www-project-secure-headers/#div-headers"
"[3. Fingerprint HTTP Response Headers]","These headers can leak information about software, versions, hostnames or IP addresses: "
"[3. Fingerprint HTTP Response Headers]","X-FB-Debug (facebook.com Platform)"
"[3. Fingerprint HTTP Response Headers]","Value: 'GOgszca9qLxKjM9jDeLK82LrEt4xNtj69s/dgFjLpcsasvcn+kOVZUlfIbDDjtTID4yeUGFkGsAYJb+hFuSrVQ=='"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","The following headers/protocols are deprecated or their values may be considered unsafe: "
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Content-Security-Policy (Deprecated Directives)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Avoid using deprecated directives: 'report-uri', 'block-all-mixed-content'"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Security-Policy"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Content-Security-Policy (Insecure Schemes)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Do not allow insecure, unencrypted schemes: 'http:'"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://www.cloudflare.com/learning/ssl/why-is-http-not-secure/"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://http.dev/wss"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Content-Security-Policy (Too Permissive Sources)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Limit these permissive origins: 'data:', 'blob:'"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://content-security-policy.com/"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Content-Security-Policy (Unsafe Values)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","'unsafe-inline' and 'unsafe-eval' negate most of the security benefits provided by this header."
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://csper.io/blog/no-more-unsafe-inline"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/eval"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Pragma (Deprecated Header)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","This header is deprecated."
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Pragma"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Report-To (Deprecated Header)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","This header is deprecated. Use instead ""Reporting-Endpoints""."
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Report-To"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Strict-Transport-Security (Recommended Values)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Add 'includeSubDomains' and 'max-age' (with 31536000 -one year- as minimum)."
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://https.cio.gov/hsts/"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Strict-Transport-Security (Required Values)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","'preload' requires 'includeSubDomains' and 'max-age' (with 31536000 -one year- as minimum)."
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Strict-Transport-Security"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Vary (Potentially Unsafe Header)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","The values of this header may expose others, facilitating attacks if user input is accepted."
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Vary"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://www.yeswehack.com/fr/learn-bug-bounty/http-header-exploitation"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","X-XSS-Protection (Deprecated Header)"
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","This header is deprecated in the three major web browsers."
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Instead, use the ""Content-Security-Policy"" header restrictively."
"[4. Deprecated HTTP Response Headers/Protocols and Insecure Values]","Ref: https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/X-XSS-Protection"
"[5. Empty HTTP Response Headers Values]","Empty HTTP headers (and are therefore considered disabled): "
"[5. Empty HTTP Response Headers Values]","Nothing to report, all seems OK!"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Cache-Control: https://caniuse.com/?search=Cache-Control"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Content-Security-Policy: https://caniuse.com/?search=contentsecuritypolicy2"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Content-Type: https://caniuse.com/?search=Content-Type"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Cross-Origin-Opener-Policy: https://caniuse.com/?search=Cross-Origin-Opener-Policy"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Origin-Agent-Cluster: https://caniuse.com/?search=Origin-Agent-Cluster"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Pragma: https://caniuse.com/?search=Pragma"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Report-To: https://caniuse.com/?search=Report-To"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Reporting-Endpoints: https://caniuse.com/?search=Reporting-Endpoints"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Strict-Transport-Security: https://caniuse.com/?search=Strict-Transport-Security"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","Vary: https://caniuse.com/?search=Vary"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","X-Content-Type-Options: https://caniuse.com/?search=X-Content-Type-Options"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","X-Frame-Options: https://caniuse.com/?search=X-Frame-Options"
"[6. Browser Compatibility for Enabled HTTP Security Headers]","X-XSS-Protection: https://caniuse.com/?search=X-XSS-Protection"
"[7. Analysis Results]","Done in 0.19 seconds! (changes with respect to the last analysis in parentheses)"
"[7. Analysis Results]","Enabled headers: 12 (First Analysis)"
"[7. Analysis Results]","Missing headers: 7 (First Analysis)"
"[7. Analysis Results]","Fingerprint headers: 1 (First Analysis)"
"[7. Analysis Results]","Deprecated/Insecure headers: 10 (First Analysis)"
"[7. Analysis Results]","Empty headers: 0 (First Analysis)"
"[7. Analysis Results]","Findings to review: 18 (First Analysis)"
"[7. Analysis Results]","Analysis Grade: D (Review 'Deprecated/Insecure headers')"
"[7. Analysis Results]","'(*)' meaning: Experimental HTTP response header"
"[7. Analysis Results]","'(*)' ref: https://mdn.io/Experimental_deprecated_obsolete"
Binary file modified screenshots/humble_csv_s.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit 8327a17

Please sign in to comment.