Skip to content

Commit

Permalink
Feature: New parameter '-if'; allows to analyze 'raw response files'.
Browse files Browse the repository at this point in the history
rfc-st committed Nov 1, 2024
1 parent 83116a8 commit ca780c0
Showing 6 changed files with 100 additions and 15 deletions.
14 changes: 11 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
@@ -6,7 +6,7 @@
<a target="_blank" href="https://www.python.org/downloads/" title="Minimum Python version required to run this tool"><img src="https://img.shields.io/badge/Python-%3E%3D3.8-blue?labelColor=343b41"></a>
<a target="_blank" href="LICENSE" title="License of this tool"><img src="https://img.shields.io/badge/License-MIT-blue.svg?labelColor=343b41"></a>
<a target="_blank" href="https://github.com/rfc-st/humble/releases" title="Latest release of this tool"><img src="https://img.shields.io/github/v/release/rfc-st/humble?display_name=release&label=Latest%20Release&labelColor=343b41"></a>
<a target="_blank" href="https://github.com/rfc-st/humble/commits/master" title="Latest commit of this tool"><img src="https://img.shields.io/badge/Latest_Commit-2024--10--31-blue.svg?labelColor=343b41"></a>
<a target="_blank" href="https://github.com/rfc-st/humble/commits/master" title="Latest commit of this tool"><img src="https://img.shields.io/badge/Latest_Commit-2024--11--01-blue.svg?labelColor=343b41"></a>
<a target="_blank" href="https://github.com/rfc-st/humble/actions?query=workflow%3ACodeQL" title="Results of the last analysis of this tool with CodeQL"><img src="https://github.com/rfc-st/humble/workflows/CodeQL/badge.svg"></a>
<a target="_blank" href="https://pkg.kali.org/pkg/humble" title="Official tool in Kali Linux"><img src="https://img.shields.io/badge/Kali%20Linux-Tool-blue?labelColor=343b41"></a>
<br />
@@ -61,6 +61,7 @@
:heavy_check_mark: Two types of analysis: brief and detailed, along with HTTP response headers.<br />
:heavy_check_mark: Can exclude specific HTTP response headers from the analysis.<br />
:heavy_check_mark: Can export each analysis to CSV, HTML5, JSON, PDF 1.4 and TXT (and in a filename and path of your choice).<br />
:heavy_check_mark: Can analyze '_raw response files_': text files containing HTTP response headers and their values. Ex: curl option '<a href="https://curl.se/docs/manpage.html" target="_blank">--dump-header<a>'.<br />
:heavy_check_mark: Each detailed analysis may include up to dozens of official links, references and technical articles.<br />
:heavy_check_mark: l10n: can display each analysis, the messages and almost all errors in English or Spanish.<br />
:heavy_check_mark: Saves each analysis, showing at the end the improvements or deficiencies in relation to the last one.<br />
@@ -96,6 +97,12 @@
<img src="https://github.com/rfc-st/humble/blob/master/screenshots/humble.PNG" alt="(Linux) - Detailed analysis in Spanish" width=70% height=70%>
</p>
<br />
.: (Linux) - Analysis of a '_raw response file_'. <a href="https://github.com/rfc-st/humble/raw/master/samples/github_input_file.txt">Example.</a><br />
<p></p>
<p align="center">
<img src="https://github.com/rfc-st/humble/blob/master/screenshots/humble_input.PNG" alt="(Linux) - Analysis of a raw response file" width=70% height=70%>
</p>
<br />
.: (Linux) - SSL/TLS checks.<br />
<p></p>

@@ -236,10 +243,10 @@ $ docker rmi humble:1.42
(Linux) $ python3 humble.py
(macOS) $ python3 humble.py

usage: humble.py [-h] [-a] [-b] [-df] [-e [TESTSSL_PATH]] [-f [FINGERPRINT_TERM]] [-g] [-grd] [-l {es}] [-lic] [-o {csv,html,json,pdf,txt}] [-of OUTPUT_FILE]
usage: humble.py [-h] [-a] [-b] [-df] [-e [TESTSSL_PATH]] [-f [FINGERPRINT_TERM]] [-g] [-grd] [-if INPUT_FILE] [-l {es}] [-lic] [-o {csv,html,json,pdf,txt}] [-of OUTPUT_FILE]
[-op OUTPUT_PATH] [-r] [-s [SKIP_HEADERS ...]] [-u URL] [-ua USER_AGENT] [-v]

'humble' (HTTP Headers Analyzer) | https://github.com/rfc-st/humble | v.2024-10-25
'humble' (HTTP Headers Analyzer) | https://github.com/rfc-st/humble | v.2024-11-01

options:
-h, --help show this help message and exit
@@ -250,6 +257,7 @@ options:
-f [FINGERPRINT_TERM] Shows fingerprint statistics; if 'FINGERPRINT_TERM' (e.g., 'Google') is omitted the top 20 results will be shown
-g Shows guidelines for enabling security HTTP response headers on popular servers/services
-grd Shows the checks to grade an analysis, along with advice for improvement
-if INPUT_FILE Analyzes 'INPUT_FILE': must contain HTTP response headers and values separated by ': '; E.g. 'server: nginx'.
-l {es} Defines the language for displaying analysis, errors and messages; if omitted, will be shown in English
-lic Shows the license for 'humble', along with permissions, limitations and conditions.
-o {csv,html,json,pdf,txt} Exports analysis to 'humble_scheme_URL_port_yyyymmdd_hhmmss_language.ext' file; csv/json will have a brief analysis
63 changes: 51 additions & 12 deletions humble.py
Original file line number Diff line number Diff line change
@@ -126,7 +126,7 @@
URL_STRING = ('rfc-st', ' URL : ', 'caniuse')

current_time = datetime.now().strftime("%Y/%m/%d - %H:%M:%S")
local_version = datetime.strptime('2024-10-31', '%Y-%m-%d').date()
local_version = datetime.strptime('2024-11-01', '%Y-%m-%d').date()


class SSLContextAdapter(requests.adapters.HTTPAdapter):
@@ -789,7 +789,7 @@ def get_epilog_content(id_mode):
with open(epilog_file_path, 'r', encoding='utf8') as epilog_source:
epilog_lines = epilog_source.readlines()
epilog_idx = epilog_lines.index(id_mode + '\n')
return ''.join(epilog_lines[epilog_idx+1:epilog_idx+12])
return ''.join(epilog_lines[epilog_idx+1:epilog_idx+14])


def get_fingerprint_headers():
@@ -816,7 +816,13 @@ def print_fingerprint_headers(headers_l, l_fng_ex, titled_fng):
def get_fingerprint_detail(header, headers, idx_fng, l_fng_ex, args):
if not args.brief:
print_fng_header(l_fng_ex[idx_fng])
if not headers[header]:
if '-if' in sys.argv:
if headers_l.get(header.lower()):
print(f" {get_detail('[fng_value]', replace=True)} \
'{headers_l[header.lower()]}'")
else:
print(get_detail('[empty_fng]', replace=True))
elif not headers[header]:
print(get_detail('[empty_fng]', replace=True))
else:
print(f" {get_detail('[fng_value]', replace=True)} \
@@ -1291,6 +1297,21 @@ def check_ru_scope():
sys.exit()


def parse_input_file(input_file):
if not path.exists(input_file):
print_error_detail('[args_inputnotfound]')
input_headers = {}
with open(input_file, 'r', encoding='utf8') as input_source:
for ln in input_source:
ln = ln.strip()
if ': ' in ln:
input_header, input_value = ln.split(': ', 1)
input_headers[input_header.title()] = input_value
reliable = False
status_code = 200
return input_headers, reliable, status_code


def get_tmp_file(args, export_date):
file_ext = '.txt' if args.output == 'txt' else 't.txt'
if args.output_file:
@@ -1439,6 +1460,9 @@ def custom_help_formatter(prog):
services")
parser.add_argument("-grd", dest='grades', action="store_true", help="Shows \
the checks to grade an analysis, along with advice for improvement")
parser.add_argument("-if", dest='input_file', type=str, help="Analyzes \
'INPUT_FILE': must contain HTTP response headers and values separated by ': ';\
E.g. 'server: nginx'.")
parser.add_argument("-l", dest='lang', choices=['es'], help="Defines the \
language for displaying analysis, errors and messages; if omitted, will be \
shown in English")
@@ -1484,6 +1508,17 @@ def custom_help_formatter(prog):

URL = args.URL

if '-if' in sys.argv:
if any([args.redirects, args.ret, args.user_agent]):
print_error_detail('[args_inputfile]')
sys.exit()
elif not args.URL:
print_error_detail('[args_urlinputfile]')
sys.exit()
else:
headers, reliable, status_code = parse_input_file(args.input_file)


if '-ua' in sys.argv:
ua_header = parse_user_agent(user_agent=True)
elif URL:
@@ -1560,11 +1595,15 @@ def custom_help_formatter(prog):
}
requests.packages.urllib3.disable_warnings()

headers, status_code, reliable, body = manage_http_request()
http_equiv = None
if body:
http_equiv = re.findall(RE_PATTERN[8], body, re.IGNORECASE)
headers_l = {header.lower(): value for header, value in headers.items()}
if '-if' not in sys.argv:
headers, status_code, reliable, body = manage_http_request()
http_equiv = None
if body:
http_equiv = re.findall(RE_PATTERN[8], body, re.IGNORECASE)
headers_l = {header.lower(): value for header, value in headers.items()}
else:
http_equiv = None
headers_l = {header.lower(): value for header, value in headers.items()}

# Export filename generation
export_filename = None
@@ -2184,9 +2223,9 @@ def custom_help_formatter(prog):
print_details('[ixcspr_h]', '[ixcspr]', 'd', i_cnt)

if 'x-content-type-options' in headers_l and '61' not in skip_list:
if ',' in headers['X-Content-Type-Options']:
if ',' in headers_l['x-content-type-options']:
print_details('[ictpd_h]', '[ictpd]', 'd', i_cnt)
elif 'nosniff' not in headers['X-Content-Type-Options']:
elif 'nosniff' not in headers_l['x-content-type-options']:
print_details('[ictp_h]', '[ictp]', 'd', i_cnt)

if headers_l.get('x-dns-prefetch-control', '') == 'on' and '62' not in \
@@ -2251,9 +2290,9 @@ def custom_help_formatter(prog):

if 'x-xss-protection' in headers_l and '74' not in skip_list:
print_details('[ixxpdp_h]', '[ixxpdp]', 'm', i_cnt)
if '0' not in headers["X-XSS-Protection"]:
if '0' not in headers_l['x-xss-protection']:
print_details('[ixxp_h]', '[ixxp]', 'm', i_cnt)
if ',' in headers['X-XSS-Protection']:
if ',' in headers_l['x-xss-protection']:
print_details('[ixxpd_h]', '[ixxpd]', 'd', i_cnt)

if args.brief and i_cnt[0] != 0:
9 changes: 9 additions & 0 deletions l10n/details.txt
Original file line number Diff line number Diff line change
@@ -1401,6 +1401,15 @@ Error: That User-Agent ID does not exist; check the available ones with '-ua 0'
[args_notestssl]
Error: The '-e' parameter requires the path of 'testssl.sh' and the parameter '-u'.

[args_inputfile]
Error: The '-if' parameter does not requires the parameters '-df', '-r' or -'ua'.

[args_urlinputfile]
Error: The '-if' parameter requires the parameter '-u'.

[args_inputnotfound]
Error: The input file is not found in the indicated path.

[args_lang]
Error: The '-l' parameter requires the parameters '-u' or '-a'.

9 changes: 9 additions & 0 deletions l10n/details_es.txt
Original file line number Diff line number Diff line change
@@ -1406,6 +1406,15 @@ Error: No existe ese ID de User-Agent; consulta los disponibles con '-ua 0'
[args_notestssl]
Error: El parámetro '-e' requiere la ruta de 'testssl.sh' y el parámetro '-u'.

[args_inputfile]
Error: El parámetro '-if' no requiere los parámetros '-df', '-r' o -'ua'.

[args_urlinputfile]
Error: El parámetro '-if' requiere el parámetro '-u'.

[args_inputnotfound]
Error: No se encuentra el fichero en la ruta indicada.

[args_lang]
Error: El parámetro '-l' requiere de los parámetros '-u' o '-a'.

20 changes: 20 additions & 0 deletions samples/github_input_file.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
HTTP/2 200
server: GitHub.com
date: Fri, 01 Nov 2024 18:40:09 GMT
content-type: text/html; charset=utf-8
vary: X-PJAX, X-PJAX-Container, Turbo-Visit, Turbo-Frame, Accept-Language, Accept-Encoding, Accept, X-Requested-With
content-language: en-US
etag: W/"72bd0f99fc2d1c295e0b5d9d09a8c35d"
cache-control: max-age=0, private, must-revalidate
strict-transport-security: max-age=31536000; includeSubdomains; preload
x-frame-options: deny
x-content-type-options: nosniff
x-xss-protection: 0
referrer-policy: origin-when-cross-origin, strict-origin-when-cross-origin
content-security-policy: default-src 'none'; base-uri 'self'; child-src github.com/assets-cdn/worker/ github.com/webpack/ github.com/assets/ gist.github.com/assets-cdn/worker/; connect-src 'self' uploads.github.com www.githubstatus.com collector.github.com raw.githubusercontent.com api.github.com github-cloud.s3.amazonaws.com github-production-repository-file-5c1aeb.s3.amazonaws.com github-production-upload-manifest-file-7fdce7.s3.amazonaws.com github-production-user-asset-6210df.s3.amazonaws.com *.rel.tunnels.api.visualstudio.com wss://*.rel.tunnels.api.visualstudio.com objects-origin.githubusercontent.com copilot-proxy.githubusercontent.com proxy.individual.githubcopilot.com proxy.business.githubcopilot.com proxy.enterprise.githubcopilot.com *.actions.githubusercontent.com wss://*.actions.githubusercontent.com productionresultssa0.blob.core.windows.net/ productionresultssa1.blob.core.windows.net/ productionresultssa2.blob.core.windows.net/ productionresultssa3.blob.core.windows.net/ productionresultssa4.blob.core.windows.net/ productionresultssa5.blob.core.windows.net/ productionresultssa6.blob.core.windows.net/ productionresultssa7.blob.core.windows.net/ productionresultssa8.blob.core.windows.net/ productionresultssa9.blob.core.windows.net/ productionresultssa10.blob.core.windows.net/ productionresultssa11.blob.core.windows.net/ productionresultssa12.blob.core.windows.net/ productionresultssa13.blob.core.windows.net/ productionresultssa14.blob.core.windows.net/ productionresultssa15.blob.core.windows.net/ productionresultssa16.blob.core.windows.net/ productionresultssa17.blob.core.windows.net/ productionresultssa18.blob.core.windows.net/ productionresultssa19.blob.core.windows.net/ github-production-repository-image-32fea6.s3.amazonaws.com github-production-release-asset-2e65be.s3.amazonaws.com insights.github.com wss://alive.github.com api.githubcopilot.com api.individual.githubcopilot.com api.business.githubcopilot.com api.enterprise.githubcopilot.com github.githubassets.com edge.fullstory.com rs.fullstory.com; font-src github.githubassets.com; form-action 'self' github.com gist.github.com copilot-workspace.githubnext.com objects-origin.githubusercontent.com; frame-ancestors 'none'; frame-src viewscreen.githubusercontent.com notebooks.githubusercontent.com www.youtube-nocookie.com; img-src 'self' data: blob: github.githubassets.com media.githubusercontent.com camo.githubusercontent.com identicons.github.com avatars.githubusercontent.com private-avatars.githubusercontent.com github-cloud.s3.amazonaws.com objects.githubusercontent.com secured-user-images.githubusercontent.com/ user-images.githubusercontent.com/ private-user-images.githubusercontent.com opengraph.githubassets.com github-production-user-asset-6210df.s3.amazonaws.com customer-stories-feed.github.com spotlights-feed.github.com objects-origin.githubusercontent.com *.githubusercontent.com; manifest-src 'self'; media-src github.com user-images.githubusercontent.com/ secured-user-images.githubusercontent.com/ private-user-images.githubusercontent.com github-production-user-asset-6210df.s3.amazonaws.com gist.github.com github.githubassets.com; script-src github.githubassets.com; style-src 'unsafe-inline' github.githubassets.com; upgrade-insecure-requests; worker-src github.com/assets-cdn/worker/ github.com/webpack/ github.com/assets/ gist.github.com/assets-cdn/worker/
set-cookie: _gh_sess=EBR96OLhPkzFd3yrgVE%2F%2FH%2F093hINxjZS75ClaEppCSZKshg8r8DUa2rMWIPwDTRWCTe%2BlDZlaAVB9UDgBHj2Me191dQ0RWSqTbfB76q3kRBG%2F3pwLYlF%2B4Zgrz0hKheNxHnYwKqD%2B8w0D4uoH%2FhCFQwcHIjEpvauG9G78TOd7o0deqI1DyLHoNoROCT2fzDLyZiXFxsSoq8lNd4axGDGW1CVbaUEgTchXi5Fg%2FSwsZsxoc6gURQ0FaDN66QAY8%2BvfAlMszEM5%2FP43QmRKFmkQ%3D%3D--ws3Hxz2PDhLKGzh8--QvyZnApl%2FZ0fGgOuBKLX%2BQ%3D%3D; Path=/; HttpOnly; Secure; SameSite=Lax
set-cookie: _octo=GH1.1.766159118.1730486412; Path=/; Domain=github.com; Expires=Sat, 01 Nov 2025 18:40:12 GMT; Secure; SameSite=Lax
set-cookie: logged_in=no; Path=/; Domain=github.com; Expires=Sat, 01 Nov 2025 18:40:12 GMT; HttpOnly; Secure; SameSite=Lax
accept-ranges: bytes
x-github-request-id: F478:38AECE:162A8934:1697BE2D:6725208C

Binary file added screenshots/humble_input.PNG
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.

0 comments on commit ca780c0

Please sign in to comment.