Continued issue with Client basic HTTP auth as per #1699 #2610
Comments
|
Can you show a example .. seems to me seems to me login is being passed. You might need to make a clientsession and add headers to it. If you need a example let me know . |
|
Test code attached below. Just the essentials in this.
There are 2 url’s.
First one (commented out) is a request for filing history - which works.
Second one is request for a document to a url contained in the filing history and the initial request to the url is then redirected to an Amazon s3 bucket. This fails.
Hope this helps.
I am away for a couple of days but if you need anything else just ask.
Cheers
Andy
… On 9 Jan 2018, at 07:03, Jasmit Tarang ***@***.***> wrote:
Can you show a example .. seems to me seems to me login is being passed. You might need to make a clientsession and add headers to it. If you need a example let me know .
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#2610 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AT6OHSFCRQimDd6UlkDVTzpCnO5wQV9Tks5tIw9agaJpZM4RDS81>.
|
|
Maybe I am going crazy but I dont see any code. |
|
Can you read the .py file attached to my reply??
A
…Sent from my iPhone
On 10 Jan 2018, at 19:24, Jasmit Tarang ***@***.***> wrote:
Maybe I am going crazy but I dont see any code.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Nope don't see it . Unfortunately. Just paste it in a gist or just markdown. |
|
Yeah fine. I will run it and add in output from success and fail so you have the full picture
A
…Sent from my iPhone
On 11 Jan 2018, at 00:40, Jasmit Tarang ***@***.***> wrote:
Nope don't see it . Unfortunately. Just paste it in a gist or just markdown.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub, or mute the thread.
|
|
Did you get my email with test code and output ok ??
A
… On 11 Jan 2018, at 00:40, Jasmit Tarang ***@***.***> wrote:
Nope don't see it . Unfortunately. Just paste it in a gist or just markdown.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#2610 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AT6OHWlYPEJ9U2Pt2sXnPKampQ2ZjaBbks5tJVh3gaJpZM4RDS81>.
|
|
Nope . Just make a gist. |
|
sent this last week
let me know if you get this email….
import asyncio
import aiohttp
import auth # own authentication class
async def test_session(loop, url, auth):
api_session = aiohttp.ClientSession(loop=loop, auth=auth)
async with api_session.get(url) as response:
return await response.read()
if __name__ == '__main__':
# metadata url
test_url = 'https://api.companieshouse.gov.uk/company/04025989/filing-history?category=accounts&items_per_page=100&start_index=0 <https://api.companieshouse.gov.uk/company/04025989/filing-history?category=accounts&items_per_page=100&start_index=0>'
# document url
# test_url = 'https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA/content <https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA/content>'
# api Oauth from secrets
oauth_path = '/Users/andyspence/Project/ccML_async/asyncImporter/config/'
oauth_file = 'ch_api_key.yaml'
# decrypt authorisation keys
decoder = auth.Auth()
BAuth = aiohttp.BasicAuth(decoder.read(oauth_path, oauth_file), '')
loop = asyncio.get_event_loop()
# fetch_task = asyncio.ensure_future(test_session(loop=loop, url=test_url, auth=BAuth))
# group = asyncio.gather(fetch_task, loop=loop, return_exceptions=False)
group = asyncio.gather(test_session(loop=loop, url=test_url, auth=BAuth), loop=loop, return_exceptions=False)
c = loop.run_until_complete(group)
print(c[0])
Run as above and Output is :
/Users/andyspence/anaconda3/bin/python3.6 /Users/andyspence/Project/ccML_async/asyncImporter/tests/trial_aioget.py
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x106ce1828>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x104b9e630>, 676.943253907)]']
connector: <aiohttp.connector.TCPConnector object at 0x106ce15c0>
b'{"total_count":18,"items":[{"action_date":"2015-12-31","date":"2016-07-27","description_values":{"made_up_date":"2015-12-31"},"paper_filed":true,"type":"AA","links":{"self":"/company/04025989/filing-history/MzE1MzcxMDI2OGFkaXF6a2N4","document_metadata":"https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA <https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA>”},……etc etc
Process finished with exit code 0
Switch url’s to the second test_url and run
Output is :
/Users/andyspence/anaconda3/bin/python3.6 /Users/andyspence/Project/ccML_async/asyncImporter/tests/trial_aioget.py
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10f18e828>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x10d93c630>, 961.485947615)]']
connector: <aiohttp.connector.TCPConnector object at 0x10f18e5c0>
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Basic NE10N3hhczlXS0FpRjFIYThmRThRUGhnQlVwWm5LZlJUZVFBMFBHYjo=</ArgumentValue><RequestId>679BB15D96599042</RequestId><HostId>hJ1M6t8yznnspSDb7LQAmQ3wz54/zxVMrimdgg3k8CSMbUasOTherfocKKhZ0wRPEOMUVK8ld7I=</HostId></Error>'
Process finished with exit code 0
… On 16 Jan 2018, at 00:11, Jasmit Tarang ***@***.***> wrote:
Nope
|
|
OK, I assume you could not read the last email either so here is code and output in this format. Harder to read but ….
import asyncio
import aiohttp
import auth # own authentication class
async def test_session(loop, url, auth):
api_session = aiohttp.ClientSession(loop=loop, auth=auth)
async with api_session.get(url) as response:
return await response.read()
if __name__ == '__main__':
# metadata url
test_url = 'https://api.companieshouse.gov.uk/company/04025989/filing-history?category=accounts&items_per_page=100&start_index=0'
# document url
# test_url = 'https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA/content'
# api Oauth from secrets
oauth_path = '/Users/andyspence/Project/ccML_async/asyncImporter/config/'
oauth_file = 'ch_api_key.yaml'
# decrypt authorisation keys
decoder = auth.Auth()
BAuth = aiohttp.BasicAuth(decoder.read(oauth_path, oauth_file), '')
loop = asyncio.get_event_loop()
# fetch_task = asyncio.ensure_future(test_session(loop=loop, url=test_url, auth=BAuth))
# group = asyncio.gather(fetch_task, loop=loop, return_exceptions=False)
group = asyncio.gather(test_session(loop=loop, url=test_url, auth=BAuth), loop=loop, return_exceptions=False)
c = loop.run_until_complete(group)
print(c[0])
Run as above and Output is :
/Users/andyspence/anaconda3/bin/python3.6 /Users/andyspence/Project/ccML_async/asyncImporter/tests/trial_aioget.py
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x106ce1828>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x104b9e630>, 676.943253907)]']
connector: <aiohttp.connector.TCPConnector object at 0x106ce15c0>
b'{"total_count":18,"items":[{"action_date":"2015-12-31","date":"2016-07-27","description_values":{"made_up_date":"2015-12-31"},"paper_filed":true,"type":"AA","links":{"self":"/company/04025989/filing-history/MzE1MzcxMDI2OGFkaXF6a2N4","document_metadata":"https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA <https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA>”},……etc etc
Process finished with exit code 0
Switch url’s to the second test_url and run
Output is :
/Users/andyspence/anaconda3/bin/python3.6 /Users/andyspence/Project/ccML_async/asyncImporter/tests/trial_aioget.py
Unclosed client session
client_session: <aiohttp.client.ClientSession object at 0x10f18e828>
Unclosed connector
connections: ['[(<aiohttp.client_proto.ResponseHandler object at 0x10d93c630>, 961.485947615)]']
connector: <aiohttp.connector.TCPConnector object at 0x10f18e5c0>
b'<?xml version="1.0" encoding="UTF-8"?>\n<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified</Message><ArgumentName>Authorization</ArgumentName><ArgumentValue>Basic NE10N3hhczlXS0FpRjFIYThmRThRUGhnQlVwWm5LZlJUZVFBMFBHYjo=</ArgumentValue><RequestId>679BB15D96599042</RequestId><HostId>hJ1M6t8yznnspSDb7LQAmQ3wz54/zxVMrimdgg3k8CSMbUasOTherfocKKhZ0wRPEOMUVK8ld7I=</HostId></Error>'
Process finished with exit code 0
… On 11 Jan 2018, at 00:40, Jasmit Tarang ***@***.***> wrote:
Nope don't see it . Unfortunately. Just paste it in a gist or just markdown.
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#2610 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AT6OHWlYPEJ9U2Pt2sXnPKampQ2ZjaBbks5tJVh3gaJpZM4RDS81>.
|
|
I can't really test this as its just not complete. Do they have a test website or something? Test creds? Constants: If not that might be the issue there . |
|
No test website for testing programmatic access - pita.
I will set up another account for this test application and we can hard code the access keys and test with both requests and aiohttp. That will removes/highlights errors elsewhere and demonstrates the anticipated
responses.
I will set that up this am and email you today. If you don’t get it today let me know
A
… On 18 Jan 2018, at 04:51, Jasmit Tarang ***@***.***> wrote:
I can't really test this as its just not complete. Do they have a test website or something? Test creds?
Constants:
requests: did you use your own auth class
If not that might be the issue there .
—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub <#2610 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AT6OHVmSY4esSzR_A-vi_jH701QZ22lPks5tLs20gaJpZM4RDS81>.
|
|
Right, as per previous email, I have set up an account specifically for this test. API key is hardcoded here. Not ideal but I will delete later and at least it allows us to exclude other code.
I have refactored to 2 test urls and perform a get on both with requests and aiohttp.
Requests behaves as expected. test_1 url is metadata for one company and is returned correctly. test_2 url is a pdf doc, also returned correctly.
Aiohttp gets the correct response from test_1 but returns '<Error><Code>InvalidArgument</Code><Message>Only one auth mechanism allowed’ for test_2 url.
Kind of it in a nutshell.
The redirect is to a Amazon aws to retrieve the document from an s3 bucket.
Test code:
""" test for the Caller class for document import"""
import asyncio
import aiohttp
import requests
# registered with the API for this app only -> use full account auth if refactoring
KEY = 'ba6xUk6iXpVg_iozY8RST8MdRgZHj_4_yrEt8__g'
PASS = ''
def requests_get(url):
r = requests.get(url, auth=(KEY, PASS))
print(r.status_code)
return r
async def aiohttp_get(loop, url, auth):
with aiohttp.ClientSession(loop=loop, auth=auth) as api_session:
print('api session context')
print(BAuth)
async with api_session.get(url) as response:
r = await response.read()
print(url)
print(r)
return
if __name__ == '__main__':
test_1 = 'https://api.companieshouse.gov.uk/company/04025989/filing-history?category=accounts&items_per_page=100&start_index=0'
test_2 = 'https://document-api.companieshouse.gov.uk/document/w3HCtbsw1OWNup0R3wbBxtbE3nixKhNwVYACspslbNA/content'
print('username: {}'.format(KEY))
print('password{}'.format(PASS))
# ----------------------------------------------------------------------------------------------------------------
# test with requests
# get the document metadata
print('\nrequests with test_1')
resp = requests_get(test_1)
print(resp.json())
# get one documents content
print('\nrequests with test_2')
resp = requests_get(test_2)
print(resp.text) # pdf file contents
# ----------------------------------------------------------------------------------------------------------------
# tests with aiohttp
BAuth = aiohttp.BasicAuth(KEY, PASS)
print('\n', BAuth)
loop = asyncio.get_event_loop()
fetch_1 = asyncio.ensure_future(aiohttp_get(loop=loop, url=test_1, auth=BAuth))
fetch_2 = asyncio.ensure_future(aiohttp_get(loop=loop, url=test_2, auth=BAuth))
group = asyncio.gather(fetch_1, fetch_2)
loop.run_until_complete(group)
|
|
Getting anywhere with this ??? |
|
The issue is on my radar. I'll try to find a time for fixing before aiogttp 3.0 |
|
This get fixed in 3.0 ?? |
|
I got the same problem.. |
|
We drop auth headers when origin changes, so I assume this is fixed. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Long story short
As before...
I was making calls with aiohttp client to a Companies House API (which requires basic HTTP authentication) and it works fine when I requested metadata but I request document download there is a redirect to amazon S3 and the request fails.
It appears that aiohttp passes authentication after redirection ?? This causes an issue as amazon requires that there should be only one auth method.
Error received includes - "Only one auth mechanism allowed; only the X-Amz-Algorithm query parameter, Signature query string parameter or the Authorization header should be specified"
I've tested with the requests lib and it works fine for identical requests.
Expected behaviour
Only one auth mechanism should be used on the redirect
Actual behaviour
end response from S3 ends with an authentication error, not entirely sure why as #1699 should have fixed this ?
Steps to reproduce
Would need to have access to server redirecting to s3?
Your environment
macOS 10.12.6
python 3.5.4
aiohttp 2.3.6
The text was updated successfully, but these errors were encountered: