Parameter validation failed using result_storage #45

eduherraiz · 2016-01-15T16:51:57Z

I'm testing this thumbor_aws fork trying to include it my docker-thumbor image:
APSL/docker-thumbor#4

The case is, is working when put a image at s3 storage, but when try to put the same with the result_storage I get the exception:

thumbor_1 | 2016-01-15 16:30:03 thumbor:ERROR ERROR: Traceback (most recent call last):
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/web.py", line 1401, in _stack_context_handle_exception
thumbor_1 |     raise_exc_info((type, value, traceback))
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/stack_context.py", line 314, in wrapped
thumbor_1 |     ret = fn(*args, **kwargs)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/thumbor/handlers/__init__.py", line 343, in save_to_result_storage
thumbor_1 |     context.modules.result_storage.put(results)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tc_aws/result_storages/s3_storage.py", line 35, in put
thumbor_1 |     self.set(bytes, path)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tc_aws/aws/storage.py", line 80, in set
thumbor_1 |     callback=self._handle_error,
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 473, in wrapper
thumbor_1 |     future.result()
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 232, in result
thumbor_1 |     raise_exc_info(self._exc_info)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado/concurrent.py", line 461, in wrapper
thumbor_1 |     result = f(*args, **kwargs)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tc_aws/aws/bucket.py", line 98, in put
thumbor_1 |     session.call(**args)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado_botocore/base.py", line 97, in call
thumbor_1 |     return self._make_api_call(operation_name=self.operation, api_params=kwargs, callback=callback)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/tornado_botocore/base.py", line 58, in _make_api_call
thumbor_1 |     request_dict = self.client._convert_to_request_dict(api_params, operation_model)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/botocore/client.py", line 369, in _convert_to_request_dict
thumbor_1 |     api_params, operation_model)
thumbor_1 |   File "/usr/local/lib/python2.7/site-packages/botocore/validate.py", line 273, in serialize_to_request
thumbor_1 |     raise ParamValidationError(report=report.generate_report())
thumbor_1 | ParamValidationError: Parameter validation failed:
thumbor_1 | Invalid type for parameter Metadata, value: <tornado.httputil.HTTPHeaders object at 0x7fefb14277d0>, type: <class 'tornado.httputil.HTTPHeaders'>, valid types: <type 'dict'>

I tried but I can't see the problem.

The text was updated successfully, but these errors were encountered:

Bladrak · 2016-01-15T16:57:00Z

Could you print out the versions of the packages you're using? I'll try to reproduce this.

eduherraiz · 2016-01-15T16:57:51Z

My requirements is:

argparse==1.2.1
envtpl==0.4.1
boto==2.38.0
dateutils==0.6.6
numpy==1.9.2
pyremotecv==0.5.0
remotecv==1.0.2
opencv-engine==1.0.1
redis==2.4.12
thumbor==5.2.1
git+https://github.com/APSL/thumbor_aws.git
tc_aws==2.0.11

The difference is when put the image to storage use:

path

And when use the result_storage:

self.context.request.url

Bladrak · 2016-01-15T16:59:29Z

Thanks, I'll try to take a look at this whenever I get the chance.

eduherraiz · 2016-01-19T11:10:42Z

@Bladrak Consider i'm using a http_loader to get a external image via URL.

Bladrak · 2016-01-20T12:51:28Z

@eduherraiz I think the issue might come from requiring boto; tc_aws now uses botocre directly (not boto), hence there might be some conflicts here.

Bladrak · 2016-01-20T12:53:03Z

Could you do a pip freeze to list the installed dependencies? Here's mine:

backports-abc==0.4
backports.ssl-match-hostname==3.5.0.1
botocore==1.2.0
certifi==2015.11.20.1
colour==0.1.2
derpconf==0.7.3
docutils==0.12
futures==3.0.4
jmespath==0.7.1
libthumbor==1.3.1
numpy==1.10.4
opencv-engine==1.0.1
pexif==0.15
Pillow==2.9.0
pycrypto==2.6.1
pycurl==7.19.5.3
python-dateutil==2.4.2
python-magic==0.4.10
redis==2.10.5
shortuuid==0.4.3
singledispatch==3.4.0.3
six==1.10.0
statsd==3.2.1
tc-aws==2.0.12
tc-core==0.3.0
tc-shortener==0.2.2
thumbor==5.2.1
tornado==4.3
tornado-botocore==1.0.2

eduherraiz · 2016-01-20T15:01:46Z

I tried removing the boto package but is still failing.
The list of packages installed now is;

backports-abc==0.4
backports.ssl-match-hostname==3.5.0.1
botocore==1.2.0
certifi==2015.11.20.1
colour==0.1.2
dateutils==0.6.6
derpconf==0.7.3
docutils==0.12
envtpl==0.4.1
futures==3.0.4
itty==0.8.2
Jinja2==2.8
jmespath==0.7.1
libthumbor==1.3.1
MarkupSafe==0.23
numpy==1.9.2
opencv-engine==1.0.1
pexif==0.15
Pillow==2.9.0
pycrypto==2.6.1
pycurl==7.19.5.3
pyremotecv==0.5.0
pyres==1.2
pystache==0.5.4
python-dateutil==2.4.2
python-magic==0.4.10
pytz==2015.7
redis==2.4.12
remotecv==1.0.2
setproctitle==1.1.9
shortuuid==0.4.3
simplejson==3.8.1
singledispatch==3.4.0.3
six==1.10.0
statsd==3.2.1
tc-aws==2.0.12
tc-core==0.3.0
tc-shortener==0.2.2
thumbor==5.2.1
thumbor-aws==1
tornado==4.3
tornado-botocore==1.0.2
virtualenv==13.1.2
wheel==0.26.0

And for more information i'm trying to get the thumbor URL:
http://localhost/unsafe/100x100/avatars0.githubusercontent.com/u/469968

Bladrak · 2016-01-20T15:02:45Z

Ok, could you print your thumbor config as well?

eduherraiz · 2016-01-20T15:46:07Z

Of course!

THUMBOR_LOG_FORMAT = '%(asctime)s %(name)s:%(levelname)s %(message)s'
THUMBOR_LOG_DATE_FORMAT = '%Y-%m-%d %H:%M:%S'
MAX_WIDTH = 0
MAX_HEIGHT = 0
MIN_WIDTH = 1
MIN_HEIGHT = 1
ALLOWED_SOURCES =  []
QUALITY = 80
WEBP_QUALITY = None
AUTO_WEBP = False
MAX_AGE = 86400
MAX_AGE_TEMP_IMAGE = 0
RESPECT_ORIENTATION = False
IGNORE_SMART_ERRORS = False
PRESERVE_EXIF_INFO = False
ALLOW_ANIMATED_GIFS = True
USE_GIFSICLE_ENGINE = False
USE_BLACKLIST = False
LOADER = 'thumbor.loaders.http_loader'
STORAGE = 'tc_aws.storages.s3_storage'
RESULT_STORAGE = 'tc_aws.result_storages.s3_storage'
ENGINE = 'thumbor.engines.pil'
SECURITY_KEY = 'MY_SECURE_KEY'
ALLOW_UNSAFE_URL = True
ALLOW_OLD_URLS = True
AWS_ACCESS_KEY = '---'
AWS_SECRET_KEY = '---'
FILE_LOADER_ROOT_PATH = '/data/loader'
HTTP_LOADER_CONNECT_TIMEOUT = 5
HTTP_LOADER_REQUEST_TIMEOUT = 20
HTTP_LOADER_FOLLOW_REDIRECTS = True
HTTP_LOADER_MAX_REDIRECTS = 5
HTTP_LOADER_FORWARD_USER_AGENT = False
HTTP_LOADER_DEFAULT_USER_AGENT = 'Thumbor/5.0.3'
HTTP_LOADER_PROXY_HOST = None
HTTP_LOADER_PROXY_PORT = None
HTTP_LOADER_PROXY_USERNAME = None
HTTP_LOADER_PROXY_PASSWORD = None
STORAGE_EXPIRATION_SECONDS = 2592000
STORES_CRYPTO_KEY_FOR_EACH_IMAGE = False
FILE_STORAGE_ROOT_PATH = '/data/storage'
UPLOAD_MAX_SIZE = 0
UPLOAD_ENABLED = False
UPLOAD_PHOTO_STORAGE = 'tc_aws.storages.s3_storage'
UPLOAD_DELETE_ALLOWED = False
UPLOAD_PUT_ALLOWED = False
UPLOAD_DEFAULT_FILENAME = 'image'
MONGO_STORAGE_SERVER_HOST = 'mongo'
MONGO_STORAGE_SERVER_PORT = 27017
MONGO_STORAGE_SERVER_DB = 'thumbor'
MONGO_STORAGE_SERVER_COLLECTION = 'images'
REDIS_STORAGE_SERVER_HOST = 'redis'
REDIS_STORAGE_SERVER_PORT = 6379
REDIS_STORAGE_SERVER_DB = 0
REDIS_STORAGE_SERVER_PASSWORD = None
MEMCACHE_STORAGE_SERVERS = ['localhost:11211']
MIXED_STORAGE_FILE_STORAGE = 'thumbor.storages.no_storage'
MIXED_STORAGE_CRYPTO_STORAGE = 'thumbor.storages.no_storage'
MIXED_STORAGE_DETECTOR_STORAGE = 'thumbor.storages.no_storage'
META_CALLBACK_NAME = None
DETECTORS = ['thumbor.detectors.feature_detector', 'thumbor.detectors.face_detector']
FACE_DETECTOR_CASCADE_FILE = 'haarcascade_frontalface_alt.xml'
OPTIMIZERS = []
JPEGTRAN_PATH = '/usr/bin/jpegtran'
PROGRESSIVE_JPEG = True
FILTERS = ['thumbor.filters.brightness', 'thumbor.filters.contrast', 'thumbor.filters.rgb', 'thumbor.filters.round_corner', 'thumbor.filters.quality', 'thumbor.filters.noise', 'thumbor.filters.watermark', 'thumbor.filters.equalize', 'thumbor.filters.fill', 'thumbor.filters.sharpen', 'thumbor.filters.strip_icc', 'thumbor.filters.frame', 'thumbor.filters.grayscale', 'thumbor.filters.rotate', 'thumbor.filters.format', 'thumbor.filters.max_bytes', 'thumbor.filters.convolution', 'thumbor.filters.blur', 'thumbor.filters.extract_focal', 'thumbor.filters.no_upscale']
RESULT_STORAGE_EXPIRATION_SECONDS = 0
RESULT_STORAGE_FILE_STORAGE_ROOT_PATH = '/data/result_storage'
RESULT_STORAGE_STORES_UNSAFE = True
REDIS_QUEUE_SERVER_HOST = 'redis'
REDIS_QUEUE_SERVER_PORT = 6379
REDIS_QUEUE_SERVER_DB = 0
REDIS_QUEUE_SERVER_PASSWORD = None
SQS_QUEUE_KEY_ID = None
SQS_QUEUE_KEY_SECRET = None
SQS_QUEUE_REGION = 'us-east-1'
USE_CUSTOM_ERROR_HANDLING = False
ERROR_HANDLER_MODULE = 'thumbor.error_handlers.sentry'
ERROR_FILE_LOGGER = None
ERROR_FILE_NAME_USE_CONTEXT = False
SENTRY_DSN_URL = ''
TC_AWS_REGION = 'eu-west-1' # AWS Region
TC_AWS_STORAGE_BUCKET = 'thumbor-travis-test' # S3 bucket for Storage
TC_AWS_STORAGE_ROOT_PATH = '/storage' # S3 path prefix for Storage bucket
TC_AWS_LOADER_BUCKET = '' #S3 bucket for loader
TC_AWS_LOADER_ROOT_PATH = '' # S3 path prefix for Loader bucket
TC_AWS_RESULT_STORAGE_BUCKET = 'thumbor-travis-test' # S3 bucket for result Storage
TC_AWS_RESULT_STORAGE_ROOT_PATH = '/result_storage' # S3 path prefix for Result storage bucket
TC_AWS_STORAGE_SSE = 'False'
TC_AWS_STORAGE_RRS = 'False'
TC_AWS_ENABLE_HTTP_LOADER = 'False'
TC_AWS_ALLOWED_BUCKETS = 'False' # List of allowed bucket to be requested
TC_AWS_STORE_METADATA = 'False' # Store result with metadata (for instance content-type)

I tried to change TC_AWS_ENABLE_HTTP_LOADER to True, but didn't solve the problem.

Bladrak · 2016-01-20T16:00:26Z

I think you may have an issue with your configuration, when you set the value False, you put it between quotes, maybe try removing them?
This doesn't mean that there's no issue, given that TC_AWS_STORE_METADATA should otherwise work properly...

Bladrak · 2016-01-20T16:17:57Z

Commit cf418fa should fix the issue. I've published version 2.1.1 with this fix.

Bladrak · 2016-01-20T18:29:06Z

@eduherraiz feel free to close this issue once you tested the fix :)

eduherraiz · 2016-01-20T23:23:05Z

Thank you!! I will test it soon.

eduherraiz · 2016-01-21T10:46:55Z

The fix works fine. Thank you @Bladrak !!

eduherraiz mentioned this issue Jan 18, 2016

Change to the offcial aws extension APSL/docker-thumbor#4

Closed

eduherraiz closed this as completed Jan 21, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Parameter validation failed using result_storage #45

Parameter validation failed using result_storage #45

eduherraiz commented Jan 15, 2016

Bladrak commented Jan 15, 2016

eduherraiz commented Jan 15, 2016

Bladrak commented Jan 15, 2016

eduherraiz commented Jan 19, 2016

Bladrak commented Jan 20, 2016

Bladrak commented Jan 20, 2016

eduherraiz commented Jan 20, 2016

Bladrak commented Jan 20, 2016

eduherraiz commented Jan 20, 2016

Bladrak commented Jan 20, 2016

Bladrak commented Jan 20, 2016

Bladrak commented Jan 20, 2016

eduherraiz commented Jan 20, 2016

eduherraiz commented Jan 21, 2016

Parameter validation failed using result_storage #45

Parameter validation failed using result_storage #45

Comments

eduherraiz commented Jan 15, 2016

Bladrak commented Jan 15, 2016

eduherraiz commented Jan 15, 2016

Bladrak commented Jan 15, 2016

eduherraiz commented Jan 19, 2016

Bladrak commented Jan 20, 2016

Bladrak commented Jan 20, 2016

eduherraiz commented Jan 20, 2016

Bladrak commented Jan 20, 2016

eduherraiz commented Jan 20, 2016

Bladrak commented Jan 20, 2016

Bladrak commented Jan 20, 2016

Bladrak commented Jan 20, 2016

eduherraiz commented Jan 20, 2016

eduherraiz commented Jan 21, 2016