Vague input error for trained TensorFlow model

[teddyward]

I am trying to retrain and deploy our team's previously-already-working TensorFlow model onto SageMaker... having no luck after several days of banging my head against it. Any help that you all can provide on how to massage this into SageMaker would be greatly appreciated... about to give up and just run an AMI

Every time that I try to make a prediction/inference request on my trained, evaluated, and deployed model, I get the error

ModelError: An error occurred (ModelError) when calling the InvokeEndpoint operation: Received server error (500) from model with message ""

Similar to #67, and probably a duplicate of #99

Serving input function is basically identical to the TensorFlow tutorial:

CSV_COLUMNS = ['reading', 'temperature', 'hour', 'weekday', 'is_holiday', 'area', 'structure_area', 'has_pv']
LABEL_COLUMN = 'reading'
CSV_COLUMN_DEFAULTS = [[0.0], [0.0], [0], [0], [0], [0.0], [2000.0], [0]]
UNUSED_COLUMNS = set(CSV_COLUMNS) - {col.name for col in INPUT_COLUMNS} - \
    {LABEL_COLUMN}

def estimator_fn(run_config, params):
 ... build and return DNNRegressor ...

def serving_input_fn():
    csv_row = tf.placeholder(
        shape=[None],
        dtype=tf.string
    )
    features = parse_csv(csv_row)
    features.pop(LABEL_COLUMN)
    return tf.estimator.export.ServingInputReceiver(features, {'csv_row': csv_row})


def parse_csv(rows_string_tensor):
    """Takes the string input tensor and returns a dict of rank-2 tensors."""

    # Takes a rank-1 tensor and converts it into rank-2 tensor
    # Example if the data is ['csv,line,1', 'csv,line,2', ..] to
    # [['csv,line,1'], ['csv,line,2']] which after parsing will result in a
    # tuple of tensors: [['csv'], ['csv']], [['line'], ['line']], [[1], [2]]
    row_columns = tf.expand_dims(rows_string_tensor, -1)
    columns = tf.decode_csv(row_columns, record_defaults=CSV_COLUMN_DEFAULTS)
    features = dict(zip(CSV_COLUMNS, columns))

    # Remove unused columns
    for col in UNUSED_COLUMNS:
      features.pop(col)
    return features

Logs:


17:04:12
Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/container_support/serving.py", line 180, in _invoke self.transformer.transform(content, input_content_type, requested_output_content_type) File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 281, in transform return self.transform_fn(data, content_type, accepts), accepts File "/usr/l
AllTraffic/i-0933877fa79ea8c2f
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/container_support/serving.py", line 180, in _invoke
self.transformer.transform(content, input_content_type, requested_output_content_type)
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 281, in transform
return self.transform_fn(data, content_type, accepts), accepts
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 206, in f
prediction = self.predict_fn(input)
File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 221, in predict_fn
return self.proxy_client.request(data)
File "/usr/local/lib/python2.7/dist-packages/tf_container/proxy_client.py", line 66, in request
request_fn = self.request_fn_map[self.prediction_type]

17:04:12
KeyError: u'tensorflow/serving/regress'
AllTraffic/i-0933877fa79ea8c2f
KeyError: u'tensorflow/serving/regress'

17:04:12
[2018-06-29 17:04:11,861] ERROR in serving: u'tensorflow/serving/regress'
AllTraffic/i-0933877fa79ea8c2f
[2018-06-29 17:04:11,861] ERROR in serving: u'tensorflow/serving/regress'

[nadiaya]

Marking as a bug since we at least need to make the error message more clear.

[nadiaya]

Traceback looks a bit odd and seems like some parts of it are missing, was it copied correctly?

Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/container_support/serving.py", line 180, in _invoke self.transformer.transform(content, input_content_type, requested_output_content_type) File "/usr/local/lib/python2.7/dist-packages/tf_container/serve.py", line 281, in transform return self.transform_fn(data, content_type, accepts), accepts File "/usr/l

[GillNitzan]

is there any update on it? we have the same issue too...

[laurenyu]

Closing due to inactivity and the introduction of script mode. With script mode, you should be able to run your TensorFlow scripts from elsewhere on SageMaker with much less modification than what was needed before. See the TensorFlow README for more information. Feel free to reopen this issue if necessary.