Error on "Refresh Druid Metadata"

[LAlbertalli]

Attempting to refresh the Metadata for a Druid Cluster returns the following error:

Traceback (most recent call last):
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1817, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1477, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1381, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1475, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python2.7/site-packages/flask/app.py", line 1461, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/usr/lib/python2.7/site-packages/flask_appbuilder/security/decorators.py", line 26, in wraps
    return f(self, *args, **kwargs)
  File "/usr/lib/python2.7/site-packages/caravel-0.9.0-py2.7.egg/caravel/views.py", line 1009, in refresh_datasources
    cluster, str(e)),
  File "/usr/lib/python2.7/site-packages/caravel-0.9.0-py2.7.egg/caravel/models.py", line 919, in __repr__
    return self.cluster_name
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 237, in __get__
    return self.impl.get(instance_state(instance), dict_)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/attributes.py", line 578, in get
    value = state._load_expired(state, passive)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/state.py", line 474, in _load_expired
    self.manager.deferred_scalar_loader(self, toload)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 664, in load_scalar_attributes
    only_load_props=attribute_names)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/loading.py", line 219, in load_on_ident
    return q.one()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2718, in one
    ret = list(self)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in __iter__
    return self._execute_and_instances(context)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances
    close_with_result=True)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session
    **kw)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 893, in connection
    execution_options=execution_options)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 898, in _connection_for_bind
    engine, execution_options)
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 313, in _connection_for_bind
    self._assert_active()
  File "/usr/lib64/python2.7/site-packages/sqlalchemy/orm/session.py", line 214, in _assert_active
    % self._rollback_exception
InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (_mysql_exceptions.IntegrityError) (1452, 'Cannot add or update a child row: a foreign key constraint fails (`caravel`.`columns`, CONSTRAINT `columns_ibfk_1` FOREIGN KEY (`column_name`) REFERENCES `datasources` (`datasource_name`))') [SQL: u'INSERT INTO `columns` (created_on, changed_on, datasource_name, column_name, is_active, type, groupby, count_distinct, sum, max, min, filterable, description, changed_by_fk, created_by_fk) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)'] [parameters: (datetime.datetime(2016, 5, 27, 18, 41, 53, 340220), datetime.datetime(2016, 5, 27, 18, 41, 53, 340250), 'client_logs_v2', 'num_recommended_unsent', 1, 'STRING', 1, 0, 0, 0, 0, 1, None, 1L, 1L)]

The problem happens with both Caravel 0.9.0 and the latest Master. The backend is MySQL.
There are actually two issues:

• The exception handling try to access the cluster object to print the error. But SQLAlchemy has invalidated the object trying to write it. Since there has been an exception, the transaction has been rolled back and the db connection is dirty.
• There's a wrong constraint on the columns table
  I'm testing a fix for both bugs and I'll issue a Pull request for both of them when it is ready.

[sid88in]

I am facing the EXACT same error while refreshing druid metadata. Are we pushing this fix to prod?

[x4base]

The constraint issue is mentioned here: #466 , but I can't get any response

[joshwalters]

I am also seeing the exact same issue.

[LAlbertalli]

This fix on the underlying two issues ( #530 and #531 ) has been accepted and merged into master.
Check out the current master and they should be fixed.

Luca