simple londiste3 replication standby server error

[ozanyurt]

I'm trying to configure simple replication and my standby server log:

2012-11-12 11:12:17,372 4435 ERROR Job st3_l3simple_db2 crashed: format requires a mapping
Traceback (most recent call last):
File "/usr/local/lib/python2.7/dist-packages/skytools/scripting.py", line 563, in run_func_safely
return func()
File "/usr/local/lib/python2.7/dist-packages/pgq/cascade/consumer.py", line 199, in work
return Consumer.work(self)
File "/usr/local/lib/python2.7/dist-packages/pgq/consumer.py", line 280, in work
self._launch_process_batch(db, batch_id, ev_list)
File "/usr/local/lib/python2.7/dist-packages/pgq/consumer.py", line 309, in _launch_process_batch
self.process_batch(db, batch_id, list)
File "/usr/local/lib/python2.7/dist-packages/pgq/cascade/consumer.py", line 172, in process_batch
self.process_remote_batch(src_db, tick_id, event_list, dst_db)
File "/usr/local/lib/python2.7/dist-packages/londiste/playback.py", line 363, in process_remote_batch
self.restore_fkeys(dst_db)
File "/usr/local/lib/python2.7/dist-packages/londiste/playback.py", line 926, in restore_fkeys
self.log.info('Creating fkey: %(fkey_name)s (%(from_table)s --> %(to_table)s)' % row)
TypeError: format requires a mapping

[ozanyurt]

Sorry

I'm using Ubuntu 12.04 and Python 2.7.3

[markokr]

please also say: skytools version, psycopg2 version, postgresql version.

and what does "select londiste.version()" show?

[ozanyurt]

Skytools 3.1.2
Postgresql 9.2.1
Python 2.7.3
python-psycopg2_2.4.5-1build1

I made some workaround maybe it helps
I commented out line 926
self.log.info('Creating fkey: %(fkey_name)s (%(from_table)s --> %(to_table)s)' % row)
and error gone.
After that
/python/londiste/compare.py:56 has throw same exception. And I change this line to not formatted version and everything works fine.

[ozanyurt]

londiste.version() show 3.1.1

[wbloos]

I have the same issue on:
Ubuntu 12.10 (Desktop) 64 bit
Skytools 2.1.13-2
Postgresql 9.1.6
python 2.7.3 and "python3" 3.2.3
python-psycopg2 2.4.5

The error is:
2012-11-27 17:29:41,377 24728 CRITICAL Job test_to_subcriber crashed: <type 'exceptions.TypeError'>: 'format requires a mapping' (<traceback object at 0x7f4598ffda28>: [' File "/usr/lib/python2.7/dist-packages/skytools/scripting.py", line 507, in run_once\n return self.work()\n', ' File "/usr/lib/python2.7/dist-packages/pgq/consumer.py", line 175, in work\n self._launch_process_batch(db, batch_id, list)\n', ' File "/usr/lib/python2.7/dist-packages/pgq/consumer.py", line 207, in _launch_process_batch\n self.process_batch(db, batch_id, list)\n', ' File "/usr/lib/python2.7/dist-packages/pgq/consumer.py", line 409, in process_batch\n self.process_remote_batch(db, batch_id, event_list, dst_db)\n', ' File "/usr/lib/python2.7/dist-packages/londiste/playback.py", line 313, in process_remote_batch\n self.restore_fkeys(dst_db)\n', ' File "/usr/lib/python2.7/dist-packages/londiste/playback.py", line 719, in restore_fkeys\n self.log.info('Creating fkey: %(fkey_name)s (%(from_table)s --> %(to_table)s)' % row)\n'])

[wbloos]

provider=# select londiste.version();

version

2.1.12
(1 row)

[markokr]

I cannot reproduce it with same version of components. Although all they are self-compiled, so maybe some ubuntu patches mess something up.

Please try to find out what kind of object is the 'row' value it complains on.

[wbloos]

Works fine (cannot reproduce) on our old:
Debian 5 (64 bit)
skytools 2.1.8 (backports)
skytools-modules-8.4 2.1.8 (backports)
PostgreSQL 8.4.7 (backports)
python 2.5.2
python-psycopg2 2.0.7-4

[wbloos]

It's the foreign key.
Please find reproduce info below.

If i don't create the foreign key in the db, the error does not occur.
If i leave either of the 2 tables out of the replication (db schema remains the same, so yes fkey), the error does not occur either.
Also, there's no problem when i add the fkey the the database after adding the tables to the replication.

Here's the test case:

• create new databases provider and subscriber
  --psql on the provider:
  create table a(aid serial primary key, aval integer);
  create table b(bid serial primary key, aid integer, bval integer, constraint aid_fkey foreign key (aid) references a(aid));
  insert into a (aval) select generate_series(1, 1000, 10);
  insert into b (aid, bval) select generate_series(1, 100), generate_series(9, 100, 10);
• on the subscriber:
  pg_dump provider|psql subscriber
• install and start pgq and londiste
• add tables on provider
• add tables on subscriber

[markokr]

I understand it's supposed to be fkey. The question was about python object - why python % operator thinks its not a "mapping"? Please add print(repr(row)) or something there, to see what kind of object it is.

[wbloos]

looks like row is:
['public.b', 'public.a', 'b_aid_fkey', 'alter table only public.b add constraint b_aid_fkey FOREIGN KEY (aid) REFERENCES a(aid)']

[wbloos]

btw for anyone reading this that has the same issue: a workaround is adding the fkey to the provider+subscriber database after you added the table to the londiste subscriber service.