run Gratipay 130

[chadwhitacre]

129

[chadwhitacre]

Reviewed 36 (26) accounts.

[chadwhitacre]

Droplet spun up and updated.

[chadwhitacre]

Backup taken and verified.

[chadwhitacre]

Script started ... aaaaaaaand it crashed:

pid-1731 thread-140389752571648 (MainThread) Traceback (most recent call last):
pid-1731 thread-140389752571648 (MainThread)   File "/home/whit537/gratipay.com/gratipay/cli.py", line 28, in payday
pid-1731 thread-140389752571648 (MainThread)     Payday.start().run()
pid-1731 thread-140389752571648 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 119, in run
pid-1731 thread-140389752571648 (MainThread)     self.payin()
pid-1731 thread-140389752571648 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 143, in payin
pid-1731 thread-140389752571648 (MainThread)     holds = self.create_card_holds(cursor)
pid-1731 thread-140389752571648 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 404, in create_card_holds
pid-1731 thread-140389752571648 (MainThread)     threaded_map(f, participants)
pid-1731 thread-140389752571648 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 35, in threaded_map
pid-1731 thread-140389752571648 (MainThread)     r = pool.map(func, iterable)
pid-1731 thread-140389752571648 (MainThread)   File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
pid-1731 thread-140389752571648 (MainThread)     return self.map_async(func, iterable, chunksize).get()
pid-1731 thread-140389752571648 (MainThread)   File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
pid-1731 thread-140389752571648 (MainThread)     raise self._value
pid-1731 thread-140389752571648 (MainThread) ProgrammingError: syntax error at or near "G"
pid-1731 thread-140389752571648 (MainThread) LINE 1: G
pid-1731 thread-140389752571648 (MainThread)         ^
pid-1731 thread-140389752571648 (MainThread)

[chadwhitacre]

ProgrammingError looks to me like something from Postgres, but the multithreading masks the actual exception.

[chadwhitacre]

I mean, WTF? G!?

[chadwhitacre]

Yeah, ack in env/lib shows ProgrammingError to be from psycopg2.

[chadwhitacre]

Since we didn't make it out of the card holds loop, we should be able to rerun without any modifications (we should be able to rerun without modifications no matter when we crash, but that's another story).

[Changaco]

I have a hack for the tracebacks in threaded_map that I didn't put in the original PR, for a reason that I can't remember:

import sys
import traceback

def threaded_map(func, iterable, threads=5):
    pool = ThreadPool(threads)
    def g(*a, **kw):
        try:
            return func(*a, **kw)
        except Exception:
            raise sys.exc_info()[0](traceback.format_exc())
    r = pool.map(g, iterable)
    pool.close()
    pool.join()
    return r

[chadwhitacre]

@Changaco We might also consider using tell_sentry at that point. It's helpful to be able to browse the context for all the stack frames, e.g.

[chadwhitacre]

At this point that info is lost and I think our options are to try to debug this blind or simply rerun.

[chadwhitacre]

@Changaco Rerunning with your hack for now ...

[chadwhitacre]

diff --git a/gratipay/billing/payday.py b/gratipay/billing/payday.py
index 0063642..28525e8 100644
--- a/gratipay/billing/payday.py
+++ b/gratipay/billing/payday.py
@@ -32,7 +32,12 @@ with open('fake_payday.sql') as f:

 def threaded_map(func, iterable, threads=5):
     pool = ThreadPool(threads)
-    r = pool.map(func, iterable)
+    def g(*a, **kw):
+        try:
+            return func(*a, **kw)
+        except Exception:
+            raise sys.exc_info()[0](traceback.format_exc())
+    r = pool.map(g, iterable)
     pool.close()
     pool.join()
     return r

[chadwhitacre]

Oops. :-)

pid-1936 thread-140001328940800 (MainThread) Traceback (most recent call last):
pid-1936 thread-140001328940800 (MainThread)   File "/home/whit537/gratipay.com/gratipay/cli.py", line 28, in payday
pid-1936 thread-140001328940800 (MainThread)     Payday.start().run()
pid-1936 thread-140001328940800 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 124, in run
pid-1936 thread-140001328940800 (MainThread)     self.payin()
pid-1936 thread-140001328940800 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 148, in payin
pid-1936 thread-140001328940800 (MainThread)     holds = self.create_card_holds(cursor)
pid-1936 thread-140001328940800 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 409, in create_card_holds
pid-1936 thread-140001328940800 (MainThread)     threaded_map(f, participants)
pid-1936 thread-140001328940800 (MainThread)   File "/home/whit537/gratipay.com/gratipay/billing/payday.py", line 40, in threaded_map
pid-1936 thread-140001328940800 (MainThread)     r = pool.map(g, iterable)
pid-1936 thread-140001328940800 (MainThread)   File "/usr/lib/python2.7/multiprocessing/pool.py", line 227, in map
pid-1936 thread-140001328940800 (MainThread)     return self.map_async(func, iterable, chunksize).get()
pid-1936 thread-140001328940800 (MainThread)   File "/usr/lib/python2.7/multiprocessing/pool.py", line 528, in get
pid-1936 thread-140001328940800 (MainThread)     raise self._value
pid-1936 thread-140001328940800 (MainThread) NameError: global name 'sys' is not defined

[chadwhitacre]

diff --git a/gratipay/billing/payday.py b/gratipay/billing/payday.py
index 0063642..95cad68 100644
--- a/gratipay/billing/payday.py
+++ b/gratipay/billing/payday.py
@@ -12,6 +12,8 @@ immediately affect the participant's balance.
 from __future__ import unicode_literals

 import itertools
+import sys
+import traceback
 from multiprocessing.dummy import Pool as ThreadPool

 from balanced import CardHold
@@ -32,7 +34,12 @@ with open('fake_payday.sql') as f:

 def threaded_map(func, iterable, threads=5):
     pool = ThreadPool(threads)
-    r = pool.map(func, iterable)
+    def g(*a, **kw):
+        try:
+            return func(*a, **kw)
+        except Exception:
+            raise sys.exc_info()[0](traceback.format_exc())
+    r = pool.map(g, iterable)
     pool.close()
     pool.join()
     return r

[chadwhitacre]

Looks like it's running fine now. Kinda bummed that we apparently hit a second exception but my bad patch didn't catch it. :-(

[chadwhitacre]

Script ran for 10 minutes (0:10:18.257751).

[chadwhitacre]

Wow, lost another big giver this week. We're almost back to $10,000!

[chadwhitacre]

Pinned https://twitter.com/Gratipay/status/537976684109381634.

[chadwhitacre]

Log downloaded and droplet destroyed.

[chadwhitacre]

MassPay done and posted back for 73 users.

[chadwhitacre]

Escrow shuffled.