run Gratipay 274

[gratipay-bot]

← Payday 273

---

Runbook

http://inside.gratipay.com/howto/run-payday

Checklist

• Review accounts
  • New givers
  • 1.0 Payout applications
• Trigger backup
• Run payday script
• Commit payday logs
• Run masspay
  • DB query to update broken paypal fees
  • Generate paypal inputs
  • Verify there is no fee mismatch
  • Submit masspay on paypal
  • Post back masspay
• Commit masspay logs
• Shuffle escrow
• Review decline rate

Rotation

Month	Pilot	Copilot
September	@whit537 ✈️	@clone1018
October	rohitpaulk	whit537
November	clone1018	rohitpaulk
December	whit537	clone1018
January	rohitpaulk	whit537
February	clone1018	rohitpaulk
March	whit537	clone1018
April	rohitpaulk	whit537
May	clone1018	rohitpaulk
June	whit537	clone1018
July	rohitpaulk	whit537
August	clone1018	rohitpaulk

[chadwhitacre]

Accounts reviewed.

[chadwhitacre]

/me sets his clock to Honolulu time ... 😁

[chadwhitacre]

Problem taking backup:

pg_dump: error reading large object 21839: SSL SYSCALL error: Operation timed out

It was pretty much done, too. :-/

And now my comment won't post.

whit537.org won't load.

Wireless on my laptop is dead?

It's off but it shows as on in the status bar?

Restarting computer.

Now getting "Wi-Fi: No hardware installed". I saw this for the first time a week ago. Re-restart cleared it up, iirc?

[chadwhitacre]

😓

[chadwhitacre]

Backup restarted ...

[chadwhitacre]

Backup finished and verified.

[chadwhitacre]

Started script, saw Sept 1 and killed it. UTC! Restarted ...

[gratipay] $ ./payday.sh 274 for_real_please
Run payday #274? (y/N) y
Run payday #274 FOR REAL?!?!?!??!?!? (y/N) y
Logging to ../logs/payday/gratipay-274.log.

Fri Sep  1 08:39:21 UTC 2017
^C
[gratipay] $ ./payday.sh 274 for_real_please
Rerun payday #274? (y/N) y
Rerun payday #274 FOR REAL?!?!?!??!?!? (y/N) y

[chadwhitacre]

Saw a TypeError, realized I might not be on master, killed it right after "Greetings, program! It's Payday!"

[chadwhitacre]

But now it looks like the script kept running? 😞

[gratipay] $ tail -n19 ../logs/payday/gratipay-274.log
pid-1658 thread-140737185432512 (MainThread) Captured 39 card holds.
pid-1658 thread-140737185432512 (MainThread) Canceling card holds.
pid-1658 thread-140737185432512 (MainThread) Canceled 0 card holds.
pid-1658 thread-140737185432512 (MainThread) Updating balances.
pid-1658 thread-140737185432512 (MainThread) Updated the balances of 270 participants.
pid-1658 thread-140737185432512 (MainThread) Taking over balances.
pid-1658 thread-140737185432512 (MainThread) Updating stats.
pid-1658 thread-140737185432512 (MainThread) Updated payday stats.
pid-1658 thread-140737185432512 (MainThread) Notifying participants.
pid-1658 thread-140737185432512 (MainThread) Traceback (most recent call last):
pid-1658 thread-140737185432512 (MainThread)   File "/Users/whit537/personal/gratipay/gratipay.com/gratipay/cli/payday.py", line 11, in main
pid-1658 thread-140737185432512 (MainThread)     Application().payday_runner.run_payday()
pid-1658 thread-140737185432512 (MainThread)   File "/Users/whit537/personal/gratipay/gratipay.com/gratipay/payday_runner.py", line 27, in run_payday
pid-1658 thread-140737185432512 (MainThread)     self._start_payday().run()
pid-1658 thread-140737185432512 (MainThread)   File "/Users/whit537/personal/gratipay/gratipay.com/gratipay/billing/payday.py", line 90, in run
pid-1658 thread-140737185432512 (MainThread)     self.notify_participants()
pid-1658 thread-140737185432512 (MainThread)   File "/Users/whit537/personal/gratipay/gratipay.com/gratipay/billing/payday.py", line 480, in notify_participants
pid-1658 thread-140737185432512 (MainThread)     _user_initiated=False
pid-1658 thread-140737185432512 (MainThread) TypeError: put() takes exactly 5 arguments (3 given)

[chadwhitacre]

I've shelved my work on gratipay/gratipay.com#4548 and am back on master.

[chadwhitacre]

I am reading payday.py to determine if I should re-run w/o any code changes (commenting out).

[chadwhitacre]

gratipay::DATABASE=> select * from paydays order by ts_end desc limit 1;
┌─────┬───────────────────────────────┬───────────────────────────────┬────────┬────────┬───────┬────────┐
│ id  │           ts_start            │            ts_end             │ volume │ nusers │ stage │ nteams │
├─────┼───────────────────────────────┼───────────────────────────────┼────────┼────────┼───────┼────────┤
│ 340 │ 2017-09-01 08:39:53.223145+00 │ 2017-09-01 08:40:37.939528+00 │ 654.94 │    499 │     2 │    231 │
└─────┴───────────────────────────────┴───────────────────────────────┴────────┴────────┴───────┴────────┘
(1 row)

gratipay::DATABASE=>

[chadwhitacre]

Since stage is 2 a re-rerun will skip payin and update_stats. We'll hit end a second time, but that'll be a no-op since there's no payday with ts_end 1970.

[chadwhitacre]

Okay, that sucked.

I re-ran ... and it started a new payday! And so I ctrl-c'd right at "It's payday!" ... and then remembered that that wasn't enough to kill the script last time! So I did a ps and kill 1823 as fast as I could ... and I think it was enough?

gratipay::DATABASE=> select * from paydays order by ts_end desc limit 1;
┌─────┬───────────────────────────────┬───────────────────────────────┬────────┬────────┬───────┬────────┐
│ id  │           ts_start            │            ts_end             │ volume │ nusers │ stage │ nteams │
├─────┼───────────────────────────────┼───────────────────────────────┼────────┼────────┼───────┼────────┤
│ 340 │ 2017-09-01 08:39:53.223145+00 │ 2017-09-01 08:40:37.939528+00 │ 654.94 │    499 │     2 │    231 │
└─────┴───────────────────────────────┴───────────────────────────────┴────────┴────────┴───────┴────────┘
(1 row)

gratipay::DATABASE=>

[chadwhitacre]

There is no log for 275. Why not? 👀

[gratipay] $ less ../logs/payday/gratipay-27
gratipay-270.log  gratipay-274.log  

[chadwhitacre]

Because I ran it as 274, so new messages are appended to 274.

[gratipay] $ ./payday.sh 274 for_real_please

[chadwhitacre]

Thankfully, it appears that I killed the process in the nick of time. I see card holds but no captures. 😓 💦

[chadwhitacre]

Sooooo, we will want to cancel all of those card holds, but we can do that later today.

[chadwhitacre]

For now we need to get back on track with notifications.

[chadwhitacre]

And then finish running payday.

[chadwhitacre]

There it is:

gratipay::DATABASE=> select * from paydays order by ts_start desc limit 2;
┌─────┬───────────────────────────────┬───────────────────────────────┬────────┬────────┬───────┬────────┐
│ id  │           ts_start            │            ts_end             │ volume │ nusers │ stage │ nteams │
├─────┼───────────────────────────────┼───────────────────────────────┼────────┼────────┼───────┼────────┤
│ 341 │ 2017-09-01 08:52:48.150756+00 │ 1970-01-01 00:00:00+00        │   0.00 │      0 │     0 │      0 │
│ 340 │ 2017-09-01 08:39:53.223145+00 │ 2017-09-01 08:40:37.939528+00 │ 654.94 │    499 │     2 │    231 │
└─────┴───────────────────────────────┴───────────────────────────────┴────────┴────────┴───────┴────────┘
(2 rows)

[chadwhitacre]

We'll have to clean that up later as well. 😞

[chadwhitacre]

... or now? I believe in order to rerun without picking up the new, wrong payday, what we need to do is delete 341, and reset ts_end on 340 back to 1970-01-01 00:00:00+00.

[chadwhitacre]

gratipay::DATABASE=> update paydays set ts_end='1970-01-01 00:00:00+00' where id=340;
ERROR:  duplicate key value violates unique constraint "paydays_ts_end_key"
DETAIL:  Key (ts_end)=(1970-01-01 00:00:00+00) already exists.
gratipay::DATABASE=> delete from paydays where id=341;
DELETE 1
gratipay::DATABASE=> update paydays set ts_end='1970-01-01 00:00:00+00' where id=340;
UPDATE 1
gratipay::DATABASE=>

[chadwhitacre]

gratipay::DATABASE=> select * from paydays order by ts_start desc limit 2;
┌─────┬───────────────────────────────┬───────────────────────────────┬────────┬────────┬───────┬────────┐
│ id  │           ts_start            │            ts_end             │ volume │ nusers │ stage │ nteams │
├─────┼───────────────────────────────┼───────────────────────────────┼────────┼────────┼───────┼────────┤
│ 340 │ 2017-09-01 08:39:53.223145+00 │ 1970-01-01 00:00:00+00        │ 654.94 │    499 │     2 │    231 │
│ 339 │ 2017-08-24 21:48:50.552623+00 │ 2017-08-24 21:49:28.799672+00 │ 680.26 │    501 │     2 │    230 │
└─────┴───────────────────────────────┴───────────────────────────────┴────────┴────────┴───────┴────────┘
(2 rows)

gratipay::DATABASE=>

[chadwhitacre]

Okay! So now a rerun will pick up 340, see stage 2, skip payin and update_stats, and proceed to end and then notify_participants.

[chadwhitacre]

😅

pid-1947 thread-140737185432512 (MainThread) Picking up with an existing payday.
pid-1947 thread-140737185432512 (MainThread) Payday started at 2017-09-01 08:39:53.223145+00:00.
count#npm-sync-lag=1
pid-1947 thread-140737185432512 (MainThread) Greetings, program! It's PAYDAY!!!!
pid-1947 thread-140737185432512 (MainThread) Notifying participants.
pid-1947 thread-140737185432512 (MainThread) Script ran for six seconds (0:00:06.933801).

[chadwhitacre]

I really don't like that we're installing cron jobs during payday (could we be flushing email from local!?), but that's another story as well.

[chadwhitacre]

Gosh! Okay!

[rohitpaulk]

That was eventful..

[chadwhitacre]

I'm working on canceling the card holds.

[chadwhitacre]

There are only ~30 card holds. I might cancel manually in the Braintree dashboard instead of trying to script it in API.

[chadwhitacre]

Although, looking at fetch_card_holds ... it's a static method and if we pass it an empty list of participant_ids it should do what we want.

I think we can just do:

$ heroku run python
[...]
>>> from gratipay.billing.payday import Payday
>>> Payday.fetch_card_holds([])
[]

[chadwhitacre]

[gratipay] $ honcho run -e defaults.env,local.env env/bin/python < cancel-holds.py 
pid-3003 thread-140737185432512 (MainThread) Fetching card holds.
[gratipay] $

# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

from gratipay import wireup
from gratipay.billing.payday import Payday

wireup.billing(wireup.env())

Payday.fetch_card_holds([])

[chadwhitacre]

Sooo ...

$ heroku run python < cancel-holds.py

You okay with that, @rohitpaulk?

[chadwhitacre]

I don't see a side-effect to importing Payday, and wireup.billing also looks safe.

[chadwhitacre]

[gratipay] $ heroku run python < cancel-holds.py > cancel-holds.log 
Running python on gratipay... up, run.5784 (Hobby)
^C[gratipay] $ cat cancel-holds.log 
# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

from gratipay import wireup

wireup.billing(wireup.env())

print('Greetings, program!')
Python 2.7.11 (default, Oct 26 2016, 16:32:16) 
[GCC 5.4.0 20160609] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> # -*- coding: utf-8 -*-
teralsom __future__ import absolute_import, division, print_function, unicode_li 
>>> 
>>> from gratipay import wireup
>>> 
>>> wireup.billing(wireup.env())
>>> 
>>> print('Greetings, program!')
Greetings, program!
>>> [gratipay] $

[chadwhitacre]

Better method:

[gratipay] $ heroku run python -c "`cat greetings.py`"
Running python -c "# -*- coding: utf-8 -*-
Running python -c "# -*- coding: utf-8 -*-
Running python -c "# -*- coding: utf-8 -*-
Running python -c "# -*- coding: utf-8 -*-
from __future__ import absolute_import, division, print_function, unicode_literals

print('Greetings, program!')" on ⬢ gratipay... up, run.8444 (Hobby)
Greetings, program!
[gratipay] $

[chadwhitacre]

Okay, getting ready to pull the trigger on:

[gratipay] $ heroku run python -c "`cat cancel-holds.py`" 2>&1 | tee cancel-holds.log

from gratipay import wireup
from gratipay.billing.payday import Payday
wireup.billing(wireup.env())
Payday.fetch_card_holds([])

[chadwhitacre]

3 ...

[chadwhitacre]

2 ...

[chadwhitacre]

1 ...

[chadwhitacre]

0!

[chadwhitacre]

Running python -c "from gratipay import wireup
from gratipay.billing.payday import Payday
wireup.billing(wireup.env())
Payday.fetch_card_holds([])" on gratipay... starting, run.4177 (Hobby)
Running python -c "from gratipay import wireup
from gratipay.billing.payday import Payday
wireup.billing(wireup.env())
Payday.fetch_card_holds([])" on gratipay... connecting, run.4177 (Hobby)
Running python -c "from gratipay import wireup
from gratipay.billing.payday import Payday
wireup.billing(wireup.env())
Payday.fetch_card_holds([])" on gratipay... up, run.4177 (Hobby)
pid-4 thread-139695064790784 (MainThread) Fetching card holds.
pid-4 thread-139695064790784 (MainThread) Canceled a $11.13 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.10 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $103.30 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $11.64 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $20.91 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $13.70 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.61 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.61 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.61 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $20.91 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.36 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $12.67 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $12.05 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.61 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.61 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $11.64 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.87 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $12.67 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.00 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.82 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.61 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.61 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $12.67 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.61 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $103.30 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $11.52 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $16.79 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.51 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $11.13 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $19.10 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $12.67 hold for [].
pid-4 thread-139695064790784 (MainThread) Canceled a $10.10 hold for [].

[chadwhitacre]

cancel-holds.log appended to gratipay-274.log in https://github.com/gratipay/logs/commit/aa24ac447d7ed8ac165531e881a9805586ca914f.

[chadwhitacre]

Still held in my bank account per bank's web interface. Will check back later on that.

[chadwhitacre]

We canceled one extra hold for 12.05 that must've been left over from ... last time? Or something? Otherwise the amounts all line up for the 31 we accidentally created.

[chadwhitacre]

Oh! Hold placement is multi-threaded so we could've created the 12.05 hold but not logged it.

[chadwhitacre]

Alright! Things to reticket:

• making it easier to kill the payday script and have it actually die → Make it easier to kill payday gratipay.com#4595
• not running cron jobs in the payday script → Don't run cron jobs during payday gratipay.com#4594
• talking about our payday rotation and whether it's time to switch that up → Revise payday rotation? #1152

[chadwhitacre]

Reticketed! Closing.