Failure with Linux and USAA

[Dameon87]

I'm unsure what the exact cause of this may be, but I'm running into trouble using this with Linux (Ubuntu 20.10) on both python3.8 and python3.9. I can get the commands to work correctly on windows. However, every command via linux is outputting:

~/.local/bin$ ofxget stmt usaa -u 011111111 --all
Traceback (most recent call last):
File "/home/wallet/.local/bin/ofxget", line 8, in
sys.exit(main())
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/scripts/ofxget.py", line 1568, in main
REQUEST_HANDLERSargs["request"]
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/scripts/ofxget.py", line 646, in request_stmt
_merge_acctinfo(args, acctinfo)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/scripts/ofxget.py", line 617, in _merge_acctinfo
acctinfos: List[AcctInfo] = sorted(extract_acctinfos(markup), key=sortKey)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/scripts/ofxget.py", line 1380, in extract_acctinfos
parser.parse(markup)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/Parser.py", line 85, in parse
self.header, message = self._read(source)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/Parser.py", line 118, in _read
header, message = parse_header(source)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/header.py", line 294, in parse_header
header, header_end_index = OFXHeaderV1.parse(rawheader)
File "/home/wallet/.local/lib/python3.9/site-packages/ofxtools/header.py", line 81, in parse
raise OFXHeaderError(f"OFX header is malformed:\n{rawheader}")
ofxtools.header.OFXHeaderError: OFX header is malformed:

<title>

        System Error
      | USAA</title>
<style media="screen" type="text/css">

This, again, only seems to happen on linux. The same command works fine on my windows install.

[csingley]

Hmm, I can't reproduce this on Python 3.9 under Linux. Is this occurring on master, or an older version of ofxtools?

[Dameon87]

I've tried this on master and the most current release version. I'm quite unsure why this is happening. Only thing I can think of is it's an issue in Ubuntu. Are you able to attempt to give this a try on Ubuntu 20.10?

[csingley]

If you're running identical Python and ofxtools on 2 different OS'es, but on one of them you're getting different results, I'd eliminate differences in user config before turning to the OS.

The problem is that USAA isn't sending you OFX. Your Ubuntu machine is sending different requests. Try comparing the output of ofxget list usaa and ofxget stmt --dryrun usaa on both hosts to find the differences.

[gthole]

I'm also encountering this issue.

Docker python:3.7-alpine
reqs.pip ofxtools
Then ofxget acctinfo usaa --user <uid> throws an invalid OFX header. It looks like USAA is returning an HTML doc in response to this endpoint, so maybe it's an issue on their end?

[csingley]

Well shoot, now it's failing for me as well. I'll try to look into this a bit more.

[jackpot51]

@csingley bad news, USAA completely changed their OFX authentication so now only Quicken can connect

https://communities.usaa.com/t5/Finances/USAA-Creates-Quicken-Monopoly/td-p/243850

[csingley]

Well that's a drag. USAA seems to be following Schwab to the "OFX Secure Plus" dark side... "You'll use Quicken and like it".

The original reports on this ticket were probably formatting differences that were fixable by tweaking connection parameters. This latest change is something qualitatively different, and it ain't gonna be fixable from my end.

[gthole]

Haven't tested this yet, but someone claims to have got this working from the master branch:
https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045704.html

[cfazzini]

@csingley I've been able to get USAA to work again with some information from the GnuCash-devel mailing list. The following config seems to work for me with latest version from here with some addl notes.

The clientuid I extracted from the URL when authenticating via https://df3cx-services.1fsapi.com/casm/usaa/enroll. After entering USAA credentials the page is redirected to https://www.usaa.com/inet/ent_oauth_consent/authorize?0&client_id=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx&.... It would seem there may be a link between the account and this clientuid. Also ClientUID must be in uppercase.

DTACCTUP of 19900101 is required to get good data. The default of 19901231 returned success, but only "Client up to date", no acct info.

pretty = true is also required (assuming the newline requirement).

I'm not sure if or where these changes should be incorporated, but hoping the information is useful none the less.

[USAA-NEW]
url = https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
clientuid = XXXXXXXX-XXXX-XXXX-XXXXX-XXXXXXX
org = USAA Federal Savings Bank
fid = 67811
bankid = 314074269
user = xxxxxxxxx
version = 103
unclosedelements = true
appid = QMOFX
appver = 2300
pretty = true

@gthole That was me. Was updating this post here at same time you posted. 👍

[jackpot51]

@cfazzini do you know what user agent is being used?

I ended up getting the IP block described on https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045690.html, thank fuck my assets are at Chase now...

[cfazzini]

@jackpot51 Best I can tell from the code in ofxget.py, default is a blank User-Agent, so it would appear I'm not sending one. But, could use confirmation on that. Sucks on the IP ban. Hopefully it's short term.

[csingley]

@cfazzini - thanks for the report! Setting CLIENTUID is not too crazy.

I'm thinking I may need to patch OFXClient to uppercase the UUID and send the truncated DTACCTUP.

Should we perhaps set up a wiki, to serve as a repository for this kind of info that's helpful to users?

[cfazzini]

Sorry just realized I wasn't too clear. It seemed to not have an issue with the full date format, the value of the date seemed to matter more. I just used the 19900101 format with the --dtacctup CLI flag and it was fine.

[csingley]

@cfazzini - thanks for the clarification.

From ofxtools.Client.OFXClient, by default useragent: str = "InetClntApp/3.0"

[cfazzini]

Thanks for clarification, I was looking at this in ofxget.py

parser.add_argument(
            "--useragent",
            dest="useragent",
            help="Value to use in HTTP 'User-Agent' header (defaults to empty string)",
        )

[gtTracy]

Well, here we are about a week later, and I'm still getting the Incapsula response using the CURL string and saved file that I had from last week. (Today is the first time I've tried to do anything with this since my last post on the 18th, and I tried only the one time.) I will try it again tomorrow night, and see if that matters (since, technically, tomorrow will be the 7th day). Not sure where to go with this if I am not able to get a connection tomorrow.

[jackpot51]

You could jump ship like I did to Chase, where they still have a working OFX endpoint

[ajam000]

I also remain blocked from home, however I set up ofxtools on an AWS instance with the exact same config, and I was able to download my account data, so it really is just blocking access from my home IP for some reason. I haven't made a decision yet whether this is an acceptable solution for me or whether just switching to a more customer-friendly bank makes more sense.

[jackpot51]

I had issues with two USAA accounts until I ensured that all *UID values were uppercase GUID's, not random strings. After that, I still get blocked downloading account data from one user, but not from the other. Downloading the list of accounts works on both users. So I think there is server-side blocking that is massively screwed up.

[gtTracy]

Well, tonight's run was rather anti-climactic. I remain blocked. So, at some point in the next week, I will probably gird my loins with excessive patience and actually call USAA and see if I can find someone there who has a clue about all this (I am not encouraged by many, many postings on their community boards) and try to get

A) unblocked
B) a successful run for account lists
C) a successful run for transactions from at least one account

Should it happen that I am successful, I will post back here with any relevant details. Should it happen that I am unsuccessful, I will at least report back that fact.

The odds are that it will be at least Tuesday or Wednesday of the coming week before I have opportunity to pursue this (given the amount of time investment required to climb through first layer tech support).

[ajam000]

I wondering whether the people who have had success getting this to work have a paid Quicken account? I have noticed that the app access for Quicken disappears from my USAA account profile after about a week. If I re-request access, I get the exact same access ID and PIN every time, and it reappears in my USAA profile. I am wondering whether Quicken is doing something to keep the token alive that is not done by ofxtools.

[cfazzini]

No paid Quicken acct, I've continued to have no issues with USAA since getting the format correct. I download transactions intermittently, sometimes more than a week apart.

[gtTracy]

@cfazzini Could you post your working configuration (without accessid and pin, of course)? Perhaps I'll see something in yours that is not in mine (or something that is in mine that is not in yours)?

At this point, it has been about two weeks (13 days, to be exact) since I last tried it. I am probably going to wait until next week (that will put me at 19 days, which should be plenty of time for an IP block to be lifted). Then if I get nowhere, I will try to figure out AWS enough to be able to spin up an instance under the free tier to test with. Then, depending on what results I get, I will likely be trying to get in touch with USAA support...

[ajam000]

FYI, AWS is not working for me either - it worked a few times and then I was blocked just like my home IP. I've tried IPs from multiple AWS regions, so either there is a very wide-ranging IP block or the blocking is tied to my account. I'm not sure what to conclude - I've not identified any difference in my setup other than my USAA account credentials.

[gtTracy]

If that is the case, then it has to be tied to the account. So, yet another thing to discuss with them once I get to that point...

[gtTracy]

OK, so it had been a couple of weeks since I messed with this, so I decided to try it again. Here's what I did, and the results that I got. I'm still not sure what these results mean, but perhaps someone here will.

First, I went to the main page of this repository, downloaded the code (using git clone https://github.com/csingley/ofxtools.git. Then i went into the ofxtools directory, and did pip install . which removed my existing ofxtools install and replaced it with what I had just downloaded (still shows version 0.9.1, but it does incorporate changes from the previously installed version, as will become evident in a moment).

Next, I went to USAA and re-authorized Quicken to access the account (to refresh all my data and access). Gave me the same access ID and access PIN as before, but I did copy the new UUID from the resulting URL into the ofxget.cfg file. Relevant portion of the ofxget.cfg file is:

[usaa1]
url = https://df3cx-services.1fsapi.com/casm/usaa/access.ofx
version = 103
unclosedelements = true
pretty = true
org = USAA Federal Savings Bank
fid = 67811
bankid = 314074269
clientuid = 1A359CF9-E558-40F9-B304-AC4777F5F55A
appid = QWIN
appver = 2700

So, after that, I tried to use ofxget acctinfo usaa1 -user <accessID> to retrieve account info. Didn't work - instead, I got:

Traceback (most recent call last):
  File "/home/tracy/.local/bin/ofxget", line 8, in <module>
    sys.exit(main())
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 1598, in main
    REQUEST_HANDLERS[args["request"]](args)
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 600, in request_acctinfo
    acctinfo = _request_acctinfo(args, password)
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/scripts/ofxget.py", line 617, in _request_acctinfo
    with client.request_accounts(
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 593, in request_accounts
    RqCls2url = self._get_service_urls(
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 420, in _get_service_urls
    profile = self.request_profile(
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Client.py", line 507, in request_profile
    parser.parse(response)
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Parser.py", line 85, in parse
    self.header, message = self._read(source)
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/Parser.py", line 118, in _read
    header, message = parse_header(source)
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/header.py", line 294, in parse_header
    header, header_end_index = OFXHeaderV1.parse(rawheader)
  File "/home/tracy/.local/lib/python3.8/site-packages/ofxtools/header.py", line 81, in parse
    raise OFXHeaderError(f"OFX header is malformed:\n{rawheader}")
ofxtools.header.OFXHeaderError: OFX header is malformed:
<html>

<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>

So I went back and did the ofxget acctinfo usaa1 --user <accessID> -n > ~/usaatest.txt command to generate the text file with the info to send via curl, which turned out as:

OFXHEADER:100
DATA:OFXSGML
VERSION:103
SECURITY:NONE
ENCODING:USASCII
CHARSET:NONE
COMPRESSION:NONE
OLDFILEUID:NONE
NEWFILEUID:76DB7950-B0FE-4EBF-BC4B-A1310641A3D2

<OFX>
<SIGNONMSGSRQV1>
  <SONRQ>
  <DTCLIENT>20210317221045.836[+0:UTC]
      <USERID>accessID
      <USERPASS>accessPIN
      <LANGUAGE>ENG
      <FI>
      <ORG>USAA Federal Savings Bank
        <FID>67811
      </FI>
      <APPID>QWIN
      <APPVER>2700
      <CLIENTUID>1A359CF9-E558-40F9-B304-AC4777F5F55A
    </SONRQ>
  </SIGNONMSGSRQV1>
  <SIGNUPMSGSRQV1>
<ACCTINFOTRNRQ>
  <TRNUID>7F4B66FF-EE7B-471C-9342-6AA59EB6A51B
      <ACCTINFORQ>
    <DTACCTUP>19900101000000.000[+0:UTC]
      </ACCTINFORQ>
    </ACCTINFOTRNRQ>
  </SIGNUPMSGSRQV1>
</OFX>

Then I tried sending that using curl using the command:

curl -isS -X POST -H "Content-Type: application/x-ofx" -A InetClntApp/3.0 --data-binary @~/usaatest.txt https://df3cx-services.1fsapi.com/casm/usaa/access.ofx

Which netted me this:

HTTP/2 200 
content-type: text/html
cache-control: no-cache, no-store
content-length: 212
x-iinfo: 7-44160127-0 0NNN RT(1616019220696 0) q(0 -1 -1 -1) r(0 -1) B10(4,289,0) U6
strict-transport-security: max-age=31536000
set-cookie: visid_incap_2454689=7ncQU8fwRnW23+kvhq6cHxR/UmAAAAAAQUIPAAAAAAApfjZcKE1eXpbwMRh34RUp; expires=Thu, 17 Mar 2022 08:21:42 GMT; HttpOnly; path=/; Domain=.1fsapi.com; Secure; SameSite=None
set-cookie: incap_ses_1167_2454689=UMitL8M5kw0O3BS1lAQyEBR/UmAAAAAAoDJkmfUKLvNnlQpx3rh08A==; path=/; Domain=.1fsapi.com; Secure; SameSite=None

<html>
<head>
<META NAME="robots" CONTENT="noindex,nofollow">
<script src="/_Incapsula_Resource?SWJIYLWA=5074a744e2e3d891814e9a2dace20bd4,719d34d31c8e3a6e6fffd425f7e032f3">
</script>
<body>
</body></html>

I did go back to USAA at one point in this process and attempt to reauthorize Quicken, and I was prompted at sign-in with a "are you human" Captcha, which I answered then it let me log in. So I'm thinking that I might have been blocked at one point, and going in and answering that Captcha unblocked me? Maybe. Anyway, this is where I'm currently at. If anyone has any additional suggestions, I'm all ears. Otherwise, I'm about to just throw my hands up at it and just forget about it again for a while (until I have another free day that I don't mind wasting on it)....

[ajam000]

I am in the same state - I consistently am blocked with this exact same error page. I have also tried testing from multiple AWS regions in order to rule out IP blocking, and did get it to work the very first time I tried sending the request from an AWS instance, but never again. I did not change anything between the successful query and the consistent failure thereafter - there must be something tied to my account blocking access, and likely yours as well.

[csingley]

Good people, you are entirely welcome to discuss the rituals required to summon lesser demons of the technofinance pantheon. Let's just, you know, take it off the bug tracker, okay? I'm pushing the "Discussions" section of the Github site, which seems like a good fit for this sort of thing.

[jackpot51]

Yeah, at this point, nothing any software you run can do will fix the issue of getting blocked. It is completely a USAA server-side issue

[UrsaNoctua]

After a good bit of testing, I think the issue is around carriage returns.

I tried @cfazzini's curl trick from #123 (comment) an on ubuntu ec2 instance. While editing in vim, I noticed that the headers had "\r\n" (shows up as a ^M at the end of the line in vim) and the lines starting with <OFX> did not.

I tried the curl test from https://lists.gnucash.org/pipermail/gnucash-devel/2021-February/045690.html and it worked fine. After reviewing the email on gnucash-devel, I edited my input file from @cfazzini's curl test to have \r\n (in vim, I removed the ^M endings and did :set ff=dos then saved). After that, the test worked and I received an OFX response.

Since @cfazzini mentioned using WSL, I'm suspecting the pretty portion of the OFX code uses native carriage returns and python on windows is using "\r\n" while python on linux is using "\n"

[UrsaNoctua]

I've had some luck with a local checkout where I edited the utils.indent() method to use "\r\n" instead of "\n". The trick is that I also need to disable cookies with OFXClient. I'm not yet sure why I have to disable cookies, but that's the only way I can get it to work.

Since ofxget doesn't have an option for disabling cookies, I'm hesitant to submit a patch for the carriage return fix - as it won't fully fix the problem.

[csingley]

Nice detective work @UrsaNoctua !

I would welcome such a patch to utils.ident() in any case, as part of the general goal of mimicking Quicken.

Along the same lines, I am puzzled why you need to disable cookies for USAA... I'd been under the impression this was a uniform sequence by Quicken. Apparently not.

If it's useful in the real world, I'd also be supportive of an option to disable cookies. I'm not able to work on it myself ATM, and generally a bit clueless about HTTP-related matters.

[jfly]

Over a year later, but I wanted to chip in in case anyone else is reading through this thread:

If I run ofxget acctinfo --dtacctup 19900101 USAA-NEW with a config file like what's described here, I see:

• It works on Python 3.9 with the requests backend, but not the urllib backend (I'm referring to USE_REQUESTS here)
• But it works on Pyton 3.10 with either the requests backend, or the urllib backend!?
• It also works on Python 3.9 and 3.10 with either requests or urllib if I disable cookies (by commenting out this requests code and this urllib code), as @UrsaNoctua mentioned above.

I spent forever trying to understand what this Python 3.9 vs 3.10 and urllib vs requests difference is all about. Here's what I tried/learned:

• There are a number of HTTP header differences when using urllib vs requests, but I don't think any of them matter. I almost thought I had something when I saw that cookies were getting ordered differently between the two, but that ultimately didn't make a difference.
• I could not reproduce the issue using https://mitmproxy.org/!
• I was able to capture the issue in wireshark, and I stared at two decrypted HTTPS captures (a working one generated with urllib on py3.9 and a failing one generated with urllib on py3.10) and I could not see any meaningful difference whatsover (the cookies were different, but I get different cookies every time I connect)

This was extra infuriating to debug because I'd often get blocked by whatever Incapsula CDN/bot protection they've got setup. I have come to the conclusion that there's something more than just the HTTP layer that's relevant here. My best guess is that Incapsula uses TLS fingerprinting to block some requests. mitmproxy/mitmproxy#4575 and https://httptoolkit.tech/blog/tls-fingerprinting-node-js/ talk about this and some workarounds. I did confirm that the TLSv1.3 Hello advertises different cipher suites when using py 3.9 vs 3.10 and requests vs urllib.

So, for now I'm happy with python 3.10 and urllib, although I bet that will break someday when Incapsula adds a block for its TLS fingerprint 😢

[csingley]

Thanks for the report @jfly ! Lots of fun writing software to talk to a party taking active countermeasures against you, when you’re paying for the privilege