Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Amazon] Amazon import source for amazon.de #144

Merged
merged 46 commits into from
Mar 28, 2022
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
46 commits
Select commit Hold shift + click to select a range
27f88c0
amazon source with locale specification for EN and DE
moritzj29 Dec 12, 2021
75097fd
update test reference output
moritzj29 Dec 12, 2021
7570a8b
addressed some PR comments, polishing
moritzj29 Dec 18, 2021
b3760b7
updated example docstring with new locale names
moritzj29 Dec 18, 2021
6501b5f
added some translations (de_DE)
moritzj29 Dec 18, 2021
c1c8af8
add ability to parse prepended 3 letter currency specification, e.g. …
moritzj29 Dec 18, 2021
93e6b11
Merge branch 'dev/amount_parse' into dev/amazonDE
moritzj29 Dec 18, 2021
e9a9c95
updated invoice sanitizer to not completely remove payment table for …
moritzj29 Dec 18, 2021
03b9a06
add some sanitized invoice tests for de_DE
moritzj29 Dec 18, 2021
02cb706
change invoice.tax from None to [] to match convention of other JSON …
moritzj29 Dec 19, 2021
1864838
de_DE correct adjustments: posttax instead of pretax
moritzj29 Dec 19, 2021
17661bf
add ability to parse gift card orders correctly
moritzj29 Dec 19, 2021
7afeaac
remove debugging logs
moritzj29 Dec 19, 2021
8c109ac
Merge branch 'dev/amazonDE' into dev/amazonDEgiftCard
moritzj29 Dec 19, 2021
0bbce16
add some test data
moritzj29 Dec 19, 2021
a562159
add test for direct debit
moritzj29 Dec 19, 2021
f4b676c
Merge branch 'dev/amazonDE' into dev/amazonDEgiftCard
moritzj29 Dec 19, 2021
3d4a077
add amazon account charge up including test case
moritzj29 Dec 19, 2021
d3e701b
remove debugging print statement
moritzj29 Dec 19, 2021
a641c94
make quantity parsing more consistent, add log message if parsing failed
moritzj29 Jan 13, 2022
a460017
fix DE shipment_quantity_pattern
moritzj29 Jan 13, 2022
d367cdb
fix shipment quantity algorithm
moritzj29 Jan 13, 2022
eda2adb
make payee match chosen locale
moritzj29 Feb 20, 2022
98f15f0
add nonshipped header translation
moritzj29 Feb 20, 2022
df350d2
add nonshipped header translation
moritzj29 Feb 21, 2022
c179a48
fix typing, remove debug print statements
moritzj29 Mar 24, 2022
1529e0b
add test cases for nonshipped headers
moritzj29 Mar 24, 2022
8b5a878
correct locale name en_EN to en_US
moritzj29 Mar 24, 2022
e8d1e83
add Locale base class
moritzj29 Mar 24, 2022
63c9a47
Merge branch 'dev/amazonDEgiftCard' into dev/amazonDE
moritzj29 Mar 24, 2022
d5263ea
clean up imports, do not create locale instance
moritzj29 Mar 24, 2022
ff21279
reduce excessive logging for amazon fresh invoices with irrelevant qu…
moritzj29 Mar 25, 2022
8715e51
correct test name
moritzj29 Mar 25, 2022
7aac6ad
add missing types
moritzj29 Mar 25, 2022
f042a9f
add docstring with hierarchy of functions
moritzj29 Mar 25, 2022
7338a7b
use static class instead of class instance as default arguments
moritzj29 Mar 26, 2022
a438613
factor out is_items_ordered_header
moritzj29 Mar 26, 2022
69d146e
clean up locales, add comments
moritzj29 Mar 27, 2022
577e196
add comments and docstrings
moritzj29 Mar 27, 2022
e4b1ee9
add logging and error messages
moritzj29 Mar 27, 2022
8b6de9c
reduce conditionals in parse_credit_card_transactions_from_payments_t…
moritzj29 Mar 27, 2022
38fe597
move order ID and date extraction to beginning of parsing method, mor…
moritzj29 Mar 27, 2022
2287972
add check and error message if no items were found for an order
moritzj29 Mar 27, 2022
4be65b3
fix handling of cases with no tax; tax on Order level is Amount (None…
moritzj29 Mar 27, 2022
41a14fa
update types, fix Shipment tax type
moritzj29 Mar 27, 2022
66298c1
make parse_gift_cards optional
moritzj29 Mar 27, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 8 additions & 2 deletions beancount_import/amount_parsing.py
Original file line number Diff line number Diff line change
Expand Up @@ -30,12 +30,18 @@ def parse_amount(x, assumed_currency=None):
if not x:
return None
sign, amount_str = parse_possible_negative(x)
m = re.fullmatch(r'(?:[(][^)]+[)])?\s*([\$€£])?((?:[0-9](?:,?[0-9])*|(?=\.))(?:\.[0-9]+)?)(?:\s+([A-Z]{3}))?', amount_str)
m = re.fullmatch(r'(?:[(][^)]+[)])?\s*([\$€£]|[A-Z]{3})?\s*((?:[0-9](?:,?[0-9])*|(?=\.))(?:\.[0-9]+)?)(?:\s+([A-Z]{3}))?', amount_str)
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

allow to specify currency with prepended three letters, e.g. EUR 20.00

if m is None:
raise ValueError('Failed to parse amount from %r' % amount_str)
if m.group(1):
currency = {'$': 'USD', '€': 'EUR', '£': 'GBP'}[m.group(1)]
# unit before amount
if len(m.group(1)) == 3:
# 'EUR' or 'USD'
currency = m.group(1)
else:
currency = {'$': 'USD', '€': 'EUR', '£': 'GBP'}[m.group(1)]
elif m.group(3):
# unit after amount
currency = m.group(3)
elif assumed_currency is not None:
currency = assumed_currency
Expand Down
21 changes: 16 additions & 5 deletions beancount_import/source/amazon.py
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@
'Gift Card Amount': 'Assets:Gift-Cards:Amazon',
'Rewards Points': 'Income:Amazon:Cashback',
},
locale='en_US' # optional, defaults to 'en_US'
)

The `amazon_account` key must be specified, and should be set to the email
Expand All @@ -54,6 +55,9 @@
specify these keys in the configuration, the generic automatic account
prediction will likely handle them.

The `locale` sets country/language specific settings.
Currently, `en_US` and `de_DE` are available.

Specifying credit cards
=======================

Expand Down Expand Up @@ -264,14 +268,15 @@
import os
import sys
import pickle
import logging

from beancount.core.data import Transaction, Posting, Balance, Commodity, Price, EMPTY_SET, Directive
from beancount.core.amount import Amount
from beancount.core.flags import FLAG_OKAY
from beancount.core.number import ZERO, ONE
import beancount.core.amount

from .amazon_invoice import parse_invoice, DigitalItem, Order
from .amazon_invoice import LOCALES, parse_invoice, DigitalItem, Order

from ..matching import FIXME_ACCOUNT, SimpleInventory
from ..posting_date import POSTING_DATE_KEY, POSTING_TRANSACTION_DATE_KEY
Expand All @@ -280,6 +285,8 @@

import datetime

logger = logging.getLogger('amazon')

ITEM_DESCRIPTION_KEY = 'amazon_item_description'
ITEM_URL_KEY = 'amazon_item_url'
ITEM_BY_KEY = 'amazon_item_by'
Expand All @@ -300,14 +307,15 @@ def make_amazon_transaction(
posttax_adjustment_accounts,
credit_card_accounts,
amazon_account: str,
payee='Amazon.com'
):
txn = Transaction(
date=invoice.order_date,
meta=collections.OrderedDict([
(ORDER_ID_KEY, invoice.order_id),
(AMAZON_ACCOUNT_KEY, amazon_account),
]),
payee='Amazon.com',
payee=payee,
narration='Order',
flag=FLAG_OKAY,
tags=EMPTY_SET,
Expand Down Expand Up @@ -387,7 +395,7 @@ def make_amazon_transaction(
(INVOICE_DESCRIPTION, adjustment.description),
]),
))
if invoice.tax is not None and invoice.tax.number != ZERO:
if len(invoice.tax)>0 and invoice.tax.number != ZERO:
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What guarantees invoice.tax is not None?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it relates to the above discussion: #144 (comment)

But I cannot recall why I removed the check on None...

For digital orders, tax=[] on order level anyway, so no need to check for None.

For regular orders, tax is set via:

tax = locale.parse_amount(
        get_field_in_table(payment_table, locale.regular_estimated_tax))

Which may return None if the field is not found...

Following up on the discussion above, I would vote for checking on None in parse_regular_order_invoice and setting tax=[] in this case. This ensures that in the resulting JSON "tax": []. Maybe also logging an error or warning might make sense, since so far all regular invoices contain tax information.

What are your thoughts on this? Did I miss something?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I tried to tackle this and stumbled upon an inconsistency in the current codebase:

According to the typedefs tax: Amount for Order and Shipment. But this is not correct. It is actually tax: Sequence[Adjustment] for Shipment. So to have an empty tax field on shipment level one needs to set tax=[], whereas on order level tax=None, since Amount can be None.

I don't know why the typedef check does not catch this...

When it comes to digital orders, the current code sets tax=[] on order level. This is not correct, since Amount cannot be [], it must be None. The code works nevertheless, since tax is always given on shipment level for digital orders and therefore tax on order level is not used anyway.

From the code perspective, I would go with the above corrections. But my dilemma arises when it comes to the JSON output for the tests: Setting tax=None translates into tax: null in the JSON. You suggested above that we should stick to the convention field=[]. I would drop this requirement since it would require additional modifications of the data (only for the test files...).

@Zburatorul are you fine with this? what are your thoughts?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I added a proposal in 4be65b3

  • fix shipment.tax to type List[Adjustment]
  • fix order.tax to type Amount
  • results in tax: null in JSON for orders with no tax transaction (tax not given or included in item price)

txn.postings.append(
Posting(
account=unknown_account_name,
Expand Down Expand Up @@ -539,6 +547,7 @@ def __init__(self,
posttax_adjustment_accounts: Dict[str, str] = {},
pickle_dir: str = None,
earliest_date: datetime.date = None,
locale='en_US',
**kwargs) -> None:
super().__init__(**kwargs)
self.directory = directory
Expand All @@ -551,6 +560,7 @@ def __init__(self,
self.pickler = AmazonPickler(pickle_dir)

self.earliest_date = earliest_date
self.locale = LOCALES[locale]

self.invoice_filenames = [] # type: List[Tuple[str, str]]
for filename in os.listdir(self.directory):
Expand All @@ -570,7 +580,7 @@ def _get_invoice(self, results: SourceResults, order_id: str, invoice_filename:
invoice = self.pickler.load(results, invoice_path) # type: Optional[Order]
if invoice is None:
self.log_status('amazon: processing %s: %s' % (order_id, invoice_path, ))
invoice = parse_invoice(invoice_path)
invoice = parse_invoice(invoice_path, locale=self.locale)
self.pickler.dump( results, invoice_path, invoice )

self._cached_invoices[invoice_filename] = invoice, invoice_path
Expand Down Expand Up @@ -605,7 +615,8 @@ def prepare(self, journal: JournalEditor, results: SourceResults):
invoice=invoice,
posttax_adjustment_accounts=self.posttax_adjustment_accounts,
amazon_account=self.amazon_account,
credit_card_accounts=credit_card_accounts)
credit_card_accounts=credit_card_accounts,
payee=self.locale.payee)
results.add_pending_entry(
ImportResult(
date=transaction.date,
Expand Down
Loading