Report summaries in JSON (API) output #10466

dacook · 2023-02-21T02:14:38Z

What? Why?

Closes Allow report summaries in JSON (API) output #10508
Also related to Allow user to show/hide totals with a checkbox toggle #6850

This change allows you to optionally enable summary rows on JSON report output, via the reports v0 API.

{
  "data": [
    {
      "hub": "Elmore Compost and Organics",
      "supplier": "Elmore Compost and Organics",
      "customer_code": null,
      "first_name": "David",
      "last_name": "Cook",
      "product": "Biochar",
      "variant": "2L",
      "quantity": 1
    },
    {
      "hub": "",
      "supplier": "",
      "customer_code": "",
      "first_name": "",
      "last_name": "",
      "product": "",
      "variant": "Total By Customer",
      "quantity": 1,
      "report_row_type": "summary"
    },
...

What should we test?

Test to ensure no change in Admin Reports:

column options in form dropdown
displayed columns
displayed header rows
displayed summary rows

Test to ensure no change in API v0 reports:

columns
header not displayed by default
summary not displayed by default

Test to enable summary rows

Add parameter display_header_row=true
Add parameter display_summary_row=true

Extra rows should be output, similar to example above

Test to enable summary row markers

Summary rows look just like normal rows, so an additional marker can be used to differentiate them:

Add parameter fields_to_show[]=report_row_type

The summary rows should have a new column/field: "report_row_type": "summary"

Release notes

Changelog Category: Technical changes

The title of the pull request will be included in the release notes.

Dependencies

Documentation updates

Yep: https://ofn-user-guide.gitbook.io/ofn-api-handbook/ofn-api-v0-unsupported/reports

abdellani · 2023-02-21T11:21:42Z

lib/reporting/report_renderer.rb

@@ -10,7 +10,13 @@ def initialize(report)
      @report = report
    end

+    # Strip headers and footers for these formats


Does footers refer to the summary rows?

if raw_render? is only used to decide about the header and summary. Maybe we can use a better name?

Yes, you're right it should say "summary rows" rather than "footers".

I agree it would be good to have a better name. I'm not sure what is better though. Maybe render_data_only?

I think this is better :)

I've updated the comment, but opted to avoid renaming the method because I hope to remove it anyway.

abdellani · 2023-02-21T11:27:17Z

Hi @dacook

Does this solve #6850?

dacook · 2023-03-03T00:29:12Z

Does this solve #6850?

Actually no, that issue is to allow summary rows in CSV output. But it seems we don't quite have consensus yet.

This PR is to allow summary rows in JSON. I don't think it depends on a decision on the above, and it does unblock our integration requirements in AU so I suggest we continue. I'll create an issue for reference.

dacook · 2023-03-03T00:42:49Z

@abdellani , I've attached an issue and addressed one of your comments. Would you be able to complete your review?

abdellani

Everything looks good :)

Thank you @dacook

mkllnk

Nice quick solution.

mkllnk · 2023-03-06T04:56:27Z

spec/lib/reports/report_spec.rb

@@ -124,7 +124,7 @@ def check_report
      end

      it "get correct data" do
-        allow(subject).to receive(:raw_render?).and_return(true)
+        allow(subject).to receive(:unformatted_render?).and_return(true)


I think this line should have been in a later commit. Too late now. 🤷

It looks like it was expected that the price and amount columns would be formatted, but this is not currently the case. So I cleaned this up to be less ambiguous. If any of these columns were to be configured for formatting, this line could be added back to test for unformatted output.

…ndependently This will allow us to change the options in the following commit.

You wouldn't want it by default, but it can be enabled with a parameter like display_summary_row=true

This can be enabled in the reports API with the hidden parameter `fields_to_show[]=report_row_type` to help with parsing summary row output.

filipefurtad0 · 2023-03-22T20:17:00Z

Hey @dacook,

I think there are only three report types, which include the header and summary row options. These are:

Bulk Co-Op
Orders & Fulfillment
Packing

For these three cases, I've tested both the API v0 requests; for all others only UI tests. A summary is found below:

API V0 tests

Bulk Co-Op 🟢

Bulk Co-op Supplier Report

https://staging.openfoodnetwork.org.uk/api/v0/reports/bulk_coop?token=****&report_subtype=supplier_report&q[completed_at_gt]=2022-02-22+00:00&q[completed_at_lt]=2023-03-23+00:00&fields_to_show%5B%5D=supplier&fields_to_show%5B%5D=product&fields_to_show%5B%5D=max_quantity_excess&fields_to_show%5B%5D=product&display_header_row=true&display_summary_row=true&fields_to_show[]=report_row_type&fields_to_show[]=bulk_unit_size&fields_to_show[]=variant&fields_to_show[]=sum_total&fields_to_show[]=unallocated&fields_to_show[]=units_required

Bulk Co-op Allocation

https://staging.openfoodnetwork.org.uk/api/v0/reports/bulk_coop?token=****&report_subtype=allocation&q[completed_at_gt]=2023-02-22+00:00&q[completed_at_lt]=2023-03-23+00:00&fields_to_show%5B%5D=supplier&fields_to_show%5B%5D=product&fields_to_show%5B%5D=max_quantity_excess&fields_to_show%5B%5D=product&display_header_row=true&display_summary_row=true&fields_to_show[]=report_row_type&fields_to_show[]=bulk_unit_size&fields_to_show[]=variant&fields_to_show[]=sum_total&fields_to_show[]=unallocated&fields_to_show[]=units_required&fields_to_show[]=customer

Orders & Fulfillment Reports 🟡

Order Cycle Supplier Totals

https://staging.openfoodnetwork.org.uk/api/v0/reports/orders_and_fulfillment?token=****&report_subtype=order_cycle_supplier_totals&q[completed_at_gt]=2023-02-22+00:00&q[completed_at_lt]=2023-03-23+00:00&fields_to_show%5B%5D=producer&fields_to_show%5B%5D=product&fields_to_show%5B%5D=variant&fields_to_show%5B%5D=quantity&fields_to_show%5B%5D=total_units&fields_to_show%5B%5D=total_cost&fields_to_show%5B%5D=sku&fields_to_show%5B%5D=producer_charges_sales_tax?&fields_to_show%5B%5D=product_tax_category&display_header_row=true&display_summary_row=true&fields_to_show[]=report_row_type&fields_to_hide%5B%5D&fields_to_show[]=

Order Cycle Supplier Totals by Distributor
https://staging.openfoodnetwork.org.uk/api/v0/reports/orders_and_fulfillment?token=****&report_subtype=order_cycle_supplier_totals_by_distributor&q[completed_at_gt]=2023-02-22+00:00&q[completed_at_lt]=2023-03-23+00:00&fields_to_show%5B%5D=producer&fields_to_show%5B%5D=product&fields_to_show%5B%5D=variant&fields_to_show%5B%5D=quantity&fields_to_show%5B%5D=total_units&fields_to_show%5B%5D=total_cost&fields_to_show%5B%5D=sku&fields_to_show%5B%5D=producer_charges_sales_tax?&fields_to_show%5B%5D=product_tax_category&display_header_row=true&display_summary_row=true&fields_to_show[]=report_row_type&fields_to_hide%5B%5D&fields_to_show[]=shipping_method

Order Cycle Distributor Totals by Supplier

I could not make this one work 🔴
I'm not sure why, I cannot spot anything wrong in the params...

I keep getting the error NameError: undefined local variable or method line_item_includes' for`

Are you able to try locally @dacook , and reproduce?

https://staging.openfoodnetwork.org.uk/api/v0/reports/orders_and_fulfillment?token=****&report_subtype=order_cycle_supplier_totals_by_supplier&q[completed_at_gt]=2023-02-22+00:00&q[completed_at_lt]=2023-03-23+00:00&display_header_row=true&display_summary_row=true&fields_to_show[]=report_row_type&fields_to_hide%5B%5D

Order Cycle Customer Totals

https://staging.openfoodnetwork.org.uk/api/v0/reports/orders_and_fulfillment?token=****&report_subtype=order_cycle_customer_totals&q[completed_at_gt]=2023-02-22+00:00&q[completed_at_lt]=2023-03-23+00:00&fields_to_show%5B%5D=hub&fields_to_show%5B%5D=product&fields_to_show%5B%5D=variant&fields_to_show%5B%5D=item_price&fields_to_show%5B%5D=billing_street&fields_to_show%5B%5D=date&display_header_row=true&display_summary_row=true&fields_to_show[]=report_row_type&fields_to_hide%5B%5D&fields_to_show[]=shipping

Packing Reports 🟢

Pack By Customer

https://staging.openfoodnetwork.org.uk/api/v0/reports/packing?token=****&q%5Border_completed_at_gt%5D=2023-02-22+00%3A00&q%5Border_completed_at_lt%5D=2023-03-23+00%3A00&q%5Border_distributor_id_in%5D%5B%5D=&report_subtype=customer&report_format=&display_summary_row=true&fields_to_show%5B%5D=hub&fields_to_show%5B%5D=supplier&fields_to_show%5B%5D=customer_code&fields_to_show%5B%5D=first_name&fields_to_show%5B%5D=last_name&fields_to_show%5B%5D=product&fields_to_show%5B%5D=variant&fields_to_show%5B%5D=quantity&fields_to_show%5B%5D=report_row_type&display_header_row=true&display_summary_row=true&fields_to_show[]=temp_controlled

Pack By Supplier
https://staging.openfoodnetwork.org.uk/api/v0/reports/packing?token=****&q%5Border_completed_at_gt%5D=2023-02-22+00%3A00&q%5Border_completed_at_lt%5D=2023-03-23+00%3A00&q%5Border_distributor_id_in%5D%5B%5D=&report_subtype=supplier&report_format=&display_summary_row=true&fields_to_show%5B%5D=hub&fields_to_show%5B%5D=supplier&fields_to_show%5B%5D=customer_code&fields_to_show%5B%5D=first_name&fields_to_show%5B%5D=last_name&fields_to_show%5B%5D=product&fields_to_show%5B%5D=variant&fields_to_show%5B%5D=quantity&fields_to_show%5B%5D=report_row_type&display_header_row=true&display_summary_row=true&fields_to_show[]=temp_controlled

Pack By Product

https://staging.openfoodnetwork.org.uk/api/v0/reports/packing?token=****&q%5Border_completed_at_gt%5D=2023-02-22+00%3A00&q%5Border_completed_at_lt%5D=2023-03-23+00%3A00&q%5Border_distributor_id_in%5D%5B%5D=&report_subtype=product&report_format=&display_summary_row=true&fields_to_show%5B%5D=hub&fields_to_show%5B%5D=supplier&fields_to_show%5B%5D=product&fields_to_show%5B%5D=variant&fields_to_show%5B%5D=customer_code&fields_to_show%5B%5D=first_name&fields_to_show%5B%5D=last_name&fields_to_show%5B%5D=quantity&fields_to_show%5B%5D=report_row_type&display_summary_row=true&fields_to_show[]=temp_controlled&fields_to_show%5B%5D=price&fields_to_show%5B%5D=phone

Manual tests
I've tested all reports manually and found the same behavior as before.

Adding the feedback label, on the red test-case above (Order Cycle Distributor Totals by Supplier).

dacook · 2023-03-22T22:45:08Z

Great thorough testing @filipefurtad0! I looked through that case and I discovered the problem in the url:

report_subtype=order_cycle_supplier_totals_by_supplier

It should be distributor :)

filipefurtad0 · 2023-03-22T23:34:36Z

Yaayy,

That's it!

https://staging.openfoodnetwork.org.uk/api/v0/reports/orders_and_fulfillment?token=****&report_subtype=order_cycle_distributor_totals_by_supplier&q[completed_at_gt]=2023-02-22+00:00&q[completed_at_lt]=2023-03-23+00:00&display_header_row=true&display_summary_row=true&fields_to_show[]=report_row_type&fields_to_hide%5B%5D&fields_to_show%5B%5D=shipping_method&fields_to_show%5B%5D=producer_charges_sales_tax?&fields_to_show%5B%5D=product_tax_category&fields_to_show%5B%5D=variant&fields_to_show%5B%5D=product&fields_to_show%5B%5D=producer&fields_to_show%5B%5D=quantity&fields_to_show%5B%5D=curr_cost_per_unit&fields_to_show%5B%5D=total_shipping_cost&fields_to_show%5B%5D=total_cost

Thanks @dacook. Merging.

dacook force-pushed the report-summaries branch from 172cce4 to 7ba8499 Compare February 21, 2023 03:02

dacook marked this pull request as ready for review February 21, 2023 03:34

abdellani reviewed Feb 21, 2023

View reviewed changes

dacook mentioned this pull request Feb 23, 2023

Allow user to show/hide totals with a checkbox toggle #6850

Closed

dacook force-pushed the report-summaries branch from 7ba8499 to 754fa54 Compare March 3, 2023 00:25

dacook requested a review from abdellani March 3, 2023 00:42

abdellani approved these changes Mar 3, 2023

View reviewed changes

mkllnk approved these changes Mar 6, 2023

View reviewed changes

sigmundpetersen assigned dacook Mar 16, 2023

filipefurtad0 self-assigned this Mar 22, 2023

filipefurtad0 added the pr-staged-uk staging.openfoodnetwork.org.uk label Mar 22, 2023

dacook added 4 commits March 22, 2023 11:47

Allow column formatting and header/footer stripping to be specified i…

db7ac90

…ndependently This will allow us to change the options in the following commit.

Allow headers and footers for JSON format

6f41a41

You wouldn't want it by default, but it can be enabled with a parameter like display_summary_row=true

Add marker for summary rows

c27974c

This can be enabled in the reports API with the hidden parameter `fields_to_show[]=report_row_type` to help with parsing summary row output.

filipefurtad0 force-pushed the report-summaries branch from 754fa54 to c27974c Compare March 22, 2023 11:47

filipefurtad0 added feedback-needed and removed pr-staged-uk staging.openfoodnetwork.org.uk labels Mar 22, 2023

dacook removed the feedback-needed label Mar 22, 2023

filipefurtad0 merged commit 9b749ee into openfoodfoundation:master Mar 22, 2023

lin-d-hop mentioned this pull request Jun 7, 2023

Add summary row data to reports API openfoodfoundation/wishlist#411

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Report summaries in JSON (API) output #10466

Report summaries in JSON (API) output #10466

dacook commented Feb 21, 2023 •

edited

Loading

abdellani Feb 21, 2023

abdellani Feb 21, 2023

dacook Feb 23, 2023

abdellani Feb 23, 2023

dacook Mar 3, 2023

abdellani commented Feb 21, 2023

dacook commented Mar 3, 2023

dacook commented Mar 3, 2023

abdellani left a comment

mkllnk left a comment

mkllnk Mar 6, 2023

filipefurtad0 commented Mar 22, 2023 •

edited

Loading

dacook commented Mar 22, 2023

filipefurtad0 commented Mar 22, 2023

Report summaries in JSON (API) output #10466

Report summaries in JSON (API) output #10466

Conversation

dacook commented Feb 21, 2023 • edited Loading

What? Why?

What should we test?

Test to ensure no change in Admin Reports:

Test to ensure no change in API v0 reports:

Test to enable summary rows

Test to enable summary row markers

Release notes

Dependencies

Documentation updates

abdellani Feb 21, 2023

Choose a reason for hiding this comment

abdellani Feb 21, 2023

Choose a reason for hiding this comment

dacook Feb 23, 2023

Choose a reason for hiding this comment

abdellani Feb 23, 2023

Choose a reason for hiding this comment

dacook Mar 3, 2023

Choose a reason for hiding this comment

abdellani commented Feb 21, 2023

dacook commented Mar 3, 2023

dacook commented Mar 3, 2023

abdellani left a comment

Choose a reason for hiding this comment

mkllnk left a comment

Choose a reason for hiding this comment

mkllnk Mar 6, 2023

Choose a reason for hiding this comment

filipefurtad0 commented Mar 22, 2023 • edited Loading

dacook commented Mar 22, 2023

filipefurtad0 commented Mar 22, 2023

dacook commented Feb 21, 2023 •

edited

Loading

filipefurtad0 commented Mar 22, 2023 •

edited

Loading