Basic tests for some website importers

[MrShark]

Hello,

I have finally had the time to sit down and hack on some unittests for the website importers, they are quite crude but they give some coverage and are something to build on.

They use locally cached versions of the webpages to not slow down testing to much with networking. My thinking is to extend them later with some "slow tests" that can be run to detect if the website have changed enough to make the plugin stop working.

I have used nose to run them, which finds some other tests that's not runnable as unittests so I will probably do some clean-up/renaming to make then usable or skipped.

[thinkle]

Note: we had some tests previously in the gourmet/test/ directory. The
testImporters.py file had a test for a website already created that had
been commented out because allrecipes.com import was broken.

If you want to use that infrastructure, I haven't looked at it in a while,
but it seems pretty simple: you set up an import and then set up attributes
that should get properly imported and test for them (so you can make sure,
for example, that you get the title you want, or whatever).

It might be worth standardizing where we put these things and how we name
them. The way you've done it certainly makes sense, though it makes it
harder to share code (for example, my importer testing code will mostly be
the same no matter what our source for import is, so moving the test code
out into the plugins for each plugin might end up being a pain, though of
course you could import a base test class from the central
tests/importerTests and then call it from test files hosted inside the
plugin folders).

Tom

On Tue, May 7, 2013 at 1:31 PM, jens persson [email protected]:

Hello,

I have finally had the time to sit down and hack on some unittests for the
website importers, they are quite crude but they give some coverage and are
something to build on.

They use locally cached versions of the webpages to not slow down testing
to much with networking. My thinking is to extend them later with some
"slow tests" that can be run to detect if the website have changed enough
to make the plugin stop working.

I have used nose https://nose.readthedocs.org/en/latest/ to run them,
which finds some other tests that's not runnable as unittests so I will

probably do some clean-up/renaming to make then usable or skipped.

You can merge this Pull Request by running

git pull https://github.com/MrShark/gourmet master

Or view, comment on, or merge it at:

#717
Commit Summary

• Add tests for ica.se plugin
• Fix buglet discovered due to tests
• Add tests for foodnetwork plugin
• Buglet found with tests and som whitespace triming in foodnetwork
  plugin

File Changes

• M
  gourmet/plugins/import_export/website_import_plugins/foodnetwork_plugin.pyhttps://github.com/Basic tests for some website importers #717/files#diff-0(4)
• M
  gourmet/plugins/import_export/website_import_plugins/ica_se_plugin.pyhttps://github.com/Basic tests for some website importers #717/files#diff-1(3)
• A
  gourmet/plugins/import_export/website_import_plugins/tests/test_foodnetwork_plugin.htmlhttps://github.com/Basic tests for some website importers #717/files#diff-2(2471)
• A
  gourmet/plugins/import_export/website_import_plugins/tests/test_foodnetwork_plugin.pyhttps://github.com/Basic tests for some website importers #717/files#diff-3(61)
• A
  gourmet/plugins/import_export/website_import_plugins/tests/test_ica_se_plugin.htmlhttps://github.com/Basic tests for some website importers #717/files#diff-4(1009)
• A
  gourmet/plugins/import_export/website_import_plugins/tests/test_ica_se_plugin.pyhttps://github.com/Basic tests for some website importers #717/files#diff-5(67)

Patch Links:

• https://github.com/thinkle/gourmet/pull/717.patch
• https://github.com/thinkle/gourmet/pull/717.diff

[ockham]

Thank you for your contribution, and sorry for taking so long to review!

I've merged your work into the master branch, but moved the tests to the gourmet/test directory and modified them quite a bit, producing a schema_org_parser from which the ica.se importer now derives, as does the updated allrecipes.com importer.

@thinkle I've tried to integrate these new/updated importer tests into gourmet/test/testImporter.py, but even with quite some effort couldn't manage to enable them to use offline copies of the sample web pages they parse which IMO is crucial (eg for testing as part of downstream packaging -- Debian can perform tests at build time, but it's mandatory that they work without an Internet connection). Unfortunately, everything's just so intertwined in the (interactive) importer/plugin/parser part of the code, so I'm sticking to MrShark's individual test files for now.

In the long run (ie when using SQLAlchemy's ORM, that is issue #712), I hope to simplify the structure of plugins and importers so we can have actual unit tests that deserve the name and don't rely on a plethora of other modules.