discussion: cucumber tests #95

emiltin · 2012-01-29T11:05:53Z

i've looked a bit more at the cucumber tests. the idea is to enable test based on small, well-known datasets. here's what tests look like:

    @routing @basic
    Feature: Basic Routing

    Scenario: Smallest possible datasat
        Given the nodes
         | a | b |

        And the ways
         | nodes |
         | ab    |

        When I route I should get
         | from | to | route |
         | a    | b  | ab    |
         | b    | a  | ba    |

    Scenario: Connected ways
        Given the nodes
         | a |   | c |
         |   | b |   |

        And the ways
         | nodes |
         | ab    |
         | bc    |

        When I route I should get
         | from | to | route |
         | a    | c  | abc   |
         | c    | a  | cba   |
         | a    | b  | ab    |
         | b    | a  | ba    |
         | b    | c  | bc    |
         | c    | b  | cb    |

    Scenario: Unconnected ways
        Given the nodes
         | a | b |
         | c | d |

        And the ways
         | nodes |
         | ab    |
         | cd    |

        When I route I should get
         | from | to | route |
         | a    | b  | ab    |
         | b    | a  | ba    |
         | c    | d  | cd    |
         | d    | c  | dc    |
         | a    | c  |       |
         | c    | a  |       |
         | b    | d  |       |
         | d    | c  |       |
         | a    | d  |       |
         | d    | a  |       |

    Scenario: Pick the fastest way type
        Given the nodes
         | a | s |
         | p | b |

        And the ways
         | nodes | highway   |
         | apb   | primary   |
         | asb   | secondary |

        When I route I should get
         | from | to | route |
         | a    | b  | apb   |
         | b    | a  | bpa   |

what goes on is:

first you define a small node map. nodes are labelled with letters.
then you define ways connecting the nodes. you can set any tags you want on ways.
cucumber then writes he corresponding osm and speedprofile files, and processes them.
the routing examples are them run one at a time, checking against expected behaviour.

here's an example run:

~/code/Project-OSRM$ cucumber --tags @basic
@routing @basic
Feature: Basic Routing

  Scenario: Smallest possible datasat # features/basic.feature:4
    Given the nodes                   # features/step_definitions/test_data.rb:124
      | a | b |
    And the ways                      # features/step_definitions/test_data.rb:139
      | nodes |
      | ab    |
    When I route I should get         # features/step_definitions/routing.rb:167
      | from | to | route                                                 |
      | a    | b  | ab                                                    |
      | b    | a  | ba                                                    |
      | a    | b  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | b    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      Tables were not identical (Cucumber::Ast::Table::Different)
      ./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
      features/basic.feature:12:in `When I route I should get'

  Scenario: Connected ways    # features/basic.feature:17
    Given the nodes           # features/step_definitions/test_data.rb:124
      | a |   | c |
      |   | b |   |
    And the ways              # features/step_definitions/test_data.rb:139
      | nodes |
      | ab    |
      | bc    |
    When I route I should get # features/step_definitions/routing.rb:167
      | from | to | route                                                 |
      | a    | c  | abc                                                   |
      | c    | a  | cba                                                   |
      | a    | c  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | a    | b  | ab                                                    |
      | c    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | b    | a  | ba                                                    |
      | a    | b  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | b    | c  | bc                                                    |
      | b    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | c    | b  | cb                                                    |
      | b    | c  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | c    | b  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      Tables were not identical (Cucumber::Ast::Table::Different)
      ./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
      features/basic.feature:27:in `When I route I should get'

  Scenario: Unconnected ways  # features/basic.feature:36
    Given the nodes           # features/step_definitions/test_data.rb:124
      | a | b |
      | c | d |
    And the ways              # features/step_definitions/test_data.rb:139
      | nodes |
      | ab    |
      | cd    |
    When I route I should get # features/step_definitions/routing.rb:167
      | from | to | route                                                 |
      | a    | b  | ab                                                    |
      | b    | a  | ba                                                    |
      | a    | b  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | c    | d  | cd                                                    |
      | b    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | d    | c  | dc                                                    |
      | c    | d  |                                                       |
      | a    | c  | (i) ""                                                |
      | d    | c  | (i) nil                                               |
      | c    | a  | (i) ""                                                |
      | a    | c  | (i) nil                                               |
      | b    | d  |                                                       |
      | c    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      | d    | c  | (i) ""                                                |
      | b    | d  | (i) nil                                               |
      | a    | d  | (i) ""                                                |
      | d    | c  | (i) nil                                               |
      | d    | a  | (i) ""                                                |
      | a    | d  | (i) nil                                               |
      | d    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]] |
      Tables were not identical (Cucumber::Ast::Table::Different)
      ./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
      features/basic.feature:46:in `When I route I should get'

  Scenario: Pick the fastest way type # features/basic.feature:59
    Given the nodes                   # features/step_definitions/test_data.rb:124
      | a | s |
      | p | b |
    And the ways                      # features/step_definitions/test_data.rb:139
      | nodes | highway   |
      | apb   | primary   |
      | asb   | secondary |
    When I route I should get         # features/step_definitions/routing.rb:167
      | from | to | route                                                     |
      | a    | b  | apb                                                       |
      | b    | a  | bpa                                                       |
      | a    | b  | Empty route: [["Head", "", 111, 0, 266, "0", "NE", 22.5]] |
      | b    | a  | Empty route: [["Head", "", 0, 0, 0, "0", "NE", 22.5]]     |
      Tables were not identical (Cucumber::Ast::Table::Different)
      ./features/step_definitions/routing.rb:180:in `/^I route I should get$/'
      features/basic.feature:69:in `When I route I should get'

Failing Scenarios:
cucumber features/basic.feature:4 # Scenario: Smallest possible datasat
cucumber features/basic.feature:17 # Scenario: Connected ways
cucumber features/basic.feature:36 # Scenario: Unconnected ways
cucumber features/basic.feature:59 # Scenario: Pick the fastest way type

4 scenarios (4 failed)
12 steps (4 failed, 8 passed)
1m56.321s

all tests are currently failing due to #88 (missing way names in routing instructions).

launching and quitting the binaries currently takes a while (perhaps due to large ram/disk allocations?) which makes the tests rather slow.

perhaps an option could be added to allocate small amounts when testing?

another idea would be to merge all three binaries into one, which could be kept running. it would then be instructed (perhaps by touching a file) to reprocess data. this would have the added benefit that request during reprocessing could be handled gracefully by sending back an appropriate status code.

The text was updated successfully, but these errors were encountered:

emiltin · 2012-01-29T11:09:51Z

test for turn restrictions:

@routing @restrictions
Feature: Turn restrictions
    OSRM should handle turn restrictions as defined by http://wiki.openstreetmap.org/wiki/Relation:restriction

    Scenario: No left turn at T-junction
        Given the nodes
         | a | j | b |
         |   | s |   |

        And the ways
         | nodes |
         | aj    |
         | jb    |
         | sj    |

        And the relations
         | from | to | via | restriction  |
         | sj   | ja | j   | no_left_turn |

        When I route I should get
         | from | to | route |
         | a    | b  | ajb   |
         | a    | s  | ajs   |
         | b    | a  | bja   |
         | b    | s  | bjs   |
         | s    | a  |       |
         | s    | b  | sjb   |

emiltin · 2012-01-29T11:46:48Z

test for oneways

@routing @oneways
Feature: Oneway streets
    Handle oneways streets, as defined at http://wiki.openstreetmap.org/wiki/OSM_tags_for_routing

    Scenario: Implied oneways
        Then routability should be
         | highway       | junction   | forw | backw |
         | motorway      |            | x    | x     |
         | motorway_link |            | x    |       |
         | trunk         |            | x    | x     |
         | trunk_link    |            | x    |       |
         | primary       | roundabout | x    |       |

    Scenario: Overriding implied oneways
        Then routability should be
         | highway       | junction   | oneway | forw | backw |
         | motorway_link |            | no     | x    | x     |
         | trunk_link    |            | no     | x    | x     |
         | primary       | roundabout | no     | x    | x     |
         | motorway_link |            | -1     |      | x     |
         | trunk_link    |            | -1     |      | x     |
         | primary       | roundabout | -1     |      | x     |

    Scenario: Handle various oneway tag values
        Then routability should be
         | highway       | oneway   | forw | backw |
         | primary       |          | x    | x     |
         | primary       | nonsense | x    | x     |
         | primary       | no       | x    | x     |
         | primary       | false    | x    | x     |
         | primary       | 0        | x    | x     |
         | primary       | yes      | x    |       |
         | primary       | true     | x    |       |
         | primary       | 1        | x    |       |
         | primary       | -1       |      | x     |

    Scenario: Disabling oneways in speedprofile
        Given the speedprofile
         | obeyOneways | no |

    Then routability should be
         | highway       | junction   | oneway | forw | backw |
         | primary       |            | yes    | x    | x     |
         | primary       |            | true   | x    | x     |
         | primary       |            | 1      | x    | x     |
         | primary       |            | -1     | x    | x     |
         | motorway_link |            |        | x    | x     |
         | trunk_link    |            |        | x    | x     |
         | primary       | roundabout |        | x    | x     |

    Scenario: Oneways and bicycles 
        Given the speedprofile
         | accessTag   | bicycle |

        Then routability should be
         | highway       | junction   | oneway | oneway:bicycle | forw | backw |
         | primary       |            |        | yes            | x    |       |
         | primary       |            | yes    | yes            | x    |       |
         | primary       |            | no     | yes            | x    |       |
         | primary       |            | -1     | yes            | x    |       |
         | motorway      |            |        | yes            | x    |       |
         | motorway_link |            |        | yes            | x    |       |
         | primary       | roundabout |        | yes            | x    |       |
         | primary       |            |        | no             | x    | x     |
         | primary       |            | yes    | no             | x    | x     |
         | primary       |            | no     | no             | x    | x     |
         | primary       |            | -1     | no             | x    | x     |
         | motorway      |            |        | no             | x    | x     |
         | motorway_link |            |        | no             | x    | x     |
         | primary       | roundabout |        | no             | x    | x     |
         | primary       |            |        | -1             |      | x     |
         | primary       |            | yes    | -1             |      | x     |
         | primary       |            | no     | -1             |      | x     |
         | primary       |            | -1     | -1             |      | x     |
         | motorway      |            |        | -1             |      | x     |
         | motorway_link |            |        | -1             |      | x     |
         | primary       | roundabout |        | -1             |      | x     |

    Scenario: Cars should not be affected by bicycle tags
        Given the speedprofile
         | accessTag   | motorcar |

        Then routability should be
         | highway | junction   | oneway | oneway:bicycle | forw | backw |
         | primary |            | yes    | yes            | x    |       |
         | primary |            | yes    | no             | x    |       |
         | primary |            | yes    | -1             | x    |       |
         | primary |            | no     | yes            | x    | x     |
         | primary |            | no     | no             | x    | x     |
         | primary |            | no     | -1             | x    | x     |
         | primary |            | -1     | yes            |      | x     |
         | primary |            | -1     | no             |      | x     |
         | primary |            | -1     | -1             |      | x     |
         | primary | roundabout |        | yes            | x    |       |
         | primary | roundabout |        | no             | x    |       |
         | primary | roundabout |        | -1             | x    |       |

emiltin · 2012-01-29T11:53:45Z

it would be very useful for testing with an option to return the ID of all the intersection nodes passed by the route.

DennisOSRM · 2012-01-31T13:25:50Z

OSM IDs are discarded. This is a design choice because otherwise it bloats the memory consumption beyond any reasonable barrier.

emiltin · 2012-01-31T13:50:35Z

right. makes sense. i guess route instructions should identify the route without ambiguity

DennisOSRM · 2012-02-01T09:21:01Z

Emil, would you be interested in documenting cucumber testing on the wiki including the necessary setup and also a self-contained pull request?

DennisOSRM · 2012-02-01T09:53:43Z

To run these tests automatically (perhaps nightly) it would be great to have cucumber generate HTML output that can be automatically pushed to the projects web site. Also, sending mails when the nightly test breaks would be awesome.

emiltin · 2012-02-01T17:20:02Z

i could definately document this on the wiki. i'm glad you think this could be useful.
but i will need to work a bit more on it before it's ready.

for sure cucumber can work with a continuous integration setup. it has various hooks etc.

emiltin · 2012-02-01T17:27:15Z

i was hoping to specify expected routes by naming nodes, but since osrm doesn't include this directly i need to think about wether nodes can be implied from the route instructions, or expected routes should instead be specified by a series of ways.

DennisOSRM · 2012-02-01T17:28:26Z

I'd suggest to use a series of ways.

emiltin · 2012-02-01T17:33:08Z

yes that seems easiest. just wondering if it's always unambiguous? i guess it is, otherwise the routing instructions would be ambiguous.

DennisOSRM · 2012-02-01T17:36:34Z

One may artificially construct such a network, but I don't think this is an issue on a well-defined test instance.

f379578a3 Release 1.7.1 040947ba2 CMake: Add language C to project() c60e87879 Update include catch.hpp to 2.13.8 6599d4828 Update change log ebd2e4f40 Allow setting C++ version to compile with in CMake config bbb2a95d0 Github actions: Only install x64 version of vcpkgs in windows build 546edf929 Avoid narrowing conversion by being explicit 91adaecd6 Fix various issues reported by clang-tidy and disable some checks 68f30a1f7 Add Github actions CI build for Linux/macOS/Windows c13886b78 Update some links bd429c52f Include tools subdirectory *after* testing is enabled 3133dc52f Use std::memcpy instead of just memcpy 7ee29422a Merge pull request #106 from daniel-j-h/fix-byteswap-ub aba0800aa CMake config: clang-tidy target needs writer_tests which need protobuf 74516e8df Modernize CMake config a little bit b4486ca7a Disable some clang-tidy warnings 19f4b75f6 Fix appveyor build 58b1a19a4 Modernize Doxygen config file 046c07d0e Update included catch2 framework to current version v2.13.7 49acea746 fix some doxygen warnings by removing obsolete config entries 0c5426df3 fix cmake dep warning: 'Compatibility with CMake < 2.8.12 will be removed from a future version of CMake' dadf7bd51 Fixes float and double byteswap undefined behavior 85db94025 Merge pull request #105 from ffontaine/master d3a35791d Merge pull request #104 from joto/master 03daae49c CMakeLists.txt: respect BUILD_TESTING 67133e362 Add missing includes 9f85f3a5e Update README.md 010ffcf69 Release 1.7.0 6ad492994 Fixed docs adde4dedf Travis: Add non-Intel architectures 37c3d0e1d Add typedefs and functions to make buffer adaptor work as container 83563acdb Remove delegating constructor because clang-tidy doesn't like it b4afc06db Use #include "" for library-internal includes d1929788e Add missing includes fae5247f0 Update change log 2abb1b5cd Travis: Test with std::string_view, not std::experimental::string_view 697bd812d Use forwarding constructor in fixed_size_buffer_adaptor c8fd2e819 Move fixed_size_buffer_adaptor into buffer_fixed.hpp abb856ecc Remove semicolon where it doesn't belong 6243855bc Change the way the customization for special buffer classes work d6a8ed098 Remove useless post-increment 184046cb0 Remove need for push_back() on custom buffer types 0a974e067 Remove templated buffer adaptor wrappers ed6ba5097 Add buffer implementation based on std::vector<char> cec309c3c Use more descriptive names for buffer test types a7b99da6f Use TEMPLATE_TEST_CASE to test different buffer implementations a0abc493c Use explicit for constructor 72850abc9 Remove broken doxygen link 1e347c620 Add more convenient fixed_size_buffer_adaptor constructor fb575e0ea Make members private 50e953b71 Make older compilers happy c850ef150 Extend tests of static buffer use f6d8394c0 Rename fixed_size_buffer to fixed_size_buffer_adaptor 3b18162e3 Make the buffer backend used by the pbf writer configurable. 6fd19c58d "Modernize" travis config 981aba084 Use explicit cast to avoid undefined behaviour 550974d5d Travis: Do not test GCC 4.7 any more 866e024fc Revert "Workaround in catch code so it compiles with older compilers" 65dfad056 Disable a clang-tidy test triggered by Catch. 02bf73df5 Workaround in catch code so it compiles with older compilers f98792a15 Travis: Do not update homebrew for faster builds 2d87da7ec Switch tests to Catch2 5dc45ac3b Avoid signed/unsigned comparison 3a93f19ba Add missing includes b49c077ac Disable clang-tidy for files where we don't have a compile command 34396fc7d Travis: Fix gcc8 build c3060101c Handle clang-tidy warnings 64ef96ff0 Revert "Initialize test messages" a0828d538 Travis: Also build with GCC 8 171c5c446 Update travis xcode versions f5a223aa7 Use "auto*" instead of just "auto" for pointer types e4fa23616 Initialize test messages e3a59454a Simpler code and avoid shadowing of external function 830f049b4 Use STL algorithms insted of raw loops faa7e6e8a Disable config settings not used in newer Doxygen versions 3b2e11438 Remove unnecessary enum name 7487f8109 Release 1.6.8 6dcaf8fde Travis config: Use "official" way to pull homebrew package. c61eb29c3 Revert "Disable warnings from clang-tidy about a missing file." 28d05a0a8 Disable warnings from clang-tidy about a missing file. 329920a3c Pesky aliases of clang-tidy warnings strike again. 79fd87922 User plain assert() instead of our own so compare() can be noexcept. 473e6ec13 Update change log. 393e279b7 Make pbf_writer destructor noexcept. 48a38b3f2 Disable clang-tidy misc-non-private-member-variables-in-classes. e9c148c8a Use no-argument version of main(). 29ba04123 Disable clang tidy checks for C arrays. 2fcfb56e2 More places to use std::array instead of a C style array. 7321761a3 Disable a clang-tidy warning. 4d9d8fff4 Make data_view::compare() noexcept. 3325364cf User uppercase integer literal suffix. df0a23c5e Use std::array instead of C arrays in some places. 8247ed76b Make clang-tidy include order check happy. f1b504e16 Update travis config to user newer compilers and operating systems. ccf692d47 Disable some clang-tidy warnings. 095abd259 CMake config: Also look for newer clang-tidy versions. 2c1f6f9c8 Use uppercase integer literal suffixes. fadd024d4 Release 1.6.7 8c6acbff7 Fix signed-unsigned comparison. b36774ccb Release 1.6.6 5a92b744f Remove useless asserts, simplify condition. 06bafb56c Fix several possible UBs. b7b290b1a Release 1.6.5 51753d514 Merge pull request #95 from tomhughes/subscript b90faaf03 Avoid out of bounds array subscript 7d418492e Merge pull request #94 from nigels-com/proto2 015f9cc5e Specify proto2 syntax to appease protoc 23d48fd2a Use universal initialization syntax in constructors. 0f610fad5 Update travis config: Use xenial for most builds. d71da0b04 Update appveyor config: Simpler builds, current MSVC, 32bit build 3ef46ba78 Release 1.6.4 3a1ef0138 Tighten some tests. 18eebb8c3 Remove unused code from tests. 29ef3e4e7 More casts to remove undefined behaviour. 6108e6480 No more bitwise operations on signed integers in zigzag encoder/decoder. 6e0d34985 Remove bitwise operations on signed integers in varint decoder. 4af65f262 Update change log. 2f82182fe Add some tips to test/create_pbf_test_data.sh. c55f4ed55 Fix some doxygen warnings. afa362a03 Add static_asserts to check movability of some classes. efeb45e0c Disable readability-implicit-bool-conversion clang-tidy warning. 78febda5b Explicit conversion and tests for new pbf_reader::data() function. 0d5492c9c Revert "Explicit conversion and tests for new pbf_reader::data() function." 43cf8fa5a Fix travis config. bd2ae4682 Explicit conversion and tests for new pbf_reader::data() function. 28cd406bd Update travis with newer compiler versions. 0555e6a1f Add function to get the not yet read data from a pbf_reader. bf4284bee Disable docker builds on travis. They are being phased out by travis. 5ffe45b71 New add_packed_fixed template function. e54cd858d Add helper function that computes the length a varint would have. 72d7e143a More consistent implementation of operators. 3a41880c2 Do not download protobuf library, it isn't found by cmake anyway. 3c662ce3c Remove comment that doesn't apply (any more). 45da6dd4d Update zigzag tests. 4ad573dbf Extra cast so we do the xor with unsigned ints. 509aec5ab Update appveyor build to current Visual Studio compiler. 67b24e1a3 Remove unnecessary workaround in Appveyor config. c559af682 Remove xcode6.4 build soon to be removed from travis. 0662dcecc Release 1.6.3 da5bfc019 Move byteswap_inplace functions from detail into protozero namespace. a44efc34e Travis: Ignore install problems on OSX. 5775b2b23 Travis update to newer OSX image. 032aa037c Special case the distance between default initialized iterators. 0ca02161e Make dereferencing operator of fixed_iterator noexcept. a0095f603 Test code must call functions that it wants to test. 6791b0bc3 Add unit tests. 191eb4004 Add some paranoia asserts. 99ca512f5 Use TEST_CASEs instead of SECTIOs in some tests. 040e2bc14 Add some asserts and tests. git-subtree-dir: third_party/protozero git-subtree-split: f379578a3f7c8162aac0ac31c2696de09a5b5f93

DennisOSRM pushed a commit that referenced this issue Feb 14, 2012

Adds the cucumber test framework referenced in issues #26, #95, #114

b210b22

DennisOSRM closed this as completed Feb 14, 2012

DennisOSRM pushed a commit that referenced this issue Feb 15, 2012

Adds the cucumber test framework referenced in issues #26, #95, #114

eeea5b0

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

discussion: cucumber tests #95

discussion: cucumber tests #95

emiltin commented Jan 29, 2012

emiltin commented Jan 29, 2012

emiltin commented Jan 29, 2012

emiltin commented Jan 29, 2012

DennisOSRM commented Jan 31, 2012

emiltin commented Jan 31, 2012

DennisOSRM commented Feb 1, 2012

DennisOSRM commented Feb 1, 2012

emiltin commented Feb 1, 2012

emiltin commented Feb 1, 2012

DennisOSRM commented Feb 1, 2012

emiltin commented Feb 1, 2012

DennisOSRM commented Feb 1, 2012

discussion: cucumber tests #95

discussion: cucumber tests #95

Comments

emiltin commented Jan 29, 2012

emiltin commented Jan 29, 2012

emiltin commented Jan 29, 2012

emiltin commented Jan 29, 2012

DennisOSRM commented Jan 31, 2012

emiltin commented Jan 31, 2012

DennisOSRM commented Feb 1, 2012

DennisOSRM commented Feb 1, 2012

emiltin commented Feb 1, 2012

emiltin commented Feb 1, 2012

DennisOSRM commented Feb 1, 2012

emiltin commented Feb 1, 2012

DennisOSRM commented Feb 1, 2012