Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

apis stuff #9

Open
bf4 opened this issue Oct 22, 2015 · 5 comments
Open

apis stuff #9

bf4 opened this issue Oct 22, 2015 · 5 comments

Comments

@bf4
Copy link
Owner

bf4 commented Oct 22, 2015

http://parley.rubyrogues.com/t/how-do-you-test-your-json-apis/2759/6

Relevant discussion from RR 176 followed by links

PETE: <snip>So, one of the techniques that we use a lot around test automation is to create fake versions of our dependent services.
<snip> if you’re using fake versions of those services then you can just stand up this fake version and configure it to simulate the scenario that you want to configure. So, it’s actually really beneficial as well in terms of being able to test edge cases and scenarios that shouldn’t happen but eventually will happen.
SARON: So, the idea of faking it makes a lot of sense to me. But at some point, you do have to test it against the actual service, right?
PETE: Yeah.
SARON: To make sure that it works the way you think it works. And I think that’s called contract tests. Can you talk a little bit about that?
PETE: Yeah. So, contract test is this idea, the challenge as you’re saying where if you’re testing against a fake version of the services, you’re essentially testing against your understanding of how this other service works. So, because you’re not actually talking to the blue team’s deal service, if they change the way their deal service works or if you just misunderstand how it works in the first place and they start returning date times as seconds from the epoch rather than ISO 8601 or something like that, they change the format of their data structures or they add a field or they remove a field, you’re not going to know from your fake services because your fake services are fake. They’re not the real deal.
So, one of the antidotes to that is to write contract tests, and these are also sometimes described as consumer-driven contracts. And the idea there is the consumer of the service writes tests against that service that basically encapsulate what they expect that service to do. So, they’re like a contract between the two. They become a contract between the two teams essentially, where the consumer of the service gives these tests to the provider of the service and says, “If you can get all these tests to pass, then as far as I’m concerned you are doing everything I need from you as a service.”
<snip> So, this is a way to encapsulate a communication mechanism between the two teams where you write the tests and give them to the team and then the team knows whether the tests are working or not.
AVDI: My experience with building fakes like this has been mixed, because I have memories of spending a lot of time just maintaining the fakes.
PETE: Yeah.
AVDI: Keeping them synced up with their real counterparts.
PETE: Yeah.
AVDI: And just sinking a lot of time into that. Have you come up with any strategies for minimizing the time spent on that?
PETE: I definitely have felt that pain as well. There’s a more sophisticated variant of the consumer contracts thing where essentially you insert a shim in between your service, or sorry, your code, let’s say if you’re a Rails app, you’re the client and the service, it’s like an HTTP proxy of some kind. And what that will do is it will record the requests coming out and be able to play them back at a later point. You can then use that same shim to verify your contract tests and use the output of those contract tests as the input for your fakes. So, this is definitely pretty high-grade sophistication.
I’ve actually never done this because it’s a lot of moving parts. But it’s become a bit simpler in the last couple of years. There were a couple of gems released by a couple of teams. So, there’s a gem called pact, P-A-C-T. And there’s another gem called pacto. And both of those are tools that you can use to do this kind of consumer contracts that also provide fakes. So, they’re two-faced if you will. As far as the consumer is concerned, they’re providing a fake service and then as far as the provider is concerned, they’re providing a set of contract tests against that provider.
AVDI: So, effectively the fake is able to test its own assumptions about the real thing.

Links:

  • https://github.com/realestate-com-au/pact Enables consumer driven contract testing, providing a mock service and DSL for the consumer project, and interaction playback and verification for the service provider project.
  • https://github.com/thoughtworks/pacto Pacto settles disputes between JSON providers and consumers http://thoughtworks.github.io/pacto
    • Pacto is a framework for Integration Contract Testing, and helps guide service evolution patterns like Consumer-Driven Contracts or Documentation-Driven Contracts. Pacto considers two major terms in order decide if there has been a breach of contract: the request clause and the response clause.
    • The consumer may also enter into an agreement with test doubles like WebMock, vcr or mountebank.
    • Consumers: Pacto helps you test consumers by both validating and stubbing services. This helps you decouple tests to reduce test times and non-deterministic test failures without losing confidence in your test suite. You can also design compatibility tests for similar consumers.
    • Providers: You can use Consumer-Driven Contracts to ensure you're providers are well tested without losing agility. Pacto will test the provider implementation and make sure it complies with the same contract used to stub the service for consumers.

http://parley.rubyrogues.com/t/json-apis-schema-validation-document-driven-consumer-driven-media-type-oh-my/3177

Continuing the discussion from How do you test your JSON APIs?5:

Warning: this is a bit of a braindump only lightly edited

I've been trying all sorts of things and have yet to find a sane solution.

Here's the problem:

  • I want to build an app that services an api.
  • I want to be able to make it easy to test by humans, e.g. via swagger6 (does anyone use jsonpath6 to test json resources?)
  • I want to be able to easily generate a client for it e.g. the heroku api toolchain 4
  • I want to be able to validate that the client and service agree on what the api is, and ideally change in sync
  • I don't want to define the api in the code, as that makes it harder to work with. e.g. if I define the api in a controller, how do I create the resource serializers and the client? I also just don't want to be stuck with a library that makes it harder to change

There are tons of tools out there that seem to do parts of this, but ultimately they either
- derive the contracts from the service, which makes testing the service a little weird
- expect you to know how to write define your json-schema1 (and know what that is) in apiblueprint2, swagger, raml1, whatever format from scratch

As https://brandur.org/elegant-apis3 writes

To recap, we’ve used JSON Schema to define the following:

  • Individual API resources (app and domain).
  • An API “super schema” that contains all resources in a single document.
  • Hyper-schema links that describe actions on those resources.
  • Schemas that validate incoming requests on each link.
  • Schemas that describe the JSON returned by each link.
  • A meta-schema that validates the conventions of our API’s schema.

Although the API itself still needs to be implemented, by combining this schema with the various packages from the HTTP toolchain, we get some nice features for free:

  • Generate API documentation with Prmd2.
  • Generate a Ruby client with Heroics4.
  • Generate a Go client with Schematic, like the one used in Heroku’s new CLI, hk1.
  • Boot a working stub with Committee2 that will validate incoming requests.
  • Insert a request validation middleware with Committee that will validate incoming request data according to schema before it reaches our stack.
  • Use Committee’s test helpers to verify that the responses from our stack conform to the schema.

The most straightforward solution to this seems to be 'documentation-driven contracts' (or feature specs), where the contract
- is a valid schema e.g. https://github.com/ruby-json-schema/json-schema https://github.com/patchboard/patchboard2
- is used to validate the client
- is used to validate the service
- is used to validate requests e.g. https://github.com/interagent/committee2

but this requires I understand how to write a json-schema and probably a hypermedia type, and how to propogate changes in a non-breaking way (versioned microservices!)

and my head just keeps exploding and I never get it the way I want it. But I know people are doing this...

A simple stupid way is the document the request/responses I want and then use https://github.com/maxlinc/json-schema-generator1 to generate a json schema from it, and then tweak that... or https://github.com/apiaryio/curl-trace-parser will let me turn a curl request to my service into an apiblueprint doc, which.. idk

I really like using swagger, but a lot of tools still use 1.2 which isn't "json-schema", and when I've tried converting a 1.2 yml to a 2.0 with https://github.com/apigee-127/swagger-tools I've had validation failures up the wazoo even though the public/api page works fine...

best I can figure right now is to take some examples and just cannibalize them.

And the appendix is then which tools do I use that don't lock me in. If I use grape and roar with hal or rails with active_model_resources or with https://github.com/RestPack/restpack_serializer and then I decide I want all resource keys to always be plural and return a collection, and there's always a meta key that we use for pagination (not in the header for whatever reason), the tooling can really slow me down... and will jsonapi1 save the day? not yet

thoughts?
@bf4
Copy link
Owner Author

bf4 commented Oct 26, 2015

http://www.vinaysahni.com/best-practices-for-a-pragmatic-restful-api

@bf4
Copy link
Owner Author

bf4 commented Oct 26, 2015

http://www.sitepoint.com/document-your-json-api-schema-with-prmd/

@bf4
Copy link
Owner Author

bf4 commented Dec 11, 2015

nice discussion in thoughtbot/guides#365

@bf4
Copy link
Owner Author

bf4 commented Dec 11, 2015

Adding https://github.com/Leventhan/api-development-tools

Related to the balanced api cucumber approach

@bf4
Copy link
Owner Author

bf4 commented May 17, 2017

Hypermedia and GraphQL stuff

that kinda thing so we're working on this we also had built a global I'm a synchronous validation middleware so that every single chasing request and I'll the system or Json packet and how the system will be valid validated with Json schema I think this is actually a really important thing because someone the bugs that we have found they're probably inherent in this sort of micro services crazy heterogeneous architecture are because love just validating data
Chad Fowler: "From Homogeneous Monolith to Radically Heterogeneous Microservices Architecture"

This is the third in a series of seven posts on service-oriented architecture derived from a workshop conducted by Cloves Carneiro and Tim Schmelmer …

JSON Validation / JSON Schema

@bf4 bf4 mentioned this issue May 17, 2017
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bf4