Schema not found, while being present

[ethanmsl]

Summary:

When running progenitor via command-line (cargo progenitor ...) against a PageDuty REST API document (raw json below) I get an error that audit_method_type/schema is missing, however, when searching the json I see "audit_method_type": ... specified.

I am not fully versed in Swagger/OpenAPI specs so I may be missing an issue with the PagerDuty OpenAPI spec itself.
(And, notably, the 52k line spec seems to produce some, but different, failures on everything that ingests it. Sometimes with circular references noted. Working with progenitor is part of how I'm trying to approach the spec. If this looks to ornery to tackle I'll also happily take any suggestions on how to piece out and sub-verify the spec myself. [and bring back any progenitor specific errors afterward])

=============================

Details:

1. nominal OpenAPI version (from json):

3.0.2

2. cargo-progenitor version (from cargo install --list):

v0.6.0

3. raw, commit-connected link:

PagerDuty/api-schema/rest/openapiv3.json

4. command line used:

RUST_BACKTRACE=1 RUST_LOG=trace cargo progenitor --input pagerduty_rest_api_public.json --output pd_rest_progenated --name pd_rest_rs --version 0.1.0

(note: the input file there reads "pagerduty_rest_api_public.json", but should be identical to the canonical "openapiv3.json")

5. output with backtrace:

[2024-05-06T13:33:41Z INFO  typify_impl::convert] convert_schema Suggested("AuditRecordMethod_truncated_token") {
      "description": "Truncated token containing the last 4 chars of the token's actual value.",
      "examples": [
        "3xyz"
      ],
      "type": "string"
    }
[2024-05-06T13:33:41Z INFO  typify_impl::convert] convert_schema Suggested("AuditRecordMethod_type") {
      "$ref": "#/components/parameters/audit_method_type/schema"
    }
thread 'main' panicked at /Users/esl/.cargo/registry/src/index.crates.io-6f17d22bba15001f/typify-impl-0.0.16/src/convert.rs:1126:32:
$ref #/components/parameters/audit_method_type/schema is missing
stack backtrace:
   0: _rust_begin_unwind
   1: core::panicking::panic_fmt
   2: typify_impl::convert::<impl typify_impl::TypeSpace>::convert_schema_object
   3: typify_impl::convert::<impl typify_impl::TypeSpace>::convert_schema
   4: typify_impl::TypeSpace::id_for_schema
   5: typify_impl::structs::<impl typify_impl::TypeSpace>::struct_property
   6: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
   7: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
   8: core::iter::adapters::try_process
   9: typify_impl::structs::<impl typify_impl::TypeSpace>::struct_members
  10: typify_impl::convert::<impl typify_impl::TypeSpace>::convert_object
  11: typify_impl::convert::<impl typify_impl::TypeSpace>::convert_schema_object
  12: typify_impl::convert::<impl typify_impl::TypeSpace>::convert_schema
  13: typify_impl::TypeSpace::id_for_schema
  14: typify_impl::structs::<impl typify_impl::TypeSpace>::struct_property
  15: <core::iter::adapters::map::Map<I,F> as core::iter::traits::iterator::Iterator>::try_fold
  16: <alloc::vec::Vec<T> as alloc::vec::spec_from_iter::SpecFromIter<T,I>>::from_iter
  17: core::iter::adapters::try_process
  18: typify_impl::structs::<impl typify_impl::TypeSpace>::struct_members
  19: typify_impl::convert::<impl typify_impl::TypeSpace>::convert_object
  20: typify_impl::convert::<impl typify_impl::TypeSpace>::convert_schema_object
  21: typify_impl::convert::<impl typify_impl::TypeSpace>::convert_schema
  22: typify_impl::TypeSpace::convert_ref_type
  23: typify_impl::TypeSpace::add_ref_types
  24: progenitor_impl::Generator::generate_tokens
  25: cargo_progenitor::main

6. what appears to be the missing schema, in the openapiv3.json:

      "audit_method_type": {
        "name": "method_type",
        "in": "query",
        "description": "Method type filter.",
        "schema": {
          "type": "string",
          "description": "Describes the method used to perform the action:\n\n`browser` -- authenticated user session. Session value is not returned in the `truncated_token` field.\n\n`oauth` -- access token obtained via the OAuth flow. Truncated token value is returned in the `truncated_token` field.\n\n`api_token` -- Pagerduty API token. Truncated token value is returned in the `truncated_token` field.\n\n`identity_provider` -- action performed by an Identity provider on behalf of a user. No value is returned in the `truncated_token` field.\n\n`other` -- Method that does not fall in the predefined categories. Truncated token value MAY be returned in the `truncated_token` field.\n",
          "enum": [
            "browser",
            "oauth",
            "api_token",
            "identity_provider",
            "other"
          ]
        }
      },

[ahl]

There was a similar issue a while ago with the PagerDuty OpenAPI doc (#591). I think both here and there, the target of the $ref is to a location we didn't expect. In particular: the current implementation assumes that all schema references will be of the form #/components/schemas/<Type>... and while that turns out to be conventionally true for many schemas, that's by no means required! e.g:

#/components/parameters/audit_method_type/schema
#/components/schemas/OrchestrationGlobal/allOf/1/properties/orchestration_path/properties/catch_all/properties/actions/allOf/1
#/components/responses/Conflict/content/application~1json/schema

I've written up some notes here regarding how we intend to address this and some related deficiencies.

[ethanmsl]

Thanks a lot, Adam.
This does seem to be a spec that's gotten complicated enough to undermine its purpose in some ways.
I'll see what I can do on my end to inline references and break apart the massive REST spec and try to re-run.

[ahl]

I don't mean to suggest that the document is overly complex or whatever... it's just uniquely complex at least in my experience. It should be possible to discover referenced subschemas and then either inline them or pull them out into some common schema from #/components/schemas... but I would like progenitor to handle these documents without requiring additional massaging. While it isn't a priority for us (and I don't see that changing) I do intend to make progress against oxidecomputer/typify#579 as time permits.