json schema support?

[e-kotov]

Hello @jonthegeek ,

the {beekeeper} homepage says that "Add support for APIs using the OAS json format" was implemented in v0.2.

However, I was not able to get past this:

rapid::as_rapid("https://api.apis.guru/v2/specs/sportsdata.io/csgo-v3-scores/1.0/openapi.json")

Error in `method(as_rapid, class_any)`:
! `x` must be comprised of properly formed, supported elements.
Caused by error in `as_rapid()`:
! 
  `"[https://api.apis.guru/v2/specs/sportsdata.io/csgo-v3-scores/1.0/openapi.json](vscode-file://vscode-app/Applications/Positron.app/Contents/Resources/app/out/vs/code/electron-sandbox/workbench/workbench.html#)"`
  must have names "info", "servers", "components", "paths", or "security".
• Any other names are ignored.

or this:

url("https://api.apis.guru/v2/specs/sportsdata.io/csgo-v3-scores/1.0/openapi.json") |> beekeeper::use_beekeeper()

Error in `as_rapid()`:
! Failed to parse paths from OpenAPI spec.

I tried with a few more random API links that I found with anyapi::find_api("data"), but to no avail.

So is the json support supposed to work in the current version or am I doing something wrong?

[e-kotov]

Update: I have already realized that I need to pass the url to rapid::as_rapid wrapped in url(), but that does not change anything.

[jonthegeek]

@e-kotov The good news is that the "json" aspect of this isn't an issue. rapid::as_rapid() works just as well for json specs as for yaml specs. That turned out to be almost a non-issue, since yaml is a superset of json.

The bad news is that rapid currently relies entirely on tibblify::parse_openapi_spec() from tibblify#191 for paths, and there's something in your spec that tibblify can't handle.

You can get the rest of the rapid functionality to work if you remove the paths like this:

spec_list <- jsonlite::read_json(
  "https://api.apis.guru/v2/specs/sportsdata.io/csgo-v3-scores/1.0/openapi.json",
  simplifyVector = TRUE,
  simplifyDataFrame = FALSE,
  simplifyMatrix = FALSE
)
spec_list$paths <- NULL
spec <- rapid::as_rapid(spec_list)

But I suspect you really want the paths to work! If nothing else that should get you a good start (with the wrappers for security and such), so hopefully that will help! You might also try running through the paths 1-by-1 to see if you can get some of them to work. I'd love to hear the specifics if you do, and I'll take a look myself once I get back to work on rapid.

I hope to dive back into paths soon (and replace the tibblify functionality), so I'll keep this open as a test case.

[e-kotov]

Thanks for the reply and explanations. All of this is a bit confusing so far, as the problems seem to be spread over several packages with potential fail points everywhere.

The actual API I was trying to use was https://mapineqfeatures.web.rug.nl/api.json (but the error was the same as I described above).

So far this is what I was able to reproduce with my investigations:

This

xurl <- "https://mapineqfeatures.web.rug.nl/api.json"
xx <- tibblify::parse_openapi_spec(xurl)

Results in

Warning message:
In readLines(file, warn = readLines.warn) :
  incomplete final line found on '[https://mapineqfeatures.web.rug.nl/api.json](vscode-file://vscode-app/Applications/Positron.app/Contents/Resources/app/out/vs/code/electron-sandbox/workbench/workbench.html#)'
Error in `map2()`:
ℹ In index: 2.
ℹ With name: /api.
Caused by error in `map2()`:
ℹ In index: 1.
ℹ With name: get.
Caused by error in `purrr::map()`:
ℹ In index: 1.
ℹ With name: 200.
Caused by error in `openapi_resolve_schema()`:
! ref does not start with #
ℹ This is an internal error that was detected in the tibblify package.
  Please report it at <[https://github.com/mgirlich/tibblify/issues](vscode-file://vscode-app/Applications/Positron.app/Contents/Resources/app/out/vs/code/electron-sandbox/workbench/workbench.html#)>
  with a [reprex](vscode-file://vscode-app/Applications/Positron.app/Contents/Resources/app/out/vs/code/electron-sandbox/workbench/workbench.html#) and the full backtrace.

I have found a problematic path as element 2, without it this code seems to work fine:

j1 <- jsonlite::fromJSON(xurl)
j1$paths[[2]] <- NULL
xx <- tibblify::parse_openapi_spec(j1)

So I guess I did find out what (but not exactly why) was causing the problem with parsing the spec, but the xx is now a data.frame and I don't think it is possible to pass it to rapid::as_rapid.

To be honest, I don't know if any of that helps you at all.

[e-kotov]

Meanwhile, rapiclient::get_api() does not seem to have any issues with any of the json OpenAPI schemas I tried, but the questions is what to do next with the object I am getting from there. I imagine it is not compatible with rapid or beekeeper.

[jonthegeek]

Meanwhile, rapiclient::get_api() does not seem to have any issues with any of the json OpenAPI schemas I tried, but the questions is what to do next with the object I am getting from there. I imagine it is not compatible with rapid or beekeeper.

Very interesting! I'll have to look into them further!

For the rest, {rapid} specifically needs the update that's mostly done in this pull request, thus the specific requirement in the DESCRIPTION. It looks like you have the CRAN version of tibblify (which is a higher version number but doesn't have the parsing enhancements from that pull request). Unfortuantely, though, this spec still doesn't work with that parser (even after removing path 2).

I want to get rid of tibblify entirely, though, so hopefully I'll get this to just work soon. I'll see if {rapiclient} has anything I can use, but, even if not, the fact that they got it working with what appears to be some relatively simple steps is inspiring me to get back to this. I was a feeling a bit overwhelmed the last time I dove in!

Sorry that the package isn't yet meeting your needs! I'll let you know when I've pushed through this and sorted it out!

Edit: I couldn't help playing a bit, so I dug into {rapiclient}. rapiclient::get_api() doesn't do much to actually parse the spec, so it isn't surprising that that works... but this also works:

xurl <- "https://mapineqfeatures.web.rug.nl/api.json"
api <- rapiclient::get_api(xurl)
operations <- rapiclient::get_operations(api)
operations$getAPI()
#> Response [https://mapineqfeatures.web.rug.nl/api]
#>   Date: 2024-12-15 15:15
#>   Status: 200
#>   Content-Type: application/json
#>   Size: 12.5 kB

It looks like they're doing a better job than tibblify at simply ignoring content that's irrelevant. The definitions you're working with really aren't that complicated, so it shouldn't be such a problem!

Edit 2: And... (temporary) fix coming in the next PR 🙃

[e-kotov]

Great to see the PR with a fix!

Just for reference, this json API that I was trying to use is super strange, it does not actually document the useful API endpoints. Details are here if you are interested e-kotov/mapineqr#3 . So I ended up making the package manually by reading the hand-written API docs instead of using this OpenAPI schema.