Improve documentation and code for 4CAT API calling

[sal-uva]

I just integrated a 4CAT data source search call in a local python script and lo and behold.. it worked great! But at the moment it's a bit of a forgotten child. We should really improve the documentation and (some of) the code on this, because it's an amazing feature.

Things to improve:

• Write documentation
  • How to retrieve an API key
  • What endpoints are available (queue-query, check-query, etc).
  • Have some kind of reference point for the options dictionaries per datasource and processors.
  • Scalar implementation?
• Make sure "-search" doesn't need to be removed for data source IDs.
• Set frontend-confirm to True by default.
• Remove some of the old code that still assumes we're searching for text for every data source.

[stijn-uva]

We have auto-generated documentation at /api/spec/tool/swagger.json (see also /api/) that addresses some of the points above. We could update it and use something like Scalar to generate pretty documentation with it...

[sal-uva]

That would be cool. And references on GitHub and 4cat.nl so people can actually find this option.

[stijn-uva]

New: https://docs.4cat.nl

[dale-wahl]

Bit of related update on /api/get-standalone-processors:

/api/get-standalone-processors did not work. I replaced some legacy code:
if not fourcat_modules.processors[processor].hasattr("datasources") and not hasattr(fourcat_modules.processors[processor], "accepts"):
but changing that to
if not hasattr(fourcat_modules.processors[processor], "datasources") and not hasattr(fourcat_modules.processors[processor], "accepts"):
seemed almost useless as most processors and even datasources evaluated incorrectly.

I first only excluded all datasources, but there are a number of processors that would fail so I made a FakeDataset to mimic what /api/process/<processor> allows/expects. Some of the processors still included likely require specific options for which the endpoint does not allow. Ideally, we could return additional information via /api/get-standalone-processors (or some other endpoint) that lets a user know what input is expected and what options are available/expected, but I do not see a non-trivial way of doing that at this point.

/api/process/<processor> itself could use some updates such as checking validate_query, but I was focused on making the endpoint available.