disable use of X11 extension MIT-SHM when using remote X server(#8956)

[allelomorph]

Description

After some research into the X11 protocol and its extensions, here's my understanding of the problem:

1. the user's SDL app, using Xlib, acts as an X client and asks the X server if the extension MIT-SHM is available
2. the server, not detecting whether the client is remote or local, answers yes if the extension is implemented
3. having permission to do so, at some point the client attempts to start sharing memory via X11_XShmAttach
4. memory sharing is doomed to fail, as the X client and server are not running on the same hardware, but the X server reports that the request has an invalid opcode (confusing given that it approved use of the extension earlier)

I could try to compile SDL with NO_SHARED_MEMORY defined, but thought it would be nice to retain use of shared memory for when it was appropriate. This PR is my attempt at creating a reasonably cross-platform way to detect at runtime the use of a remote X server, and disable memory sharing via MIT-SHM.

Existing Issue(s)

This is a workaround to #8956, but does not address the original request for cleaner SDL exit on relevant Xlib errors.

[slouken]

It might also be worth investigating if we can catch the error at attach and fall back to the normal path in that case. That might be more robust than just checking an environment variable at startup.

[allelomorph]

@slouken Thanks for the quick response.

It's true that I was focused on a particular use case, and maybe as a first time contributor I wanted to be non-invasive.
Agreed that ultimately it would be better to just consider a remote X server one of several reasons that X11_XShmAttach could fail, and improve SDL handling of that failure.

Currently most of my experience with X11 programming is with xcb, not Xlib, so I would have to do a bit more reading to look into it. If I do come up with a more robust proposal, how best to submit it? As added commits to this PR or as a new one?

[eafton]

@slouken Thanks for the quick response.

It's true that I was focused on a particular use case, and maybe as a first time contributor I wanted to be non-invasive. Agreed that ultimately it would be better to just consider a remote X server one of several reasons that X11_XShmAttach could fail, and improve SDL handling of that failure.

Currently most of my experience with X11 programming is with xcb, not Xlib, so I would have to do a bit more reading to look into it. If I do come up with a more robust proposal, how best to submit it? As added commits to this PR or as a new one?

Sorry for barging in, but a better way of checking would be to set a custom error handler with XSetErrorHandler, checking the result of XShmAttach or if an error occurs, and setting the old error handler again.