I'm sorely tempted to take the backend of a Wayland compositor and write a new X server on top of it, with a focus on deprecating everything not actually used by modern X apps that don't have alternatives, which would be a lot.
I bet you could benefit from quite a bit of the Wayland compositor work on modernising the lower levels, and end up with something much simpler than current Xorg without ditching much compatibility.
I think XWayland is basically an X server that supports Wayland. It is based on Xorg though. But if you really think it has a bunch of features nobody uses, you could try to delete those simplify that way.
I don't want an X server that supports Wayland. I want a leaner, simpler X server without Wayland.
EDIT: To clarify, the reason I mentioned Wayland compositors is that it'd be an opportunity to pick a low-level rendering backend that has been written from scratch without the baggage of Xorg.
The "good parts" of X that modern apps actually use are comparatively simple compared to the low level bits - the protocol is trivial-ish, and you can get 90% there by implementing a small-ish subset of the protocol.
XWayland can be run in 'rootless' and 'rootful' modes.
Rootless is what it is typically ran as and allows you to integrate X11 apps into your Wayland desktop environment.
Rootful would allow you to run X11 desktop with X11 Window manager and the whole ten yards.
I don't know how well rootful mode is working because it is rarely used. I imagine it would take some work to make it fully functional. But it something that exists and I would expect that the Xorg/Wayland devs would like to see it fully fleshed out.
But that essentially gets you most of the way there.
I bet you could benefit from quite a bit of the Wayland compositor work on modernising the lower levels, and end up with something much simpler than current Xorg without ditching much compatibility.