I was a little disappointed to see that there wasn't a discussion of how Netflix provides (or doesn't!) a high-fidelity visual experience. I imagine certain artists require fluid-motion and previews without visual artifacting, so I was hoping there was some discussion about how they provide that. I've used a handful of remote desktop systems, and I haven't found anything that's able to provide even a passable video watching experience. The issue I've run into is that the technology designed for low-latency video heavily utilizes compression, which falls apart when looking at text.
I really wish Microsoft would just throw a ton of money at making RDP an absolutely delightful experience. Last time I checked, RDP wasn't able to use your GPU at all unless you had a very specific, no-longer-maintained version of Windows 10 Pro (the Workstation variant).
I work on Jump Desktop (https://jumpdesktop.com) that's being used for remote video editing @ 60fps on up to 3 monitors. It supports very low latency remote desktop and even works in environments that have high latency. For example here's Jump connected from the east coast to a physical server in Paris with 150ms ping times between the two: https://twitter.com/jumpdesktop/status/1356679263806763013?s...
It works on Macs and Windows. It works really well with text as well as full motion video.
Currently we use a modified WebRTC stack with VP8/VP9 to encode the video stream. This is similar to what quite a few game streaming services are doing like Stadia and even Amazon's new Luna(?) service. There are lots of platform specific optimizations up and down the webrtc stack to make sure latency is kept to a minimum while preserving quality. Things like only encoding pixels that change, progressively encoding over multiple frames to sharpen quality, keeping jitter buffers to a minimum.
Recent advancements in hardware video encoding have made things like this significantly easier to achieve. If you're jumping into a device that has the power for video editing tasks, then it most certainly also has great video encoding capabilities.
What’s the actual latency you get between pressing a key and seeing the result on the screen? And how is it if the connection is good and the server is close?
This is a good question. I guess what you're actually interested in is the latency between the physical display updating vs the local display updating, right? I just did a quick test under good circumstances using this video (machines on the same lan): https://www.youtube.com/watch?v=OLxY0HDakRk with the physical and remote display side by side. I noticed an approximately 3 frame lag between the physical display and the local / client display update.
Well what I really meant was the time between pressing a key and seeing a character on the screen. It’s easy to measure (except for deciding when the key is pressed. I go for hitting the key fast and timing from when it stops going down) by using some high speed camera (or medium speed, see https://apps.apple.com/gb/app/is-it-snappy/id1219667593 ) and it’s the latency that actually affects how usable something is for typing. The problem is that it’s only really good for comparing things as lots of devices in the middle add latency, eg keyboards can have 15-50ms of latency[1], typical deferred rendering can give you something like a frame of latency, monitors may add latency or just have slower switching times (it takes longer to switch a pixel to an accurate colour), and some pixels at the bottom of the screen will make it out of the computer 16ms (or whatever a frame is) later than those at the top.
For comparison with my (not particularly optimized) set up going over a fast internet connection, I get something like 150ms between pressing a key and the character starting to show up in emacs (it takes a few ms for the pixels to finish switching). 10 frames feels like a lot to me. My best guess is that without changing anything drastic or eg reducing resolution, I might be able to get that down to 100ms which is still looks like a pretty big number (eg a round trip between London and New York is something like 70ms).
Anyway, thank you for investigating this. A few frames from the network seems pretty good, especially with a reasonably large delta between frames in that video)
I think the gold standard here is what Chromecast does with YouTube— tell the cast device directly what video to show instead of re-encoding and passing the whole thing over the network.
But of course, doing that requires application level support, or at least low-level hooks into the desktop rendering system to extract video streams and treat them differently from everything else. And if the video is being generated on the fly or is an uncompressed local preview, then you may be looking at an intermediate compression step regardless, but at least it could be compression suited for video instead of for text.
Well, that works because there is a pre-rendered video that one can just point the Chromecast to most of the time. If you cast a tab or whole desktop to display anything that's not pre-rendered (which is what workstations would fall into) then Chromecast runs into the same problems as anything else.
> But of course, doing that requires application level support, or at least low-level hooks into the desktop rendering system to extract video streams and treat them differently from everything else.
That's exactly what RDP does on Windows! I don't know the extent to which controls it supports rendering on the client, but the concept is that RDP is aware of things like text, buttons, title bars, etc., and is able to tell the client things like "draw a button at (20, 140)" instead of sending the raw pixel data. With enough engineering effort (and some standardization on the application UI side) I totally think Microsoft could make RDP the protocol of choice for virtual desktops.
It makes total sense; OTOH I'm kind of cracking up here because isn't this what a web browser has become? A portal for viewing a remote application state, that can be serialized into a blob of semi-structured XML and JSON?
It's hard to imagine that retrofitting such a serialization onto the total wild west that is desktop applications could be any more successful or consistent than what the web has already got.
> but the concept is that RDP is aware of things like text, buttons, title bars, etc
The fun bit, at least as I understand it, with the move to more and more Electron based apps, this is somewhat moot since those apps draw their own thing. There are still performance improvements RDP brings over a simple "blit everything" protocol, but they're smaller and smaller.
My biggest complaint with RDP in general is how many Windows apps react poorly to changing resolutions and DPI. I'm practically an expert now in rescuing windows that have decided "I was over at here on the right side of your ultra-wide monitor, so I'm still over here in this no man's land on this tiny RDP resolution."
When I worked at a TV station, all the editors had capture and output hardware in their systems. The output only displayed what was in the video preview window on the screen, and it sync'd with what was being displayed in the editor. Most video editors are likely already set up to do this.
Teradici clients for realtime. For smooth playback they export to a mp4. For real-time smooth playback, they do SDI to a streaming box that provides low latency.
Hey, we launched Renderro - cloud computers for filmmakers, graphic designers and animators in October 2020, and since then we are helping creatives all around the globe. Check us out here: https://renderro.com/
We are supporting up to 4 monitors, each up to 4k res, and the quality we are offering (60 Hz and low latency) allows you to do real-time editing, eg. when using Premiere Pro or Avid MC running on our Cloud Computers. We are also used a lot for 3D design and rendering with Blender.
If you have any questions, don't hesitate to contact us at info@renderro.com
Please note, that with us you don't need any additional tools to connect to our Cloud Computers (no need for Teradici or Parsec) - we've got VDI solutions built in Renderro. We are focusing on delivering Renderro as super easy to use, out-of-the-box solution with clear pricing and superb quality, so I hope that you will enjoy it ;)
> Last time I checked, RDP wasn't able to use your GPU at all unless you had a very specific, no-longer-maintained version of Windows 10 Pro
Not sure where you got that idea. RDP has been able to use the GPU for both encoding and the apps since Windows 7 era and both certainly still work on the current version of Windows 10. The main limitation I've run into is it's still limited to 30 FPS though.
During the Windows 7 era it required Enterprise or Server, during the Windows 10 era it requires Pro, Enterprise, or Server (or that Pro for Workstations version you mention). Remember you have to actually enable it though, it's not configured by default.
Perhaps you're thinking of the deprecation of the RemoteFX guest vGPU for the Hyper-V role?
This has been a quest of mine for a while now, “native” (or close enough that it does not matter) display and input over a data network.
You can get very close today, but you need both very low network latency (<8ms or half a 16ms frame at a minimum) and a hardware + software stack that
can quickly and efficiently dump it’s display framebuffer to the network.
Getting all that to work at the same time is a real trick. Getting it to all work consistently has so far not proven worth the effort vs some other solution like just using another device.
I really wish Microsoft would just throw a ton of money at making RDP an absolutely delightful experience. Last time I checked, RDP wasn't able to use your GPU at all unless you had a very specific, no-longer-maintained version of Windows 10 Pro (the Workstation variant).