[Feature] Apple Screen Sharing High Performance

[Chapoly1305]

Background

High Performance mode extends remote desktop with a display pipeline that behaves more like a native, GPU-driven output. It targets modern workflows such as high-resolution content, HDR, real-time interaction, and multi-display setups. It does so by adding a stream-oriented path with virtual displays, hardware-accelerated encoding, dynamic resolution, and full audio and video, which together support consistent low-latency interaction and accurate rendering. It complements the existing screen-sharing model rather than replacing it, giving sessions a higher-performance option when the workload calls for it.

Design

Under heavy research, it seems no publicly accessible official specification, nor has anyone knowingly implemented this (at the time this ticket was originally created). You can check RFC-style Manual for more hard-to-understand details. Here I present some abstract information for easy understanding.

Authentication

Apple's screen sharing supports a family of authentication methods, built on standard mechanisms such as Diffie-Hellman, RSA, SRP, and Kerberos. This lets it accept real macOS user accounts, Apple Remote Desktop credentials, and directory or single-sign-on logins. During the initial handshake the client and server agree on one method and complete it before the session continues.

Two points are useful for an implementer:

• The set is pluggable. The server advertises the methods it supports and the client selects one. (Our proof-of-concept uses the RSA/SRP method; the per-method details are in the RFC.)
• After authentication, the control channel is encrypted. Everything that follows (screen requests, input, stream setup) is carried inside that encrypted session, so support involves the post-login transport as well as the login step itself.

Full Quality Mode (VNC RFB Style)

This mode follows the standard RFB model and reuses the existing VNC machinery. The server tracks changes to the screen and sends the changed regions back as compressed, pixel-accurate images; the client requests updates and draws them, the same way an ordinary VNC session works.

Characteristics:

• It is lossless or near-lossless, so text stays sharp, images are exact, and there are no video-compression artifacts.
• It needs no special hardware on either side, so any machine, including older Intel Macs, can both send and receive it. This makes it a broadly compatible baseline, and it is the path the current proof-of-concept uses for HiDPI (sharp Retina output) and dynamic resolution.

Because each update carries a compressed image of the changed region, the amount of data scales with how much of the screen changes between frames. For mostly static content such as documents, code, or design work, this is efficient and exact. When a large area changes continuously, such as video playback, scrolling, or dragging windows, the data volume and update rate rise, which increases bandwidth use and latency. The framerate and latency is the most significant downside of this mode. That continuous-motion case is what the next mode is designed for.

Adaptive Quality Mode (Streaming Style)

This mode encodes the screen and its audio as a real-time video stream, similar to a video call or a game stream, and sends it over a separate low-latency channel. The quality and bitrate adapt to the network, and the client decodes the picture on the GPU.

This goes beyond the standard RFB model: it is a streaming pipeline added alongside it, set up through a short negotiation after the session is established. It is well suited to continuous motion, video, and scrolling, and keeps latency low when much of the screen is changing.

In short, this mode primary utilize HEVC and supports 60 fps 4:4:4, pretty good performance and competitive against the native Windows RDP. On HiDPI mode, it can utilize up-to 300Mbps for bitrate; on not HiDPI mode, up-to 60 Mbps.

There have good and bad perspectives of using this mode.

Good. Apple did not design a special circuit for this feature, meaning it actually runs on other OS than macOS + Apple Silicon.

Bad.

1. The most significant downside for this mode is compatibility. 4:4:4 is not commonly supported in multiple perspectives. AMD Consumer GPUs, e.g., 6800XT, does not have the capability to hardware decode the stream. As the result, user can only fallback to software decoding.
2. Some user might use Intel CPU which may supports 4:4:4 decoding by utilizing QSV. The video quality is fine most of the time but there has been observed some random distortions (incorrect brightness, corruption), and latency is higher than software decoding on 12900KS.
3. Libav and FFmpeg on Windows do not support hardware decoding 4:4:4 at this moment so whom wish to develop needs to wire the decoding manually to the hardware decoder.
4. There is one hidden experimental direction AVC H264 and 4:2:0, which is much more compatible with most modern GPUs, even ancient ones like 6800XT. However, there seems to be a bug on M1 MacBook Air that makes video quality significantly lower than HEVC, and the image is blurry.

The two modes are complementary: Full Quality is the universal, pixel-accurate baseline; Adaptive Quality adds a low-latency streaming path tuned for motion, at the cost of some fidelity and a hardware-decode requirement.

Progress

I have conducted preliminary study and created a minimal proof-of-concept at https://github.com/Chapoly1305/libvncserver and https://github.com/Chapoly1305/iShareScreen. If folks interested in this feature, we can make it matter before finally merge into the main stream.

Tasks

• Authentication Types. PR Adding support for ARD AuthType 33/35/36 #698 adds Authentication Type 33 for RSA/SRP, Type 35 for Kerberos, and Type 36 for SRP.
• HiDPI. Very sharp font and images.
• Dynamic Resolution. User may resize view, full screen on various resolutions with HiDPI support.
• Performance Optimization. The performance is basically aligned with native client.
• Latency Optimization. The performance is basically aligned with native client.
• Backport into this project.

Demo

---

Usage

For the fork of this project I made,

https://github.com/Chapoly1305/iShareScreen is mature, and https://github.com/Chapoly1305/libvncserver is less mature.

From the repo root:

cmake -S . -B build
cmake --build build --target client_examples_applehpdebug -j4

The binary ends up at:

• libvncserver/build/examples/client/applehpdebug

Basic use:

build/examples/client/applehpdebug <host> [port] [seconds]

For the Apple high-performance path, use env like:

VNC_APPLE_HP=1 VNC_AUTH_SCHEMES=33 VNC_USER='your_user' VNC_PASS='your_pass' VNC_LIVE_VIEW=1 \
build/examples/client/applehpdebug 100.96.0.1 5900 30

Useful knobs:

• VNC_LIVE_VIEW=1 opens the SDL viewer window
• VNC_LIVE_VIEW_VSYNC=0|1 controls vsync
• VNC_LIVE_VIEW_LOW_LATENCY_INPUT=0|1 controls the short post-input low-latency polling path
• VNC_APPLE_HP_REGION_W / VNC_APPLE_HP_REGION_H can force refresh region size for experiments

[bk138]

Interesting. Is ARD any different on ARM Macs as compared to Intel Macs? I can connect to both with the code from the master branch though performance is admittedly a little worse than using Tight encoding. Would you like to document your findings wrt differences here?

[Chapoly1305]

Thank you for asking.

1. Screen Sharing has two modes. Officially, only both ends are ARM Mac (both host and client) can use the high performance mode. The most obvious is, on High Performance mode, the physical display will be black/blank; while the old standard mode will sync the mouse movement to the physical display. ARD (Type 30) authentication is traditional / standard mode. For this high performance mode, Apple used a different authentication design, i.e., RSA/SRP (Type 33).
2. I am interested in sharing as a wiki style so folks can aid to correct it. Currently trying to find a place.

[bk138]

1. Thanks for sharing that link. What I get from it:
   • ARM/Apple-silicon requirement: some hardware encoding might be going on
   • UDP is in use; what for? streaming? input? meta-data-signaling?
2. I think you can keep that draft PR Add Apple high-performance viewer example #697 as a means of "looking at things together" which will not be merged but we can split out small meaningful changes over time and get them in. You can also add any draft docs to that PR, we can merge bit by bit.

[Chapoly1305]

@bk138
I tested on Intel as viewer, I am quite satisfied with the performance. During on-going experiments, no UDP is observed. All connections captured used TCP. I only observed UDP is when the connection is through iCloud.

Additional, some updates:

• The authentication mechanism appears to be independent of High Performance Mode, which is a positive finding. This indicates that all supported authentication types (30, 33, 35, and 36) are compatible with High Performance Mode. These authentication types fall under the broader Apple Remote Desktop (ARD) framework. I have ported and tested types 33, 35, and 36, and verified that they function correctly in this context.
  Users can therefore choose between enhanced security (using the newer authentication types) or maintaining compatibility with the legacy authentication type 30 when using ARD.
• The dynamic resolution feature now triggers automatically when the user resizes the window or enters full-screen mode, which significantly improves usability. One minor issue remains: I observed colored bars appearing along the edges of the display. I am not certain what this abnormally is called, and ai did not resolve this after many repeated attempts.

[bk138]

If you want, you can already file the support for sectypes 33, 35, and 36 in a split-out PR :-)

[Chapoly1305]

This PR refactors the existing ARD design by abstracting ARD authentication, removing its dependency on Type 30 and enabling support for additional types (33, 35, and 36). Type 34 is theoretically feasible, but since key exchange does not occur during connection establishment, its implementation is deferred. Additionally, Type 35 is not purely password-based—it is Kerberos-driven, so users are expected to provide the appropriate realm. #698

[popbones]

Thank you sir. Looking forward to this!

[Chapoly1305]

It has been a while, so I wanted to share some progress.

The authentication variants are now mostly complete. Beyond that, I can confirm that High Performance mode actually consists of two different modes.

The first is the Remote Framebuffer protocol, commonly abbreviated as RFB. This mode is straightforward to integrate into the project and is the most compatible option. It works by transmitting bitmaps for active pixels. However, the tradeoff is performance: it is relatively slow and laggy, so it does not deliver a native-feeling experience. I have done this part.

The second is an Apple Silicon-exclusive streaming mode. This is the area where I have been blocked for quite some time. It appears to be the mode responsible for the advertised 60 FPS performance and near-zero observable latency. The hard part is determining how to properly decode the video stream.

[popbones]

It has been a while, so I wanted to share some progress.

The authentication variants are now mostly complete. Beyond that, I can confirm that High Performance mode actually consists of two different modes.

The first is the Remote Framebuffer protocol, commonly abbreviated as RFB. This mode is straightforward to integrate into the project and is the most compatible option. It works by transmitting bitmaps for active pixels. However, the tradeoff is performance: it is relatively slow and laggy, so it does not deliver a native-feeling experience. I have done this part.

The second is an Apple Silicon-exclusive streaming mode. This is the area where I have been blocked for quite some time. It appears to be the mode responsible for the advertised 60 FPS performance and near-zero observable latency. The hard part is determining how to properly decode the video stream.

Well we know the high performance mode is done via virtual display and probably based one the screen capture kit installed in recent version. And we also know it seems to limit to 4K at most. Requiring Apple Silicon means it uses some hardware encoding/decoding capability that's available in all the Apple Silicon, it's probably some HEVC variant or some Apple codec, maybe similar to Sidecar? Also I wonder if it actually streams the whole screen since it's has been glitch with ARD official client and Screen Sharing.app in a way make me feel like it might stream the desktop and active app separately (which I think was also introduced in recent version of the screen capture related frameworks).

Anyway, I haven't dig in myself, so if anything I said doesn't make sense, please ignore.

Again, appreciate your work.

[Chapoly1305]

It has been a while, so I wanted to share some progress.
The authentication variants are now mostly complete. Beyond that, I can confirm that High Performance mode actually consists of two different modes.
The first is the Remote Framebuffer protocol, commonly abbreviated as RFB. This mode is straightforward to integrate into the project and is the most compatible option. It works by transmitting bitmaps for active pixels. However, the tradeoff is performance: it is relatively slow and laggy, so it does not deliver a native-feeling experience. I have done this part.
The second is an Apple Silicon-exclusive streaming mode. This is the area where I have been blocked for quite some time. It appears to be the mode responsible for the advertised 60 FPS performance and near-zero observable latency. The hard part is determining how to properly decode the video stream.

Well we know the high performance mode is done via virtual display and probably based one the screen capture kit installed in recent version. And we also know it seems to limit to 4K at most. Requiring Apple Silicon means it uses some hardware encoding/decoding capability that's available in all the Apple Silicon, it's probably some HEVC variant or some Apple codec, maybe similar to Sidecar? Also I wonder if it actually streams the whole screen since it's has been glitch with ARD official client and Screen Sharing.app in a way make me feel like it might stream the desktop and active app separately (which I think was also introduced in recent version of the screen capture related frameworks).

Anyway, I haven't dig in myself, so if anything I said doesn't make sense, please ignore.

Again, appreciate your work.

I recently discovered a similar project (https://github.com/renegadelink/iShareScreen) that actually resolved the decoding challenge. It appears that I had made some incorrect assumptions. I will complete the document and first add the dynamic resolution feature to this repository. Then, I will port it back here.

[Chapoly1305]

I have updated the ticket description in the abstract and concluded the observations for those reading this thread in the future.

The PoC is basically done on my personal fork https://github.com/Chapoly1305/iShareScreen, which is tested on macOS (w/ hardware acceleration) and Windows (Intel QSV as hardware acceleration and software for fallback). I added the "RFC" style documentation and it is basically out-of-the-box ready to use. I will treat that as the starting point and back port here soon.