GUACAMOLE-288: Add support for multi-monitor connections on RDP.

[corentin-soriano]

Anyone who wishes can test and feedback will be appreciated!

Requires using this branch on guacamole-server: apache/guacamole-server#560

• Display each monitor in separate window.
• Each window has its own dimensions and Y position.
• Display user consent button to switch all window on fullscreen when the main window fullscreen is enabled.
• Monitors are ordered from left to right (left monitor is the primary).
• Handle CTRL+ALT+SHIFT in secondary monitors (open the menu on the main window).
• Close all secondary monitors on client disconnect.
• Admin can limit the usage of multi-monitors on each connection.
• Drag and drop between windows without having to release the click and resume on the other side.

WIP:

• Reorder monitors vertically if needed.
• Use a dedicated BroadcastChannel for each connection.
• Handle cases where secondary windows remain open after disconnect.
• Display bug on copy operations from one screen to another.

Admin settings:

Add new monitor:

Multiple monitors opened:

Different sizes:

[stephzero1]

wow this is huge! i will test asap

[neandrake]

Exciting. Could you write up the technical implementation here? Here's what I gather from looking at the changes here and in the guacamole-server pull request.

• It looks like all the drawing instructions will now set an optional x-offset for the base/default layer. All other layers are relative to the base layer which is why they don't need the x-offset (correct?).
• A new optional parameter is added to the size instruction for indicating number of monitors.
• A new optional parameter for maximum number of secondary monitors is added to the connection parameters.

That leads me to understand,

1. There is a constraint that monitors must be laid out horizontally and be the same dimension.
2. Each window/tab/connection to the server is receiving graphical updates for all monitors but only displaying the subset for its selected monitor.

Could you clarify if that is correct? If so, how is each tab specifying which monitor/offset to use?

[corentin-soriano]

Exciting. Could you write up the technical implementation here? Here's what I gather from looking at the changes here and in the guacamole-server pull request.

• It looks like all the drawing instructions will now set an optional x-offset for the base/default layer. All other layers are relative to the base layer which is why they don't need the x-offset (correct?).

Only the main layer is affected by the offset because the others (eg: mouse) are relative to the browser window instead of the remote screen.

• A new optional parameter is added to the size instruction for indicating number of monitors.

With FreeRDP, the number of monitors is set via the display update channel (the same as for the dimensions). The setting is optional for backward compatibility.

• A new optional parameter for maximum number of secondary monitors is added to the connection parameters.

You can define it in your user-mapping.xml or in guacamole settings with a jdbc extension:

<param name="secondary-monitors">3</param>

In the example above I can open 3 additional monitors and then the button is grayed out (guacd would ignore the instruction):

That leads me to understand,

1. There is a constraint that monitors must be laid out horizontally and be the same dimension.

For now, all monitors have the same dimensions but there is client-side scaling if one of the windows needs to have different dimensions:

2. Each window/tab/connection to the server is receiving graphical updates for all monitors but only displaying the subset for its selected monitor.

Exactly, each window displays its visible part according to its x-offset. Each time a monitor is added/removed, an instruction is sent by sendMonitorsInfos() to all the windows to know their new position (allow to close a middle window).
The x-offset corresponds to the position * width of the monitor.

Could you clarify if that is correct? If so, how is each tab specifying which monitor/offset to use?

Secondary windows knows their position thanks to an ID defined in the route:

In the screenshot above:

• Main window is at the position 0 => x-offset = 0*width = 0 px
• monitor with ID 3 is at the position 1 => x-offset = 1*width.

Does this answer your questions?

[mike-jumper]

@corentin-soriano Rather than build on the size instruction, what about leveraging layer parameters and the established set instruction (similar to the multi-touch support)?

For example, we could do something like:

/* Hint that some_layer is best represented as a separate monitor/display */
guac_protocol_send_set(socket, some_layer,
    GUAC_PROTOCOL_LAYER_PARAMETER_PRESENTATION,
    GUAC_PROTOCOL_LAYER_PARAMETER_PRESENTATION_DETACHED);

/* Hint that the default layer will only contain other layers and need not have 
 * a graphical surface */
guac_protocol_send_set(socket, default_layer,
    GUAC_PROTOCOL_LAYER_PARAMETER_PRESENTATION,
    GUAC_PROTOCOL_LAYER_PARAMETER_PRESENTATION_VOID);

to advise compatible clients that a particular layer should be ideally rendered as a separate display in its own right, with its position within any parent layers to be interpreted as hinting for relative location.

That would make the same semantics available in a backward-compatible manner, while also allowing the concept of separating out parts of the display to be more broadly applicable (such as for popping out RemoteApp windows).

[gerdesj]

I'd like to test your code but I am not a github expert. I have checked out both guacamole-client and guacamole-server locally.

How do I put them in the correct state? I suspect --branch is involved.

I'd be grateful if you could note how to get the starting line set up and then I'll run with it.

[necouchman]

I'd like to test your code but I am not a github expert. I have checked out both guacamole-client and guacamole-server locally.

How do I put them in the correct state? I suspect --branch is involved.

@gerdesj

Should be something like:
git clone https://github.com/apache/guacamole-client
cd guacamole-client
git fetch origin pull/1061/head:test/GUACAMOLE-288
git checkout test/GUACAMOLE-288
mvn clean package

(and similarly for guacamole-server, just substituting in the correct pull request number for that repo, then autoreconf, configure, make, etc.)

[gerdesj]

Wow! It works perfectly. I'd say it is better than stock mstsc because you can add and remove screens at your whim.

So far I have tested with Firefox on Kubuntu 24.04, connecting to a Windows 2022 RDS box. Guac + Tomcat are running on Ubuntu 22.04. I only have two identical screens so haven't looked into what happens with varying sizes.

I'll move this to our live box soon and get some more testing done with real users.

[gerdesj]

I'll move this to our live box soon and get some more testing done with real users.

I actually cracked out the compiler and moved it to our live box the same day I made that comment. We are an IT firm so our users generally work around issues ... of which we have found none.

One of my office admins uses our Guacamole so she can rock a Mac and still run all the Windows based stuff like Outlook. Bloody millennials! However, that shows that you cannot know exactly how your code will end up being used. Her feedback has been that I will not be allowed to revert this functionality.

In the spirit of enquiry, I am currently using a laptop with just the local panel. I started another screen up and it works exactly as I would expect. I have a second floating window with a start menu. I've just passed an app from one screen to the other.

Most of my firm use Windows and a Mac or two and I exclusively use Linux and Wayland these days. We use nearly all mainstream browsers. Feedback about multi screen support has been overwhelmingly positive.

My only suggestion is to consider changing the word monitor in the option Add an additional monitor to window or perhaps screen. My personal preference is window.

[Vertganti]

I have done a little testing using docker images built from your two branches. Great work so far!

By default the resize method configuration option is set to a blank entry. If you specify a maximum number of secondary monitors and leave the resize method blank you will only get a white screen when trying to open a secondary monitor. This is presumably why you added needs display-update to the label, but I just overlooked it the first time, because i didn't know what was meant. Maybe you could create an Enable secondary monitors checkbox, which I think would greatly improve the UX for administrating this feature. Checking the checkbox would:

• fix the resize method to display-update (set the value and disable the dropdown if that is possible, maybe show a warning about the required RDP version)
• enable the text field for the maximum secondary monitors (and set a default value?)
• cause the button for opening another monitor to be displayed during the connection

Another thing I noticed testing a connection from Chrome on Windows 10 to a Windows 11 PC is that dragging a window to the secondary monitor will create visual artifacts as in the screenshot below. It uses a Chrome window as an example but the same happens for other windows, e.g. the file explorer or CMD.

1. If I drag the window from primary to secondary without releasing the mouse button, I get the artifacts on the window in the secondary monitor.
2. If I drag the window from secondary to primary without releasing the mouse button, I do not get the artifacts on the window in the primary monitor.
3. If I drag the window from primary to the edge of the screen, release the mouse button and move the mouse to the secondary monitor, then grab the window on the secondary monitor and drag it in, I get the artifacts on the window in the secondary monitor.
4. If I drag the window from secondary to the edge of the screen, release the mouse button and move the mouse to the primary monitor, then grab the window on the primary monitor and drag it in, I get the artifacts on the window in the primary monitor.

Note how only dragging without releasing to the primary does not generate artifacts. Moving windows around in the monitor they were created in does not create artifacts for either monitor. I have only tested with two monitors (primary + secondary).

Otherwise it has been working great so far. I will continue testing it over the next days.

[corentin-soriano]

I have done a little testing using docker images built from your two branches. Great work so far!

By default the resize method configuration option is set to a blank entry. If you specify a maximum number of secondary monitors and leave the resize method blank you will only get a white screen when trying to open a secondary monitor. This is presumably why you added needs display-update to the label, but I just overlooked it the first time, because i didn't know what was meant. Maybe you could create an Enable secondary monitors checkbox, which I think would greatly improve the UX for administrating this feature. Checking the checkbox would:

• fix the resize method to display-update (set the value and disable the dropdown if that is possible, maybe show a warning about the required RDP version)
• enable the text field for the maximum secondary monitors (and set a default value?)
• cause the button for opening another monitor to be displayed during the connection

Another thing I noticed testing a connection from Chrome on Windows 10 to a Windows 11 PC is that dragging a window to the secondary monitor will create visual artifacts as in the screenshot below. It uses a Chrome window as an example but the same happens for other windows, e.g. the file explorer or CMD.

1. If I drag the window from primary to secondary without releasing the mouse button, I get the artifacts on the window in the secondary monitor.
2. If I drag the window from secondary to primary without releasing the mouse button, I do not get the artifacts on the window in the primary monitor.
3. If I drag the window from primary to the edge of the screen, release the mouse button and move the mouse to the secondary monitor, then grab the window on the secondary monitor and drag it in, I get the artifacts on the window in the secondary monitor.
4. If I drag the window from secondary to the edge of the screen, release the mouse button and move the mouse to the primary monitor, then grab the window on the primary monitor and drag it in, I get the artifacts on the window in the primary monitor.

Note how only dragging without releasing to the primary does not generate artifacts. Moving windows around in the monitor they were created in does not create artifacts for either monitor. I have only tested with two monitors (primary + secondary).

Otherwise it has been working great so far. I will continue testing it over the next days.

I haven't had time to research a fix yet, but I've already identified this issue.
These are copies of a piece of image from one screen to another screen that do not work (and leave the old image).
The bug occurs because each window ignores the instructions sent to other screens, so the part to be copied is not known.

Regarding the other comments, I will try to improve them. Thanks for taking the time to test and provide feedback!