Use of SSH ProxyJump and LocalForward to access lab devices

[a-v-popov]

It is not an idea for a feature, but for documentation probably. Not sure in which category it should go.
Looking at https://netlab.tools/example/external/#connecting-to-lab-devices somehow I do not see option to use a built-in SSH functionality to access lab devices. For example one can use the following ansible playbook and jinja template to generate ssh config file:

# cat ssh_config.yml
---
- name: Generate SSH config 
  hosts: localhost
  gather_facts: no

  tasks:
  - name: Get my public IP
    community.general.ipify_facts:

  - name: Render a config template
    template:
      src: ssh_config.j2
      dest: ./ssh_config

# cat ssh_config.j2 
# --- start of netlab config ---

Host netlab
    Hostname {{ ipify_public_ip }}
{% for host in groups['fortios'] %}
{%   set hv = hostvars[host] %}
{%   if hv.get('provider') != 'clab' %}
    LocalForward {{ 8000 + hv.id }} {{ hv.ansible_host }}:443
{%   endif %}
{% endfor %}

{% for host in groups['all'] %}
{%   set hv = hostvars[host] %}
Host {{ host }}
{%   if hv.get('provider') == 'clab' %}
    Hostname netlab
    RequestTTY yes
    RemoteCommand docker exec -it {{ hv.ansible_host }} sh -il
{%   else %}
    Hostname {{ hv.ansible_host }}
    User {{ hv.ansible_user }}
    ProxyJump netlab
{%   endif %}

{% endfor %}


# --- end of netlab config ---

[ipspace]

Thank you, definitely an interesting topic ;) The https://netlab.tools/example/external/#connecting-to-lab-devices section was concerned more with having direct access to the lab devices, what you seem to be proposing here is the lab server acting as a sort of a bastion host, right?

Anyway, we could do the same thing as a netlab report, but then of course you'd have to save it (or append it) to ssh_config file yourself.

[a-v-popov]

I guess it would depend on the definition of a bastion host and a direct access.
The generated config allows me to use scp to lab devices directly from my laptop, thanks to ProxyJump.
With LocalForward I can open Fortinet web interface in my browser on my laptop using https://localhost:80xx as URL.
All connections are encrypted inside SSH tunnel, no need to expose anything externally.
I also have a WireGuard tunnel, but practically have no use for it.

[ipspace]

I would suggest we turn your template into a text report, make that part of netlab package, and add a short note to the connectivity section, maybe pointing to this discussion (or somewhere else) for more in-depth information?

[a-v-popov]

I am not sure what you mean by "turn your template into text report". This is the first time I am using netlab and I haven't come up with any sort of established procedure quite yet. Plus, I am using GCP, which might not be everyone's favorite. Personally, I am already very happy I can use standard open source tools to get whatever I need with netlab. Right now I can tell that ssh built-in features, like LocalForward, ProxyJump, and potentially ProxyCommand are certainly the way to go for me, it just a question how exactly they will be wrapped up. As you mentioned the ssh_config file generated by the playbook somehow needs to get to the local ssh config. Right now I am using "gcloud compute ssh..." to connect initially and then at some point "gcloud compute scp ..." to pull that ssh_config file and drop it into "~/.ssh/config.d/" from where I am pulling it from the main ssh config file with Include directive. What it means for integration into netlab itself, from the perspective of my use case:

• not sure what report is, but I would prefer to have it as a plain text file immediately suitable for inclusion into ssh config
• ideally ssh_config file should be (re)generated automatically each time topology is rebuilt
• its content might depend on how people use it, e.g. ProxyJump vs ProxyCommand, or ipify use - I am still trying to figure out the best way to update it after I suspend/resume the netlab VM, etc.

[DanPartelly]

Netlab has the ability to generate reports, useful bits of information which are extracted from a topology and outputted on stdout by default. There are wiring reports, addressing reports, nodes reports (I love this one as it allows me using a bash script to connect to many devices with a single command). It looks like a nice fit for this functionality too.

For example, a wiring report for a lab would look like:
netlab create netlab report wiring

LAB TOPOLOGY SUMMARY

LAB NAME: sr

POINT-TO-POINT LINKS:

|-----------------|---------------|-------------|--------------------|------------------|
| Link Name       | Origin Device | Origin Port | Destination Device | Destination Port |
|-----------------|---------------|-------------|--------------------|------------------|
|                 | pe1           | Ethernet0/1 | h1                 | eth1             |
|                 | pe2           | Ethernet0/1 | h2                 | eth1             |
|                 | pe1           | Ethernet0/2 | p                  | Ethernet0/1      |
|                 | pe2           | Ethernet0/2 | p                  | Ethernet0/2      |
|-----------------|---------------|-------------|--------------------|------------------|

[a-v-popov]

Then I would run something like

gcloud compute ssh ... -- "cd lab_path; /home/apopov/venv/bin/netlab report ssh" > ssh_config

right? And I should be able to simply drop a template in ~/.netlab/repots. Sounds good to me.
But if there are any comments it would be better to escape them, or have an option to escape them, with hashes.

[ipspace]

@a-v-popov The proof-of-concept code is in ssh-config branch (you'll have to run netlab from local Git repo).

I decided to use RemoteCommand for all devices to get around the password/identity file morass. Works well for terminal access, but OTOH you cannot execute a command on a device or do scp.

Would appreciate your feedback; I'll add port forwarding once we get the initial concept to a usable state.

[a-v-popov]

I guess it would depend on individual needs, but for interactive sessions, like configuration and troubleshooting, I would be more likely to run tmux on a remote netlab host. So getting interactive SSH session from local machine would not be my priority.
My priorities would be:

1. Local port forwarding to get access to Fortigate WebUI and API.
2. SSH Proxy to get SCP working directly from local machine for config backups.

It should noted that I am considering the whole netlab host VM expendable.
Also there could be multiple instances of netlab running simultaneously.
Generally, I feel more inclined to do work locally rather then try pushing things into netlab VMs.

As for public_ip it would seem that at least for now I am running local script on my laptop to update it from gcloud output anyway, so it is less important for me now.

[ipspace]

So you're effectively saying "this is useless for me." ;)

Another idea: what if we define two host names, one useful for terminal sessions that starts netlab connect (so you don't have to deal with passwords) and another one (only for devices with SSH access) that would be a passthrough SSH session for SCP and stuff. The first hostname would come from netlab topology, the second one would have a suffix like .ssh or .scp (or maybe someone has a better idea /cc @DanPartelly @jbemmel)

The port forwarding is the next step ;)

[a-v-popov]

I think I like the .scp idea. There should be no harm in having a second Host, and I can easily add it where I need it.

[ipspace]

I updated the ssh-config branch (and opened #2593). The "ssh_config" report now generates two hostnames for devices with SSH access, and port forwarding rules if the nodes have clab.ports or libvirt.ports forwarding specs.

[a-v-popov]

Does it mean that to get ssh port forwarding in report I would need to expose the port to the Internet on the netlab host as well? Would there be a way to get local port forwarding in the ssh report only?

[ipspace]

Does it mean that to get ssh port forwarding in report I would need to expose the port to the Internet on the netlab host as well?

That was the easiest way to do it ;)

Would there be a way to get local port forwarding in the ssh report only?

Any suggestions?

[a-v-popov]

Technically ssh port forwarding is not related to host port forwarding. In fact, one reason I am using the former is to avoid the latter.
So if the question is what would suffice, I think, a key which I can put into ~/.netlab.yml to disable all host level port forwarding would do it.
If the question is what I would like to have in a perfect world, then I guess a config on a (custom) group level or per device type specifying a base port and the type of forwarding would be preferable, i.e. something like:

groups:
  secure_hosts:
    ports:
      - https:8000  # forwarded port is 8000 + id
        protocol: tcp   # should be default   
        forwarder: [ssh, host] # could be default
  insecure_hosts:
    ports:
      - https: 9000
        forwarder: [ssh]

[ipspace]

I think, a key which I can put into ~/.netlab.yml to disable all host level port forwarding would do it.

No port forwarding is enabled by default (at least not by netlab).

[ddutt]

Instead of adding multiple options into netlab, why not just support passing ssh_config that enables whatever ssh allows?

[ipspace]

Instead of adding multiple options into netlab, why not just support passing ssh_config that enables whatever ssh allows?

We already have that, but on the netlab-host-to-device level. What @a-v-popov would like to have is a way to generate ssh_config file for his laptop that would include port forwarding laptop->netlab-host->device.

I'll probably go with netlab_ssh_forward hack that would contain a list of laptop:device port pairs. Node parameters starting with "netlab_" are ignored during the validation and it would be used in a single report anyway.

[ddutt]

He wants ssh_config generated? I see.

[ipspace]

@a-v-popov I changed the report to use netlab_ssh_forward node parameter for forwarded SSH ports. It's not ideal, but it was the minimally intrusive solution. Switch to the ssh-config branch to try it out.

I have no idea how many other people use port forwarding; if it turns out that port forwarding is popular then we can start thinking about the "perfect world" solution.

[a-v-popov]

Thank you. If you have doubts about usefulness of this feature for others, I am fine with leaving it out from netlab to avoid creating undocumented "nerd knobs".

[ipspace]

No worries. I like the feature, it's just that I don't expect it to be used much. I will make sure it's mentioned somewhere, I just didn't want to do that before we had the concept hammered out.

[ipspace]

@a-v-popov Wrote the documentation. Would love to hear your feedback on the "Using SSH Port Forwarding" section (I also fixed a few other things based on persistent Grammarly nagging).

[a-v-popov]

I would suggest to mention ssh_config keywords which might be useful for remote access to VMs, in my opinion these are:

• LocalForward
• ProxyJump
• ProxyCommand
• RemoteCommand

This would allow people to do their own research as needed, or just copy-paste them into any LLM to get a full explanation.
Also a link to a page describing custom reports might be helpful in my opinion
https://netlab.tools/outputs/report/

[ipspace]

I would suggest to mention ssh_config keywords which might be useful for remote access to VMs, in my opinion these are:

Will add a footnote.

Also a link to a page describing custom reports might be helpful in my opinion https://netlab.tools/outputs/report/

Absolutely not a problem. Thank you!

[ipspace]

Implemented in #2593