Playing with Fortigate

[sdargoeuves]

I don't know how many users are using fortigate in their netlab environment? I know @petercrocker did, and it was time I started to play with it myself. I'll be curious to understand what version is being used, as I downloaded 7.4.7 and realised their Permanent trial mode is quite restrictive: 3 interfaces max 🫤.

❤️ Thanks to this fix: being able to NOT have a loopback, means I can connect my firewall to 2 devices 🥳!

Maybe a version prior to 7.2.1 with the 15-day trial license is more usable... but it will be quite annoying to have to rebuild the vagrant box frequently.

FortiOS 7.2.1 introduces a new permanent trial license, and moves away from the 15-day trial license. See Permanent trial mode for FortiGate-VM.

Anyway, so far so good with the 7.4.7 despite the restrictions, I need to understand better the rest of the restrictions to make my mind up!

---
message: "Test Fortigate..."
plugin: [ multilab ]
defaults.multilab.id: 2
defaults.addressing.mgmt.start: 10
defaults.devices.eos.clab.image: "ceos:4.29.9.1M"
defaults.devices.fortios.libvirt.image: "fortinet/fortios:7.4.7"
provider: libvirt

nodes:
  fw1:
    device: fortios
    module: [ ospf ]
    libvirt.uuid: eb0603f4-0d27-43a5-b294-682dec123456
    loopback: false
  d1:
    device: eos
    module: [ ospf ]
    provider: clab
  d2:
    device: eos
    module: [ ospf ]
    provider: clab

[ipspace]

I don't know how many users are using fortigate in their netlab environment?

I know at least @melvync did (based on #1933). At that time, I set up a FortiOS instance with an evaluation license, fixed the tasks, and never started it again.

Anyway, so far so good with the 7.4.7 despite the restrictions, I need to understand better the rest of the restrictions to make my mind up!

That's interesting. IIRC, at the time when I was fixing the fortinet tasks, anything beyond 7.2.something required token-based authentication, and I decided to implement that only when someone asks for it. Are you saying I dodged that bullet? 😎

[jbemmel]

I don't know how many users are using fortigate in their netlab environment?

I have used it, very briefly

Each instance requires an evaluation license, the license is tied to the VM UUID, and you only get 1 free license. So I decided not to bother - we have HA use cases that would have been nice to verify virtually, but I refuse to invest time in a company that poses that many restrictions.

[sdargoeuves]

(based on #1933)
thanks, there's definitely some interesting info in there, with the same license dilemma...

Are you saying I dodged that bullet?
I think you did! You can still use the username/password, but the license needs to be applied. I had to link the license to the vm before creating the vagrant box. Then keeping the UUID (using the awesome fix you did I can specify the uuid in the topology file), it means it now works as expected, as the license is applied correctly.

I refuse to invest time in a company that poses that many restrictions.
I wanted to add the HA use case in a later phase, but I underestimated the restrictions of that license... I will keep exploring what is doable with so much restrictions, and will probably give up after a while 😄 or at least keep it to a very basic use case.

Thanks for your comments!

[sdargoeuves]

I've created this PR to fix the weird MTU behaviour I saw, by forcing it to 1500, if not set in the interface parameters and refresh the docs regarding the licensing with the current latest version.

[sdargoeuves]

Still playing with the Fortigate... (I know I should stop and move on 😄) I have now got a local branch working with the separate vdom for management and traffic.
The problem I am now facing, is that I can't seem to create a Loopback interface in the traffic vdom:

TASK [Configure loopback interface] *************************************************************************************
fatal: [fw1]: FAILED! => changed=false
  meta:
    action: ''
    build: 3510
    cli_error:
    - current vf=netlab:1
    - object set operator error, -4 discard the setting
    - Command fail. Return code -4 (reached the maximum number of entries)
    - Command fail. Return code 1
    error: -4
    http_method: POST
    http_status: 500
    name: interface
    path: system
    revision: 1eea39d94db11659b554254b3f3d4ea6
    revision_changed: false
    serial: FGVMEVJCWIP2TN78
    status: error
    vdom: netlab
    version: v7.6.3
  msg: Error in repo

It works great if I add the loobpack: false to the node definition.

With the limitation of 3 interfaces, one being the management, it doesn't leave much space for a loopback interface if you wanted to connect your fw 'in-line'. In the topology I am planning to use, the FW would have the management interface, and 2 interfaces connected: north and south.

@ssasso @ipspace which option do you think is best:

1. keep the current state, where there is a single vdom (and I think the loopback creation is working -- although I don't remember testing)
2. use the multi-vdom, one for traffic and one for management, and
   1. add to the notes that Loopback will not work due to license constraints (unless it's another issue on my side, I should probably mentioned: I've never played with Fortinet, apart from adding rules in the UI >5 years ago) and suggest adding loopback: false in the node definition,
   2. or, remove the loopback creation from the ansible tasks
3. something else 🙃

[ssasso]

@sdargoeuves with both vagrant and Containerlab you have the "limitation" that the first interface must be dedicated for management, and that's unfortunately counting as one for fortigate VM... Regardless of using a dedicated vdom for it or not.

[ipspace]

1. keep the current state, where there is a single vdom (and I think the loopback creation is working -- although I don't remember testing)

I think that's the default setting, so one would expect the loopback to be created. While it would be nice to have a management VDOM, I don't care too much if we don't have it.

2. use the multi-vdom, one for traffic and one for management, and

As I said, up to you and @ssasso ;)

1. add to the notes that Loopback will not work due to license constraints

You can also do that.

unless it's another issue on my side

Well, the error message is pretty explicit, although I wonder why it works with when the loopback interface is in the management VDOM.

and suggest adding loopback: false in the node definition,

That would be mandatory. In that case, I would also add loopback: false to device node data (hoping that copy is done before the loopbacks are initialized ;)

2. or, remove the loopback creation from the ansible tasks

Nope. There might be a netlab user that has a "proper" license and would like to have a loopback.

[ipspace]

@sdargoeuves with both vagrant and Containerlab you have the "limitation" that the first interface must be dedicated for management, and that's unfortunately counting as one for fortigate VM... Regardless of using a dedicated vdom for it or not.

That has always been true, but it looks like @sdargoeuves can configure management interface, two data-plane interfaces, and a loopback if he uses default VDOM, but not if he has two VDOMs.

[sdargoeuves]

@sdargoeuves can configure management interface, two data-plane interfaces, and a loopback if he uses default VDOM, but not if he has two VDOMs.

this is what I am seeing so far, but as mentioned, I could either be tricked by the limited license, or facing a human error when trying to configure it!

Nope. There might be a netlab user that has a "proper" license and would like to have a loopback.

good point!

That would be mandatory. In that case, I would also add loopback: false to device node data (hoping that copy is done before the loopbacks are initialized ;)

Yes that looks good 😌

[sdargoeuves]

I think I have an idea so that someone could use either type of vagrant box, the one with multi-vdom, or the single vdom one. This way it won't be a breaking change.

By default the vdom we will use everywhere will be the vdom root, so in a single vdom mode that's fine. Now if someone wishes to use the multi-vdom box, they can simply set the data_vdom: netlab in the node parameters (or traffic_vdom, or something like this). In this situation the playbook will use this vdom for the interfaces and configuration but not for the management part, which will always be in the root vdom.

If you think this is a good idea, I would just need some help on how can I add a new variable to the node parameters, so that I can use it in the playbook? Unless there is a different way I am not aware of to achieve this.

In the topology file I would envision something like this:

nodes:
  fw1:
    device: fortios
    module: [ ospf ]
    libvirt.uuid: xxxxx
    data_vdom: "netlab" # if not set, it will default to "root"

and I should then be able to use it in the netsim/devices/fortios.yml file like this:

  vdom: "root"
  data_vdom: "root"

so the data_vdom is either root for single vdom, or the name coming from the node paramter from the topology file if set.

I could potentially investigate in the future the Fortigate ansible module, to see how I could configure the fortigate to take this variable into account and enable multi-vdom, and if not, to disable it... then the default box would work directly.

[ipspace]

If the admin VDOM is hardcoded as "root" (I have no idea whether that's true), then you could just use a single VDOM variable to specify the data-plane VDOM (with "root" being the default value).

The only thing I would do would be to rename devices.fortios.group_vars.vdom to netlab_vdom so you can specify it on a node without triggering "invalid attribute" error.

[sdargoeuves]

@ipspace thank you so much! Quick test this morning seems to work as expected!!
I didn't know I can actually spin up more than one fortigate device, I thought the licensing would block me, but it seems to be fine.

So I have two vagrant boxes, one configured with the single vdom (root), and the other one configured with multi-vdom enabled with root as the management vdom.

PLAY RECAP ***********************************************************************************************************
fw1                        : ok=21   changed=4    unreachable=0    failed=0    skipped=11   rescued=0    ignored=0
fw2                        : ok=21   changed=4    unreachable=0    failed=0    skipped=11   rescued=0    ignored=0
r1                         : ok=23   changed=3    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0
r2                         : ok=23   changed=3    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0
r3                         : ok=23   changed=3    unreachable=0    failed=0    skipped=6    rescued=0    ignored=0

[SUCCESS] Lab devices configured

Topology file I am using:

---
message: "Fortigate test"

plugin: [ multilab ]
defaults.multilab.id: 2
defaults.addressing.mgmt.start: 10

defaults.devices.eos.clab.image: "ceos:4.29.9.1M"
provider: libvirt

groups:
  routers:
    _auto_create: true
    device: eos
    provider: clab
    module: [ ospf ]
    members: [ r1, r2, r3 ]
  fw:
    device: fortios
    module: [ ospf ]
    loopback: false # can be removed the release after 2.0.1
    members: [ fw1, fw2 ]
nodes:
  fw1:
    libvirt.uuid: xxxxx # uuid for box 7.6.30 (single-vdom)
    image: "fortinet/fortios:7.6.30" # single-vdom (it's actually 7.6.3, but the tag is 7.6.30 to identify the correct vagrant box)
  fw2:
    libvirt.uuid: xxxxx # uuid for box 7.6.3 (multi-vdom)
    image: "fortinet/fortios:7.6.3"
    netlab_vdom: "netlab"
links:
  - fw1:
    r1:
  - fw1:
    fw2:
  - fw2:
    r2:
  - r2:
    r3:

I am slowing getting there, I just need to recreate the vagrant box with the multiple vdom enabled without the traffic vdom set to netlab, and let the playbook create it based on the name provided in the node parameter. In my current setup, the vdom netlab was configured in the vagrant box...

Once I have this, I might even be able to use the default box, and create switch the setting multi-vdom based on the topology information.

Regarding your question

If the admin VDOM is hardcoded as "root" (I have no idea whether that's true)

The doc for the permanent trial license specifies:

Support for a maximum of two virtual domains (VDOM). When using multi-VDOM mode, the root VDOM must be an admin type and the other can be a traffic VDOM.

and in my small experience with fortinet, vdom root is used as management, but is it possible to have it NOT as the management in multi-vdom, I am not sure... I'll try to dig more through the documentation.

[ssasso]

@sdargoeuves thanks for digging into this!

If you have time and willingness to experiment a bit more, I would suggest:

• Avoid having two separate images, and keep only one simple box with only one vdom (default config).
• IF a group var called traffic_vdom exists, THEN
  • Create the new vdom (with that name) as part of the initial configuration tasks
  • Assign the different interfaces to it still during the initial configuration tasks
  • Use the var on the other modules, defaulting it to 'root' if not present

[ssasso]

On a side note @ipspace custom templates are not working with fortios, since the custom (and plugins) expect to find whatever.j2 - while fortios needs to be configured with custom "tasks".

If you agree I will try to address this in a generic way, so potentially any device can use either .j2 files or .yml files as custom configs.

If a .J2 is found, it is applied as per today code.

If a .yml file is found, Ansible include_tasks can be used.

[sdargoeuves]

@sdargoeuves thanks for digging into this!

If you have time and willingness to experiment a bit more, I would suggest:

• Avoid having two separate images, and keep only one simple box with only one vdom (default config).

• IF a group var called traffic_vdom exists, THEN

  • Create the new vdom (with that name) as part of the initial configuration tasks
  • Assign the different interfaces to it still during the initial configuration tasks
  • Use the var on the other modules, defaulting it to 'root' if not present

Thanks for the tips, I totally agree! I was slowly moving in that direction, just need to look at the ansible docs and play some more with the initial configuration. I'll create a PR once I've got something working 😃

[a-v-popov]

while fortios needs to be configured with custom "tasks".

Not sure if you still care, but there is another way, sort of.
https://docs.fortinet.com/document/fortigate/7.6.3/administration-guide/780930/configuration-scripts
It can use keywords like purge.
It is exposed as API endpoint /system/config-script/upload
which can be accessed with ansible
https://ansible-galaxy-fortios-docs.readthedocs.io/en/latest/fortios_monitor.html
Here is an example which purges all static routes in netlab vdom and add default gateway:

# cat route.yml 
---
- name: set default route
  hosts: b2
  gather_facts: no

  tasks:
  - name: set default route
    vars:
      gateway: 10.1.0.8
      port: port2
    fortinet.fortios.fortios_monitor:
        selector: upload.system.config-script
        params:
          filename: set_default_route
          file_content: "{{ lookup('template', 'route.j2') | b64encode }}"

# cat route.j2
config vdom
edit {{ netlab_vdom}}
config router static
    purge
    edit 1
        set gateway {{ gateway }}
        set device "{{ port }}"
    next
end
end

[ipspace]

Not sure if you still care, but there is another way, sort of. https://docs.fortinet.com/document/fortigate/7.6.3/administration-guide/780930/configuration-scripts It can use keywords like purge. It is exposed as API endpoint /system/config-script/upload which can be accessed with ansible https://ansible-galaxy-fortios-docs.readthedocs.io/en/latest/fortios_monitor.html

Thanks a million!

If that thingy works reliably across the range of Fortinet software versions we're working with, it might be a better choice than the various task lists. However, I'm guessing we're pretty low on the mental power to tackle that migration considering the crazy licensing restrictions we have to deal with.

[a-v-popov]

pretty low on the mental power

Totally understandable.
Right now I am merely sharing more or less generic pieces I come across while playing with netlab.
I might eventually come back with a couple of requests but they will be placed explicitly.

[sdargoeuves]

I'm quite happy with how it currently works: PR:

• based on the node parameter netlab_vdom, if not set, we will use the default which is set to root, and will keep the fortigate in the no-vdom mode. If set, root will be the management vdom, with port1 the mgmt interface, the rest will be configured in the 2nd vdom.

• I initially wanted to use traffic_vdom oro data_vdom, but it was not accepted. I haven't figured out why, hence I'm staying with netlab_vdom

  [ERRORS]  Errors found in topology.yml
  [ATTR]    nodes: Invalid node attribute 'data_vdom' found in nodes.fw2
  [HINT]    use 'netlab show attributes node' to display valid attributes
  [FATAL]   Cannot proceed beyond this point due to errors, exiting
  

• tested ok with 7.4 and 7.6. I have a 7.0, but license has expired, so I am not able to test previous versions.

• Initialisation: when changing the mode to multi-vdom, it takes some time before the FW can be configured. I haven't figured a better way than waiting 60seconds... I've tried a few options to get something that would tell the code it can carry on, but no luck...

  • The reason why I am not using a pause, is that in my test setup, I am spinning two FW (which weirdly works, despite only having one license: it was assigned to the first box during its creation, then removed, and assigned to the next box), and the pause is not applicable per host. So it would fail to pause if I was trying to spin up one no-vdom and one multi-vdom fortigate.

[ipspace]

I tried to run multiple fortios 7.6.3 boxes with FortiFlex licenses over the weekend and maybe some notes for the record.

Thanks a million for the update!

netlab down rip out the images and they would need to be re-licenses. It might be a good thing if it would stop FortiFlex licenses in the process to reduce billing.

I could add custom hooks in the netlab up/down process that would call bash scripts that could do whatever. Would that help?

[sebastien-dargoeuves-ipf]

Thanks for sharing, I didn't know about those licenses, quite interesting!

netlab down rip out the images and they would need to be re-licenses.

If you want to bring back the "same" vm, or at least the same Serial Number, you can keep note of the UUID when creating the box, and use the libvirt.uuid in the topology file to force the same uuid to be used when performing netlab up.

I've added those steps in this blog post:

• using a shell command: virsh dominfo fortios748-vm | grep UUID
• directly from the fortinet, although it's not formatted nicely: diagnose hardware sysinfo vm full

Then you just have to add under the node, the libvirt.uuid: <uuid> parameter:

nodes:
  fw1:
    device: fortios
    module: [ ospf ]
    libvirt.uuid: eb0603f4-0d27-43a5-b123-682dec123456 # 7.4.8
    netlab_vdom: netlab

[a-v-popov]

I could add custom hooks in the netlab up/down process that would call bash scripts that could do whatever. Would that help?

Thank you, I think this should work. But if you don't have anyone else asking for this feature let me first figure out how I would do it and then I will come back to you with a better understanding of my needs.

[a-v-popov]

If you want to bring back the "same" vm...

I suspect disk should be the one used in activation process. I actually tried to set libvirt.uuid, but without corresponding disk image it didn't do me any good. I would guess it is also checking the "HW" so if UUID, and maybe something else, changes this will invalidate the license. I assume in your approach you always have the same disk image, i.e. vagrant box = disk used for the specific license, and then keeping UUID is critical.

However in my case I don't want want to create a box per license, so I do not apply a license before creating a vagrant box. I simply followed the instruction in the documentation to build it: https://netlab.tools/labs/fortios/
Plus with FortiFlex licenses I shouldn't be accumulating "points" until license is activated. My UUIDs is changing, but as long as it is not changing after license activation it should be OK. However, we will see if someone comes after me to tell that I am abusing the service.

[sdargoeuves]

I assume in your approach you always have the same disk image, i.e. vagrant box = disk used for the specific license, and then keeping UUID is critical.

That's right — I maintain a box per license, so keeping the UUID is essential in my setup.

However, we will see if someone comes after me to tell that I am abusing the service.

Your method of applying the license after bringing the lab up makes a lot of sense — no need to pay or accumulate points for a box that’s not in use. It’s a slightly different story on my side, but your approach definitely looks a lot cleaner than mine 🙈🙉