Using MikroTik hEX as a router with VLANs at maximum throughput

The challenge

Today I wanted to install a MikroTik hEX router into my lab environment to act as a core router (without NAT), one hop before my firewall (see diagram below).

MikroTik hEX as core router

So in a scenario like this, we want the hEX to be a router to allow intra-VLAN routing between the VLANs 1001, 1002 and 1003. Of course, the hardware firewall in our diagram could probably do that for us as well, but that’s not always something one wants or needs or maybe the firewall in charge even isn’t as capable in terms of throughput and one doesn’t want to put that additional load of potentially full Gigabit between the VLANs on it. In this scenario I didn’t plan for any firewalling on the hEX router but we could, in theory, deploy some ACLs which should be processed in hardware (iirc) to allow for some segregation in our segmentation 😉

Also, using a second router to hold all our internal VLANs gives us a separated routing instance which might come in handy with later, more complex scenarios. Not to mention the security benefit of being able to create VLANs and networks for DMZ purposes on the internet-facing firewall which effectively means that traffic literally wouldn’t even aim towards a physical cable leading to your internal network, but staying physically (on Layer 2) behind a firewall, where external traffic belongs. We all do that like this, right? Right?

While being extremely powerful and versatile, the correct configuration for a specific tasks doesn’t always appear obvious with MikroTik devices. Sometimes it even depends on the type of hardware MikroTik is using for a given device model. MikroTik has tons of documentation on everything but still, this took me two or three hours to figure out, being a complete newcomer to anything labeled MikroTik.

The solution

Now, enough of the babbling. In this scenario, we have configured bridge1 to contain the interfaces ether2-ether5, leaving ether1 separated for the WAN side of the router. You could in theory use ether2-ether5 as separate interfaces, all with their own VLANs but in this case I wanted to run all VLANs through a single cable regardless of which port (ether2-ether5) I connect the uplink cable to. In theory, we could do some port aggregation later on, running all VLANs through some port-aggregated uplink. For now, this is a classical router layout where you have a LAN side switch with some ports (ether2-ether5) and one WAN port (ether1).

/interface bridge port
add bridge=bridge interface=ether2
add bridge=bridge interface=ether3
add bridge=bridge interface=ether4
add bridge=bridge interface=ether5

Because as of firmware 6.48.6* the hEX does not support VLAN filtering on the bridge level while keeping hardware offloading enabled, the approach here is a bit different than on other devices like the CRS3xx switch series. For the hEX router (in fact for any device using the MT7621 chipset) we need to configure the VLANs on the “Switch/VLAN” level instead. Note: switch1-cpu needs to be a member of all VLANs for routing to work properly. It is recommended to implement appropriate firewall rules to prevent unwanted access to the router (we will cover that later below).

* According to MikroTik documentation as of writing this post, Bridge VLAN Filtering in hardware has been added in RouterOS 7.1rc5 for MT7621 based devices

/interface ethernet switch vlan
add independent-learning=no ports=ether2,ether3,ether4,ether5,switch1-cpu \
switch=switch1 vlan-id=1001
add independent-learning=no ports=ether2,ether3,ether4,ether5,switch1-cpu \
switch=switch1 vlan-id=1002
add independent-learning=no ports=ether2,ether3,ether4,ether5,switch1-cpu \
switch=switch1 vlan-id=1003

Set VLAN mode to “secure” for our LAN interfaces so only appropriately tagged traffic can enter the so secured ports. Other traffic will be dropped.

/interface ethernet switch port
set 0 default-vlan-id=1 vlan-mode=fallback
set 1 default-vlan-id=1 vlan-mode=secure
set 2 default-vlan-id=1 vlan-mode=secure
set 3 default-vlan-id=1 vlan-mode=secure
set 4 default-vlan-id=1 vlan-mode=secure
set 5 default-vlan-id=1 vlan-mode=secure

Additionally, we need to disable any spanning-tree protocol on the bridge, otherwhise hardware offloading will get disabled.

/interface bridge set protocol-mode=none

Interfaces for all VLANs have to be created, so we can assign IP addresses to them and actually start routing data:

/interface vlan
add interface=bridge name=VLAN1001 vlan-id=1001
add interface=bridge name=VLAN1002 vlan-id=1002
add interface=bridge name=VLAN1003 vlan-id=1003

Assign IP addresses to the new interfaces:

/ip address
add address=10.1.0.2/24 interface=ether1 network=10.1.0.0
add address=10.0.1.1/24 interface=VLAN1001 network=10.0.1.0
add address=10.0.2.1/24 interface=VLAN1002 network=10.0.2.0
add address=10.0.3.1/24 interface=VLAN1003 network=10.0.3.0

Now lets restrict traffic to the management services a bit and disable some legacy services. It is advised to create an SSL certificate and only manage over encrypted protocols like https or api-ssl. We could as well create some firewall filter rules to lock the router down even more but for now let’s start with this. Let’s assume that 10.0.1.0/24 is our management VLAN, so access to the management services is only allowed from there. Keep in mind, that we are restricting the source IPs of the management traffic, so if you are on a management workstation which has e.g. the IP address of 10.0.1.100, you can still access the router on 10.0.2.1 and 10.0.3.1 because traffic is being routed from 10.0.1.100 (which is still the source of the request and therefore allowed) to 10.0.2.1 or 10.0.3.1. Restricting management traffic to 10.0.1.1 only would require firewall filtering rules or configuring the management services not to listen on the other interfaces (which I am not aware of whether that’s possible).

/ip service
set telnet address=10.0.1.0.0/24 disabled=yes
set ftp address=10.0.1.0/24 disabled=yes
set www address=10.0.1.0/24
set ssh address=10.0.1.0/24
set www-ssl address=10.0.1.0/24 tls-version=only-1.2
set api address=10.0.1.0/24 disabled=yes
set winbox address=10.0.1.0/24
set api-ssl address=10.0.1.0/24 disabled=yes

Note, that even disabled services have their allowed address range configured, just as a good measure.

Now let’s organize our interfaces into two lists (LAN and WAN) and restrict access to the MAC server and MAC Winbox only to internal interfaces:

/interface list
add name=WAN
add name=LAN

/interface list member
add interface=bridge list=LAN
add interface=VLAN1001 list=LAN
add interface=ether1 list=WAN

/tool mac-server
set allowed-interface-list=LAN
/tool mac-server mac-winbox
set allowed-interface-list=LAN

Results

As you can see below, this configuration leads to active hw-offload on the bridged interfaces:

Hardware offload is active

Here we can see a throughput of around 800 Mbits/sec with one single stream via iperf3. The CPU load on the hEX was around 20% during the iperf run:

Throughput of around 800 Mbits/sec with one single stream

Conclusion

Proper configuration of the hEX router and making use of its hardware features allows us to achieve high throughput rates without running into the limits of the CPU. This post does not claim to be a complete guide for setting up a hEX router, many more configuration options may apply to your environment. AFAIK, MikroTik also recommends to enable Ingress Filtering on the Bridge Ports to increase security, but I haven’t digged into that yet.

If you have any suggestions, improvements or any other feedback, leave it in the comments down below.