Precise Time Sync & iSCSI

It’s a good feeling to kill two birds with one stone (figuratively) when delivering an IT project. I had the opportunity to do just that recently when two separate business units offered up two very different problems that I could solve at the same time. The first requirement was time synchronisation using PTPv2, or IEEE1588-2008 Precision Time Protocol, to synchronise devices in multiple buildings within our campus (approximately 2000 fibre route meters from end to end). The second requirement was an iSCSI network between the same locations for SAN replication between multiple pairs of disparate buildings. 

Our current LAN network at the time just didn’t support PTP. I’m sure we could have made it work somehow by using multicast routing, but the sub 50 microsecond precision required for the application wouldn’t have been achievable. It was obvious that we needed some new switches. Initially I was looking at Cisco’s industrial switches since we already had some deployed in the network.

As for the iSCSI requirement, we needed multiple 10Gig uplinks to every SAN. Again, our current LAN network at the time didn’t have the capacity to accommodate that many 10Gig ports, so it was also apparent that new switches were required to run a separate SAN network. 

After a lot of Googling for high density 10Gig switches, I came across some Nexus switches that also happened to support PTP. They had 32 x 10Gig SFP+ ports each. My thought process ended up something like: Ideal. WOW Expensive! Let’s try Ebay. Ideal.

We ended up buying 8 x Nexus 5548up switches from Ebay, with the L3 daughter board and L3 license (i’ll explain why later). The switches ended up costing around £1,500 each, instead of £25k+ per switch new.

To provide an accurate time source we also purchased a Meinberg GPS / GNSS PTP grandmaster clock. This also turned out to provide NTP as well, so our domain has a pretty precise time source now, but that’s a different story.

Physical Topology

Now the topology of the switches is as follows. topology_1.jpg We decided to go with this topology to allow us to add an additional grandmaster clock at a later date in the second “hub” location. Since both hubs are on different substations etc, this would provide some resilience if and when required. Wed also have an abundance of disparate fibre between all of the facilities and the two hubs, as these are are main datacenter locations. 

PTP

The PTP logical topology looks a little something like this.topology_2.jpg
Each switch is configured to be a boundary clock in the PTP domain. The PTP priorities of the switches mean that if the grandmaster goes offline for whatever reason, the switches will all synchronise to the lowest priority switch. This was done to ensure the equipment in all facilities were in sync with each other, even if they weren’t in sync with actual time. If a facility becomes isolated from he rest of the network, then at least its clocks will all remain in sync with each other since the switch in that facility has a lower priority than the default PTP priority.

The PTP ports are on a vlan local to the switch. Since the switches are boundary clock, there doesn’t need to be layer 2 adjacency between the grandmaster clock and the clocks (devices) for PTP to work. The port configuration is simple and is as follows.

You don’t even need to configure ip addresses on the local vlan. The devices will use link local addresses and multicast to communicate with the switch. No pesky VRF’s required!

iSCSI

The iSCSI topology ended up being Layer 3 between the buildings. We didn’t want to stretch Vlan’s between the buildings to help contain outages. The operational facilities are test & validation facilities with a lot of moving mass or high voltage, so the likelihood of a switch being taken out is pretty high. To propagate routes we used OSPF, with the hubs being area 0 and each facility having it’s own area. The logical iSCSI topology looks like this.topology_1.jpg I’ve only included the information for two of the four facilities to make it more legible.

I’m aware that some people don’t like using layer 3 in iSCSI networks. In our situation, the replication is asynchronous. In each building the iSCSI initiator and target are both on the same subnet / vlan. Only the replication traverses layer 3 boundaries. We haven’t seen any performance or reliability issues with this topology yet so all is good. 

Final Observations / Notes

One caveat of the Nexus 5548up switches is that the ports cannot run at 100Mbit/s. This is an issue for some PTP devices as they only have 100Mbit/s ports. To get around this we used cheap media converters from FS.com. These media converters will happily have a 1Gbit/s SFP in the SFP port, and a 100Mbit/s connection in the RJ45 port. 

Another caveat of the Nexus 5548up switches is that if you want to run a port at 1Gbit/s with an RJ45 SFP, you must force the port to 1000Mbit before it will come up. Just add speed 1000 to the port configuration.

Buying Cisco SFP’s isn’t really worth it when using used switches. Even on Ebay a used one is around £35-60 and you don’t know what sort of environment it’s been in. FS.com sell compatible optics for £19 with a 5 year warranty.

We also purchased spare switches for a quick response to a failure. We could have doubled down and deployed two per facility, but didn’t think it was necessary. The clock devices only have a single port at the end of the day, and SAN replication can catch up.

That’s it. I just thought I’d share one of those little last minute projects that needs to be delivered on a  tight budget.

Cisco Router Dual WAN Uplinks with NAT

Dual WAN uplinks for resilience are a common request when configuring small business routers. I’ll work with the topology below and go through the configuration.

Screenshot 2020-02-13 at 20.56.40.png

The only devices I’ll be going over are R1 and PC1. In the real world, the rest would be out of your control anyway. I’ll include the GNS3 project file at the end of this post if you would like to play with it.

Configuration

The first thing we need to do is configure the interfaces of the two ISP connections.

interface GigabitEthernet0/0
  description ISP1
  ip address 100.1.1.2 255.255.255.252
  ip nat outside
!
interface GigabitEthernet1/0
  description ISP2
  ip address 200.2.2.2 255.255.255.252
  ip nat outside
!

And the LAN interface

interface GigabitEthernet6/0
  description LAN
  ip address 192.168.1.1 255.255.255.0
  ip nat inside
!

Next we’ll configure our routes via both ISP’s. To make the failover work we need to track some objects on the primary connection. This will make failover occur if the internet connection goes down. I would advise you track reachability of a couple of hosts to avoid failing over if somebody else is having an issue. I would also advise against tracking the next hop, as an issue within the ISP network wouldn’t cause failover to occur but may prevent you from reaching the internet. We do this using ip sla to two different known hosts (I used Google’s public DNS servers for this demo).

ip sla 100
icmp-echo 8.8.8.8 source-interface GigabitEthernet0/0
threshold 1000
timeout 1000
frequency 10
ip sla schedule 100 life forever start-time now
ip sla 101
icmp-echo 8.8.4.4 source-interface GigabitEthernet0/0
threshold 1000
timeout 1000
frequency 10
ip sla schedule 101 life forever start-time now

Next we create two track objects to monitor the ip sla’s for reachability.

track 100 ip sla 100 reachability
!
track 101 ip sla 101 reachability

Then a track object to track the first two objects. By using the boolean or option, the track will go down if all of the tracked objects go down but will not if only one goes down.

track 105 list boolean or
object 100
object 101

Now we will add our default routes via both ISP’s. We will use the track 105 object to determine if ISP1 is up and add it to the routing table. Otherwise we will add ISP2 with a metric of 10.

ip route 0.0.0.0 0.0.0.0 100.1.1.1 track 105
ip route 0.0.0.0 0.0.0.0 200.2.2.1 10

That should be the routing done, now we will need to configure NAT to allow the LAN clients to access the internet. We’ll create an access list to define the LAN traffic that should be translated.

ip access-list standard NAT-INSIDE
permit 192.168.1.0 0.0.0.255
!

And we’ll use some route-maps to match the LAN traffic on the outside interfaces for translation.

route-map RM-NAT-ISP2 permit 20
match ip address NAT-INSIDE
match interface GigabitEthernet1/0
!
route-map RM-NAT-ISP1 permit 10
match ip address NAT-INSIDE
match interface GigabitEthernet0/0
!

And finally, the NAT configuration commands.

ip nat inside source route-map RM-NAT-ISP1 interface GigabitEthernet0/0 overload
ip nat inside source route-map RM-NAT-ISP2 interface GigabitEthernet1/0 overload

Verification

Now we can do some testing. If we ping from PC1 to 8.8.8.8, the ping should succeed. You can also perform a traceroute from PC1 to 8.8.8.8 to verify the route the traffic flows.

PC1#ping 8.8.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 48/85/196 ms

PC1#traceroute 8.8.8.8
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.1.1 16 msec 48 msec 28 msec
2 100.1.1.1 104 msec 72 msec 44 msec
3 1.2.1.1 56 msec 60 msec 68 msec

You can use show ip nat translations on R1 to verify the NAT translations over ISP1.

R1#show ip nat translations
Pro Inside globalInside local Outside localOutside global
icmp 100.1.1.2:1025192.168.1.11:168.8.8.8:16 8.8.8.8:1025
udp 100.1.1.2:4501 192.168.1.11:49157 8.8.8.8:334378.8.8.8:33437
udp 100.1.1.2:4502 192.168.1.11:49158 8.8.8.8:334388.8.8.8:33438
udp 100.1.1.2:4503 192.168.1.11:49159 8.8.8.8:334398.8.8.8:33439
udp 100.1.1.2:4504 192.168.1.11:49160 8.8.8.8:334408.8.8.8:33440
udp 100.1.1.2:4505 192.168.1.11:49161 8.8.8.8:334418.8.8.8:33441
udp 100.1.1.2:4506 192.168.1.11:49162 8.8.8.8:334428.8.8.8:33442

And also, show ip route on R1 should show the next hop as ISP1.

S*    0.0.0.0/0 [1/0] via 100.1.1.1

Now, if we suspend a link connected to the ISP1 route, it doesn’t matter which one, our topology should failover.

Screenshot 2020-02-13 at 21.44.38.png

First thing you should notice is the track objects going down on R1.

*Feb 13 21:44:28.847: %TRACKING-5-STATE: 100 ip sla 100 reachability Up->Down
*Feb 13 21:44:28.851: %TRACKING-5-STATE: 101 ip sla 101 reachability Up->Down
*Feb 13 21:44:29.835: %TRACKING-5-STATE: 105 list boolean or Up->Down

And the default route on R1 should have changed to ISP2.

S*0.0.0.0/0 [10/0] via 200.2.2.1

And if we repeat the same ping and traceroute from PC1 to 8.8.8.8, they should still work fine, but the route should show ISP2 as the second hop instead of ISP1.

PC1#ping 8.8.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/56/84 ms

PC1#traceroute 8.8.8.8
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.1.1 12 msec 88 msec 24 msec
2 200.2.2.1 52 msec 32 msec 36 msec
3 1.2.2.1 64 msec 64 msec 36 msec

show ip nat translations on R1 should also show the NAT translations to ISP2 now.

R1#show ip nat translations
Pro Inside globalInside local Outside localOutside global
icmp 200.2.2.2:1024192.168.1.11:2 8.8.8.8:28.8.8.8:1024
udp 200.2.2.2:4501 192.168.1.11:49167 8.8.8.8:334378.8.8.8:33437
udp 200.2.2.2:4502 192.168.1.11:49168 8.8.8.8:334388.8.8.8:33438
udp 200.2.2.2:4503 192.168.1.11:49169 8.8.8.8:334398.8.8.8:33439
udp 200.2.2.2:4504 192.168.1.11:49170 8.8.8.8:334408.8.8.8:33440
udp 200.2.2.2:4505 192.168.1.11:49171 8.8.8.8:334418.8.8.8:33441
udp 200.2.2.2:4506 192.168.1.11:49172 8.8.8.8:334428.8.8.8:33442

Now if we resume the link to the ISP1 route, the track objects will come back up and everything should fail back.

*Feb 13 21:50:43.871: %TRACKING-5-STATE: 100 ip sla 100 reachability Down->Up
*Feb 13 21:50:43.871: %TRACKING-5-STATE: 101 ip sla 101 reachability Down->Up
*Feb 13 21:50:44.839: %TRACKING-5-STATE: 105 list boolean or Down->Up

Notes

There are a couple of things to note with this configuration:

  • There is zone-based firewall configuration in this demo. I highly recommend one if you aren’t using a dedicated firewall.
  • Inbound connections using port forwarding and the primary connection IP address will not failover if ISP1 fails.
  • The IOS image used in this GNS project is c7200-advipservicesk9-mz.152-4.S5.bin. You’ll have to provide that yourself.

Downloads

GNS3 Project – Dual WAN

Playing with interface templates in Cisco IOS XE

I recently learned of the existence of interface templates. Previously I had been using int range a lot to make mass updates to our access ports, but this wasn’t really scaling well and took ages to do on some of our 7 switch stacks.

For anybody who hasn’t used interface templates before, they are a mechanism to configure once and apply to many interfaces. Not only is this convenient, it reduces the size of your configuration quite a bit.

We built a number of templates for use in our network, one for each vlan we assign ports to. Since we standardised the access vlans across all switches (e.g. vlan 100 is always for users, vlan 200 is always for voice, vlan 600 is always for guest access etc), this makes all the templates across all switches identical.

What goes into a template? pretty much any configuration you can apply to an interface, you can put in a template. Heres an example of a template for a user port.

template USER
 storm-control broadcast level pps 1k
 storm-control action shutdown
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
 switchport access vlan 100
 switchport mode access
 switchport voice vlan 200
 switchport port-security maximum 2
 switchport port-security violation protect
 switchport port-security aging time 3
 switchport port-security
 service-policy input MARKING
 service-policy output 2P6Q3T-WRED
 description USER
!

And then to apply it to an interface:

interface Gi1/0/25
 source template USER
!

Thats literally all of the configuration you need on an interface. The problem now is, if you do a show run on an interface, all you get is “source template USER”. There is a different show command to solve that problem.

SW1#show derived-config interface gi1/0/25
Building configuration...

Derived configuration : 501 bytes
!
interface GigabitEthernet1/0/25
 description USER
 switchport access vlan 100
 switchport mode access
 switchport voice vlan 200
 switchport port-security maximum 2
 switchport port-security violation protect
 switchport port-security aging time 3
 switchport port-security
 storm-control broadcast level pps 1k
 storm-control action shutdown
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
 service-policy input MARKING
 service-policy output 2P6Q3T-WRED
end

You can also mix and match on an interface, and have configuration directly on the interface. The the interface configuration will override the template configuration. It still looks a bit weird in show run though.

template USER
 storm-control broadcast level pps 1k
 storm-control action shutdown
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
 switchport access vlan 100
 switchport mode access
 switchport voice vlan 200
 switchport port-security maximum 2
 switchport port-security violation protect
 switchport port-security aging time 3
 switchport port-security
 service-policy input MARKING
 service-policy output 2P6Q3T-WRED
 description USER
!
interface Gi1/0/25
 description Marketing Laptop
 source template USER
!

SW1#show run int gi1/0/25
Building configuration...

Current configuration : 92 bytes
!
interface GigabitEthernet1/0/25
 description Marketing Laptop
 source template USER
end

SW1#show derived-config interface gi1/0/25
Building configuration...

Derived configuration : 506 bytes
!
interface GigabitEthernet1/0/25
 description Marketing Laptop
 switchport access vlan 100
 switchport mode access
 switchport voice vlan 200
 switchport port-security maximum 2
 switchport port-security violation protect
 switchport port-security aging time 3
 switchport port-security
 storm-control broadcast level pps 1k
 storm-control action shutdown
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
 service-policy input MARKING
 service-policy output 2P6Q3T-WRED
end

I found interface templates really useful during a recent network upgrade. It wasn’t too long ago I became CCNA certified and these weren’t even mentioned on the courses, which is a shame.

These templates are configured on 9200L switches, if you are interested.

Nest Cam Outdoor – My experiences

So a couple of months ago some bastard keyed my car. My initial knee jerk reaction was to buy a CCTV camera to watch over it in case it happened again, so I knew who’s knees to break. So the search began, but I wanted something fast.

I looked at a couple of offerings but I wanted something easy and quick to install, and preferably without too many cables to run around the house. I settled in the Nest Cam Outdoor, after owning their thermostat for a year and being very pleased with its performance and functionality.

When the camera arrived it was very well packaged with the usual plethora of mounting hardware. The kit comes with long enough cables for most installations and all the required fixing hardware, including cable clips, which was nice.

The mounting system for the camera uses strong magnets to hold it in place. While this is a novel idea, in practice a well-aimed football will knock the camera off the wall and render it useless. Points lost there unfortunately.

The cable attached to the nest has a ruggedized USB connector on it that’s approximately 18mm diameter. All well and good until you need to run it through a wall. I know this camera is meant for American homes which I presume mostly have outdoor power outlets, but that’s an uncommon facility in the UK. Drilling an 18mm+ hole in a double skinned brick wall is no simple task. More points lost there.

Once installed the camera is easy to set up using the Nest iOS or Android apps. Scan the bar code on the app and away you go. Don’t forget you’ll need a constant internet connection for the camera to work though.

The video quality of the camera even in daylight is shit considering it is allegedly 1080p. While the camera sensor probably is 1080p, the camera compressed the nuts off the video to upload it to Google’s cloud platform, losing most of the quality. I honestly can’t even read the number plate of my car from the video, and the car is parked maybe 15 feet from the camera, if that. Image quality is even worse at night.

Remember Google’s server I just mentioned? They only allow the camera to store 3 hours of footage, unless you buy a Nest Aware subscription to store up to 10 days, or 30 days, whichever tickles your fancy. This isn’t particularly well communicated by Nest either, to the point that when my “Free Trial” expired, I didn’t even get a notification to say “Hey, your camera has just deleted all video over three hours old! Best break the credit card out!” Thinking about this logically, if somebody broke into my car between 1am and 3am, I wouldn’t have any footage of it at all. What’s the point?

In conclusion, don’t buy this camera. It’s effectively a glorified, very expensive, video doorbell without a subscription, which wouldn’t be as bad if the cost of the camera wasn’t so fucking much to start with. Even then, you can’t make out facial features unless the camera is installed at a level where the magnets make it easy to sabotage. Again, what’s the point?

Honestly. Don’t waste your money. This device has the potential to be awesome, but is used as a cash cow by Nest instead.

Tips For Working From Home

Some people have the fortune of being able to work from home as part of their employment. Working from home, or telecommuting, can be extremely productive for a lot of people. It can also be counter productive if you’re not experienced at telecommuting. As a regular home worker, I have learned a lot of ways to make the valuable, non-interrupted time as productive as possible.

Routine

It’s important to keep your usual morning routine (apart from the travelling to the office part, obviously) to help switch your brain to work mode. Lounging around in your dressing gown with your laptop is something you can do on a Sunday morning, and you’ll struggle to associate the time with work.

My normal morning routine looks something like this:

  • 6:00 – Alarm goes off.
  • 6:05 – Have a cuppa and breakfast with the wife and kids.
  • 6:30 – Shepherd the kids into their uniforms for school.
  • 7:00 – Get a shower and get dressed for work.
  • 7:20 – Shepherd the kids into coats and shoes ready to leave.
  • 7:30 – Leave to drop the kids off at various locations.
  • 7:50 – Leave the hometown en route to work.
  • 8:20 – Arrive at work.

My “working from home” routine is pretty much the sames:

  • 6:00 – Alarm goes off.
  • 6:05 – Have a cuppa and breakfast with the wife and kids.
  • 6:30 – Shepherd the kids into their uniforms for school.
  • 7:00 – Get a shower and get dressed for work.
  • 7:20 – Shepherd the kids into coats and shoes ready to leave.
  • 7:30 – Leave to drop the kids off at various locations.
  • 7:55 – Arrive back home ready for a day of work.

Since I usually wear a work branded polo shirt and jeans for work, I usually just wear them when working from home. Your mileage may vary if you have to wear a suite though.

Workspace

It’s important to have somewhere to work from while telecommuting. Sitting on your sofa with your laptop might seem like a great idea, but take it from me, you will soon get too comfortable and start slouching. Then you start to become distracted and turn the TV on. This is fine if you can work like that, but I bet your productivity is affected.

The best scenario is having a desk and a comfortable office chair to sit in, preferably in a different room to the hustle and bustle of your house, if there are other family members home. If you don’t have a desk, the dining table is a good alternative, providing enough space to spread your work out a little and key coffee cup support within arms reach.

Connectivity

The vast majority of work is performed on laptops these days. Unless you are simply writing a proposal and you don’t need internet connectivity, then you need to think about connectivity.

In some cases, a straight WiFi connection will suffice for access to emails and the internet, but you might need to consider access to internal business systems. Since each company is different, contact you ICT department to find out what options are available to connect to the resources you require for your job.

The company I work for offer a device called a Teleworkers Gateway. This device plugs into my own router and gives me the options of either a “work” WiFi connection, or a cabled Ethernet connection. They also offer a software VPN solution but the Teleworkers Gateway is the simpler and more convenient option.

Welfare

The beauty of working from home is having unlimited coffee without feeling obliged to make one for everybody else in your office each time. Just make sure you stop for lunch at your normal time.

Another benefit of working from home is your bed. Seriously. Feeling a bit burnt out by lunchtime? Have a 45 minute nap. You’ll be nice and refreshed and more productive in the afternoon.

Moderation

Like all good things, telecommuting should be used in moderation if it isn’t your primary working arrangement. Working from home is great when you need to focus on a particular task without the interruptions of the office. Do it too much though, and people get used to you not being in the office and start calling, emailing and instant messaging you more, thus increasing the interruptions you were trying to avoid.