Precise Time Sync & iSCSI

It’s a good feeling to kill two birds with one stone (figuratively) when delivering an IT project. I had the opportunity to do just that recently when two separate business units offered up two very different problems that I could solve at the same time. The first requirement was time synchronisation using PTPv2, or IEEE1588-2008 Precision Time Protocol, to synchronise devices in multiple buildings within our campus (approximately 2000 fibre route meters from end to end). The second requirement was an iSCSI network between the same locations for SAN replication between multiple pairs of disparate buildings. 

Our current LAN network at the time just didn’t support PTP. I’m sure we could have made it work somehow by using multicast routing, but the sub 50 microsecond precision required for the application wouldn’t have been achievable. It was obvious that we needed some new switches. Initially I was looking at Cisco’s industrial switches since we already had some deployed in the network.

As for the iSCSI requirement, we needed multiple 10Gig uplinks to every SAN. Again, our current LAN network at the time didn’t have the capacity to accommodate that many 10Gig ports, so it was also apparent that new switches were required to run a separate SAN network. 

After a lot of Googling for high density 10Gig switches, I came across some Nexus switches that also happened to support PTP. They had 32 x 10Gig SFP+ ports each. My thought process ended up something like: Ideal. WOW Expensive! Let’s try Ebay. Ideal.

We ended up buying 8 x Nexus 5548up switches from Ebay, with the L3 daughter board and L3 license (i’ll explain why later). The switches ended up costing around £1,500 each, instead of £25k+ per switch new.

To provide an accurate time source we also purchased a Meinberg GPS / GNSS PTP grandmaster clock. This also turned out to provide NTP as well, so our domain has a pretty precise time source now, but that’s a different story.

Physical Topology

Now the topology of the switches is as follows. topology_1.jpg We decided to go with this topology to allow us to add an additional grandmaster clock at a later date in the second “hub” location. Since both hubs are on different substations etc, this would provide some resilience if and when required. Wed also have an abundance of disparate fibre between all of the facilities and the two hubs, as these are are main datacenter locations. 

PTP

The PTP logical topology looks a little something like this.topology_2.jpg
Each switch is configured to be a boundary clock in the PTP domain. The PTP priorities of the switches mean that if the grandmaster goes offline for whatever reason, the switches will all synchronise to the lowest priority switch. This was done to ensure the equipment in all facilities were in sync with each other, even if they weren’t in sync with actual time. If a facility becomes isolated from he rest of the network, then at least its clocks will all remain in sync with each other since the switch in that facility has a lower priority than the default PTP priority.

The PTP ports are on a vlan local to the switch. Since the switches are boundary clock, there doesn’t need to be layer 2 adjacency between the grandmaster clock and the clocks (devices) for PTP to work. The port configuration is simple and is as follows.

You don’t even need to configure ip addresses on the local vlan. The devices will use link local addresses and multicast to communicate with the switch. No pesky VRF’s required!

iSCSI

The iSCSI topology ended up being Layer 3 between the buildings. We didn’t want to stretch Vlan’s between the buildings to help contain outages. The operational facilities are test & validation facilities with a lot of moving mass or high voltage, so the likelihood of a switch being taken out is pretty high. To propagate routes we used OSPF, with the hubs being area 0 and each facility having it’s own area. The logical iSCSI topology looks like this.topology_1.jpg I’ve only included the information for two of the four facilities to make it more legible.

I’m aware that some people don’t like using layer 3 in iSCSI networks. In our situation, the replication is asynchronous. In each building the iSCSI initiator and target are both on the same subnet / vlan. Only the replication traverses layer 3 boundaries. We haven’t seen any performance or reliability issues with this topology yet so all is good. 

Final Observations / Notes

One caveat of the Nexus 5548up switches is that the ports cannot run at 100Mbit/s. This is an issue for some PTP devices as they only have 100Mbit/s ports. To get around this we used cheap media converters from FS.com. These media converters will happily have a 1Gbit/s SFP in the SFP port, and a 100Mbit/s connection in the RJ45 port. 

Another caveat of the Nexus 5548up switches is that if you want to run a port at 1Gbit/s with an RJ45 SFP, you must force the port to 1000Mbit before it will come up. Just add speed 1000 to the port configuration.

Buying Cisco SFP’s isn’t really worth it when using used switches. Even on Ebay a used one is around £35-60 and you don’t know what sort of environment it’s been in. FS.com sell compatible optics for £19 with a 5 year warranty.

We also purchased spare switches for a quick response to a failure. We could have doubled down and deployed two per facility, but didn’t think it was necessary. The clock devices only have a single port at the end of the day, and SAN replication can catch up.

That’s it. I just thought I’d share one of those little last minute projects that needs to be delivered on a  tight budget.

Cisco Router Dual WAN Uplinks with NAT

Dual WAN uplinks for resilience are a common request when configuring small business routers. I’ll work with the topology below and go through the configuration.

Screenshot 2020-02-13 at 20.56.40.png

The only devices I’ll be going over are R1 and PC1. In the real world, the rest would be out of your control anyway. I’ll include the GNS3 project file at the end of this post if you would like to play with it.

Configuration

The first thing we need to do is configure the interfaces of the two ISP connections.

interface GigabitEthernet0/0
  description ISP1
  ip address 100.1.1.2 255.255.255.252
  ip nat outside
!
interface GigabitEthernet1/0
  description ISP2
  ip address 200.2.2.2 255.255.255.252
  ip nat outside
!

And the LAN interface

interface GigabitEthernet6/0
  description LAN
  ip address 192.168.1.1 255.255.255.0
  ip nat inside
!

Next we’ll configure our routes via both ISP’s. To make the failover work we need to track some objects on the primary connection. This will make failover occur if the internet connection goes down. I would advise you track reachability of a couple of hosts to avoid failing over if somebody else is having an issue. I would also advise against tracking the next hop, as an issue within the ISP network wouldn’t cause failover to occur but may prevent you from reaching the internet. We do this using ip sla to two different known hosts (I used Google’s public DNS servers for this demo).

ip sla 100
icmp-echo 8.8.8.8 source-interface GigabitEthernet0/0
threshold 1000
timeout 1000
frequency 10
ip sla schedule 100 life forever start-time now
ip sla 101
icmp-echo 8.8.4.4 source-interface GigabitEthernet0/0
threshold 1000
timeout 1000
frequency 10
ip sla schedule 101 life forever start-time now

Next we create two track objects to monitor the ip sla’s for reachability.

track 100 ip sla 100 reachability
!
track 101 ip sla 101 reachability

Then a track object to track the first two objects. By using the boolean or option, the track will go down if all of the tracked objects go down but will not if only one goes down.

track 105 list boolean or
object 100
object 101

Now we will add our default routes via both ISP’s. We will use the track 105 object to determine if ISP1 is up and add it to the routing table. Otherwise we will add ISP2 with a metric of 10.

ip route 0.0.0.0 0.0.0.0 100.1.1.1 track 105
ip route 0.0.0.0 0.0.0.0 200.2.2.1 10

That should be the routing done, now we will need to configure NAT to allow the LAN clients to access the internet. We’ll create an access list to define the LAN traffic that should be translated.

ip access-list standard NAT-INSIDE
permit 192.168.1.0 0.0.0.255
!

And we’ll use some route-maps to match the LAN traffic on the outside interfaces for translation.

route-map RM-NAT-ISP2 permit 20
match ip address NAT-INSIDE
match interface GigabitEthernet1/0
!
route-map RM-NAT-ISP1 permit 10
match ip address NAT-INSIDE
match interface GigabitEthernet0/0
!

And finally, the NAT configuration commands.

ip nat inside source route-map RM-NAT-ISP1 interface GigabitEthernet0/0 overload
ip nat inside source route-map RM-NAT-ISP2 interface GigabitEthernet1/0 overload

Verification

Now we can do some testing. If we ping from PC1 to 8.8.8.8, the ping should succeed. You can also perform a traceroute from PC1 to 8.8.8.8 to verify the route the traffic flows.

PC1#ping 8.8.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 48/85/196 ms

PC1#traceroute 8.8.8.8
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.1.1 16 msec 48 msec 28 msec
2 100.1.1.1 104 msec 72 msec 44 msec
3 1.2.1.1 56 msec 60 msec 68 msec

You can use show ip nat translations on R1 to verify the NAT translations over ISP1.

R1#show ip nat translations
Pro Inside globalInside local Outside localOutside global
icmp 100.1.1.2:1025192.168.1.11:168.8.8.8:16 8.8.8.8:1025
udp 100.1.1.2:4501 192.168.1.11:49157 8.8.8.8:334378.8.8.8:33437
udp 100.1.1.2:4502 192.168.1.11:49158 8.8.8.8:334388.8.8.8:33438
udp 100.1.1.2:4503 192.168.1.11:49159 8.8.8.8:334398.8.8.8:33439
udp 100.1.1.2:4504 192.168.1.11:49160 8.8.8.8:334408.8.8.8:33440
udp 100.1.1.2:4505 192.168.1.11:49161 8.8.8.8:334418.8.8.8:33441
udp 100.1.1.2:4506 192.168.1.11:49162 8.8.8.8:334428.8.8.8:33442

And also, show ip route on R1 should show the next hop as ISP1.

S*    0.0.0.0/0 [1/0] via 100.1.1.1

Now, if we suspend a link connected to the ISP1 route, it doesn’t matter which one, our topology should failover.

Screenshot 2020-02-13 at 21.44.38.png

First thing you should notice is the track objects going down on R1.

*Feb 13 21:44:28.847: %TRACKING-5-STATE: 100 ip sla 100 reachability Up->Down
*Feb 13 21:44:28.851: %TRACKING-5-STATE: 101 ip sla 101 reachability Up->Down
*Feb 13 21:44:29.835: %TRACKING-5-STATE: 105 list boolean or Up->Down

And the default route on R1 should have changed to ISP2.

S*0.0.0.0/0 [10/0] via 200.2.2.1

And if we repeat the same ping and traceroute from PC1 to 8.8.8.8, they should still work fine, but the route should show ISP2 as the second hop instead of ISP1.

PC1#ping 8.8.8.8
Type escape sequence to abort.
Sending 5, 100-byte ICMP Echos to 8.8.8.8, timeout is 2 seconds:
!!!!!
Success rate is 100 percent (5/5), round-trip min/avg/max = 28/56/84 ms

PC1#traceroute 8.8.8.8
Type escape sequence to abort.
Tracing the route to 8.8.8.8
VRF info: (vrf in name/id, vrf out name/id)
1 192.168.1.1 12 msec 88 msec 24 msec
2 200.2.2.1 52 msec 32 msec 36 msec
3 1.2.2.1 64 msec 64 msec 36 msec

show ip nat translations on R1 should also show the NAT translations to ISP2 now.

R1#show ip nat translations
Pro Inside globalInside local Outside localOutside global
icmp 200.2.2.2:1024192.168.1.11:2 8.8.8.8:28.8.8.8:1024
udp 200.2.2.2:4501 192.168.1.11:49167 8.8.8.8:334378.8.8.8:33437
udp 200.2.2.2:4502 192.168.1.11:49168 8.8.8.8:334388.8.8.8:33438
udp 200.2.2.2:4503 192.168.1.11:49169 8.8.8.8:334398.8.8.8:33439
udp 200.2.2.2:4504 192.168.1.11:49170 8.8.8.8:334408.8.8.8:33440
udp 200.2.2.2:4505 192.168.1.11:49171 8.8.8.8:334418.8.8.8:33441
udp 200.2.2.2:4506 192.168.1.11:49172 8.8.8.8:334428.8.8.8:33442

Now if we resume the link to the ISP1 route, the track objects will come back up and everything should fail back.

*Feb 13 21:50:43.871: %TRACKING-5-STATE: 100 ip sla 100 reachability Down->Up
*Feb 13 21:50:43.871: %TRACKING-5-STATE: 101 ip sla 101 reachability Down->Up
*Feb 13 21:50:44.839: %TRACKING-5-STATE: 105 list boolean or Down->Up

Notes

There are a couple of things to note with this configuration:

  • There is zone-based firewall configuration in this demo. I highly recommend one if you aren’t using a dedicated firewall.
  • Inbound connections using port forwarding and the primary connection IP address will not failover if ISP1 fails.
  • The IOS image used in this GNS project is c7200-advipservicesk9-mz.152-4.S5.bin. You’ll have to provide that yourself.

Downloads

GNS3 Project – Dual WAN

Playing with interface templates in Cisco IOS XE

I recently learned of the existence of interface templates. Previously I had been using int range a lot to make mass updates to our access ports, but this wasn’t really scaling well and took ages to do on some of our 7 switch stacks.

For anybody who hasn’t used interface templates before, they are a mechanism to configure once and apply to many interfaces. Not only is this convenient, it reduces the size of your configuration quite a bit.

We built a number of templates for use in our network, one for each vlan we assign ports to. Since we standardised the access vlans across all switches (e.g. vlan 100 is always for users, vlan 200 is always for voice, vlan 600 is always for guest access etc), this makes all the templates across all switches identical.

What goes into a template? pretty much any configuration you can apply to an interface, you can put in a template. Heres an example of a template for a user port.

template USER
 storm-control broadcast level pps 1k
 storm-control action shutdown
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
 switchport access vlan 100
 switchport mode access
 switchport voice vlan 200
 switchport port-security maximum 2
 switchport port-security violation protect
 switchport port-security aging time 3
 switchport port-security
 service-policy input MARKING
 service-policy output 2P6Q3T-WRED
 description USER
!

And then to apply it to an interface:

interface Gi1/0/25
 source template USER
!

Thats literally all of the configuration you need on an interface. The problem now is, if you do a show run on an interface, all you get is “source template USER”. There is a different show command to solve that problem.

SW1#show derived-config interface gi1/0/25
Building configuration...

Derived configuration : 501 bytes
!
interface GigabitEthernet1/0/25
 description USER
 switchport access vlan 100
 switchport mode access
 switchport voice vlan 200
 switchport port-security maximum 2
 switchport port-security violation protect
 switchport port-security aging time 3
 switchport port-security
 storm-control broadcast level pps 1k
 storm-control action shutdown
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
 service-policy input MARKING
 service-policy output 2P6Q3T-WRED
end

You can also mix and match on an interface, and have configuration directly on the interface. The the interface configuration will override the template configuration. It still looks a bit weird in show run though.

template USER
 storm-control broadcast level pps 1k
 storm-control action shutdown
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
 switchport access vlan 100
 switchport mode access
 switchport voice vlan 200
 switchport port-security maximum 2
 switchport port-security violation protect
 switchport port-security aging time 3
 switchport port-security
 service-policy input MARKING
 service-policy output 2P6Q3T-WRED
 description USER
!
interface Gi1/0/25
 description Marketing Laptop
 source template USER
!

SW1#show run int gi1/0/25
Building configuration...

Current configuration : 92 bytes
!
interface GigabitEthernet1/0/25
 description Marketing Laptop
 source template USER
end

SW1#show derived-config interface gi1/0/25
Building configuration...

Derived configuration : 506 bytes
!
interface GigabitEthernet1/0/25
 description Marketing Laptop
 switchport access vlan 100
 switchport mode access
 switchport voice vlan 200
 switchport port-security maximum 2
 switchport port-security violation protect
 switchport port-security aging time 3
 switchport port-security
 storm-control broadcast level pps 1k
 storm-control action shutdown
 spanning-tree portfast
 spanning-tree bpduguard enable
 spanning-tree guard root
 service-policy input MARKING
 service-policy output 2P6Q3T-WRED
end

I found interface templates really useful during a recent network upgrade. It wasn’t too long ago I became CCNA certified and these weren’t even mentioned on the courses, which is a shame.

These templates are configured on 9200L switches, if you are interested.

Nest Cam Outdoor – My experiences

So a couple of months ago some bastard keyed my car. My initial knee jerk reaction was to buy a CCTV camera to watch over it in case it happened again, so I knew who’s knees to break. So the search began, but I wanted something fast.

I looked at a couple of offerings but I wanted something easy and quick to install, and preferably without too many cables to run around the house. I settled in the Nest Cam Outdoor, after owning their thermostat for a year and being very pleased with its performance and functionality.

When the camera arrived it was very well packaged with the usual plethora of mounting hardware. The kit comes with long enough cables for most installations and all the required fixing hardware, including cable clips, which was nice.

The mounting system for the camera uses strong magnets to hold it in place. While this is a novel idea, in practice a well-aimed football will knock the camera off the wall and render it useless. Points lost there unfortunately.

The cable attached to the nest has a ruggedized USB connector on it that’s approximately 18mm diameter. All well and good until you need to run it through a wall. I know this camera is meant for American homes which I presume mostly have outdoor power outlets, but that’s an uncommon facility in the UK. Drilling an 18mm+ hole in a double skinned brick wall is no simple task. More points lost there.

Once installed the camera is easy to set up using the Nest iOS or Android apps. Scan the bar code on the app and away you go. Don’t forget you’ll need a constant internet connection for the camera to work though.

The video quality of the camera even in daylight is shit considering it is allegedly 1080p. While the camera sensor probably is 1080p, the camera compressed the nuts off the video to upload it to Google’s cloud platform, losing most of the quality. I honestly can’t even read the number plate of my car from the video, and the car is parked maybe 15 feet from the camera, if that. Image quality is even worse at night.

Remember Google’s server I just mentioned? They only allow the camera to store 3 hours of footage, unless you buy a Nest Aware subscription to store up to 10 days, or 30 days, whichever tickles your fancy. This isn’t particularly well communicated by Nest either, to the point that when my “Free Trial” expired, I didn’t even get a notification to say “Hey, your camera has just deleted all video over three hours old! Best break the credit card out!” Thinking about this logically, if somebody broke into my car between 1am and 3am, I wouldn’t have any footage of it at all. What’s the point?

In conclusion, don’t buy this camera. It’s effectively a glorified, very expensive, video doorbell without a subscription, which wouldn’t be as bad if the cost of the camera wasn’t so fucking much to start with. Even then, you can’t make out facial features unless the camera is installed at a level where the magnets make it easy to sabotage. Again, what’s the point?

Honestly. Don’t waste your money. This device has the potential to be awesome, but is used as a cash cow by Nest instead.

Meters Music OV-1 – My Opinion

For anybody that has been living under a rock since CES 2017, there’s a new kid on the block when it comes to prosumer headphones. The Meters Music OV-1 by Ashdown Engineering.

I recently purchased a set of these headphones after watching countless interviews with Ashdown at CES, and reading some reviews of the headphones by tech websites. The reviews weren’t all 100% but that’s expected in this day and age, and I tend to listen to different styles of music than the reviewers, so I decided to give them a try all the same.

Packaging

Opening the packaging wasn’t awe-inspiring like some high-end products I receive, but it was a pleasant experience all the same. The inclusion of two cables, one including an inline mic and one not, was a nice touch and they were decently built cables. The included Micro USB for charging on the other hand was just some generic cable you can probably buy from eBay for a few pence.

The headphones themselves were in the included hard case but lacked any additional packaging. The inclusion of some sticky protective plastic on the alloy cups would have been nice, and would have probably prevented the small scuff mark on the left cup. It looks like the box might have been dropped in transit and the holder for the cup has hit the cup. It’s hard to see under normal use, but I know it’s there!

OV-1 Scuff mark on cup.jpg

Aesthetic & Build Quality

The headphones are pleasing on the eye with good industrial design, and the inclusion of the VU line meters on the ear cups is cool but a bit of a gimmick. The overall look of the sweeping, single mount cup holder, coupled with the stylish, albeit plastic, hinge mechanism is a well designed and engineered look for these alloy headphones. The transition from alloy to protein leather, with its neatly stitched seems, is also well executed.

The biggest let down from an aesthetic perspective is the use of black plastic on the hinge and hangar mechanism. even just colour coding the plastic to the alloy would have served their vision better in my opinion.

The headphones feel well constructed though and I can’t really knock the build quality other than the slightly over tightened hex screw holding the left ear cup in, which I had to file a couple of burrs off to appease my OCD.

IMG_1907.jpg

A bit more consideration of the positioning of the three-way switch (EQ, Passive, Active Noise Cancelling) would have been nice. It is currently located under the ear cup holder, meaning you have to remove the headphones to change the mode. I feel it would have made the experience just that little bit more pleasing if it had been somewhere a little easier to reach during use.

Comfort

From a comfort perspective, they take a bit of adjustment and getting used to, but after a couple of days you barely even notice they are on your head. During the initial honeymoon period where they were on and off my head every few minutes for adjustment, the clamping force of the sprung steel headband was a bit overwhelming and didn’t ease any until I had forced the headband apart a few times over the course of a day.

The headband also applied pressure to the top of my head until I found the correct hight for the height adjustable cup sliders. Unfortunately the mechanism on the sliders is too week and the cups tend to lose their position whenever I stretch the headphones enough to put them over my head. This is annoying and needs to be fixed in any future release.

IMG_1906.jpg

To give Meters their due, the fake “protein” leather on the ear cups is comfortable, but isn’t sweat proof as claimed. However the sweat is a lot less noticeable to the point it didn’t bother me like it does on other headphones.

Noise Cancellation

In the interest of full disclosure, I am yet to try the noise cancelling feature (ANS) on any mode of transport, which is the main use case for such technology. I have, however, tried it out in some server rooms where the constant drone of server fans are drowned out to the point of being almost silent while using the headphones without an audio source connected. The slight hiss of the ANS system doing it’s thing is just audible though, until I start playing some music, of course.

All in all I’m happy with the ANS system and would even stretch to call it the best I have used yet, although I’ve probably only experienced a total of five other noise cancelling systems in the past.

Sound Quality

I’m no audiophile, but I do appreciate great sounding music. I also listen to a wide variety of music so I’ve put the headphones though their paces with a few genres, from Heavy Metal to UK Hardcore and everything in between.

The headphones sound great and have decent volume when listening in passive mode (or off). They offer crisp high ends, prominent mid range and deep bass. Switching the ANS on reduces volume somewhat, and dampens the high ends a little but still maintains a decent sound quality with adequate volume.

EQ mode, the final sound option, sounds terrible. It’s pointless beating around the bush, the bottom end is far too overpowering and crushes the high-end. I can’t find a single song that sounds good while using the mode so I just don’t use it. I seriously wish the headphones allowed the user to tune and tweak the EQ setting using the USB connection.

Summary

All things considered, I like the OV-1 headphones. They are the best sounding headphones I have used (apart from the EQ setting) and they are extremely comfortable for prolonged use at work. As a developer I often spend hours on end listening to music to drown out the bustle of the office. They also give a bit more depth to podcasts and significantly improve the usually mediocre sound quality of them.

I’m still not sure If I would wear them in public though. Maybe if Meters allowed you to control whether the VU meters were on or off independently of the noise cancelling feature I might be more inclined to, but as they stand, they could look a bit obnoxious in certain environments.

In conclusion, if you are an audiophile, don’t bother. If you are a run of the mill prosumer who likes gadgets and great sounding music, then these headphones are good bang for your buck. Plus they don’t feed more of you hard-earned cash into Apple’s coffers.