slazer2au

When using new wireless kit, never assume the vendor knows what they are doing, most of the time they do not know what the local laws regarding wireless equipment even are. We have some vendors ignore standards while others follow the standard so closely the kit becomes unusable.

We installed a new 900Mhz radio to a customer who was in a particular bad spot. All seem well, the customer was getting the speed over the wireless and the latency was rather good.

A few weeks after install I get a call from the customer.

ring ring

Me: G'day slazer speaking.
Cus: Hi, this is [manager] calling from [customer] we have a guy here saying the radio on our roof is interfering with [national mobile carrier] in the area.
Me: Ooook, that doesn't sound good. Can I talk with him?
Cus: Sure. I'll shoot the call down to reception where he is.

call transfer

Me: G'day this is Slazer, we run the kit on the roof, what is the issue?
CarrierTech: This is CarrierTech from [contracting firm] we have been sent out by [national carrier] to find out why their customers are experiencing call problems in this area.
Me: I see, is [Cus] still hanging around?
CarrierTech: Yes,
Me: Sweet, I need to have a quick word with him and we can sort this out.

Phone passed back to Cus

Me: Hi mate, Thanks for calling us. We will handle everything from here and you wont have to do anything.
Cus: Ok, sounds good, I will pass you back to CarrierTech

Phone ping pong finishes.

Me: Right mate, lets get this sorted. What are you seeing and how can we resolve it.
CarrierTech: I noticed the radio on this roof and our kit is saying it is running in the 900Mhz band. What brand and model is the radio?
Me: It is a Ubiquiti Nanobridge M900.
CarrierTech: Is the firmware up to date and you are running in the Australian country code?
Me: Yes.
CarrierTech: Ok, so it looks like it currently isn't complying with Aussie rules because it is sitting in the middle of the 900Mhz band assigned to [national carrier].
Me: Not good, What is there band?
CarrierTech: [freq band]
Me: Yea, we are sitting in the middle of that, luckily this is a backup link so I can mess with it during business hours. Let me lock out those frequencies and reboot the unit.

few min later

Me: Ok, I have gone as far away as I can from their band, how is it looking?
CarrierTech: I will have to check from outside. Can I have a number I can call you back on?
Me: sure, [insert company number]
CarrierTech: OK, I will call back a little later.

About 20 min later he calls back.

CarrierTech: It looks like that has cleared up the problem. Where does this link go back to?
Me: [insert address from city 10Km away]
CarrierTech: sigh I spent the entire day there yesterday chasing down the same problem and narrowed it down to that street. I should of started at this end.
Me: Well, my apologies mate, I will have to get in touch with the vendor and get this fixed for the next firmware release.
CarrierTech: Yes. I am sure [National Carrier] will also push them and the ACMA about it.
Me: On that note. I assume because the problem is fixed we won't be getting a call from them?
CarrierTech: No, if they complained to the ACMA it would be 6 months before they could do anything about it.
Me: Sounds about right for a government department, just out of curiously how many sites were affected by this?
CarrierTech: About 20 to 30 sites.
Me: wow, now I am really glad you called us first.

insert ending formalities

/End call

I let the boss know what happened and he was glad how it worked out.

Last time we had a run in with the ACMA it ended badly for them, but that is another tale for another time.

cross-posted from: https://lemmy.world/post/19180701

My incident over ~~2~~ 9 years ago involves the federal regulator making impossible claims.

Working in the wonderful world of Wireless Internet Service Provider (WISPs), you get those calls once in a blue moon that makes you question everything.

phone rings

Me: G'day, this is slazer.
Caller: Hi, this is Fred calling from the ACMA (the Aussie version of the FCC). Can I talk to your senior radio engineer please.
Me: We don't have one, but I am the senior network engineer. I will do what I can do help.
Fred: Ok, I am at [site] and we are detecting some interference on the local council 80Mhz band and we believe your equipment is responsible.
Me: I am sorry, run that by me again.
Fred: We believe the equipment operated by your company on [site] is interfering with the local councils 80Mhz emergency push to talk system.
Me: Ooook. That sounds impossible our equipment is running at 5Ghz. How did you get to that conclusion?
Fred: Well, we have shut down all the other wireless operators on the tower but the interference is still there. In your cabinet there is what looks like an amp which takes up about the bottom 6RU. Would you be able to turn that off?
Me: We don't have an amp in our cabinet. That is our UPS in case there is a power outage.
Fred: A UPS? That explains why your equipment didn't go down when we turned off your breaker.
Me: It also kept beeping at you till you turned the power back on didn't it?
Fred: Yes. So is there a way we can turn your kit off so we can finish our tests?
Me: Not at this time of the day. We have clients actively using the service.
Fred: Ok, I will run some more tests and get back to you.

/call

I take down his number in case he calls back and let my minions know that if he calls put him directly though to me. I call our vendor rep, just to make sure I am correct.

Vendor: Hello this is (dude) from (vendor)
Me: G'day , it is slazer from (WISP). Do you have some time to chat, I just got off the phone with the ACMA.
Vendor: Oh boy, whats up?
Me: Well one of the ACMA "engineers" have said the kit we have installed is interfering with an 80Mhz push to talk system.
Vendor: That doesn't sound possible. If it were possible, we would have people all over the world complaining.
me: I know, just doing a sanity check. I will let you know if it turns out to be your stuff, which I doubt.
Vendor: No worries mate, thanks.

/call

I also call the boss and let him know what is going on. He has the same mind set as the vendor, impossible for us to interfere with an 80Mhz system.

A couple hours pass and he calls back.

Me: g'day mate, how did you go?
Fred: You have a radio pointed between 50 and 60 degrees off the tower, I think that is responsible for the problem.

I look up the radio in question and it is a 5.4Ghz radio.

Me: That can't be. It is a 5Ghz radio.
Fred: can you turn it off so see if the interference goes away?
Me: Like I said before I can't turn off any of our radios unexpectedly during the day, that particular radio goes to the school in [suburb].
Fred: Hmm, when can we turn it off to test?
Me: provided the school is OK with the outage, 2 weeks from now at 3AM.
Fred: Your shitting me?
Me: No, part of the contact we have with the school says we have to give 2 weeks notice for any planed maintenance that could impact their service.
Fred: But why 3AM?
Me: Because that is the time when it will disrupt the schools service the least.
Fred: There has to be a better time then 3AM.
Me: Not really, the schools nightly backup goes from 8PM till 2AM.
Fred: Seriously?
Me: Yes. I will call the school now and organise the outage. I will give you a call back when I have confirmed everything.

/call

I organised the outage with the customer and kept everyone in the loop.

Outage window came along and I got a call from Fred.

Fred: How far off are you?
Me: I am ready to go.
Fred: Eh? Aren't you meeting us here?
Me: No, why spend 2 hours travelling up there at night when I can do it from the comfort of my home?
Fred: OK, well lets get started.
I turn off all the radios except the the one I am using to log into the site via.
Me: They are all off except one, how is it looking?
Fred: Still seeing the interference. When you say they are off, I am still seeing the same amount of lights on your gear in the hut.
Me: I have turned off the radio unit on the outdoor unit. So at the moment all our radios bar one are not transmitting.
Fred: Which one is on?
Me: Our backhaul, if I turn it off I wont be able to turn it back on remotely. What I can do is bounce it. Have are you looking at your kit?
Fred: Yes.

I reboot the final backhaul radio.

Me: OK, you have about 2 min before it comes back online. How is it looking?
Fred: No different...... What in the world is causing this interference.
Me: No clue mate, we operate in the 5Ghz band. Seeing as you haven't found anything I am going to turn our kit back on now.
Fred: but we haven't finished testing yet.
Me: Yes we have, all our kit was off and you said there was no difference in the interference.
Fred: It must be your kit. It is the only unlicensed kit in the area. Everyone else is using licensed spectrum.
Me: ............. I would ask how you came to the conclusion of they don't use licensed spectrum so they must be the problem, but it is 3AM and I would like to go back to bed.
Fred: But we aren't done yet.
Me: Yes, we are. Good night.

/call

I turn on our equipment again and write up a report for the boss, then return to bed.

A couple days later, we received a warning notice from the ACMA about the events that transpired. Sadly, this is where my part in the story ends and the boss picks it up.

After several back and forth between the boss, our lawyers, and the ACMA rep. The warning is withdrawn and the 80Mhz kit gets moved to another tower a couple hundred meters down the road only to run into the same interference problem.

I don't know if they ever fixed the problem, it has been a few years and it doesn't bother me.

These stories are originally posted over the past decade on Reddits TalesfromTechSupport so I am copying over to Lemmy to help bring some life into this /c/

Sigh, I had one of those Mondays. As per the rules all names are replaced to protect the identity of the stupid and ill informed.

Some auzzie slang/humour may come off as offensive, I apologise, its just how we roll in the land down under

Back story, I work for a fixed wireless ISP. I deal anywhere between integration firms and the onsite IT bloke. This particular incident took place at the HQ of a multi site medical center group

Get a call at 6:30AM

Me: G'day slazer speaking.
Customer IT guy (Lets call him Steve): Hi mate, its Steve from Medical Group our head office is offline at the moment. We had a really bad storm go though last night, it may just be power but can you guys be on stand by just in case?
Me: nyaaa, all right. I'll do my usual morning stuff and get into the office asap. Can you check out HQ and let me know?
Steve: no worries mate.

2 min later

imessage from the boss: slazer, the HQ of Medical Group is down. whats going on.
imessage to the boss: Just got off the phone with their IT bloke and he is going in to checking power. I'll get to the office early and prep our spare radios.

no reply from the boss.
[insert usual morning stuff of shower, shave, and shi....]
While driving to the office I get another call from Steve

Me: G'day Steve, how is it looking on your end?
Steve: Well, we have lost a UPS and a switch to last nights storm, you may of lost your radio though, there is no up-link light on your Cisco NTU.
Me: bugger, I guess you have tried power cycling it?
Steve: Yea, the light is on the power injector but no light on the NTU. Our sparkie (aussie slang for an electrician) is coming in to check everything else is OK, I'll get him to check your cable too.
Me: Cheers mate, I'll get a spare radio configured and head straight up to you.

3 accidents on the motorway D: a normal 45 min trip takes 2 hours but I get there eventually.

CIT: You took your time mate.
Me: Traffics a female dog.
CIT: Fair call, the sparkie had a look at the run from the server room to the radio on the roof, he said everything is fine. where do you want to start?
Me: Well lets make sure the POE injector is OK first.

We head to the server room and I notice there is no light on the POE injector. I do the usual troubleshooting and the light on the POE will only stay on while the cable to the radio is not plugged in. I check the injector by plugging in the replacement radio, lights stays on and the radio turns on and starts squawking while it searches for a base station to connect to. The port on the NTU also comes on ruling out the POE and NTU as the cause of the fault.

Me: Well the problem is not down here. Lets go for a sticky beak on the roof.
just as i finish saying the sentence, the sparkie appears out of nowhere.
Sparkie: Everything is fine on the roof, I have checked the cable and the radio is powered up
Me: ......... its not that I don't believe you, its just that..... no bugger it, I don't believe you.
Sparkie: hmmf

the sparkie walks off.

Steve: Little rude there mate? Me: Only because he lied.

Stevelooks confused

Me: By how the light was behaving on the injector, there is no way everything is fine.
Steve: Fair enough mate, let me know what you find.

He goes back to checking the servers and I head up to the roof alone. Once I get onto the roof I notice there is no light on the bottom of the radio...

I remove the waterproof bung and saw the rj45 head had been...... I don't have a word that will get passed the profanity filter for how the head looked.

Now, I have seen RJ45 heads shorted before from either over voltage (doing 54v to a 24v device) or water getting into the bung but nothing this bad.

It takes me a moment to collect myself and I begin repairing the cable. YAY for service loops!!! I install the replacement radio and get off the roof to make sure the customer is back online.

Warning: PUT YOUR DRINKS DOWN BEFORE OPENING THE PICTURE

I find the Steve in his "office" (read cubby hole)

Steve: back online are we? Good, What was the problem.
Me: May wana get the sparkie in for this.

Stevelooks confused, but pages him to his "office".

Sparkie: Whats up?
Me: When you said you checked the cable, what did you do?
Sparkie: I put a RJ45 tester on both ends and it tested OK.
Me: Again, I do not believe you. Tell me, how did you "test" this?

I gave both Steve and the sparkie a moment to collect their jaws from the table.

Steve: You can go slazer, thanks for getting the connection working. May I keep that head?
Me: Sure mate, I have a pic, that is all we require.

I am not sure if I will find out what happens with that sparkie, but I doubt I want to. On the bright side, because I had to travel before 7AM the company paid for my breakfast :D

To those of you who saw the pic before my warning of putting your drinks down, I am sorry. For those of you who blandly ignored it.... well, I am still sorry, but you were warned.

Update Time

So it turns out the sparkie vocabulary is smaller than both myself and Steve thought most sparkies have. When he was told to check the cable going to the radio on the roof he thought they were talking about the Wifi Access Point on the 3th floor.

His reasoning: Because ground, 1st and 2nd have floors above them they have ceilings. 3rd floor is the top floor so it is not a ceiling, it is a roof...... I'll let that logic sink in for the rest of you too.

This is a more recent story while working for an MSP in Europe compared to my time working for an ISP in Australia

the cast:
Me: Slazer
OT: Other Tech

I get a message on slack

OT: Hey, I am seeing something weird in the French office for customer, can you help me look into it?
Me: Sure

Queue the Teams call.

OT: So all the Access Points in that office are reported as offline in cloud.vendor.com portal but the customer is not reporting an issue.
Me: Ok, that is odd. What is the monitoring system saying?
OT: Monitoring says everything is OK, I can ping them and do SNMP calls to all the AP, they are just reporting as offline in the portal.
OT: The other thing is the firewall says the AP are trying to access cloud.vendor.com but the local in policy is denying the traffic.
Me: That is rather strange.

I log into the firewall and check the logs and see the APs are in fact trying to access cloud.vendor.com but the destination is 255.255.255.255. Not the expected IP from the vendors documentation.

Me: Well I want to say it's a DNS issue what happens when you reboot the AP?
OT: Rebooting from the portal doesn't work but I rebooted on from the switchport and the same thing happens.
Me: Is the on prem DNS server working?
OT: Yea, the domain controller is the DHCP/DNS server and it has no issue with access, the customer hasn't reported connection issues. It looks to be just the APs.
Me: Ok then, are they being allocated the right DNS servers?

OT logs into the domain controller and everything is looking good.

Me: dafuq?..... Wait, do these even use the DNS server from DHCP or do we set one via the device template?
OT: Not sure, never had this happen before. When we provision these they are plug and play.

I log into the vendor portal and start poking around and notice all the APs have the same DNS server of 208.67.222.222 (OpenDNS)

Me: Ok, well the AP aren't using the local DNS server they are using openDNS. Lets start a packet capture to see what is going on.

I setup a packet capture on the firewall and limit it to the IP of the AP we are looking at and let it run for a bit and crack open the capture in Wireshark.
I just start laughing at the error

OT: I know that laugh, what did you find?
Me: what do you make of this error?

Every single DNS query had this as the response.

The OpenDNS service is currently unavailable in France and some French territories due to a court order under Article L.333-10 of the French Sport code. See https://support.opendns.com/hc/en-us/

OT: Wha????
Me: Yea.... Now for the hard part.
OT: Hard part?
Me: How do we fix this? There is no ssh logins to the AP, we can't push config because the devices are offline according to the portal, and there is no way we are getting console to each of those units.
OT: I see.

Then the dumb idea occurred to me.

Me: I have a dumb idea. We DNAT any traffic destined for OpenDNS to Googles DNS so we can reconfigure the units to use the local DNS servers.
OT: Would that work?
Me: It should.... I hope.

We then setup DNAT for the AP specifically to rewrite the DNS request destined for OpenDNS and forward it to Googles DNS.
After activating the config we start seeing the devices come online in the portal as if nothing happened to them.

OT: Hey, it worked.
Me: omg, it actually worked...

I am somewhat sill shocked it worked.

At some point I will get some time to clean up that DNAT and finish reconfiguring the APs.

This is a repost of a story I posted on Reddit a few years ago.

Story participants
Me: Slazer
Boss: the boss
T1: Tech 1
T2 Tech 2

Backstory

The boss is all about redundancy and backup. If he finds a single point of failure that I have missed he lets us know and sets a time frame for when he wants it resolved along with a when the failover testing should be done. Because an untested backup is worse than no backup.

To spare the boring BGP details
We have 2 data centres in our closest state capitol. With transit multihomed transit through a single level 2 carrier (while not true multihomed we have transit of last resort through one of our layer 2 customers).

One day the boss arrives in the office around 10:30 AM after being in a huff about hearing of a major outage in a competitors network.

Boss: Slazer, did you get our traffic balanced over our 2 transit paths like we discussed a while ago?
Me: Yes, DC1 advertises prefix 1,3,5 and the aggregate. DC2 advertises prefixes 2,4,6 and the aggregate.
Boss: What happens when one of the transit fails?
Me: I am advertising the DC2 prefixes out DC1 with the backup BGP community. Then doing the same thing for DC1 prefixes over DC2. In the event of a transit failure the upstream has a backup path ready to go. Boss: and it works?
Me: Yes, last time I tested it was about 2 or 3 months ago and it failover over correctly.
Boss: Why haven't you tested it sooner?
Me: RANCID hasn't reported a configuration change since the last test. I only test it if there has been a config change on and of those routers.
Boss: But how can you be sure it still works?
Me: Shall I force a failover now to show it works?
Boss: Sure. (which I assume he said with sarcasm)

Me: Starts logging to DC1 core router

T1 seeing me do my configuration change face.

T1: If you are doing that I am going for a break.

I shutdown our transit interface for DC1 and wait for BGP to time out.
After about 10 min with no calls the boss turns around and continues the conversation.

Boss: So when will you be testing the failover?
Me: We are, right now.
Boss: What??!! as his face drops.
Me: You agreed. Plus this way now you know for sure it works because the phones haven't started ringing.
T2: Slazer is right. The graphs show how an increase in traffic on DC2 transit.

Boss slides over to T2 desk. Sure enough, the graph for DC1 transit is reading zero traffic and the graph for DC2 is showing all the transit traffic for the state.

Boss: That doesn't looks like much traffic.
Me: Only about 20-30% of our traffic goes via Transit, the rest goes via the various IXs we are on.
Boss: Who don't we get via the IX?
Me: Customers of our transit provider who aren't on any IX, Telstra and Optus as they aren't on any IX, and any international site that doesn't use a CDN.

We continue discussing for a good 20 - 30 min about where we get various traffic from and further redundancy in the core networks. During which time T1 returns from his break.

T1: Phones are quiet?
Me: Yes.
Boss: Can you turn the DC1 transit back on?

I walk back to my desk and turn the transit interface back on and see the BGP peer back on. While T2 and the boss are watching the graph for DC2 transit it drops about 2/3 of traffic and that appears back on DC1 transit.

And from that day the Boss hasn't asked about the transit failover because now he knows it works.

Quite an old joke.
I do miss my old Blackberry 9900