Page 1 of 1

Site shedding subcribers

Posted: Sun Feb 19, 2017 7:03 am
by Bill_G
This involves a Cassidian P25 phase 1 multi-site trunking system. Very intermittently, one site will suddenly shed two thirds of its users to the other sites for about 2 minutes, and then start picking them back up again like nothing ever happened.

From the technical perspective, no alarms are thrown when this happens. Nothing from the base stations, nothing from the site controllers, nothing from the central controller. All the machinery is happy. I have verified that during the events, the transmitter stays keyed, that the power level remains constant, that RSL at far points remains constant, and that the OSW is always present.

From the dispatch perspective, it is transparent. They don't know it happened until users call in afterwards.

From the user perspective, their mobile / portable suddenly starts beeping, and displaying NO SERVICE for a few seconds. That is followed by a reaffilliation beep, and a new site name. Sometimes it can take over a half minute for individual subscribers to reaffilliate because of the hundreds involved.

The things I have done to isolate this are - swap everything in the site transmit path with no change in the problem. We've changed the control channel base station. We've changed the controller. We've changed ports on the combiner. We've changed to different antennas and mainlines. None of that has changed the frequency or magnitude of the problem. We've pulled error logs from subscribers immediately after an event. I have had a SDR running with DSD+ decoding the control channel. My next step is to find something better than DSD+ to decode and display every byte sent out.

So far, we have no breadcrumbs leading us to the culprit. I'm open to ideas.

Thank you

Re: Site shedding subcribers

Posted: Tue Feb 21, 2017 1:41 pm
by com501
Look at the controller buffer. This might be an issue where the controller overflowed one of the buffers and shed load for a restart. Make sure all your versions match, system wide.

Re: Site shedding subcribers

Posted: Tue Feb 21, 2017 3:49 pm
by Bill_G
I have seen buffer overload messages. I'll pass it to the engineers. Thanks!

PS - we have not seen any controller restarts, but they may be able to find process restarts that I haven't.

Re: Site shedding subcribers

Posted: Tue Feb 21, 2017 4:50 pm
by com501
The software may not log a restart if it is a glitch in the compiler, either. Send them all the logs and make sure they name the next 'upgrade' after you.

Re: Site shedding subcribers

Posted: Tue Feb 21, 2017 7:23 pm
by Bill_G
Ah yes. Bask in the glow of my name immortalized by AirBus. Next stop, Hollywood.

Re: Site shedding subcribers

Posted: Fri Mar 03, 2017 10:34 am
by KE7JFF
Interesting Bill...this explains why users of said system I know of were telling me "It gets angry sometimes"

DSD is pretty decent, however you might to look at using OP25 with its detailed P25 abilities; but that requires you to setup a Linux box.

Re: Site shedding subcribers

Posted: Sat Mar 04, 2017 5:16 am
by Bill_G
Thanks. Had not run across that while searching for SDRSharp apps. DSD+ is fine for listening to voice, but it's decode of the OSW is scant.

Re: Site shedding subcribers

Posted: Mon Mar 06, 2017 6:51 pm
by Bill_G
A quick update - with multiple mobiles in various locations around the city logging the RSSI and QOS (amount of error correction being applied), we can say that the transmitter is neither dropping out nor corrupting the OSW. Power remains constant, and the error rate remains less than 1e6. Likewise, we are not seeing a rogue transmitter coming up to interfere.

Re: Site shedding subcribers

Posted: Tue Mar 07, 2017 1:56 pm
by wavetar
What are you running for subscribers? Motorola has a "Radio Xtractor" software that can be used with XTS/XTL/APX radios to capture internal data processed during the event that could be helpful. It's available on MOL...mine, at least.

Have you tried 'site locking' a subscriber to the site to see if it still goes no service?

Re: Site shedding subcribers

Posted: Tue Mar 07, 2017 7:35 pm
by Bill_G
This is a Tait system, and yes - I've tried site locking - no joy.

Another event today duplicating the results we have so far - no change in rsl, and the OSW never stops. I truly believe the subscribers are being booted off. But, Tait says no such command exists. They are still chasing after an external interference that somehow evades detection by measuring rsl and logging the OSW. What I want next is a full literal decode of the OSW.

I think we are experiencing something similar to my BBS days with dialup modems. This one file would fail to download over my Supra modem. The modem would drop out at the same amount of time each attempt, but successfully passed through my USR. We recompressed the file from zip to lha, and it passed through the Supra. Something in the zip compression made a string the forced the Supra into command mode, and then hang up. I think that is what is happening. A string of characters in the OSW from this site is faking out the subscribers making them think the site is gone.

Interesting tidbit from Tait about their system - we have probably identified the two minute thing - it's the subscriber's internal blacklist. They won't reveal the complete set of rules to me, but if a site qualifies the blacklist, it stays there for two minutes before it will be looked at again.

Re: Site shedding subcribers

Posted: Wed Mar 08, 2017 8:53 am
by wavetar
Ask TAIT if they have something similar to the Radio Xtractor software to interface to their subscribers. If they do, they should be able to see exactly what is causing the radio to decide the site is gone.

Re: Site shedding subcribers

Posted: Fri Mar 10, 2017 4:35 am
by Bill_G
As a matter of fact, they did just that this week. It's a lab tool v0.4.0, but it can decode the entire OSW using a mobile as the receiver. It can also log in real time to a flat file for later "playback", or line by line inspection. Those logs get huge fast - about 8Mb an hour. We managed to capture several events, but the results are inconclusive - nothing is jumping out at us. You can get blurry eyed very quickly working with this tool.

Re: Site shedding subcribers

Posted: Wed Mar 15, 2017 6:23 pm
by Bill_G
After a week's drought of these shedding events, we enjoyed several in a row in the early afternoon today. The logs were dutifully forwarded to Tait for analysis, and they hinted at a Eureka! just a bit ago.

Waiting with great anticipation.

Re: Site shedding subcribers

Posted: Thu Mar 16, 2017 2:29 am
by wavetar
Bill_G wrote:Waiting with great anticipation.
As am I. We also have a Cassidian based system with TAIT subscribers in another Province, this resolution will be good to know in case we come across the issue.

Re: Site shedding subcribers

Posted: Thu Mar 16, 2017 11:44 am
by Bill_G
Tait has been able to reproduce it in the lab. It appears that when an emergency input is detected while a mobile is turned off, unregistered and unaffiliated, it will covertly still attempt to send in the emergency. How that causes the system to deregister other subscribers is still TBD, but it may have something to do with processing a priority message from an "unknown" unit. IE: It's making time for the unit to fully affiliate and register before processing other units.

Re: Site shedding subcribers

Posted: Fri Mar 17, 2017 2:45 am
by Bill_G
.... and they've opened a case with Airbus.
We shall see.

Re: Site shedding subcribers

Posted: Fri Mar 17, 2017 11:21 am
by wavetar
Bill_G wrote:.... and they've opened a case with Airbus.
We shall see.
You gotta love "open architecture" systems!

Re: Site shedding subcribers

Posted: Fri Mar 24, 2017 9:42 am
by Bill_G
A quick update - based on the lab results, we've been able to recreate the conditions in the field on the live system. This forces it back to their engineering dept to come up with a solution.