Site shedding subcribers

Moderator: Queue Moderator

Post Reply
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Site shedding subcribers

Post by Bill_G »

This involves a Cassidian P25 phase 1 multi-site trunking system. Very intermittently, one site will suddenly shed two thirds of its users to the other sites for about 2 minutes, and then start picking them back up again like nothing ever happened.

From the technical perspective, no alarms are thrown when this happens. Nothing from the base stations, nothing from the site controllers, nothing from the central controller. All the machinery is happy. I have verified that during the events, the transmitter stays keyed, that the power level remains constant, that RSL at far points remains constant, and that the OSW is always present.

From the dispatch perspective, it is transparent. They don't know it happened until users call in afterwards.

From the user perspective, their mobile / portable suddenly starts beeping, and displaying NO SERVICE for a few seconds. That is followed by a reaffilliation beep, and a new site name. Sometimes it can take over a half minute for individual subscribers to reaffilliate because of the hundreds involved.

The things I have done to isolate this are - swap everything in the site transmit path with no change in the problem. We've changed the control channel base station. We've changed the controller. We've changed ports on the combiner. We've changed to different antennas and mainlines. None of that has changed the frequency or magnitude of the problem. We've pulled error logs from subscribers immediately after an event. I have had a SDR running with DSD+ decoding the control channel. My next step is to find something better than DSD+ to decode and display every byte sent out.

So far, we have no breadcrumbs leading us to the culprit. I'm open to ideas.

Thank you
com501
Posts: 1088
Joined: Fri Nov 02, 2001 4:00 pm
What radios do you own?: Over 50 - All Motorola

Re: Site shedding subcribers

Post by com501 »

Look at the controller buffer. This might be an issue where the controller overflowed one of the buffers and shed load for a restart. Make sure all your versions match, system wide.
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

I have seen buffer overload messages. I'll pass it to the engineers. Thanks!

PS - we have not seen any controller restarts, but they may be able to find process restarts that I haven't.
com501
Posts: 1088
Joined: Fri Nov 02, 2001 4:00 pm
What radios do you own?: Over 50 - All Motorola

Re: Site shedding subcribers

Post by com501 »

The software may not log a restart if it is a glitch in the compiler, either. Send them all the logs and make sure they name the next 'upgrade' after you.
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

Ah yes. Bask in the glow of my name immortalized by AirBus. Next stop, Hollywood.
KE7JFF
Posts: 270
Joined: Sun Nov 02, 2008 7:27 pm
What radios do you own?: MX300 lunchbox

Re: Site shedding subcribers

Post by KE7JFF »

Interesting Bill...this explains why users of said system I know of were telling me "It gets angry sometimes"

DSD is pretty decent, however you might to look at using OP25 with its detailed P25 abilities; but that requires you to setup a Linux box.
Mountain Wave Search & Rescue http://www.mwave.org
Support Search & Rescue: Get Lost!
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

Thanks. Had not run across that while searching for SDRSharp apps. DSD+ is fine for listening to voice, but it's decode of the OSW is scant.
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

A quick update - with multiple mobiles in various locations around the city logging the RSSI and QOS (amount of error correction being applied), we can say that the transmitter is neither dropping out nor corrupting the OSW. Power remains constant, and the error rate remains less than 1e6. Likewise, we are not seeing a rogue transmitter coming up to interfere.
User avatar
wavetar
Administrator
Posts: 7340
Joined: Sun Sep 09, 2001 4:00 pm

Re: Site shedding subcribers

Post by wavetar »

What are you running for subscribers? Motorola has a "Radio Xtractor" software that can be used with XTS/XTL/APX radios to capture internal data processed during the event that could be helpful. It's available on MOL...mine, at least.

Have you tried 'site locking' a subscriber to the site to see if it still goes no service?
No trees were harmed in the posting of this message...however an extraordinarily large number of electrons were horribly inconvenienced.

Welcome to the /\/\achine.
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

This is a Tait system, and yes - I've tried site locking - no joy.

Another event today duplicating the results we have so far - no change in rsl, and the OSW never stops. I truly believe the subscribers are being booted off. But, Tait says no such command exists. They are still chasing after an external interference that somehow evades detection by measuring rsl and logging the OSW. What I want next is a full literal decode of the OSW.

I think we are experiencing something similar to my BBS days with dialup modems. This one file would fail to download over my Supra modem. The modem would drop out at the same amount of time each attempt, but successfully passed through my USR. We recompressed the file from zip to lha, and it passed through the Supra. Something in the zip compression made a string the forced the Supra into command mode, and then hang up. I think that is what is happening. A string of characters in the OSW from this site is faking out the subscribers making them think the site is gone.

Interesting tidbit from Tait about their system - we have probably identified the two minute thing - it's the subscriber's internal blacklist. They won't reveal the complete set of rules to me, but if a site qualifies the blacklist, it stays there for two minutes before it will be looked at again.
User avatar
wavetar
Administrator
Posts: 7340
Joined: Sun Sep 09, 2001 4:00 pm

Re: Site shedding subcribers

Post by wavetar »

Ask TAIT if they have something similar to the Radio Xtractor software to interface to their subscribers. If they do, they should be able to see exactly what is causing the radio to decide the site is gone.
No trees were harmed in the posting of this message...however an extraordinarily large number of electrons were horribly inconvenienced.

Welcome to the /\/\achine.
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

As a matter of fact, they did just that this week. It's a lab tool v0.4.0, but it can decode the entire OSW using a mobile as the receiver. It can also log in real time to a flat file for later "playback", or line by line inspection. Those logs get huge fast - about 8Mb an hour. We managed to capture several events, but the results are inconclusive - nothing is jumping out at us. You can get blurry eyed very quickly working with this tool.
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

After a week's drought of these shedding events, we enjoyed several in a row in the early afternoon today. The logs were dutifully forwarded to Tait for analysis, and they hinted at a Eureka! just a bit ago.

Waiting with great anticipation.
User avatar
wavetar
Administrator
Posts: 7340
Joined: Sun Sep 09, 2001 4:00 pm

Re: Site shedding subcribers

Post by wavetar »

Bill_G wrote:Waiting with great anticipation.
As am I. We also have a Cassidian based system with TAIT subscribers in another Province, this resolution will be good to know in case we come across the issue.
No trees were harmed in the posting of this message...however an extraordinarily large number of electrons were horribly inconvenienced.

Welcome to the /\/\achine.
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

Tait has been able to reproduce it in the lab. It appears that when an emergency input is detected while a mobile is turned off, unregistered and unaffiliated, it will covertly still attempt to send in the emergency. How that causes the system to deregister other subscribers is still TBD, but it may have something to do with processing a priority message from an "unknown" unit. IE: It's making time for the unit to fully affiliate and register before processing other units.
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

.... and they've opened a case with Airbus.
We shall see.
User avatar
wavetar
Administrator
Posts: 7340
Joined: Sun Sep 09, 2001 4:00 pm

Re: Site shedding subcribers

Post by wavetar »

Bill_G wrote:.... and they've opened a case with Airbus.
We shall see.
You gotta love "open architecture" systems!
No trees were harmed in the posting of this message...however an extraordinarily large number of electrons were horribly inconvenienced.

Welcome to the /\/\achine.
User avatar
Bill_G
Posts: 3087
Joined: Thu Sep 17, 2009 5:00 am

Re: Site shedding subcribers

Post by Bill_G »

A quick update - based on the lab results, we've been able to recreate the conditions in the field on the live system. This forces it back to their engineering dept to come up with a solution.
Post Reply

Return to “Smartnet/Smartzone/Project 25 (Phase I/II) Systems”