Announcement

Collapse
No announcement yet.

Data quality assessment

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #16
    Originally posted by g7ruh View Post
    Interesting idea to check quality.
    Thanks; quality ≠ quantity, but the latter is much easier to measure.

    The code linked is very much exploratory/experimental in nature because I don't yet know how best to characterise the quality of ADS-B messages, so check back periodically and look at the commit log where I explain what changes I've made and why in some detail.

    I only know that, when I look at the dump1090 web page, some tracks are mostly unbroken and some tracks are mostly broken up linked by dotted estimation lines. It's the latter I want to characterise, hence looking at mean and standard deviations. They won't yet mean much on their own, not until we have a better idea of what to expect from a "good" installation (which mine is not because the antenna is in the loft, not on the roof).

    So far, I've found that while the s.d. of displacement ratio (against displacement at last known airspeed) seemed plausible, the inter-arrival s.d. was wildly variable and very difficult to make any sense of. I could speculate why that might be but it would be pure guesswork.

    Yesterday afternoon, I pushed a change to adsb_msg_dist that significantly changes the way that intervals are calculated. Whereas before I used to look consider intervals between any type 3 or 4 message (position and velocity respectively), I now consider only intervals between position messages.

    Since then, the interval s.d. has dropped to a much more sensible level and in fact, mean interval ≈ mean displacement and interval s.d. ≈ displacement s.d. This is not surprising; they are, after all, distributions of the same data (position messages), but congruence validates the displacement ratio as a metric (as opposed to raw displacement in n.m. or feet) is working as intended.

    The next step is probably to try binning a/c by range from receiver on the basis that longer range messages are likely to experience a higher degree of loss than closer range messages and, currently, there is no distinction made between them. If the bins are small enough (maybe 25 nm might be a reasonable size), then that gives a parametric profile of any given receiver from which receiver pattern might be deduced.

    Question is whether the bins ought to be linearly spaced or whether they should be proportional to the inverse square of range on the basis that signal strength falls off in proportion to the square of distance. In terms of comparing like with like, that would seem sensible to me.

    I elaborate further on interpretation of the mean and s.d. in the README.

    Originally posted by g7ruh View Post
    I recently updated to an orange FA dongle from the standard TV DVB type. I got 50% more aircraft tracked and 3 times increase in position reports.
    I upgraded from a plain NooElec SDR to the blue FA SDR (ie the one that has the bandpass filter in it also).

    What gain are you using? I noticed that my SNR went down a bit afterward but I seem to get a substantially higher peak (valid) message rate and my range and count statistics improved by between 50–100%.

    Originally posted by g7ruh View Post
    I had to upgrade the Pi to a Pi3 as the single core CPU was overloaded given the increase to c 600,000 positions per day!
    The other Munin plugin, dump1090_, might also be of interest to you, but it only works with dump1090-mutability and related forks that write JSON data. Neither antirez' nor MalcolmRobb's fork upon which the bundled decoder is based do.

    Originally posted by g7ruh View Post
    This got me thinking somewhat the way you have described. I have started to wonder how many "data packets" are lost because of collisions or are being drowned out by stronger transmissions.
    I think both are highly likely although, in the case of stronger signals overpowering weak ones, the weak signal will appear as noise and as long as the demodulator can still make out 1s and 0s, chances are the stronger message will be received and the weaker will never be acknowledged at all.

    Originally posted by g7ruh View Post
    As I am also wondering if a pair of antenna each covering 180 degrees would throw any light on this: if the merged data from 2 dongles and 2 antenna results in significantly more decoded data, that could also be a pointer to the issue.
    That might improve things somewhat, but the problem of multipath interference remains. Suppose you have east and west facing receivers: the west facing receiver will see echos of messages from east sector traffic reflected off obstructions to the west and, weak as echos are, they may still be strong enough to overpower (or at least corrupt) traffic from distant westward a/c.

    IIRC, ADS-B is pulse-position modulated which is essentially still amplitude modulated binary bits. Because there's no active symbol for a binary 0, (only the position of the 1 relative in time), it's not straight forward to discern the difference between noise of a '0' and '1' from a more remote transmitter.

    Dump1090 currently records the mean and peak signal strength in dBFS (dB Full-Scale) and at least -mutability reports noise floor, but it doesn't attempt to detect collisions. One possible way to do that might be to look at the amplitude of where a '0' ought to be relative to the noise floor, but this is complicated by phase differences between different a/c.

    Originally posted by g7ruh View Post
    I will download and have a look at your code.
    Let me know what you think! (And be sure you have the latest version.)

    Comment


    • #17
      Thanks for the reply and for the description of your recent updates and thinking. I agree bins based on inverse square would make sense and worth trying first.

      I am using --gain -10 at present as it was the best I found with limited testing. I am waiting for a few more days to get a reasonable history with the dongle before changing the gain gradually from the max supported to where it drops below the results at -10. That way I will be able to see small variations in the results. I have only just got everything else working in a consistent manner, the Pi 2 fails to do mlat very well on the 4.9.x kernel so had several attempts at fixing and reverting to the 4.5.x kernel. In the end I replaced the Pi2 with a Pi3 using 4.9.x kernel and mlat works fine. That increased my stats so I am still waiting for the baseline data.

      I am using -mutability fork, I changed from MR for the graph capability as it helps determine some of the basis of gain v SNR etc.

      My thinking for 2 antenna was to try to reduce the overload from strong signals 360 degrees and see if there was any increase in distance / position reports having done so. My antenna is a home made one here (modified url as I still can't post links) www sprut de/electronic/pic/projekte/adsb/adsb_en.html#antenne and it is mounted on the roof of a single storey house and has a 360 degree outlook but obstructed in lots of places by big oak trees to the north and 2 storey houses on the south. My max range is 190-210 NM in a few directions between the buildings.

      I agree reflections and multipath will be an issue but that may be a smaller one than the one we are trying to measure. Also weather conditions will change things too, high pressure or rain clouds. I guess aircraft bounce will happen too: it works at 144 MHz so since the wavelength is a lot less, there is more surface area in the airframe.

      If you take into account all the variables and modifers then you could say the analysis would be rather akin to herding cats, nigh on impossible! Still we would never learn anything if we took that view!

      Another way may be to isolate the variables as much as possible by taking a narrow view of the sky by using a yagi with 15 or so elements and looking for signal quality using the 360 degree one as the base for data comparison. A test one out of wood and hardened copper wire would be easy enough to make for testing purposes even if not weather proof. Of course running 2 dongles and 2 Pi3s. I might make one and use a rotator to allow directional testing without climbing on the roof to adjust it. I will consider this task.

      Now back to installing your code: having trouble getting munin to work.
      Last edited by g7ruh; 2017-05-12, 14:08. Reason: post did not happen first time

      Comment


      • #18
        Originally posted by g7ruh View Post
        Thanks for the reply and for the description of your recent updates and thinking. I agree bins based on inverse square would make sense and worth trying first.
        Good enough, I'll for that in the first instance.

        Originally posted by g7ruh View Post
        I am using --gain -10 at present as it was the best I found with limited testing.
        That's interesting. I had completely the opposite results with that setting. -10 is a magic number to dump1090 and enables one of the two types of AGC (the --enable-agc flag enables the other type).

        This worked well enough for my generic SDR, but it seemed to go badly wrong with my FA dongle, perhaps because there is a cell base station about 100 yards away. Despite the FA Pro+ having a bandpass filter, its bandwidth and degree is broad enough to include at least some of GSM 900 signal. The result was a massively high noise floor and most of the signal driven into saturation because of the +20 dB auxiliary LNA that the FA SDRs have, so I had to manually fiddle with the gain controls. I settled on a slightly conservative gain of +25 dB.

        Originally posted by g7ruh View Post
        I am waiting for a few more days to get a reasonable history with the dongle before changing the gain gradually from the max supported to where it drops below the results at -10.
        That's wise, and the dump1090_ munin plugin will be very handy for that when you get it working.

        My approach was to choose a gain where the max signal mostly remains maybe -5 dBFS and seldom clips to 0 dbFS. With the Munin plugin, you only need to wait a matter of 10-30 minutes between adjustments to become apparent.

        Originally posted by g7ruh View Post
        the Pi 2 fails to do mlat very well on the 4.9.x kernel so had several attempts at fixing and reverting to the 4.5.x kernel.
        Any clue why the kernel should make a difference? The demodulation and I/O is almost exclusively in user space, after all. Perhaps your pi2 became I/O or CPU bound? Munin's good for monitoring that, too. And temperature. The pi3 can readily be driven past its throttling threshold of 82°C without a heatsink. Mine has a heatsink but, in the height of the summer, I fancy I will have to fan force cool it because my pi is in the loft.

        Originally posted by g7ruh View Post
        In the end I replaced the Pi2 with a Pi3 using 4.9.x kernel and mlat works fine.
        How do you know whether MLAT is working or not? There's nothing in the fr24feed logs that I can see that positively identify when MLAT is working. For example, it never complained that it wasn't getting timestamps from dump1090 (ie dump1090 lacked the --mlat flag; but then, mine is a DIY configuration and yours might be using the Beast feed rather than AVR).

        Originally posted by g7ruh View Post
        My thinking for 2 antenna was to try to reduce the overload from strong signals 360 degrees and see if there was any increase in distance / position reports having done so.
        In practice, it probably won't work that way because the only way multiple signals could result in saturation is when they transmit at once, but colliding messages are garbled anyway so it doesn't matter what the gain is set to or whether the receiver is saturated or not.

        Therefore, the dominant factor for saturation is gain and the power received from any one given a/c. I gather that ADS-B transmitters can range from 75-500 Watts (omni) so a 500W a/c might could overload a receiver when much further away than a nearer, lower power a/c.

        Originally posted by g7ruh View Post
        My antenna is a home made one
        As is mine, but I used the coax core-to-shield method minus the terminator and plus a 1/4w decoupling balun inverted and half way up the bottom element so that it forms part of the lowest active 1/2w element.

        Originally posted by g7ruh View Post
        My max range is 190-210 NM in a few directions between the buildings.
        Not bad! Munin reports my max-max range at 233 nm and my average max at 140 nm (for any given five minute period). And that's in the loft, not outside. But my FR24 dash consistently reports a max range of 160 nm or so, which tells me that they're doing some filtering of messages to exclude low quality samples.

        Originally posted by g7ruh View Post
        I agree reflections and multipath will be an issue but that may be a smaller one than the one we are trying to measure.
        We can't measure any individual contributor, all we can do is measure the sum of all influences and get an over-all index of the quality of messages received. That said, I agree; signal obstructions and collisions are more likely influential than multipath interference.

        Originally posted by g7ruh View Post
        Also weather conditions will change things too, high pressure or rain clouds.
        I can't see how pressure would affect signal at all. Doubtless Wx has some influence, but I'm unsure of what magnitude. Bond resonance in H2O is in the infra-red (IIRC), so attenuation due to cloud will be through dialectric loss, same principle as microwave ovens.

        Question is whether H2O density is enough to appreciably change the dialectric constant of air. OTOH, they're already weak signals (probably µW) so even a small amount of attenuation could be the difference between getting a signal and not. Table 5.3 of this paper suggests probably not; it quotes at most -0.06 dB for 4km of cloud at 2.3 GHz. Overall, it's probably never more than -0.1 dB.

        Originally posted by g7ruh View Post
        I guess aircraft bounce will happen too: it works at 144 MHz so since the wavelength is a lot less, there is more surface area in the airframe.
        Bounce? Do you mean reflections and multipath? I guess, but in relative terms even an A380 is a tiny spec at 50-100nm.

        Originally posted by g7ruh View Post
        Another way may be to isolate the variables as much as possible by taking a narrow view of the sky by using a yagi with 15 or so elements and looking for signal quality using the 360 degree one as the base for data comparison.
        Kind of how like VOR stations work. It would be an interesting test for a given location, but not one readily repeatable across multiple radars.

        Originally posted by g7ruh View Post
        Of course running 2 dongles and 2 Pi3s.
        The Pi3 has more than enough processing power to run multiple SDRs. Mine does ADS-B and airband on separate SDRs and load averages 0.65 (for four cores) and CPU idle averages around 350%, and that's with the overhead of Munin and icecast (sorry, not publicly accessible!)

        When I was running only dump1090, load averaged around 0.4.

        Originally posted by g7ruh View Post
        I might make one and use a rotator to allow directional testing without climbing on the roof to adjust it. I will consider this task.
        Sounds like fun! Use a 360° servo, perhaps like this connected to the pi's GPIOs and it'd be trivial to write some python to sweep the horizon at, say, one revolution per minute and plot the differences.

        Short of hacking dump1090, about your only way of getting signal strength data with fine enough resolution is probably to monitor /run/dump1090-mutability/aircraft.json which only gets updated roughly once a second. This Python watchdog library would probably reduce load rather than polling the file. It needn't be multithreaded as such; it's probably enough to wait on a new JSON file and then advance the yagi to its next orientation.

        The only downside to all of this is that you'd have to sweep the horizon back and forth rather than constantly rotating. That'd require some sophisticated, low-loss rotating RF coupling and a shaft encoder. Primary surveillance radars have it easy: they use hollow waveguides.

        Originally posted by g7ruh View Post
        Now back to installing your code: having trouble getting munin to work.
        Any particular problem? IME, the distribution packages work well, but configuring the web server can be a bit of a pain. I included an example for lighttpd if you're using that for dump1090. I just pushed a minor fix for that; previously the URL needed a trailing slash. Now it doesn't.

        /var/log/munin/munin-{update,node}.log can be helpful in diagnosing problems with collectors.
        Last edited by Strix technica; 2017-05-12, 16:26.

        Comment


        • #19
          Originally posted by Oblivian View Post
          So Windows + beast + feeder via USB = no mlat
          Pi + fr24feed windows raw out via TCP =
          Not sure what you mean. You got mlat working by changing something?

          I have no experience from any windows sw, I'm talking about the Rpi/Debian sw only.

          /M
          F-ESDF1, F-ESGG1, F-ESGP1, F-ESNK1, F-ESNV2, F-ESNV3 F-ESSL4, F-ESNZ7, F-LFMN3
          T-ESNL1, T-ESNL2, T-ESGR15
          P-ESIA, P-ESIB, P-ESGF, P-ESSN, P-EFMA
          mrmac (a) fastest.cc

          Comment


          • #20
            Originally posted by MrMac View Post
            Not sure what you mean. You got mlat working by changing something?

            I have no experience from any windows sw, I'm talking about the Rpi/Debian sw only.

            /M
            Just an observation.

            Due to lack of changelog, the last heard was beast MLAT support was still a WIP. It sure doesn't activate natively as it doesn't call dump1090 when you select that type of feeder (rather using direct ftdi calls). But when tapping into the active windows uploader raw relay out from a 2nd instance of feed software (Pi in my case) it calls Dump1090 for reading and therefore activates MLAT tagged uploading.

            Yet to try it in reverse. To see if the windows client port is just not capable, or a limitation of USB direct reading rather than Dump1090 doing the work
            Posts not to be taken as official support representation - Just a helpful uploader who tinkers

            Comment


            • #21
              I don't know why the kernel casues issues on Pi2. Certainly not CPU overload / heat related CPU max 30 % The -mutability graphs provide that data as well.
              The clue came from the FA forum, see MLAT Anomolies. I am not going to pursue the issue further, I will just use the Pi2 elsewhere.

              MLAT: I feed several sites and they report that MLAT is working fine (the way they have set it up). But I see the same thing in my FA log as you posted above. -mutabilty also shows mlat stats in the graphs.

              Pressure and airframes: we radio amateurs look for high pressure as it provides all sorts of enhanced propagation, and I would expect that ASDB would also benefit: greater distance would be the best sign. Does not happen often, but would skew results if it occured during testing. A passenger jet airframe is many times larger than the wavelength of ADSB so acts as a reflector (with some signal loss) but a strong signal from one aircraft to another not too far away could be reflected, just another form of multipath.

              2 PIs and dongles: keep the "production system" separate from test and development. That way you can mess with the test system while your live one does its thing. Development Rule No 1: do not experiment on a live system. Rule No 2: see rule No 1 !!!

              With the yagi I would point in one direction for an hour or more to collect data then move it to another. That way we would be sampling data from different direction and be able to see any differences.

              I got munin installed and working. I have your plugin running but no output.

              Code:
               $ systemctl status adsb-msg-dist.service
              ● adsb-msg-dist.service - ADS-B message distribution monitor
                 Loaded: loaded (/etc/systemd/system/adsb-msg-dist.service; enabled)
                 Active: active (running) since Fri 2017-05-12 16:53:15 UTC; 16h ago
                Process: 6679 ExecStartPost=/bin/sleep 0.3 (code=exited, status=0/SUCCESS)
                Process: 6673 ExecStart=/usr/local/share/munin/plugins/adsb_msg_dist daemon (code=exited, status=0/SUCCESS)
               Main PID: 6678 (adsb_msg_dist)
                 CGroup: /system.slice/adsb-msg-dist.service
                         └─6678 /usr/bin/python /usr/local/share/munin/plugins/adsb_msg_dist daemon
              I will check paths etc in the files and see if there is an error.

              How does the data show up in munin web page?

              UPDATE: munin stoped working, investigating that first

              Further update: removed munin, munin-node AND old data and web files and reinstalled. Now I see the ASDB data pages and it is collecting data. So far so good.
              Last edited by g7ruh; 2017-05-13, 11:38.

              Comment


              • #22
                Getting good data now. Not enough yet for detailed analysis

                First thought: single point tracks: I think it may be helpful to know these by distance, those on the extremity of range would be expected, closer to the receiver may be indicating quality issues. What do you think?

                Comment


                • #23
                  Originally posted by g7ruh View Post
                  -mutabilty also shows mlat stats in the graphs.
                  These graphs must be from Joe Prochazka's project? Because I can't see any graphs in my build of -mutability from his github master.

                  Originally posted by g7ruh View Post
                  Pressure and airframes: we radio amateurs look for high pressure as it provides all sorts of enhanced propagation, and I would expect that ASDB would also benefit
                  Oh, tropospheric refraction? TIL.

                  Originally posted by g7ruh View Post
                  Does not happen often, but would skew results if it occured during testing.
                  Even better argument for binning by distance.

                  Apropos, it occured to me that it might make more sense to bin by received signal strength in dBFS because given two planes at the same range but with substantially different tx power, the radar will receiver better quality messages from one than the other even though they're at the same range and are therefore not comparable.

                  Against that, dBFS values depend on gain settings and, as I mentioned, my FA SDR seems to give lower SNR than my generic SDR, even though the FR SDR gives a higher message rate and longer range. Also, since dBFS will be affected by the same factors that affect message quality, it's probably a poor reference. Worse, SBS data don't include signal strength IIRC.

                  All in all, it sounds like distance is still the better bin. What do you think?

                  Originally posted by g7ruh View Post
                  A passenger jet airframe is many times larger than the wavelength of ADSB so acts as a reflector (with some signal loss) but a strong signal from one aircraft to another not too far away could be reflected, just another form of multipath.
                  Understood. Two a/c nearby, one transmitting the other reflecting will have similar signal strengths and ADS-B pulse width is 1 µs or about 1000 feet. It's plausible that multipath from other a/c could cause a message to be garbled. Of course, the reflecting a/c would have to be located along a very narrowly defined corridor to reflect a signal to a particular radar and given the speed with which they move, I don't imagine that this would happen very often or for very long.

                  An extended squitter + preamble is 120 us or 20 nm away so to receive the same message twice via multipath would require enough separation that the power at the reflected aircraft is already about -21 dB down relative to 1 bit's distance worth and at that distance represents a tiny reflector in terms of area anyway.

                  Originally posted by g7ruh View Post
                  2 PIs and dongles: keep the "production system" separate from test and development. That way you can mess with the test system while your live one does its thing. Development Rule No 1: do not experiment on a live system.
                  In mission critical stuff I would agree, but I fool around with airband on the same pi without risk to ADS-B.

                  Originally posted by g7ruh View Post
                  With the yagi I would point in one direction for an hour or more to collect data then move it to another. That way we would be sampling data from different direction and be able to see any differences.
                  Perfectly reasonable approach; the reason I proposed a sweep of 1 minute was because message rates change so much even minute to minute, never mind hour to hour. The distribution for each sector hours apart are likely not comparable.

                  Originally posted by g7ruh View Post
                  I got munin installed and working. [...] Now I see the ASDB data pages and it is collecting data. So far so good.
                  Great!

                  Comment


                  • #24
                    Originally posted by g7ruh View Post
                    First thought: single point tracks: I think it may be helpful to know these by distance, those on the extremity of range would be expected, closer to the receiver may be indicating quality issues. What do you think?
                    That's a very good idea.

                    Naïvely, I had expected that the majority of nearer a/c would dominate multi-point tracks (assuming a good install), but it might turn out that so much traffic just grazes the outer most range and given its circumference (or whatever shape it is) compared to the number of a/c contained within the volume of space with reasonable reception, single-point tracks are always likely to be a high proportion of the total. So this strengthens your call for tracks and range.

                    That's a clumsy way of saying that there are always large numbers of a/c beyond your reliable range and it only takes one position message in any 5 minute period to qualify as a single point track, which is much easier and perhaps more likely than the relatively small number of a/c within your reliable range which will count as multi-point tracks.

                    (What's weird is when the ratio of single point tracks briefly exceeds 100%!)

                    So perhaps that statistic is not very interesting.

                    Unfortunately, the single point track data are aggregates from stats.json. In order to get range for single point tracks, I'd have to replicate dump1090's track assembly logic by analysing every single position message. Doable, but not trivial, and I (hope) that once adsb_msg_dist has been modified to bin for range, we'd get equivalent or better data anyway.

                    Comment


                    • #25
                      have replied but needs moderator approval as it has an attachment

                      Comment


                      • #26
                        Originally posted by g7ruh View Post
                        have replied but needs moderator approval as it has an attachment
                        @ Moderators; please feel free to email g7ruh with my email address if you prefer us to take such detailed discussions out of the forums. Since there's no PM facility, there's no other way I can share my address with him.

                        Comment


                        • #27
                          @g7ruh, I did so within the time frame you mentioned. Hope you got my email.

                          Comment


                          • #28
                            Originally posted by g7ruh
                            Plan B: Are you a member of the Raspberry Pi forum?
                            I have the same user name there and you can PM me from there.
                            Private message sent via RPi forums.

                            Comment


                            • #29
                              Thanks got it

                              Comment


                              • #30
                                ADSB-tool 2018

                                Originally posted by Strix technica View Post
                                Good enough, I'll for that in the first instance.
                                Dear sir,
                                I've just setup your solution in my system and I'm having problems with some of the collected data.
                                Are you still supporting this software?
                                Thank you very much,
                                Francesco
                                T-NOOB-001

                                Comment

                                Working...
                                X