Announcement

Collapse
No announcement yet.

Stats hard to justify

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • #31
    Originally posted by Khan View Post
    as you mentioned, it requires quite a bit of processing doing it that way. It's been under discussion internally to see if we can somehow make it fairer
    PM or email me if you'd like to collaborate on any of this in any more detail. I don't know if you saw my earlier post but my background (which includes scientific analysis and also systems and operations) may be of some use to FR24 for such matters.

    I don't know if you saw my thread on data quality assessment, but I've implemented what I described there and will let you know in due course what the results look like, if you like.

    Trouble is, I've got no known-good receiver with which to compare my results. If anybody with a known-good receiver would be willing to give me access to the SBS port on their dump1090 instance (I can give you a static IP for your firewall), I'd have a much better idea of how it compares.

    One thing that has already become apparent to me is that a good receiver that can hear a/c 250 nm out will still inevitably experience lost messages, so I'll have to think of a way to mitigate that or otherwise figure out how to turn the mean and s.d. figures into some measure of quality.

    I'm contemplating pushing the code to github (once I'm reasonably sure there are no bugs in the analysis code) so people can try it for themselves, but that'll only help if I get to see the results.

    S.
    Last edited by Strix technica; 2017-05-04, 13:33. Reason: Add mention of data quality assessment & SBS port

    Comment


    • #32
      Originally posted by Strix technica View Post
      PM or email me if you'd like to collaborate on any of this in any more detail. I don't know if you saw my earlier post but my background (which includes scientific analysis and also systems and operations) may be of some use to FR24 for such matters.

      I don't know if you saw my thread on data quality assessment, but I've implemented what I described there and will let you know in due course what the results look like, if you like.

      Trouble is, I've got no known-good receiver with which to compare my results. If anybody with a known-good receiver would be willing to give me access to the SBS port on their dump1090 instance (I can give you a static IP for your firewall), I'd have a much better idea of how it compares.

      One thing that has already become apparent to me is that a good receiver that can hear a/c 250 nm out will still inevitably experience lost messages, so I'll have to think of a way to mitigate that or otherwise figure out how to turn the mean and s.d. figures into some measure of quality.

      I'm contemplating pushing the code to github (once I'm reasonably sure there are no bugs in the analysis code) so people can try it for themselves, but that'll only help if I get to see the results.

      S.
      I will send you an email shortly
      --

      Comment


      • #33
        Just one further comment, probably reiterating what I previously said.
        There are two viewpoints here.
        From FR24 viewpoint - how useful is this feeder regarding quality of data and coverage of activity.
        From a feeder point of view, who is hugely affected by whether there is a lot of activity or not, how many hours is my station actually operational (in case there is activity) and how far away (range) am I getting signals,
        This from a feeder perspective gives you assistance regarding self improvement and tuning of antenna etc.

        Whats on the Stats page now goes a long way to providing this, it would just make it easier if the comparative stats analysis incorporated it as well.

        As Strix says activity is the greatest influence on your rating. If situated say 200nm from a busy airway corridor with little local traffic you can expect to score maximum points on the present system.

        Maybe there are two forms of stats needed to be available.

        Comment


        • #34
          Originally posted by JohnSunnyhills View Post

          Maybe there are two forms of stats needed to be available.
          That might just happen if a more fairer formula can't be agreed upon. Of course, if any changes are to be made in this regard, they will first be put here for discussion.
          --

          Comment


          • #35
            Originally posted by Khan View Post
            if any changes are to be made in this regard, they will first be put here for discussion.
            That will be a First!
            Mike


            www.radarspotting.com

            Radarspotting since 2005

            Comment


            • #36
              Originally posted by JohnSunnyhills View Post
              As Strix says activity is the greatest influence on your rating. If situated say 200nm from a busy airway corridor with little local traffic you can expect to score maximum points on the present system.
              Just in case you missed it, I noted in a later post that (assuming the formula Khan provided is accurate, which it seems to be) uptime vastly dominates max range by something like 130:1 or more because uptime is measured in minutes, not hours. Perfect uptime alone accounts for the first 8640 points of your score, which is about 76% of the current top ranked radar (T-EDDK145).

              Note that you probably only need one a/c and a few positional reports to qualify as 'up' at any given minute, so the actual number of a/c in view probably doesn't matter unless it's zero.

              So what MrMac and I were arguing for — more weighting on uptime — is already in effect. I think that's the right approach because the objective should first be to encourage as much uptime as possible. JMO, of course. The balance of the score then ranks those with the "best" receiver amongst those with the best uptime.

              That said, I take your point about activity given that only uptime with messages counts. It's easy for people in the northern hemisphere to assume there will always be some level of high altitude traffic, but that assumption is probably not true of central Australia or anywhere south of the Bombay hills in New Zealand, to pick two examples at random.

              For northern hemisphere radars, there's always going to be some high altitude activity so uptime is entirely in the hands of the feeder (a pi is cheap and more than capable!) Southern hemisphere radars are a bit of a conundrum.

              Although it's really a binary problem of whether or not to consider the station up when there really is no traffic about, perhaps some sort of scaling of the following form applied to stations south of the equator might just compensate for the reduced apparent uptime where there is likely to be no traffic:

              uptime_scale = F.cos(2.pi.(t/(24*60*60) + Tmin/24 + lon/360))^o

              where:
              • F is the magnitude of the compensation factor, which will depend very much on how and where this compensation is applied,
              • Tmin is the central hour of the traffic minimum; if my data are representative, that's around 3 am (low is tmin ± 3 hours),
              • o is in the range of [1,16] depending on how sharply you want to curb the nocturnal compensation. At the 4 hour mark, for example, the function outputs .75, .56, .32, .1 and .01 for o=1, 2, 4, 8 and 16 respectively
              • lon is the degrees longitude of the station, and
              • t is seconds since midnight UTC.


              F, Tmin and o would have to be found empirically based on datasets from around the world. It wouldn't be hard to script distributions if I had access to those data, but I guess I would have to be an FR24 employee to do that and get the co-operation of the FR24 ops team to instrument inbound data if they don't already have such instrumentation. (FR24, fancy offering me a contract job? )

              The idea behind this is that traffic seems to follow a roughly sinusoidal diurnal pattern (in its troughs, anyway; peaks are bimodal) and so this applies a complementary (cosinusoidal) compensation factor F that falls off at a rate of order o. In practical terms, you'd probably want a fairly sharp order because the objective is to limit the compensation to those hours where there is genuinely no traffic. The second term of the argument to cos() compensates for the fact that the traffic trough isn't centred around midnight and the third term is a simplified way of dealing with time zones that is easy to implement.


              JohnSunnyhills, assuming your station is active 24x7, what does your apparent uptime say and, if you don't mind saying, what's your nearest airport? Even simpler, what's your radar ID?

              The only alternative is to reduce the weight of uptime, but that defeats the objective of encouraging uptime. OTOH, there's already an argument to do that: there's no way that T-VHHH82 deserves a rank of 6865 with an average and max range of 1 and 10. That might be fixed by weighting average range higher but, for reasons explained below, max range will not likely help.

              I'm going to guess that there are more radars in the northern hemisphere because there's easily more population in the north, so the harm this does probably outweighs the benefits — especially as average and max range aren't very reliable indicators of quality, for either feeder or FR24. That brings us onto your other point:

              Originally posted by JohnSunnyhills View Post
              From FR24 viewpoint - how useful is this feeder regarding quality of data and coverage of activity.
              From a feeder point of view, who is hugely affected by whether there is a lot of activity or not, how many hours is my station actually operational (in case there is activity) and how far away (range) am I getting signals,
              This from a feeder perspective gives you assistance regarding self improvement and tuning of antenna etc.
              The two need not be mutually exclusive. They might even be more likely than not to be coincidental, unless you're a Dx'er only interested in, er, length competitions. There's nothing wrong with that, but since Dx is mostly a competition with yourself and other Dx'ers, there's little point in including max distance in the score except to separate otherwise equal radars.

              Everyone else's interests coincide with those of FR24 — getting the best possible data for the best possible coverage. An individual feeder interested in improving his station wants to know the quality of the data he's feeding (and max distance tells you very little about this). Other feeders a) want fair ranking and b) want the best quality feeds for the best quality coverage on fr24.com, which is also what FR24 itself wants.

              The reason max range isn't terribly helpful is because, as I've already remarked, you only need to get a few percent or probably much less of a distant a/c's positional messages to register max distance on FR24, but losing even 90% of that a/c's messages hardly constitutes quality and isn't in anybody's interests, including the feeder's own interests because he's fooling himself that his radar's performance is better than it is.

              You may have noticed, if you keep records of data taken directly from your feed, that FR24's "max" is consistently less than the actual longest self-reported range, and this is probably why. FR24 probably require more than a single point at a given distance to qualify it as the radar's maximum.

              If you aren't monitoring your own station, my dump1090 monitoring code is now online. It is fairly easy to install and will happily run on a Pi.
              Last edited by Strix technica; 2017-05-05, 21:19. Reason: Elaborate on uptime_scale; fix mistakes in it

              Comment


              • #37
                @Khan, @Oblivion,

                In case you already saw my reply to JohnSunnyhills when it was originally posted, I've developed and elaborated on the compensation factor for zero traffic, in case you're interested in that sort of detail.
                Last edited by Strix technica; 2017-05-05, 17:31.

                Comment


                • #38
                  Originally posted by Strix technica View Post
                  JohnSunnyhills, assuming your station is active 24x7, what does your apparent uptime say and, if you don't mind saying, what's your nearest airport? Even simpler, what's your radar ID?
                  .
                  I am based about 10km NE of NZAA, obviously in Auckland New Zeaaland. I am running two radars NZAA4 (Beast) which is my main one and NZMB22 (R820T2Dongle) which is more test and is running MLAT.
                  Unlike Europe NZAA virtually shuts down at various times between midnight and dawn. Both my units are running 24x7 on separate Raspberry PIs, feeding down my home broadband connection. The difference between the two is range, as they are on separate antenna. NZAA4 shows 690 to 694 uptime currently, back in March it was up to 704 (based on Jollain stats). The only difference is activity levels. The personal stats pages invariably show up time as 100%.

                  As an aside I have noticed a significant falloff of the level of hits, positions and contacts in the last month which I have so far failed to explain. Either due to changes to the feeder stats pages or the effect of a new Vodafone home wireless broadband service recently launched.

                  Comment


                  • #39
                    Originally posted by Anmer View Post
                    That will be a First!
                    Now, now. Be nice

                    Comment


                    • #40
                      Originally posted by JohnSunnyhills View Post
                      I am based about 10km NE of NZAA, obviously in Auckland New Zeaaland. I am running two radars NZAA4 (Beast) which is my main one and NZMB22 (R820T2Dongle) which is more test and is running MLAT.

                      Both my units are running 24x7 on separate Raspberry PIs, feeding down my home broadband connection. The difference between the two is range, as they are on separate antenna. NZAA4 shows 690 to 694 uptime currently, back in March it was up to 704 (based on Jollain stats). The only difference is activity levels. The personal stats pages invariably show up time as 100%.
                      Oblivian's (NZCH1) is comparable to NZMB22 and, between the three of them, amount to 1–3 hours without traffic registered. So a 6 hour window centred around the quietest time (3am) would seem to be a reasonable place to start for those three locations. That assumes FR24 would entertain the idea of a quiet-time compensation in the first place.

                      Just curious, what antennae are you using? If they're both at your home they must be essentially co-located, in which case how come they're named for different airfields and see different traffic (other than range of antennae)? Different sides of the building? NZMB and NZAA aren't exactly close to each other, and there's One Tree Hill in the way...

                      Originally posted by JohnSunnyhills View Post
                      As an aside I have noticed a significant falloff of the level of hits, positions and contacts in the last month which I have so far failed to explain. Either due to changes to the feeder stats pages or the effect of a new Vodafone home wireless broadband service recently launched.
                      There is a distinct possibility that RF intermodulation is responsible, particularly if you don't have a 1090 MHz filter. My locally-recorded stats improved anywhere between 30% and 100% depending on where, when and what you look at after I installed a FA pro plus (RTL + LNA + BP filter). There's a cell base station about 100 yards from us, 4G is in the 800 MHz band and, of course, there's the original GSM900 band.

                      Oddly, the track count (a high proportion of which are single-point tracks) started to follow a diurnal pattern in a way it hadn't since I installed my DIY co-linear (which itself was a vast improvement on the stock 1/4w whip supplied with the SDR which itself outperformed the 1/2w dipole I constructed, though that lacked a proper balun), strongly suggesting RF shenanigans.

                      Originally posted by JohnSunnyhills View Post
                      Unlike Europe NZAA virtually shuts down at various times between midnight and dawn.
                      Yeah. I remember arriving at NZAA in the middle of the night (I forget why) and having to wait hours with my luggage before I could check in. This was long before self-check in came about.

                      Idk about other European airports but EGLL (Heathrow) and EGSS (Stansted) also operate similar hours, for outbound flights at least. That surprised me, given how much Heathrow complain about capacity. But then, the residents of Hounslow also vocally complain about noise, so that probably has a lot to do with it, as does availability of public transport.

                      I guess I assumed that Auckland would see at least some traffic all the time as I do, but I guess I see a lot of transiting traffic in our domestic quiet hours. That might also explain why my max distance is greatest and bad message percentage is lowest during those periods. That's rather counter-intuitive, but I conclude that fewer a/c mean fewer message collisions and more good messages (by proportion).

                      Comment


                      • #41
                        Originally posted by Strix technica View Post

                        Just curious, what antennae are you using? If they're both at your home they must be essentially co-located, in which case how come they're named for different airfields and see different traffic (other than range of antennae)? Different sides of the building? NZMB and NZAA aren't exactly close to each other, and there's One Tree Hill in the way...
                        With respect, Strix, you are clearly a guru, but your comments re One Tree Hill are astray. One Tree Hill may well be in line with the two "airports". I say "airports" as NZMB is a inner city heliport. I am about 10k to the west of One Tree Hill so I have a clear line of sight view of both facilities, or at least the airspace above them. [Alternative fact: Several years ago the tree was actually chopped down by a local activist]

                        My two radars were originally both on NZAA nomenclature. When I registered to start providing MLAT data I was allocated the NZMB moniker and ended up with NZMB22. I don't know whether that is a standard naming convention that FR24 are using for MLAT.

                        Normally both radar share the same antenna, which is a simple coaxial colinear antenna that I made by cutting up a piece of RG6. I used a great description written by Dusan Balara. I have been switching NZMB22 around to other antennae options this week and the other day the NZMB22 antenna was lying on the garage floor and it still showed as 100% uptime with zero activity. Last night I switched them so they are both using the same colinear antenna.

                        For me, I am only using the stats to see if by heightening my antenna or relocating it to different locations or swapping out bits of cable, or rotating it around the mast, I can increase the range, and thus reduce my level of inactivity predawn (and other things). I only use the stats for tuning.

                        As I said before there are two uses of the stats, and right now your analytics seem to be foccussing on usefulness to FR24.

                        From a usefulness perspective why bother with compensation. The compensation today comes in the form of FR24 providing feeder kit to those liviing in isolated locations, adjusting the ratings for those same people is compromising reality.
                        If one is based in an isolated location too bad, in the end the usefulness rating reflects that.
                        If living in a quiet location, to improve your stats just move to a better location, build a bigger antennae etc.

                        Clearly there is a "feel good" element from monitoring the current stats. It is much the same when trying to see your own Radar code on the aircraft tracking panel. That feeling is gold when you get a hit.

                        That would be an interesting stat though. How many aircraft have you tagged each day and for how long, that would reflect your moments of glory.

                        From a MLAT perspective we see no stats. What contribution to MLAT activity did your radar make, how many aircraft did you track.

                        A typical feeder needs a bit more from FR24 for their committments, the free sub to the Business plan is good but I feel that more stats could be provided.
                        Last edited by JohnSunnyhills; 2017-05-06, 20:00.

                        Comment


                        • #42
                          Just a further thought..
                          I notice NZAA8 and NZAA22 are both registering uptime of arount 716/717. Both in ths same city to me. Why introduce compensation where clearly inthe right location with the right gear you can achieve a good ranking. What gear are they using?
                          Maybe they are enjoying panoramic views from homes on the uppermost slopes of One Tree Hill. I would be interested to know these answers too.

                          Comment


                          • #43
                            Originally posted by JohnSunnyhills View Post
                            Why introduce compensation where clearly in the right location with the right gear you can achieve a good ranking
                            You are missing the point. The score should reflect the parameters that a feeder can change, improve, and also see on the individual stats page. If the range is poor, it will already be reflected in the stats as it reduces both max and avg range. If I provide a bad internet connection, or turn the box off at night, it should be visible in the uptime stats. But, if I provide power and internet 24/7, and always get 100% uptime in the individual stats, the public stats SHOULD show 720 hours.

                            And don't start with the "build a better antenna", there is an area right now in northern Scandinavia the size of NZ south island whithout any flights whatsoever. 500 km radius, no antenna or height will make any difference.


                            So. my suggestion is simply, don't complicate it, just change the uptime to mean the same as the 100% individual uptime, i. e. the connected time. If FR24 then wants to increase the appreciation of good range compared to uptime, just change the score formula from the current "2x uptime + 2x max + avg" to " Uptime + 2x max + 2x avg".

                            That would make a much more transparent stats calculation without any complicated compensation formulas.

                            /M
                            F-ESDF1, F-ESGG1, F-ESGP1, F-ESNK1, F-ESNV2, F-ESNV3 F-ESSL4, F-ESNZ7, F-LFMN3
                            T-ESNL1, T-ESNL2, T-ESGR15
                            P-ESIA, P-ESIB, P-ESGF, P-ESSN, P-EFMA
                            mrmac (a) fastest.cc

                            Comment


                            • #44
                              Originally posted by JohnSunnyhills View Post
                              your comments re One Tree Hill are astray. One Tree Hill may well be in line with the two "airports". I say "airports" as NZMB is a inner city heliport. I am about 10k to the west of One Tree Hill so I have a clear line of sight view of both facilities, or at least the airspace above them.
                              Fair cop, I was lazy, just looked at the relative locations of the two airfield (and yes, I knew that NZMB is a heliport, though not until I looked it up) and didn't consider your actual location.

                              Originally posted by JohnSunnyhills View Post
                              Alternative fact: Several years ago the tree was actually chopped down by a local activist
                              Yeah, I remember. Mike Smith, Benjamin Nathan and several others whose names I now can't remember are complete nutters. In many ways, they're amongst the least dangerous. Seabed and foreshore debacle or tino rangatiratanga, anyone? But back on topic...

                              Originally posted by JohnSunnyhills View Post
                              When I registered to start providing MLAT data I was allocated the NZMB moniker and ended up with NZMB22. I don't know whether that is a standard naming convention that FR24 are using for MLAT.
                              Ack. I've wondered about reregistration should you move, and similar.

                              Originally posted by JohnSunnyhills View Post
                              Normally both radar share the same antenna, which is a simple coaxial colinear antenna that I made by cutting up a piece of RG6.
                              Same as me, then. Mine's an 8 segment with a 1/4w decoupler half way up the first element (actually part of the feeder). Mine is based on Balarad's with some modifications.

                              Originally posted by JohnSunnyhills View Post
                              For me, I am only using the stats to see if by heightening my antenna or relocating it to different locations or swapping out bits of cable, or rotating it around the mast, I can increase the range, and thus reduce my level of inactivity predawn (and other things). I only use the stats for tuning.
                              Have a look at my Munin plugins to get your own local data. FR24 filter their stats somewhat, whereas those Munin plugins record raw data from the antenna. They've been most useful in the tweaks I've made to my antenna.

                              Originally posted by JohnSunnyhills View Post
                              right now your analytics seem to be foccussing on usefulness to FR24.
                              As I said, I don't think the two are mutually exclusive. What's good for you is good for FR24 and visa versa, which is not to say that FR24 couldn't do more in terms of feeder quality data, hence my data quality thread.

                              Originally posted by JohnSunnyhills View Post
                              From a usefulness perspective why bother with compensation.
                              The reason for paying attention to the uptime is to mitigate the disadvantage of radars with periods of zero activity in terms of public ranking, on the grounds that the public ranking is likely to be a strong motivator to improve one's antenna. That's why weighting max distance, for example, is a mistake (IMO) because it gives a poor idea of your installation's actual performance.

                              Originally posted by JohnSunnyhills View Post
                              The compensation today comes in the form of FR24 providing feeder kit to those liviing in isolated locations
                              Different sort of compensation!

                              Originally posted by JohnSunnyhills View Post
                              adjusting the ratings for those same people is compromising reality.
                              The point of standardising uptime data is to make them comparable across the globe, namely that there's a difference between no data because not up and no data because no a/c to report. I'm not suggesting that range metrics be adjusted, and I'm working on some alternative measures of quality that may be of use to individual radar operators.

                              Originally posted by JohnSunnyhills View Post
                              If one is based in an isolated location too bad, in the end the usefulness rating reflects that.
                              In terms of uptime, I argue that the most important consideration is availability as opposed to reported data. Consider someone who lives in Queenstown: they're not likely to see much nocturnal traffic so what incentive have they (in terms of ranking) to do anything about the quality of the data they report if they're disadvantaged for want of recognised uptime? Can't have it both ways.

                              Originally posted by JohnSunnyhills View Post
                              If living in a quiet location, to improve your stats just move to a better location, build a bigger antennae etc.
                              Moving is unreasonable and building a bigger antenna is implausible. You can increase gain, but only at the cost of pattern and altitudinal coverage. For signals that move in three dimensions, this does not a good combination make.

                              Originally posted by JohnSunnyhills View Post
                              From a MLAT perspective we see no stats. What contribution to MLAT activity did your radar make, how many aircraft did you track.
                              I have a cynical suspicion that MLAT sources are considerably more limited than saying 'yes' to MLAT in your fr24feed installation would suggest, at least in the general case, as this moderately heated discussion would suggest.

                              Originally posted by JohnSunnyhills View Post
                              A typical feeder needs a bit more from FR24 for their committments, the free sub to the Business plan is good but I feel that more stats could be provided.
                              Idk. You were operating the radar and collecting the data anyway, what else does FR24 owe you other than a business account? If you want better data, it's better to rely on something you know and control rather than some third party's statistics. That's why I wrote the Munin plugins I did.

                              I don't have to feed FR24, but business plan access makes it worth my while to do so. I don't know what else they could offer that I can't do for myself, though I admit my background disposes me to this sort of analysis.

                              I guess one thing they could do more is make more raw data available, even if in anonymised form. That's standard practice in the academic research community, though I guess there are commercial considerations. FR24 have to make money, and giving their competitors data for free would not advance their cause any.

                              Comment


                              • #45
                                Originally posted by MrMac View Post
                                You are missing the point. The score should reflect the parameters that a feeder can change, improve, and also see on the individual stats page. If the range is poor, it will already be reflected in the stats as it reduces both max and avg range. If I provide a bad internet connection, or turn the box off at night, it should be visible in the uptime stats. But, if I provide power and internet 24/7, and always get 100% uptime in the individual stats, the public stats SHOULD show 720 hours.

                                And don't start with the "build a better antenna", there is an area right now in northern Scandinavia the size of NZ south island whithout any flights whatsoever. 500 km radius, no antenna or height will make any difference.


                                So. my suggestion is simply, don't complicate it, just change the uptime to mean the same as the 100% individual uptime, i. e. the connected time. If FR24 then wants to increase the appreciation of good range compared to uptime, just change the score formula from the current "2x uptime + 2x max + avg" to " Uptime + 2x max + 2x avg".

                                That would make a much more transparent stats calculation without any complicated compensation formulas.

                                /M
                                I don'y think you have understood my posts. As a feeder you only need basic stats as you say. I am thinking do IU buy/setup a new antenna, do I apply for a internet fibre service to be installed. the simple stats assist here.

                                All the other ones are academic

                                Comment

                                Working...
                                X