Announcement

Collapse
No announcement yet.

Unnecessary NTP traffic from fr24feed... (should probably get a vendor zone)

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Unnecessary NTP traffic from fr24feed... (should probably get a vendor zone)

    Hello,

    I was noticing occasional odd DNS lookups from my Pi that's running fr24feed - they were for NTP pool servers that are nowhere near me, and it was happening every 10min or so. This seemed odd, since all my machines are NTP synced to my local stratum 1 server.

    I was seeing these hosts being looked up:

    1.europe.pool.ntp.org
    1.north-america.pool.ntp.org
    1.asia.pool.ntp.org
    1.oceania.pool.ntp.org
    1.south-america.pool.ntp.org
    1.africa.pool.ntp.org


    I then realized that the fr24feed process tries to Sync time every 10 min or so, as per the log file:

    2018-01-15 06:51:00 | [time][i]Synchronizing time via NTP

    This is probably a setting that the end user should be able to disable, or you should alter this so that it just checks to see if the local clock is sync'd first, before trying to force it.... Or perhaps simply an enumerated option like:

    Set system clock:
    • Every 10 minutes
    • Once per hour
    • Once per day
    • Only at process start


    Your current utilization is also - technically - against the NTP Pool terms of use (I'd link to it, but I was warned not to post links as I'm a newly-registered user -- Suffice it to say you can find it on the "ntppool DOT org" web page ), as you should probably have created a vendor pool zone for fr24, and used that. Also, forcing every user to poll pools that are all over the world is silly, and often some of these pools may be small and not have as much capacity as larger pools. You are generating unnecessary load on the global pools, especially since normal NTP query backoff would settle around 1024 seconds (~17min), and you're polling more frequently than that (and to an unnecessarily large number of servers).

    Sure, i t's smart to have a "sanity check" to see if the clock is correct, but other than at initial process startup, you really probably don't need to check more often than every hour, tops.

    Checking to see if the system's clock is generally synched already (at least at process start) should be sufficient to avoid unnecessary traffic and unnecessary use of "all" the various global pools (which might traditionally not be expecting much traffic). Again, you really probably should not be specifying all of the various regional pools - just use the [0-3].pool.ntp.org hosts... but better yet getting a "vendor id" created for you, and using that.

    Anyway, just trying to make sure the fr24 process is being as "friendly" as it should be, when dealing with the volunteer NTP pools

    (n.b. I run several public NTP servers and have been contributing to the global NTP pools for a while now)

    -Taner

  • #2
    https://forum.flightradar24.com/thre...pool-addresses
    Posts not to be taken as official support representation - Just a helpful uploader who tinkers

    Comment


    • #3
      Ahh, I tried searching for "ntp" and "dns" but those were "too short" to use for search terms I also tried "Synchronizing" in the hopes someone might have pasted that line from the logs, too, but nothing.

      Sorry for the dupe thread, I guess.

      That said, I think I state the problem in a different manner, and it's actually really important that people do not abuse the public NTP pools like fr24 is (technically) doing. They really should apply for a vendor zone (it's pretty trivial to do).

      On a per-client basis, yes, it's a trivial amount of traffic every 10 min. But multiply that by the number of clients, and then you turn into a DDoS ;-) (I mean, not really, but I think you understand my point).

      There is also no reason to try to sync the system clock so often. I realize MLAT calcs are very time-sensitive, but even rPi clocks don't drift that much, and especially for those of us who "know what we're doing" and have meticulously set up ntp synchronization on all of our machines, this is just annoying to have going on.

      -Taner

      Comment


      • #4
        Theres another related one somewhere else I can't find off hand that had similar thoughts.

        I cheated and did a google site:forum.flightradar24.com <term> for shorter than 4-5 char
        Posts not to be taken as official support representation - Just a helpful uploader who tinkers

        Comment


        • #5
          A number of people have raised this issue with FR24 support - they agreed to look at it at one stage but nothing has changed. The best way to raise it is to head over to the flightradar24.com site and use the "Contact Us" link under "Social" - maybe the team has changed and someone will fix it this time... It seems like whoever first coded it really misunderstood how the NTP pool works!

          If you've got a local NTP server (GPS etc) you can redirect the lookups locally by adding a line to /etc/hosts:

          <ip of your ntp> 1.africa.pool.ntp.org 1.asia.pool.ntp.org 1.europe.pool.ntp.org 1.north-america.pool.ntp.org 1.oceania.pool.ntp.org 1.south-america.pool.ntp.org

          If you use planefinder you can reduce the load on the pool by adding this to /etc/hosts too:

          <ipv4 of your local ntp> 0.planefinder.pool.ntp.org 1.planefinder.pool.ntp.org 2.planefinder.pool.ntp.org 3.planefinder.pool.ntp.org
          <ipv6 of your local ntp> 0.planefinder.pool.ntp.org 1.planefinder.pool.ntp.org 2.planefinder.pool.ntp.org 3.planefinder.pool.ntp.org

          Comment


          • #6
            No reason to do that for planefinder. They have their own vendor zone and are only querying it once every 15 minutes.
            Literally one NTP query (packet) every 15 minutes sent, not more.

            (They also ask on their forum not to do a redirect via DNS entries in /etc/hosts.
            https://forum.planefinder.net/thread...p-server.1130/
            https://forum.planefinder.net/thread...1130/post-8328 )


            FR24 on the other hand sends 2 ntp queries and 2 ping packets to every of the regional pools. (every 10 minutes)
            That is a whopping 24 packets being sent.

            That is against the terms of the ntp pool and at least for me more than enough reason to lighten the load on the pools at least a bit by redirecting the regional pools that are not being used by any other application to localhost (just installing ntp will do the trick, it runs very nicely with the default settings from raspbian).

            Comment


            • #7
              I can understand the desire to have a consistent configuration, but NTP over the internet is I think exceedingly unlikely to be as accurate as a local Stratum one server. The number of servers in the pool is generally decreasing (link to the Pool stats) so lightening the load and using an inherently more accurate time source makes sense to me, but each to their own. Once per 15 minutes is just within the Pool guidelines. Maybe what we need is a push for more servers to join the NTP Pool. I did get a persuade The Register to run a story a while back but it didn't seem to make much difference

              FR24 - yes, as I say the person who originally implemented the code obviously completely misunderstood how the pool works. Requesting time every ten minutes directly from a server on all but one continent just doesn't make sense, especially given the load on the few servers in the South American (37) and Africa (42) pools. I think it's time for another email to Support to try and convince them to update the code...

              Comment


              • #8
                Ok, email sent to FR24 support.

                Comment


                • #9
                  Originally posted by elljay View Post
                  I think it's time for another email to Support to try and convince them to update the code...
                  Any response from them on this?

                  Thanks...

                  --

                  Comment


                  • #10
                    Hi, yes I got a reply "We are aware of that and our developers will try to look into it as soon as they can.", so followed up with the three quick, easy steps needed (get a free vendor zone; update the list of pool servers; change to once per 15 mins) and got acknowledgement of that. That was about a week ago. The ticket ref is #274996 if anyone wants to follow up.

                    Comment


                    • #11
                      I usually have about 21 IP addresses issued by my home router (computers, phones, tablets, gizmos). Looking at the logs for a week, the R Pi running fr24 is the noisiest by a factor of 3 over the next noisiest (my main computer, on most of the time).

                      Code:
                      Top 10 domains for top 10 clients:
                       --------------------------------------------------------
                      
                       192.168.29.41, daphnis:
                       --------------------------------------------------------
                       21618  1.oceania.pool.ntp.org                    
                       21614  1.asia.pool.ntp.org                       
                       21614  1.africa.pool.ntp.org                     
                       21612  1.south-america.pool.ntp.org              
                       21610  1.north-america.pool.ntp.org              
                       21606  1.europe.pool.ntp.org                     
                       2370   feed.flightradar24.com
                      I can see why an ntp pool member (thanks for your efforts) might consider this traffic from every fr24 feeder a little unfortunate. So, I'll add my voice to the request to the fr24 developers (thanks for your efforts) to become more pool friendly in 2019.

                      Comment


                      • #12
                        For anyone who has this thread bookmarked, v1.0.24-2 just released updates the NTP code:

                        By default only one geographically close server queried every twenty minutes instead of six servers, one on each continent every ten minutes. Still a little often, but MUCH better than before!

                        I also noticed "Synchronization delayed, received back-off response" in the log file, which seems to cause it to wait 30 seconds.

                        And the icing on the cake - the option to use a local server without faffing around with etc/hosts ! (ntp-server="<ip address>" in /etc/fr24feed.ini)

                        Well done, FR24!

                        Comment

                        Working...
                        X