Announcement

Collapse
No announcement yet.

Excessive DNS queries for NTP pool addresses

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Excessive DNS queries for NTP pool addresses

    Is there any way to stop the excessive ping requests that FR24 is doing on an almost constant basis to NTP pool servers?

    My local DNS server has seen 1.5 million queries over the last 15 hours just from the FR24 Raspberry Pi alone. Looking into it further I can see that it's basically just launching a "ping -c 1 1.<continent>.pool.ntp.org" over and over again. This then gets multiplied because of the IPv4/IPv6 addresses of the client and server, meaning several DNS queries per ping, and then further because FR24 tries every major continent in the pool. It's currently doing 50-100 DNS queries per second just to ping NTP servers.

    As someone who hosts an FR24 feed and also maintains two of the NTP servers in the UK pool I find both sides of this concerning. Locally my DNS is getting hammered and my bandwidth wasted and at the data centre side I have NTP servers that will be receiving large amounts of unsolicited ICMP traffic from FR24 feeds.

  • #2
    Originally posted by skslater View Post
    My local DNS server has seen 1.5 million queries over the last 15 hours just from the FR24 Raspberry Pi alone.
    Assuming your local DNS is caching, as it should be, then I don't see why that matters much as excessive pings. You could always install a caching-only DNS on the pi if internal network traffic bothers you.

    But even that won't help much if your DNS A records have small TTLs. A bit of digging suggests that they're set to 150 seconds. Unless IP addresses change frequently, might it not be worth getting the ntpns.org admins to increase their TTLs? That's something only your org can do.

    Originally posted by skslater View Post
    Looking into it further I can see that it's basically just launching a "ping -c 1 1.<continent>.pool.ntp.org" over and over again.
    Is that based on ICMP seen at your NTP server or observed at your pi? If the latter, did you determine whether the ICMP messages are being generated by fr24feed or by a script?

    My pi feeder is self-assembled from Raspbian, dump1090-mutability and the fr24feed deb rather than FR24's pi image, and I'm not seeing any ICMI traffic on my pi's network i/f other than those I send from my desktop. That suggests that it's not being generated by fr24feed and, instead, is something to do with FR24's pi image. If so, then you could find the ping command in whatever shell script and get rid of it.

    That doesn't mitigate the pings from the thousands (?) of others running unmodified FR24 image, of course. It'd be fair to ask FR24 dev/ops to look into that.

    Originally posted by skslater View Post
    As someone who hosts an FR24 feed and also maintains two of the NTP servers in the UK pool I find both sides of this concerning. Locally my DNS is getting hammered and my bandwidth wasted and at the data centre side I have NTP servers that will be receiving large amounts of unsolicited ICMP traffic from FR24 feeds.
    From an NTP server ops pov, I'd have thought ICMP was the least of your concerns wrt unwanted traffic. Perhaps some creative routing might redirect ICMP to a different machine to handle such traffic? Better still, set your border router to route all traffic to another machine save for NTP traffic. You'd be limited to ssh from within your network, but you're doing that anyway, right?

    Comment


    • #3
      I looked at it in some more detail a few hours after I posted this and it looks like this isn't a "normal" feature of the FR24 client but rather a failure to handle DNS problems gracefully. An issue with a local DNS server meant it was returning "domain not found" for some domains, including the *.pool.ntp.org domains.

      It seems the FR24 client has embedded calls to system programs to would find the first working (i.e successful ping) NTP host, sync up time from there and then carry on, but in this case as each host fails it tries the next in the list and when the list is exhausted it starts again from the top. Thus firing off an endless stream of DNS queries at a high rate until it manages to connect to an NTP server. The ICMP traffic wasn't actually generated in this case as the ping command fails because in my specific case the host address couldn't be found, for NTP servers that are in the pool but block ICMP this would generate traffic but still fail a ping.

      The local DNS problem turned out to be a misbehaving Pi-Hole installation on my home network refusing some DNS queries but it would be nice to see some kind of backoff algorithm or delay in the client, or preferably just insist on a working NTP client on the feeder rather than forced NTP syncs via external commands and a liberal use of ping.

      Comment


      • #4
        Originally posted by skslater View Post
        I looked at it in some more detail a few hours after I posted this and it looks like this isn't a "normal" feature of the FR24 client but rather a failure to handle DNS problems gracefully.
        Ah "good", but perhaps still something to look into. I don't imagine it would be a high priority for FR24, though.

        Originally posted by skslater View Post
        for NTP servers that are in the pool but block ICMP this would generate traffic but still fail a ping.
        Yeah, I assumed that NTP servers would have to be pingable (because TCP SYN makes for a terrible way to see if a host is reachable), but it doesn't necessarily have to be the actual machine running ntpd that sends the ICMP echo reply packet

        Comment


        • #5
          Hi, see the discussion in this thread.

          It may be that the latest release addresses it but so far the details of the changes haven't been released. I did email support a week or so ago to ask but no reply so far...

          Comment


          • #6
            Reply from FR24 as follows:

            So sorry for the late reply. We had missed this email.

            This version has a small change made to it and that's for the decoder to use libbds. The next version where there will be improvements for MLAT will be a very important update.

            Unfortunately, I am not sure about timeline for linux versions as we are working on these improvements at the moment.


            Suggest emailing FR24 support if you are concerned about the way the client does NTP lookups.

            Comment


            • #7
              An update here.

              On neither of our test platforms there is anything close to being significant request rate. Hence most probably it is some custom user installations causing these problems. E.g. hacking /etc/hosts, disabling local DNS cache, etc. Therefore, this will not affect many people to be fair, so for the time being we are not going to make any changes here.
              --

              Comment


              • #8
                I noticed dns floods because of ntp requests also some weeks ago after the first tries of the updated feeder for RasPi. I noticed because I'm running PiHole on the same raspi and the logs exploded with requests for dns of all those *.pool.ntp.org servers, each of them requests every second. Nameresolution for every other software worked, also for those domains.

                I then resolved the problem after some trying by falling back to an old start-of-december backup of my SD-Card, because the updated feeder software didn't work for me. So probably there is some not-so-chilled error handling somewhere hidden.
                But as you said, probably doesn't affect lots of ppl, so no priority but perhaps to be kept in mind.
                T-EDLP28 / Paderborn / Germany

                Comment


                • #9
                  Hi Khan, thanks for the update, but the DNS requests are the least impact part of using the NTP Pool. fr24feed appears to be doing 12 NTP queries direct to servers around the world every ten minutes, which is about three times as much load as the pool recommends and appears to be a very high load on the parts of the world where there are few volunteers running NTP servers - e.g. Oceania only has 112 volunteer servers, South America only 44 servers and Africa on 36 servers. The NTP Pool is geo aware so the client only needs to look up appropriate servers and the pool will automatically find NTP servers that are local to the client.

                  Looking at queries to my local DNS server it appears to look these up every ten minutes or so:
                  Code:
                  Feb  3 09:15:52 sam named[23329]: client 192.168.123.142#37701 (1.europe.pool.ntp.org): query: 1.europe.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:15:52 sam named[23329]: client 192.168.123.142#37701 (1.europe.pool.ntp.org): query: 1.europe.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:52 sam named[23329]: client 192.168.123.142#58623 (1.north-america.pool.ntp.org): query: 1.north-america.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:52 sam named[23329]: client 192.168.123.142#58623 (1.north-america.pool.ntp.org): query: 1.north-america.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:15:52 sam named[23329]: client 192.168.123.142#52859 (1.asia.pool.ntp.org): query: 1.asia.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:52 sam named[23329]: client 192.168.123.142#52859 (1.asia.pool.ntp.org): query: 1.asia.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:15:53 sam named[23329]: client 192.168.123.142#44644 (1.oceania.pool.ntp.org): query: 1.oceania.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:53 sam named[23329]: client 192.168.123.142#44644 (1.oceania.pool.ntp.org): query: 1.oceania.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:15:54 sam named[23329]: client 192.168.123.142#53584 (1.south-america.pool.ntp.org): query: 1.south-america.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:54 sam named[23329]: client 192.168.123.142#53584 (1.south-america.pool.ntp.org): query: 1.south-america.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:15:54 sam named[23329]: client 192.168.123.142#60561 (1.africa.pool.ntp.org): query: 1.africa.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:54 sam named[23329]: client 192.168.123.142#60561 (1.africa.pool.ntp.org): query: 1.africa.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:15:55 sam named[23329]: client 192.168.123.142#41974 (1.europe.pool.ntp.org): query: 1.europe.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:55 sam named[23329]: client 192.168.123.142#41974 (1.europe.pool.ntp.org): query: 1.europe.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:15:55 sam named[23329]: client 192.168.123.142#45883 (1.north-america.pool.ntp.org): query: 1.north-america.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:55 sam named[23329]: client 192.168.123.142#45883 (1.north-america.pool.ntp.org): query: 1.north-america.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:15:55 sam named[23329]: client 192.168.123.142#51100 (1.asia.pool.ntp.org): query: 1.asia.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:55 sam named[23329]: client 192.168.123.142#51100 (1.asia.pool.ntp.org): query: 1.asia.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:15:56 sam named[23329]: client 192.168.123.142#42347 (1.oceania.pool.ntp.org): query: 1.oceania.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:15:56 sam named[23329]: client 192.168.123.142#42347 (1.oceania.pool.ntp.org): query: 1.oceania.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:16:06 sam named[23329]: client 192.168.123.142#38346 (1.south-america.pool.ntp.org): query: 1.south-america.pool.ntp.org IN A + (192.168.123.186)
                  Feb  3 09:16:06 sam named[23329]: client 192.168.123.142#38346 (1.south-america.pool.ntp.org): query: 1.south-america.pool.ntp.org IN AAAA + (192.168.123.186)
                  Feb  3 09:16:07 sam named[23329]: client 192.168.123.142#48021 (1.africa.pool.ntp.org): query: 1.africa.pool.ntp.org IN A + (192.168.123.186)
                  Then running tcpdump on the Pi these are the corresponding NTP requests (which are also made every ten minutes or so):
                  Code:
                  2018-02-03 09:15:52.236424 IP 192.168.123.142.38636 > 193.224.45.106.123: NTPv3, Client, length 48
                  2018-02-03 09:15:52.288467 IP 193.224.45.106.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:15:52.581242 IP 192.168.123.142.38636 > 104.131.139.195.123: NTPv3, Client, length 48
                  2018-02-03 09:15:52.740122 IP 104.131.139.195.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:15:53.199883 IP 192.168.123.142.38636 > 211.233.40.78.123: NTPv3, Client, length 48
                  2018-02-03 09:15:53.483171 IP 211.233.40.78.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:15:53.851808 IP 192.168.123.142.38636 > 103.242.68.68.123: NTPv3, Client, length 48
                  2018-02-03 09:15:54.122658 IP 103.242.68.68.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:15:54.449351 IP 192.168.123.142.38636 > 146.164.48.5.123: NTPv3, Client, length 48
                  2018-02-03 09:15:54.684385 IP 146.164.48.5.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:15:54.991350 IP 192.168.123.142.38636 > 196.223.19.3.123: NTPv3, Client, length 48
                  2018-02-03 09:15:55.185590 IP 196.223.19.3.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:15:55.337883 IP 192.168.123.142.38636 > 193.224.45.106.123: NTPv3, Client, length 48
                  2018-02-03 09:15:55.388413 IP 193.224.45.106.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:15:55.572828 IP 192.168.123.142.38636 > 144.217.65.183.123: NTPv3, Client, length 48
                  2018-02-03 09:15:55.664567 IP 144.217.65.183.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:15:55.882842 IP 192.168.123.142.38636 > 80.241.0.72.123: NTPv3, Client, length 48
                  2018-02-03 09:15:56.014709 IP 80.241.0.72.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:16:06.115416 IP 192.168.123.142.38636 > 13.55.50.68.123: NTPv3, Client, length 48
                  2018-02-03 09:16:06.410727 IP 13.55.50.68.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:16:06.756386 IP 192.168.123.142.38636 > 164.73.227.4.123: NTPv3, Client, length 48
                  2018-02-03 09:16:07.007620 IP 164.73.227.4.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  2018-02-03 09:16:07.352752 IP 192.168.123.142.38636 > 41.204.120.137.123: NTPv3, Client, length 48
                  2018-02-03 09:16:07.581649 IP 41.204.120.137.123 > 192.168.123.142.38636: NTPv3, Server, length 48
                  I suspect the pool has some logic to protect the areas of the world with few volunteer servers, but fr24feed should really follow the pool guidelines, request it's own vendor zone and query around four servers that the pool offers, not hard code servers on every continent. The NTP pool maintainer, Ask, I'm sure will be happy to help set up a vendor zone for you and answer any set up queries - his contact details are on the NTP Pool site here.

                  Comment


                  • #10
                    For what it's worth, I own my own Stratum 0 (GPS) NTP server so wanted to use that one, I use a local DNS server with RPZ functionality to redirect all *.pool.ntp.org queries to my local NTP server.

                    Comment


                    • #11
                      Originally posted by Sweetpants View Post
                      For what it's worth, I own my own Stratum 0 (GPS) NTP server so wanted to use that one, I use a local DNS server with RPZ functionality to redirect all *.pool.ntp.org queries to my local NTP server.
                      I think you will find you have a stratum 1 NTP server, only government and very large private entities would have a stratum 0 server (the GPS satellites have stratum 0 servers, anything synced to that would be stratum 1)
                      I also run a stratum 1 GPS NTP server using a STM32 based system, things like the Raspberry pi don’t have the speed or precision to handle the required timing.
                      FR24 F-EGLF1, Blitzortung station 878, OGN Aldersht2, PilotAware PWAldersht, PlanePlotter M7.

                      Comment


                      • #12
                        Hmm, this is what ntpq -pn shows on my clock

                        remote refid st t when poll reach delay offset jitter
                        ================================================== ============================
                        127.127.1.0 .LOCL. 10 l 265m 64 0 0.000 0.000 0.000
                        o127.127.20.0 .GPS. 0 l 8 16 377 0.000 0.002 0.002
                        -193.67.79.202 .GPS. 1 u 61 64 377 26.226 2.585 0.895
                        +131.188.3.221 .DCFp. 1 u 58 64 377 21.905 1.676 1.460
                        *131.188.3.222 .GPS. 1 u 39 64 377 22.745 1.701 1.520
                        +131.188.3.223 .PZF. 1 u 47 64 377 22.390 1.934 0.589

                        Comment


                        • #13
                          I guess it means the time source you're using (the GPS) itself is stratum 0. (Wackypedia gives a good write up).

                          I took another route and added all the ntp.pool hostnames FR24feed looks up to /etc/hosts and pointed them to my local GPS NTP server.

                          Maybe it's time to email FR24 again as the latest client still does a bunch of NTP lookups every ten minutes, looking up hosts on every continent! Doh!!!

                          Comment


                          • #14
                            For anyone who has this thread bookmarked, v1.0.24-2 just released updates the NTP code:

                            By default only one geographically close server queried every twenty minutes instead of six servers, one on each continent every ten minutes. Still a little often, but MUCH better than before!

                            I also noticed "Synchronization delayed, received back-off response" in the log file, which seems to cause it to wait 30 seconds.

                            And the icing on the cake - the option to use a local server without faffing around with etc/hosts ! (ntp-server="<ip address>" in /etc/fr24feed.ini)

                            Well done, FR24!

                            Comment

                            Working...
                            X