Announcement

Collapse
No announcement yet.

Feed server instability

Collapse
X
 
  • Filter
  • Time
  • Show
Clear All
new posts

  • Feed server instability

    Seems the feed servers are having some hiccups at the moment.

    One possible reason:
    Code:
    2019-07-09 19:39:06 | [feed][i]filtering out 98 overlapping AC (saving bandwidth)
    2019-07-09 19:39:10 | [feed][i]removed 1 of 106 AC
    2019-07-09 19:39:10 | [feed][i]sent 8,85 AC
    2019-07-09 19:39:13 | [time][i]Synchronizing time via NTP
    2019-07-09 19:39:13 | [stats][i]sent 32536 bytes
    2019-07-09 19:39:14 | [time][i]Time synchronized correctly, offset +0.0000 seconds, drift 0.0000 seconds/minute
    2019-07-09 19:39:15 | [feed][i]sent 7,87 AC
    2019-07-09 19:39:18 | [feed][i]removed 2 of 106 AC
    2019-07-09 19:39:20 | [feed][i]sent 8,83 AC
    2019-07-09 19:39:22 | [feed][i]removed 2 of 104 AC
    2019-07-09 19:39:25 | [feed][i]sent 7,86 AC
    2019-07-09 19:39:25 | [mlat][i]Pinging the server
    2019-07-09 19:39:30 | [feed][i]removed 1 of 102 AC
    2019-07-09 19:39:30 | [feed][i]sent 7,85 AC
    2019-07-09 19:39:35 | [feed][i]sent 7,83 AC
    2019-07-09 19:39:40 | [feed][i]sent 8,82 AC
    2019-07-09 19:39:45 | [mlat][i]Stats 18425636/2007
    2019-07-09 19:39:45 | [feed][i]sent 9,80 AC
    2019-07-09 19:39:47 | [mlat][i]Pinging the server
    2019-07-09 19:39:47 | [mlat][i]Stats 18429584/2026
    2019-07-09 19:39:50 | [feed][i]removed 1 of 103 AC
    2019-07-09 19:39:50 | [feed][i]sent 8,80 AC
    2019-07-09 19:39:55 | [feed][i]sent 9,80 AC
    2019-07-09 19:40:00 | [feed][i]sent 10,81 AC
    2019-07-09 19:40:05 | [feed][i]sent 10,80 AC
    2019-07-09 19:40:06 | [feed][i]removed 1 of 104 AC
    2019-07-09 19:40:07 | [mlat][i]Pinging the server
    2019-07-09 19:40:07 | [mlat][i]Stats 18431345/1761
    2019-07-09 19:40:10 | [feed][i]sent 10,80 AC
    2019-07-09 19:40:15 | [feed][i]sent 11,79 AC
    2019-07-09 19:40:18 | [feed][i]removed 2 of 107 AC
    2019-07-09 19:40:20 | [feed][i]sent 13,77 AC
    2019-07-09 19:40:25 | [feed][i]sent 12,79 AC
    2019-07-09 19:40:26 | [feed][i]removed 1 of 105 AC
    2019-07-09 19:40:27 | [mlat][i]Pinging the server
    2019-07-09 19:40:27 | [mlat][i]Stats 18433071/1726
    2019-07-09 19:40:30 | [feed][i]removed 1 of 104 AC
    2019-07-09 19:40:30 | [feed][i]sent 14,75 AC
    2019-07-09 19:40:36 | [feed][i]sent 87,0 AC
    2019-07-09 19:40:38 | [feed][i]removed 3 of 105 AC
    2019-07-09 19:40:41 | [feed][i]sent 89,0 AC
    2019-07-09 19:40:46 | [feed][i]sent 92,0 AC
    2019-07-09 19:40:47 | [mlat][i]Pinging the server
    2019-07-09 19:40:47 | [mlat][i]Stats 18434788/1717
    2019-07-09 19:40:51 | [feed][i]sent 91,0 AC
    2019-07-09 19:40:56 | [feed][i]sent 90,0 AC
    2019-07-09 19:40:58 | [feed][i]removed 1 of 106 AC
    2019-07-09 19:41:01 | [feed][i]sent 88,0 AC
    2019-07-09 19:41:06 | [feed][i]sent 89,0 AC
    2019-07-09 19:41:07 | [feed][i]filtering out 97 overlapping AC (saving bandwidth)
    Note that the filtering information is missing at 19:40 so the filtering times out and all planes get sent to the server for around 30 seconds.
    This of course increases load on the servers which can cause all kinds of problems if many feeders are not getting filter list either.

    Just in case anyone from FR24 is reading here
    Good luck fixing it.

    My feed isn't cutting out but i see quite a few syncing stream timeouts:
    Code:
    Jul 09 17:21:28 pi fr24feed[826]: [feed][n]syncing stream: timeout
    Jul 09 17:31:20 pi fr24feed[826]: [feed][n]syncing stream: timeout
    Jul 09 17:35:43 pi fr24feed[826]: [feed][n]syncing stream: timeout
    Jul 09 17:35:43 pi fr24feed[826]: [feed][i]Feed disconnected
    Jul 09 17:35:43 pi fr24feed[826]: [feed][n]disconnected
    Jul 09 17:35:43 pi fr24feed[826]: info | [feed][n]waiting 7 seconds
    Jul 09 17:35:50 pi fr24feed[826]: [feed][n]connecting
    Jul 09 17:35:50 pi fr24feed[826]: [feed][n]connected via UDP (fd 15)
    Jul 09 17:35:50 pi fr24feed[826]: [feed][i]Feed connected
    Jul 09 17:35:50 pi fr24feed[826]: info | Network thread connecting to 185.218.24.23:8099 for feed EDQG49
    Jul 09 17:53:21 pi fr24feed[826]: [feed][n]syncing stream: timeout
    Jul 09 18:10:46 pi fr24feed[826]: [feed][n]syncing stream: timeout
    Note that this is filtered to remove all the spammy output and catch only abnormals.

  • #2
    Oh and if someone would like to have fr24feed errors in their systemd journal, this is my modified service file:
    /etc/systemd/system/fr24feed.service

    Code:
    [Unit]
    Description=Flightradar24 Decoder & Feeder
    After=network-online.target
    
    [Service]
    Type=simple
    Restart=always
    ExecStartPre=-/usr/lib/fr24/install_dump1090.sh
    ExecStartPre=-/usr/lib/fr24/create_missing_directories.sh
    ExecStart=/bin/bash -c "/usr/bin/fr24feed | sed -u -e 's/[0-9,-]* [0-9,:]* | //' | grep -F -e imeout -e rror -e info -e main -e onnect -e reader --line-buffered"
    User=fr24
    Group=fr24
    PermissionsStartOnly=true
    StandardOutput=syslog
    SyslogIdentifier=fr24feed
    
    [Install]
    WantedBy=multi-user.target
    The StandardOutput goes to syslog instead of nowhere.
    But to reduce the spamminess only certain keywords are picked up.
    (modified ExecStart line)

    Comment

    Working...
    X