How to pipe dump1090 decoded data to a file?

[r_inzerillo]

Hi everybody,
I'm new here and eager to learn more about FR24, navigation and communications.
I have a running Pi24 system at the moment, feeding data to FR24.

I'd like to plot down some statistics about the data I am collecting (and sending tofr24) and I wonder which could be the best approach.

I see dump1090's Mode-s "decoded data" output on port 30003, which could be a good starting point.

I wonder if it's possible/feasable to pipe the same kind of data out of dump1090 to a file on a local/remote machine for offline processing.

Still, I'm not that familiar with dump1090 and linux pipes. So, maybe someone here did something similar and may help me with a few preliminary considerations before delving too much into the wrong path.
Suggestions?

Thx :-)

[abcd567]

.
.
Exploring Port 30003 and 30106

.

[r_inzerillo]

Nice read :-)

I already "explored" port 30003 output, that was my starting point and I'm already comfortable in logging that output with Putty or similar.
That implies a remote putty client (or similar, even something running on the same Pi24 machine) accessing dump1090's output.

I'd like dump1090 to be the main "actor" though, I'd like it to be the one eporting the data directly (if possible). I'm reading about dump1090 debug mode, maybe that's more like it. Will dig further.

"Exploring Port 30003 and 30106" is a nice read anyway, it even gave me new insights into Mlat. Thx :-)

[wiedehopf]

You could look at this script to get an idea of how to do it:

https://discussions.flightaware.com/t/dump1090-mysql-database-script-with-alert-and-filter/26619

I mean you can also connect to port 30003 with any program you write or with nc:
nc pi 30003

Gives you the data from that port on the command line for further processing. You probably want some possibility to reconnect in case of a connection loss, using a while loop and making sure nc quits when the connection is lost.

Just grabbing the json data every second might be the easiest and is very resistant against a connection loss.

http://pi/dump1090/data/aircraft.json

pi is the IP address of the pi you run your dump1090 on.
Might be pi/dump1090-mutability or something on your system, you can check the lighttpd configuration.


Data gathering from another system is not a bad idea, the pi could be overwhelmed by the data depending on how you handle them.

You might also want to look at making yourself a new feed system like OPTION 2 from this guide https://discussions.flightaware.com/t/bake-a-pi/19886
Installing piaware can easily be left out if you only want to feed fr24.
There are notes on install fr24 in that guide as well but just as a heads up: chosing DVBT when asked during fr24-feeder installation will screw up your system.

[wiedehopf]

I'd like dump1090 to be the main "actor" though, I'd like it to be the one eporting the data directly (if possible).

port 30003 is a direct output of dump1090. Sockets / networking is the preferred way of handing data from one program to another on unix even on the same computer.

Anyway the json file method in my other post might be easier for you, it only gives you momentary snapshots of a state though, not when an altitude change was received for example.

[r_inzerillo]

Just grabbing the json data every second might be the easiest and is very resistant against a connection loss.

That's nice :-) Very usefull if data processing has to be made with JavaScript (not only JS though)!

Might be pi/dump1090-mutability ...

I'm using stock Pi24 img.

Data gathering from another system is not a bad idea, the pi could be overwhelmed by the data depending on how you handle them.

I'm considering sending the data to a remote host for further processing. But I really like the idea that dump1090 is the one sending the data somehwre (being it local file/ or a remote host), and not a remote client requesting for it. I'm open to alternatives but that's the approach I'm investigating for now.

I'm going to separate data harvesting and data processing anyway.

... just as a heads up: chosing DVBT when asked during fr24-feeder installation will screw up your system.

I think I already know about that :-(

[r_inzerillo]

Sockets / networking is the preferred way of handing data from one program to another on unix even on the same computer.

You mean I should learn about sockets on Linux, right?
Btw, you're talking here about Internet Sockets ... not Unix Sockets, correct?

Will read about that and consider pros and cons.

[wiedehopf]

I'm considering sending the data to a remote host for further processing. But I really like the idea that dump1090 is the one sending the data somehwre (being it local file/ or a remote host), and not a remote client requesting for it. I'm open to alternatives but that's the approach I'm investigating for now.

I don't see an advantage to that. dump1090 can just output to console but redirecting that to a local file is just messy. Also that output is raw messages and not decoded positions/altitudes i believe.
Other than that you could always change the source code of dump1090 and make it write the SBS output to a file i guess.

Using the json file served via http or the network sockets is a much more elegant approach.
I'm not sure why you are opposed to it

Unix Sockets and network sockets shouldn't be much different performance wise on a Linux system (well apart from the IP filter chain i guess). Anyway we are not talking amounts of data where it matters.

By the way dump1090 is sending the data somewhere, it's just being asked to do so by the client connecting via TCP.
The remote client just establishes the connection and then the data flow is mostly from dump1090 to the client.

[r_inzerillo]

I don't see an advantage to that. dump1090 can just output to console but redirecting that to a local file is just messy. Also that output is raw messages and not decoded positions/altitudes i believe.
Other than that you could always change the source code of dump1090 and make it write the SBS output to a file i guess.

Using the json file served via http or the network sockets is a much more elegant approach.
I'm not sure why you are opposed to it

I wouldn't say I'm opposed to that.

As of a now I am at very early stage and don't know much about the implications of each approach. I'm quite open to every possible scenario. Let's just say I imagine how it could work based on previous experiences and presuppositions that could very well be inaccurate if not plain wrong. I'm collecting ideas, suggestions and reading docs in order to have the gist of what's involved, pros and cons. Not to mention the limitations imposed by my programming skills which are not hard core at all (so, no chance of adding my own diffs to the code)!

Unix Sockets and network sockets shouldn't be much different performance wise on a Linux system (well apart from the IP filter chain i guess). Anyway we are not talking amounts of data where it matters.

By the way dump1090 is sending the data somewhere, it's just being asked to do so by the client connecting via TCP.
The remote client just establishes the connection and then the data flow is mostly from dump1090 to the client.

Cool. Will focus on that :-)

[wiedehopf]

I'm collecting ideas, suggestions and reading docs in order to have the gist of what's involved, pros and cons. Not to mention the limitations imposed by my programming skills which are not hard core at all (so, no chance of adding my own diffs to the code)!

You could just try yourself at installing this:

https://discussions.flightaware.com/t/dump1090-mysql-database-script-with-alert-and-filter/26619

It is a script which basically scrapes up the json file every second and makes entries in a mysql database.

[r_inzerillo]

I mean you can also connect to port 30003 with any program you write or with nc:
nc pi 30003

I'm enjoying the way nc handles data transfer with ease. I'm not a bash scripting guru ... but I think I'll figure a way to deal with it in a robust way.
At the moment it seems to me the most elegant/concise solution for this specific need.

[flygirl20]

Hello! I have a similar issue/question since I am quite new to coding. I have a Raspberry Pi 3 and have set it up to be an ADS-B receiver with mode-S. I can see the data it writes and when I use the command nc localhost 30003, it starts showing all the collected data of the current time, until I tell it to cut. Is there a way for it to write this data that it is recording directly to a csv file for me to use to analyze the data it collects? I have found options of crontab and an option of having a daily email log, but using the Port 30003 seems to be what I want to have written into a folder for me to view. Is there a simple script or prewritten one anyone knows about that would allow me to output the data from Port 30003 exactly as it is being recorded to a file that I can access? I read the "Exploring Port 30003 and 30106" and the PuTTy option would not be best for my situation because I want to access the files from the Raspberry Pi, not from an outside source such as using PuTTy or a USB port.
I have the program running automated in the background, so even when it is not streaming the data to my screen, the program itself is running in the background of the Raspberry Pi being on and tracking aircraft.