xoolive / traffic Goto Github PK
View Code? Open in Web Editor NEWA toolbox for processing and analysing air traffic data
Home Page: https://traffic-viz.github.io/
License: MIT License
A toolbox for processing and analysing air traffic data
Home Page: https://traffic-viz.github.io/
License: MIT License
Using commit ccc46c4, upon calling:
ext_flight = flight.query_ehs()
Got one of the following exceptions:
File "[...]\traffic\core\flight.py", line 376, in query_ehs
flight = t[self.callsign]
File "[...]\traffic\core\traffic.py", line 123, in __getitem__
subset = list(index) # noqa: F841
TypeError: 'int' object is not iterable
TypeError: 'float' object is not iterable
Recently, more tables have been added to the Impala database, sometimes redundant with the REST API interface.
This may call for new methods in opensky_impala.py
inspired by the following link for this fork for flights_data4
, allcall_replies_data4
,
[hadoop-1:21000] > show tables;
+--------------------------+
| name |
+--------------------------+
| acas_data4 |
| allcall_replies_data4 |
| flarm_raw |
| flights |
| flights_data4 | <= ?
| identification_data4 | <= ?
| operational_status_data4 | <= ?
| position_data4 |
| rollcall_replies_data4 |
| sensor_visibility |
| sensor_visibility_data3 |
| state_vectors |
| state_vectors_data3 |
| state_vectors_data4 |
| velocity_data4 |
+--------------------------+
Deal with the following warnings and check the changelog.
DeprecationWarning: alt40fms() has been renamed to selalt40fms(). It will be removed in the future.
DeprecationWarning: alt40mcp() has been renamed to selalt40mcp(). It will be removed in the future.
Relevant commits seem to be junzis/pyModeS@6f6b507 and junzis/pyModeS@8ded350
ping @junzis
traffic/data/adsb/opensky.py
has been left out the change to UTC aware datetimes.
The following settings in the configuration file should have precedence over environment variables.
[proxy]
http: xxx
https: xxx
Hi,
If I try to follow the example here: https://traffic-viz.github.io/airports.html?highlight=airport
from traffic.data import airports
from traffic.data.samples import belevingsvlucht
Then I get the following error:
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name 'belevingsvlucht'
Any ideas what is wrong? This happens with the latest traffic from github.
The method predict
of the class Clustering
inside traffic/algorithms/clustering.py, should use transform
instead of fit_transform
We need to rethink the whole workflow to interpret new incoming data as timezone aware UTC datetimes, yet respect backward compatibility with timezone naive datetimes.
Taking summer time into account if applicable in the current timezone would be great!
Feed traffic.data.samples
with:
Interpolation on angles should probably be based on the unwrapped signals.
The part of the QS that says
If you have access to AIRAC files (published by Eurocontrol), you may get information about more complex zones, here the Terminal Maneuvering Area (TMA) for Toulouse.
from traffic.data import nm_airspaces
nm_airspaces['LFBOTMA']
fails for me...(and subsequent ones where nm_airspaces
is used)
Does it need some additional data sources?
Probably a link to Data Sources would help...
Under discussion with @ramondalmau
Current syntax obliges the user to manually cascade filters:
flight.filter().filter(altitude=53)
It would be beneficial, esp. with default parameters to cascade filters in one command:
flight.filter(altitude=(17, 53))
I have been playing around with the airport and runway guesses. For the landing runway guess, an error is generated for the selection of the last 10 minutes:
Error from pandas:
File "C:\Users\olsc\AppData\Local\Continuum\anaconda3\lib\site-packages\pandas\core\computation\ops.py", line 426, in convert_values
if v.tz is not None:
AttributeError: 'NaTType' object has no attribute 'tz'
Triggered by:
File "C:/EmpaDaten/Idealisierte_Spuren_aus_Radar/opensky_adsb/main_opensky_zrh18.py", line 79, in
o=flight.guess_landing_runway()File "C:\Users\olsc\AppData\Local\Continuum\anaconda3\lib\site-packages\traffic\core\flight.py", line 1054, in guess_landing_runway
.last(minutes=10)File "C:\Users\olsc\AppData\Local\Continuum\anaconda3\lib\site-packages\traffic\core\flight.py", line 558, in last
return self.class(self.data.query("timestamp > @bound"))
traffic/traffic/plugins/cesiumjs.py
Line 29 in 0122e05
should be : .isoformat()[:-6] + 'Z'
Trying to load data via the opensky/hadoop method causes traffic to hang.
I run this command: flight = opensky.history("2017-02-05", callsign="EZY158T")
And I see:
0%| | 0/24 [00:00<?, ?it/s]
This display stays on the screen forever (or, at least for the 2 hours I left it!) and the data never downloads.
Any ideas?
Line 1010 of opensky_impala.py
:
callsigns += "and callsign='{:<8s}' ".format(callsign)
triggers UnboundLocalError: local variable 'callsigns' referenced before assignment
upon call like :
df = opensky.rawdata(start=start_date, stop=end_date, callsign=callsign, cached=False)
HOW TO FIX : change Line 1010 of opensky_impala.py
to :
callsigns = "and callsign='{:<8s}' ".format(callsign)
Hi again,
I am looking at some historical data and see a different number of unique icao24 registrations between traffic and impala.
Here's what I do in traffic:
from traffic.data import opensky
from datetime import datetime
bounds = (-74.7,40.3,-72.9,41.2)
start_dt = datetime(2019,2,28,17,0)
end_dt = datetime(2019,2,28,18,0)
flights = opensky.history(start=start_dt,stop=end_dt, bounds=bounds)
print('Number of unique icao24: ',flights.data['icao24'].nunique())
Which gives: Number of unique icao24: 133
But if I run this query in impala:
SELECT COUNT(DISTINCT icao24) FROM state_vectors_data4 WHERE time>=1551373200 AND time<=1551376800 AND hour>=1551373200 AND hour<=1551376800 AND lat>=40.3 AND lat<=41.2 AND lon>=-74.7 AND lon<=-72.9;
I get:
+------------------------+
| count(distinct icao24) |
+------------------------+
| 319 |
+------------------------+
So impala is returning around 2.4 times more unique icao24 addresses than traffic.
Is traffic doing some pre-filtering to this data to remove bad position reports, or have I misunderstood how something works? :)
Future works:
Several use cases result in weird pandas DataFrame due to the naive implementation of the (precious) Flight.agg_time
method.
from traffic.data.samples import belevingsvlucht
belevingsvlucht.agg_time('5T', altitude=('max', 'mean')).data.columns[-3:]
produces:
Index([ 'rounded',
('altitude_('max', 'mean')', 'max'),
('altitude_('max', 'mean')', 'mean')],
dtype='object')
expected: ['rounded', 'altitude_max', 'altitude_mean']
belevingsvlucht.agg_time('5T', altitude=lambda x: x.sum()).data.columns[-2:]
produces:
Index(['rounded', 'altitude_<function <lambda> at 0x15a24b670>'], dtype='object')
expected: ['rounded', 'altitude_<lambda>']
.
def shh(x): x.sum()
belevingsvlucht.agg_time('5T', altitude=shh).data.columns[-2:]
produces:
Index(['rounded', 'altitude_<function shh at 0x15a4dcee0>'], dtype='object')
expected (use of function.__name__
):
Index(['rounded', 'altitude_shh'], dtype='object')
Some operations on Flight
and Traffic
structures require the underlying DataFrame to match a given number of properties (data sorted according to timestamp, data matching a given sampling rate, etc.). One way of ensuring the data matches the specification is to apply the treatment once more, but the overhead may be problematic. In some situations, the choice has been made to delegate the responsibility to the end user.
It could be beneficial to add a set of properties to these structures (Sorted
, Sampled['1s']
, etc.) and propagate them through all methods
A first use case to consider could be:
Flight[Any].resample('1s')
returns aFlight[Sampled['1s']]
, orFlight[Sorted].sort_values('timestamp')
returnsself
The implementation of such mechanism as a dictionary seems reasonable (beware of __slots__
though...). A further improvement would be to leverage these behaviours at the typing
level. (Check documentation here for a good reference)
Hello, I recently needed to access the data on the opensky network. After installing the traffic module according to the requirements, running the sample program in the document found that the traffic module was unavailable and many errors occurred. Just enter the code of the imported module and one line of code will give an error.
import traffic
The errors that appear are as follows, what is causing this, thank you very much! Other examples on the official documentation will not work.
C:\Users\Administrator\Anaconda3\envs\evn-traffic\python.exe C:/Users/Administrator/Desktop/traffic-test/traffic-ce001.py
Traceback (most recent call last):
File "C:/Users/Administrator/Desktop/traffic-test/traffic-ce001.py", line 4, in <module>
import traffic
File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\traffic\__init__.py", line 8, in <module>
import pkg_resources
File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\pkg_resources\__init__.py", line 80, in <module>
from pkg_resources.extern import appdirs
File "<frozen importlib._bootstrap>", line 971, in _find_and_load
File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
File "<frozen importlib._bootstrap>", line 656, in _load_unlocked
File "<frozen importlib._bootstrap>", line 626, in _load_backward_compatible
File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\pkg_resources\extern\__init__.py", line 43, in load_module
__import__(extant)
File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\pkg_resources\_vendor\appdirs.py", line 560, in <module>
import win32com.shell
File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\win32com\__init__.py", line 6, in <module>
import pythoncom
File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\pythoncom.py", line 2, in <module>
import pywintypes
File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\win32\lib\pywintypes.py", line 137, in <module>
__import_pywin32_system_module__("pywintypes", globals())
File "C:\Users\Administrator\Anaconda3\envs\evn-traffic\lib\site-packages\win32\lib\pywintypes.py", line 40, in __import_pywin32_system_module__
for suffix_item in imp.get_suffixes():
AttributeError: module 'importlib' has no attribute 'get_suffixes'
Data from OpenSky has position and velocity frames merged. Sometimes, position is lost for a while and interpolation does not detect the data has been repeated. Fields from the position frames should be set to None before interpolation to avoid future errors.
Check with distances between the flight in the following Traffic
data:
opensky.history(
'2017-06-14 18:00',
'2017-06-14 21:00',
callsign=['AEA1518', 'AFL2604']
)
Hi Xavier,
I'm trying to download some icao24-specific flights from opensky, but get Nonetype as a return.
My code, with the specific icao24 code removed here:
ic24 = 'icao_removed'
start_dt = datetime(2020, 2, 23, 18, 30, tzinfo = pytz.utc)
end_dt = datetime(2020, 2, 23, 18, 50, tzinfo = pytz.utc)
flights = opensky.history(start=start_dt, stop=end_dt, icao24=ic24)
print(type(flights))
My debug:
INFO:root:Sending request between time 2020-02-23 18:30:00+00:00 and 2020-02-23 18:50:00+00:00 and hour 2020-02-23 18:00:00+00:00 and 2020-02-23 19:00:00+00:00
INFO:root:Sending request: select time, icao24, lat, lon, velocity, heading, vertrate, callsign, onground, alert, spi, squawk, baroaltitude, geoaltitude, lastposupdate, lastcontact, hour from state_vectors_data4 where hour>=1582480800.0 and hour<1582484400.0 and time>=1582482600.0 and time<1582483800.0 and icao24='icao_removed'
The print
output: <class 'NoneType'>
If I log in to the opensky impala shell via ssh and run the select
request shown in the debug prompt above then I do receive the data. It just doesn't seem to get parsed properly in traffic
. In case it is important, a cache file is produced with size 141kb
Any idea what the problem might be?
Hi!
First of all, thanks for the library, it is really awesome!
I were getting errors when tried to make a query to Opensky historical database
with airport, arrival_airport or departure_airport parameters.
The error is related to Impala shell query:
opensky.history(
"2019-01-19", "2019-01-20",
airport="LOWS")
ImpalaError: time icao24 lat lon velocity heading vertrate callsign onground alert spi squawk baroaltitude geoaltitude lastposupdate lastcontact hour
select time, icao24, lat, lon, velocity, heading, vertrate, c
callsign, onground, alert, spi, squawk, baroaltitude, geoaltitude, lastposupdate,
, lastcontact, hour from state_vectors_data4 as sv join (select icao24, firstseen
n, estdepartureairport, lastseen, estarrivalairport, callsign, day from flights_d
data4 where estarrivalairport ='LOWS' and (1553904000 <= day and day <= 155407680
00)) as est on sv.icao24 = est.icao24 and sv.callsign = est.callsign and est.firs
stseen <= sv.time and sv.time <= est.lastseen where hour>=1553904000.0 and hour<1
1553907600.0 and time>=1553904000.0 and time<1553990400.0 ;
ERROR: AnalysisException: Column/field reference is ambiguous: 'icao24'
So I've assigned labels for 'icao24' and 'callsign' in flights_data4 table in opensky_impala.py in parts of the query related to airports (rows 654 - 685) and put same labels in join part (rows 625 - 629).
It works now for me.
All the best.
Consider checking the following database instead:
https://opensky-network.org/datasets/metadata/aircraftDatabase.csv
When following the Quick Start steps I found that if the claim is to start from the cloned repo
I have to correct the path to the sample data from
from traffic.core import Traffic
t = Traffic.from_file('../data/sample_opensky.pkl')
to (one dot, i.e. data
directory under current working directory)
from traffic.core import Traffic
t = Traffic.from_file('./data/sample_opensky.pkl')
Example attached with one flight intersecting LFBBBDX as a point.
The following could be included in a patch to the clip method:
(
f.shape is not None and
not isinstance(
f.shape.intersection(nm_airspaces["LFBBBDX"].shape),
shapely.geometry.point.Point
)
)
Implement a parser for files provided according to this standard.
Example file: FAA CIFP
Sample decoder (in Go): sv18/arinc424
Hi Xavier,
I have this code:
from traffic.data import opensky, airports
from datetime import timedelta, datetime
lat_ave,lon_ave = airports['VABB'].latlon
bounds = [lon_ave - 0.45, lat_ave - 0.45,
lon_ave + 0.45, lat_ave + 0.45]
st_time = datetime(2019,8,19,11,8,0)
flights = opensky.history(start=st_time,
stop=st_time+timedelta(minutes=1),
bounds=bounds,
other_params=" and time-lastcontact<=15 ")
Which I think should give me flights at 11:08 UTC
on 19th Aug 2019. But it actually gives me flights for 10:08 UTC
.
I guess this is some kind of timezone thing (although I'm on UTC). Any ideas?
I think in joss/paper.md:
... indexed by time, which derivate as ground speed, track ...
should be reworded to something like
... indexed by time, with derived ones such as ground speed, track ...
It would probably make sense to annotate each column of the pandas dataframe with physical units. A basic improvement could be a basic annotation for features; a more advanced improvement would check consistency and convert units if need be.
Reference:
It seems that memory is leaking.
When several big data files are handled in chain, memory seems to never be completely freed. I suspect @lru_cache
decorators on methods to be responsible for that as they seem to store results and instances at the class level.
This could be a starting point to investigate:
https://stackoverflow.com/a/33672499/1595335
See also: https://github.com/youknowone/methodtools/ where you just need to update:
from methodtools import lru_cache
Following discussion with @JarryGabriel it would be helpful to get a cumulative distance wrt the end of the trajectory, esp. in the context of approach analyses.
Include the AIRAC decoder by @obbe79
Try to keep the inclusion as smooth as possible, with:
Hello,
When I try to resample flight data using the built-in method I get a ValueError.
To reproduce:
from traffic.data import opensky
flight = opensky.history("2017-02-05", callsign="EZY158T")
flight2=flight.resample()
Gives this error:
File "<stdin>", line 1, in <module>
File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/traffic/core/flight.py", line 628, in resample
.resample(rule)
File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/generic.py", line 6835, in interpolate
**kwargs)
File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 519, in interpolate
return self.apply('interpolate', **kwargs)
File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/internals/managers.py", line 395, in apply
applied = getattr(b, f)(**kwargs)
File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/internals/blocks.py", line 1845, in interpolate
limit=limit),
File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/arrays/datetimelike.py", line 769, in fillna
value, method = validate_fillna_kwargs(value, method)
File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/util/_validators.py", line 348, in validate_fillna_kwargs
method = clean_fill_method(method)
File "/network/aopp/apres/users/proud/conda/lib/python3.6/site-packages/pandas/core/missing.py", line 91, in clean_fill_method
raise ValueError(msg)
ValueError: Invalid fill method. Expecting pad (ffill) or backfill (bfill). Got linear
This is using the latest build of traffic (obtained with: pip install git+https://github.com/xoolive/traffic
), Pandas version 0.24.1 and python version 3.6.7.
Any ideas what's happening? I tried to diagnose it but have not had any success.
When an Impala request returns None
, there is currently no way to know whether no data has been fetched or whether a Syntax error caused an error. The output should be raised as a Python Exception.
altair seems a good way to display data, although not great with geographical representations. It is easy to reimplement Flight.plot_time
to altair with decent default options. This could be added to the default API.
Another scenario that would be nice to work on would be to the possibility to project trajectories to a 2D space (anything, basic projections, t-SNE, PCA, etc.) and be able by an altair-like selector to come back to a Traffic subset.
PermissionError is raised on Windows when trying to remove file still open, at:
traffic/traffic/data/adsb/opensky_impala.py
Line 118 in 5a38c69
PermissionError: [WinError 32] Der Prozess kann nicht auf die Datei zugreifen,
da sie von einem anderen Prozess verwendet wird
I think the message being passed about Python, Pandas and Shapely is that these are used by traffic
, so the paragraph
Python is a popular language among academics and data scientists for its active
community and large catalogue of libraries fitting the needs of most in their
respective areas of research.
Pandas is a natural solution to represent time series,
Shapely [@shapely_2007] a package of choice to address geometrical operations
(intersections, inclusion, cascaded joins among others).
is coming IMHO too early...
Maybe something like the following will make that para less standalone:
The traffic library uses the Python language to reach to its large base of academics and data scientists, to serve and benefit its active community and to build on top of a
large catalogue of libraries.
So trajectories are modeled on top of Pandas dataframe, a natural solution to
represent time series, while airspaces delegate to Shapely [@shapely_2007] geometrical operations (intersections, inclusion, cascaded joins among others).
t = opensky.history(
'2017-06-14 18:00',
'2017-06-14 21:00',
callsign=['AEA1518', 'AFL2604']
)
t['AFL2604']
returns a Flight but t['AEA1518']
returns None in spite of being present in the dataframe.
DeprecationWarning: '+init=<authority>:<code>' syntax is deprecated. '<authority>:<code>' is the preferred initialization method
Error downloading a large chunk of data:
File "pandas/_libs/parsers.pyx", line 2172, in pandas._libs.parsers.raise_parser_error
ParserError: Error tokenizing data. C error: Expected 17 fields in line 23725, saw 31
thrown by
File "C:\Users\olsc\AppData\Local\Continuum\anaconda3\lib\site-packages\traffic\data\adsb\opensky_impala.py", line 115, in _read_cache
df = pd.read_csv(s, dtype={"icao24": str, "callsign": str})
example (error occurs after a few minutes):
data_raw = opensky.history(
"2018-01-01 00:01",
"2018-02-01 00:00",
bounds=(
7.99182, 47.09367, #sw
9.24188, 47.83655 #ne
),
)
Adding error_bad_lines=False or engine='python' to read_csv did not help.
Any way to skip these erroneous lines?
(Interestingly enough, putting the start time at hour 00:00 yields a different error.)
Hello,
I would like to clip so6 trajectories to specific FIRs.
For instance, I have collected all the trajectories crossing a specific FIR:
bdx_so6_LIMM = so6_m1.inside_bbox(eurofirs['LIMM'])
and now I would like to clip them.
Unfortunately the clip command does not work with the eurofirs objects (I don't have AIRAC files):
bdx_so6_LIMM['BCS361'].clip(eurofirs['LIMM'].flatten())
I get the following error:
NotImplementedError: Multi-part geometries do not provide a coordinate sequence
I'm probably missing something. Is there a better way to do what I'm trying to do?
Thank you,
Rob
Hi,
I've been following the examples here:
https://traffic-viz.github.io/opensky_usage.html
import random
sample_callsigns = random.sample(sv.callsigns, 6)
sample_callsigns
['RCH319', 'KZR921', 'SKW5996', 'N21714', 'JSA538', 'DAL2225']
flight = sv[sample_callsigns[5]]
flight
returns the following error:
HTTPError: 404 Client Error: for url: https://opensky-network.org/api/tracks/?icao24=acbe7a
Any hints on how to overcome this?
Many thanks
Hello,
I'm trying to make use of traffic to download data from OpenSky. I have already configured my credentials in the configuration file, but now I get an error regarding the AIRAC directory:
RuntimeError: Edit file /Users/jad/Library/Application Support/traffic/traffic.conf with AIRAC directory
I really do not know how to proceed with this, as I am not very familiar with Air Traffic information systems. How should I proceed in order to be able to use traffic as an easy access point to OpenSky data?
I appreciate you help with this matter.
Thanks,
Jorge.
Typing mistake not detected because buffer_to_iter
is not typed either. But @obbe79 found it!
traffic/traffic/data/so6/so6.py
Line 370 in 5f881e8
Should probably be replaced (but not before proper unit testing) by:
yield self.__class__(...)
I get the following error message when importing SO6.
ModuleNotFoundError Traceback (most recent call last)
in
----> 1 from traffic.data.so6 import so6
2 #so6 = SO6.from_file('20190708_m1.so6')
3 #so6.to_pickle('./data/sample.pkl')
ModuleNotFoundError: No module named 'traffic.data.so6'
Have you also had this issue ?
Best,
Yves
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.