[BUG] unable load historical data

[nmatton]

Hello,

I want to expand upon a discussion I initiated earlier (referenced below).

From my tests, the traffic library appears to struggle with loading historical data. To demonstrate this, I compared fetching data using the traffic library to a custom SQL query. The custom SQL query returned results in just a few seconds, whereas the request via the traffic library seems to stall (as detailed in the earlier discussion).

Comparison Code:

using custom sql query

import paramiko
callsign = "%BEL%"
icao24 = "%"
start = datetime.datetime(2023, 6, 1, 8, 0, 0)
end = datetime.datetime(2023, 7, 1, 9, 5, 0)
lat_min, lat_max = 50.6041, 51.1106
lon_min, lon_max = 3.9189, 4.9705
start_time = int(start.timestamp())
start_hour = start_time - (start_time % 3600)
end_time = int(end.timestamp())
end_hour = end_time - (end_time % 3600)

request = (
    f"-q select * from state_vectors_data4"
    f" where callsign like '{callsign}'"
    f" and icao24 like '{icao24}'"
    f" and time>={start_time} and time<={end_time}"
    f" and hour>={start_hour} and hour<={end_hour}"
    f" and lat>={lat_min} and lat<={lat_max}"
    f" and lon>={lon_min} and lon<={lon_max}"
)
# request = "-q select * from state_vectors_data4 where callsign like '%BEL%' and icao24 like '%' and time>=1685599200 and time<=1688195100 and hour>=1685599200 and hour<=1688194800 and lat>=50.6041 and lat<=51.1106 and lon>=3.9189 and lon<=4.9705"
p = paramiko.SSHClient()
p.set_missing_host_key_policy(paramiko.AutoAddPolicy())
p.connect("data.opensky-network.org", port=2230, username="my_username", password="my_password")
stdin, stdout, stderr = p.exec_command(request)
opt = stdout.readlines()

using traffic library

from traffic.data import opensky
result_data = opensky.history(
    start='2023-06-01 08:00:00',
    end='2023-07-01 09:05:00',
    callsign='%BEL%',
    icao24 = '%',
    bounds =(3.9189, 50.6041, 4.9705,51.1106),
    cached=False
    )

In this case, even if the process completed, the result of result_data is None

Discussed in #374

^{Originally posted by nmatton October 17, 2023}
Hi there !
I try to load historical data but the process is going veeery slow, and most of the time ends up to be stuck.
Here is a small portion of code that reproduce the issue :

from traffic.data import opensky
from traffic.data import eurofirs
ebbu = eurofirs['EBBU']
flights = opensky.history(
    start='2023-07-15 00:00'
    airport = "EBBR",
    bounds=ebbu
)

I usually get the 2 or 3 first iterations but it then get stucks, or take a very long time (>10 minutes) per next iteration.

Is it an issue of the opensky service or I misuse it somewhere ?

Thanks for your help !

[nmatton]

possibly the bounds are in fact not in the right order. but I followed the doc here (west, south, east, north)
using

from traffic.data import eurofirs
ebbu = eurofirs['EBBU']
ebbu = eurofirs['EBBU']
result_data = opensky.history(
    start='2023-06-01 08:00:00',
    end='2023-07-01 09:05:00',
    callsign='%BEL%',
    icao24 = '%',
    bounds =ebbu,
    cached=False
    )

still returns None however...

[xoolive]

Thanks for the report. Am quite too busy until November to look into it. Could you please activate logging and check which query is generated to try to understand what could be wrong?

import logging

# before running the query
logging.basicConfig(level=logging.INFO)

[xoolive]

Hello, I found the issue, it was not in the bounds parameter but in the callsign code and icao code: icao24 = "%" is not supported (you just do not put the parameter for the same effect), and there was a bug if the callsign started with a %. With the same version, you can use callsign="BEL%" (which is probably what you want), but I am pushing a fix (on pyopensky) to support that.