Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Nativeformats #652

Merged
merged 62 commits into from
May 29, 2019
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
43c87ae
add iqarray
jopohl May 21, 2019
c5e357f
use iqarray
jopohl May 21, 2019
aac1319
fix some tests
jopohl May 21, 2019
f8c828c
fix some more tests
jopohl May 21, 2019
a2d5165
allow loading of complex16 file
jopohl May 21, 2019
3e85825
fix view
jopohl May 22, 2019
5962197
min max implicit
jopohl May 22, 2019
9f9d400
Merge remote-tracking branch 'origin/nativeformats' into nativeformats
jopohl May 22, 2019
a7e97eb
fix test
jopohl May 22, 2019
c87d93e
fix test
jopohl May 22, 2019
8eaa480
fix estimation
jopohl May 22, 2019
ad90756
fix psk
jopohl May 22, 2019
c5f02ae
fix filter tests
jopohl May 22, 2019
a0250b0
use 2d array
jopohl May 23, 2019
42b2fdd
fix type
jopohl May 23, 2019
fb8d2a1
remove fromarray calls
jopohl May 23, 2019
512462e
fix insert sine
jopohl May 23, 2019
302fb89
fix wav import
jopohl May 23, 2019
5fb35e4
fix modulation test
jopohl May 23, 2019
8673f16
fix tests
jopohl May 23, 2019
693904c
fix more stuff
jopohl May 23, 2019
f40c99d
fix sniffer
jopohl May 23, 2019
8bdf549
use more iqarray
jopohl May 23, 2019
9388536
fix typo
jopohl May 23, 2019
1700942
use more iqarray
jopohl May 23, 2019
2b95c08
fix test
jopohl May 23, 2019
926780b
fix more
jopohl May 23, 2019
d07dc33
fix tests
jopohl May 23, 2019
2535a8b
restore
jopohl May 23, 2019
a8a95d7
remove _fulldata
jopohl May 23, 2019
b22d6d5
first shot scaling function
jopohl May 24, 2019
ce5ad03
fix condition
jopohl May 24, 2019
a86f71e
add tests and shift
jopohl May 25, 2019
cbbde37
add conversion for uint8
jopohl May 25, 2019
7da05bf
add stuff
jopohl May 25, 2019
6378abd
add conversion and tests
jopohl May 25, 2019
045645b
add axis
jopohl May 25, 2019
d464cd5
airspy rx
jopohl May 26, 2019
827499c
bladerf rx
jopohl May 26, 2019
3ccd9db
complex32 support
jopohl May 26, 2019
6199d99
relative noise threshold
jopohl May 26, 2019
8b2098f
fix tooltip
jopohl May 26, 2019
9fdd130
fix center for complex16 and 32
jopohl May 26, 2019
4090b6c
bladerf tx
jopohl May 26, 2019
635673e
hackrf rx tx
jopohl May 26, 2019
3b1795d
fix data type
jopohl May 26, 2019
769fe8b
no percent
jopohl May 26, 2019
4dd918f
improve performance
jopohl May 26, 2019
5b2caad
fix filter
jopohl May 26, 2019
2949308
fix tests
jopohl May 26, 2019
b391923
fix sniffer
jopohl May 26, 2019
b410051
fix test
jopohl May 26, 2019
8d50e2b
limesdr rx tx
jopohl May 26, 2019
dd295b7
rx tx pluto
jopohl May 26, 2019
622450e
rx rtlsdr
jopohl May 26, 2019
2b0173c
sdrplay rx
jopohl May 26, 2019
8197acd
soundcard rx tx
jopohl May 26, 2019
9b3a922
usrp rx tx
jopohl May 26, 2019
8dd7ebe
set default data type
jopohl May 26, 2019
7c0d840
make modulation accuracy configurable
jopohl May 27, 2019
1bf70b8
fix cont modulator
jopohl May 27, 2019
df884d7
fix rx scene in simulator
jopohl May 27, 2019
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
137 changes: 87 additions & 50 deletions data/ui/options.ui
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@
<rect>
<x>0</x>
<y>0</y>
<width>803</width>
<width>814</width>
<height>822</height>
</rect>
</property>
Expand All @@ -21,58 +21,95 @@
<item>
<widget class="QTabWidget" name="tabWidget">
<property name="currentIndex">
<number>1</number>
<number>0</number>
</property>
<widget class="QWidget" name="tabGeneration">
<attribute name="title">
<string>Generation</string>
</attribute>
<widget class="QWidget" name="layoutWidget">
<property name="geometry">
<rect>
<x>20</x>
<y>20</y>
<width>324</width>
<height>102</height>
</rect>
</property>
<layout class="QGridLayout" name="gridLayout_4">
<item row="1" column="1">
<widget class="QLabel" name="labelFuzzingSamples">
<property name="text">
<string>Samples</string>
</property>
</widget>
</item>
<item row="0" column="0" colspan="2">
<widget class="QCheckBox" name="checkBoxDefaultFuzzingPause">
<property name="toolTip">
<string>&lt;html&gt;&lt;head/&gt;&lt;body&gt;&lt;p&gt;If you disable the default pause, the pause of the fuzzed message will be used.&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
</property>
<property name="text">
<string>Use a default pause for fuzzed messages</string>
</property>
</widget>
</item>
<item row="1" column="0">
<widget class="KillerDoubleSpinBox" name="doubleSpinBoxFuzzingPause">
<property name="decimals">
<number>3</number>
</property>
<property name="maximum">
<double>999999999.000000000000000</double>
</property>
</widget>
</item>
<item row="2" column="0" colspan="2">
<widget class="QCheckBox" name="checkBoxMultipleModulations">
<property name="text">
<string>Enable modulation profiles</string>
</property>
</widget>
</item>
</layout>
</widget>
<layout class="QVBoxLayout" name="verticalLayout_9">
<item>
<layout class="QGridLayout" name="gridLayout_4">
<item row="1" column="1">
<widget class="QLabel" name="labelFuzzingSamples">
<property name="text">
<string>Samples</string>
</property>
</widget>
</item>
<item row="0" column="0" colspan="2">
<widget class="QCheckBox" name="checkBoxDefaultFuzzingPause">
<property name="toolTip">
<string>&lt;html&gt;&lt;head/&gt;&lt;body&gt;&lt;p&gt;If you disable the default pause, the pause of the fuzzed message will be used.&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
</property>
<property name="text">
<string>Use a default pause for fuzzed messages</string>
</property>
</widget>
</item>
<item row="1" column="0">
<widget class="KillerDoubleSpinBox" name="doubleSpinBoxFuzzingPause">
<property name="decimals">
<number>3</number>
</property>
<property name="maximum">
<double>999999999.000000000000000</double>
</property>
</widget>
</item>
<item row="2" column="0" colspan="2">
<widget class="QCheckBox" name="checkBoxMultipleModulations">
<property name="text">
<string>Enable modulation profiles</string>
</property>
</widget>
</item>
</layout>
</item>
<item>
<widget class="QGroupBox" name="groupBoxModulationAccuracy">
<property name="title">
<string>Modulation Accuracy</string>
</property>
<layout class="QVBoxLayout" name="verticalLayout_7">
<item>
<widget class="QRadioButton" name="radioButtonLowModulationAccuracy">
<property name="text">
<string>Low (2x8 bit) - Recommended for HackRF and RTL-SDR</string>
</property>
</widget>
</item>
<item>
<widget class="QRadioButton" name="radioButtonMediumModulationAccuracy">
<property name="text">
<string>Medium (2x16 bit) - Recommended for BladeRF, PlutoSDR and SDRPlay</string>
</property>
</widget>
</item>
<item>
<widget class="QRadioButton" name="radioButtonHighModulationAccuracy">
<property name="text">
<string>High (2x32 bit) - Recommended if you are not sure what to choose</string>
</property>
</widget>
</item>
</layout>
</widget>
</item>
<item>
<spacer name="verticalSpacer">
<property name="orientation">
<enum>Qt::Vertical</enum>
</property>
<property name="sizeHint" stdset="0">
<size>
<width>20</width>
<height>500</height>
</size>
</property>
</spacer>
</item>
</layout>
</widget>
<widget class="QWidget" name="tabView">
<attribute name="title">
Expand Down Expand Up @@ -244,8 +281,8 @@
<rect>
<x>0</x>
<y>0</y>
<width>723</width>
<height>404</height>
<width>762</width>
<height>397</height>
</rect>
</property>
<layout class="QVBoxLayout" name="verticalLayout_4"/>
Expand Down
5 changes: 4 additions & 1 deletion data/ui/send_recv_device_settings.ui
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<x>0</x>
<y>0</y>
<width>860</width>
<height>668</height>
<height>711</height>
</rect>
</property>
<property name="windowTitle">
Expand Down Expand Up @@ -501,6 +501,9 @@ QGroupBox::indicator:checked {
<property name="text">
<string>Apply DC correction</string>
</property>
<property name="checked">
<bool>true</bool>
</property>
</widget>
</item>
</layout>
Expand Down
2 changes: 1 addition & 1 deletion data/ui/send_recv_sniff_settings.ui
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<x>0</x>
<y>0</y>
<width>482</width>
<height>388</height>
<height>424</height>
</rect>
</property>
<property name="windowTitle">
Expand Down
7 changes: 5 additions & 2 deletions data/ui/signal_frame.ui
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@
<x>0</x>
<y>0</y>
<width>1057</width>
<height>539</height>
<height>566</height>
</rect>
</property>
<property name="sizePolicy">
Expand Down Expand Up @@ -173,6 +173,9 @@
<property name="toolTip">
<string>&lt;html&gt;&lt;head/&gt;&lt;body&gt;&lt;p&gt;Set the &lt;span style=&quot; font-weight:600;&quot;&gt;noise magnitude&lt;/span&gt; of your signal. You can tune this value to mute noise in your signal and reveal the true data.&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
</property>
<property name="suffix">
<string/>
</property>
<property name="decimals">
<number>4</number>
</property>
Expand Down Expand Up @@ -437,7 +440,7 @@ If you want your protocol to be better seperated, edit the PauseLen using right-
</sizepolicy>
</property>
<property name="toolTip">
<string>&lt;html&gt;&lt;head/&gt;&lt;body&gt;&lt;p&gt;Choose the view of your signal. Analog, Demodulated or Spectrogram.&lt;/p&gt;&lt;p&gt;The quadrature demodulation uses a &lt;span style=&quot; font-weight:600;&quot;&gt;treshold of magnitude,&lt;/span&gt; to &lt;span style=&quot; font-weight:600;&quot;&gt;supress noise&lt;/span&gt;. All samples with a magnitude lower than this treshold will be eliminated (set to &lt;span style=&quot; font-style:italic;&quot;&gt;-127&lt;/span&gt;) after demod.&lt;/p&gt;&lt;p&gt;Tune this value by selecting a &lt;span style=&quot; font-style:italic;&quot;&gt;noisy area&lt;/span&gt; and mark it as noise using &lt;span style=&quot; font-weight:600;&quot;&gt;context menu&lt;/span&gt;.&lt;/p&gt;&lt;p&gt;Current noise treshold is: &lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
<string>&lt;html&gt;&lt;head/&gt;&lt;body&gt;&lt;p&gt;Choose the view of your signal. Analog, Demodulated or Spectrogram.&lt;/p&gt;&lt;p&gt;The quadrature demodulation uses a &lt;span style=&quot; font-weight:600;&quot;&gt;threshold of magnitudes,&lt;/span&gt; to &lt;span style=&quot; font-weight:600;&quot;&gt;supress noise&lt;/span&gt;. All samples with a magnitude lower than this treshold will be eliminated after demodulation.&lt;/p&gt;&lt;p&gt;Tune this value by selecting a &lt;span style=&quot; font-style:italic;&quot;&gt;noisy area&lt;/span&gt; and mark it as noise using &lt;span style=&quot; font-weight:600;&quot;&gt;context menu&lt;/span&gt;.&lt;/p&gt;&lt;/body&gt;&lt;/html&gt;</string>
</property>
<item>
<property name="text">
Expand Down
23 changes: 14 additions & 9 deletions src/urh/ainterpretation/AutoInterpretation.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,6 +10,7 @@
from urh.cythonext import auto_interpretation as c_auto_interpretation
from urh.cythonext import signal_functions
from urh.cythonext import util
from urh.signalprocessing.IQArray import IQArray


def max_without_outliers(data: np.ndarray, z=3):
Expand Down Expand Up @@ -59,7 +60,7 @@ def detect_noise_level(magnitudes):

mean_values = np.fromiter((np.mean(chunk) for chunk in chunks), dtype=np.float32, count=len(chunks))
minimum, maximum = util.minmax(mean_values)
if minimum / maximum > 0.9:
if maximum == 0 or minimum / maximum > 0.9:
# Mean values are very close to each other, so there is probably no noise in the signal
return 0

Expand Down Expand Up @@ -175,10 +176,11 @@ def detect_modulation(data: np.ndarray, wavelet_scale=4, median_filter_order=11)
return "OOK"


def detect_modulation_for_messages(signal: np.ndarray, message_indices: list) -> str:
def detect_modulation_for_messages(signal: IQArray, message_indices: list) -> str:
modulations_for_messages = []
complex = signal.as_complex64()
for start, end in message_indices:
mod = detect_modulation(signal[start:end])
mod = detect_modulation(complex[start:end])
if mod is not None:
modulations_for_messages.append(mod)

Expand Down Expand Up @@ -348,28 +350,31 @@ def get_bit_length_from_plateau_lengths(merged_plateau_lengths) -> int:
return int(result)


def estimate(signal: np.ndarray, noise: float = None, modulation: str = None) -> dict:
magnitudes = np.abs(signal)
def estimate(iq_array: IQArray, noise: float = None, modulation: str = None) -> dict:
if isinstance(iq_array, np.ndarray):
iq_array = IQArray(iq_array)

magnitudes = iq_array.magnitudes
# find noise threshold
noise = detect_noise_level(magnitudes) if noise is None else noise

# segment messages
message_indices = segment_messages_from_magnitudes(magnitudes, noise_threshold=noise)

# detect modulation
modulation = detect_modulation_for_messages(signal, message_indices) if modulation is None else modulation
modulation = detect_modulation_for_messages(iq_array, message_indices) if modulation is None else modulation
if modulation is None:
return None

if modulation == "OOK":
message_indices = merge_message_segments_for_ook(message_indices)

if modulation == "OOK" or modulation == "ASK":
data = signal_functions.afp_demod(signal, noise, 0)
data = signal_functions.afp_demod(iq_array.data, noise, 0)
elif modulation == "FSK":
data = signal_functions.afp_demod(signal, noise, 1)
data = signal_functions.afp_demod(iq_array.data, noise, 1)
elif modulation == "PSK":
data = signal_functions.afp_demod(signal, noise, 2)
data = signal_functions.afp_demod(iq_array.data, noise, 2)
else:
raise ValueError("Unsupported Modulation")

Expand Down
4 changes: 3 additions & 1 deletion src/urh/cli/urh_cli.py
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,8 @@

import numpy as np

from urh.signalprocessing.IQArray import IQArray

DEFAULT_CARRIER_FREQUENCY = 1e3
DEFAULT_CARRIER_AMPLITUDE = 1
DEFAULT_CARRIER_PHASE = 0
Expand Down Expand Up @@ -221,7 +223,7 @@ def modulate_messages(messages, modulator):

cli_progress_bar(0, len(messages), title="Modulating")
nsamples = sum(int(len(msg.encoded_bits) * modulator.samples_per_bit + msg.pause) for msg in messages)
buffer = np.zeros(nsamples, dtype=np.complex64)
buffer = IQArray(None, dtype=np.float32, n=nsamples)
pos = 0
for i, msg in enumerate(messages):
# We do not need to modulate the pause extra, as result is already initialized with zeros
Expand Down
16 changes: 10 additions & 6 deletions src/urh/controller/GeneratorTabController.py
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,7 @@
from urh.plugins.NetworkSDRInterface.NetworkSDRInterfacePlugin import NetworkSDRInterfacePlugin
from urh.plugins.PluginManager import PluginManager
from urh.plugins.RfCat.RfCatPlugin import RfCatPlugin
from urh.signalprocessing.IQArray import IQArray
from urh.signalprocessing.Message import Message
from urh.signalprocessing.MessageType import MessageType
from urh.signalprocessing.Modulator import Modulator
Expand Down Expand Up @@ -391,26 +392,29 @@ def generate_file(self):
except Exception as e:
logger.exception(e)
sample_rate = 1e6
FileOperator.save_data_dialog("generated.complex", modulated_samples, sample_rate=sample_rate, parent=self)
FileOperator.save_data_dialog("generated", modulated_samples, sample_rate=sample_rate, parent=self)
except Exception as e:
Errors.generic_error(self.tr("Failed to generate data"), str(e), traceback.format_exc())
self.unsetCursor()

def prepare_modulation_buffer(self, total_samples: int, show_error=True) -> np.ndarray:
memory_size_for_buffer = total_samples * 8
def prepare_modulation_buffer(self, total_samples: int, show_error=True) -> IQArray:
dtype = Modulator.get_dtype()
n = 2 if dtype == np.int8 else 4 if dtype == np.int16 else 8

memory_size_for_buffer = total_samples * n
logger.debug("Allocating {0:.2f}MB for modulated samples".format(memory_size_for_buffer / (1024 ** 2)))
try:
# allocate it three times as we need the same amount for the sending process
np.zeros(3*total_samples, dtype=np.complex64)
IQArray(None, dtype=dtype, n=3*total_samples)
except MemoryError:
# will go into continuous mode in this case
if show_error:
Errors.not_enough_ram_for_sending_precache(3*memory_size_for_buffer)
return None

return np.zeros(total_samples, dtype=np.complex64)
return IQArray(None, dtype=dtype, n=total_samples)

def modulate_data(self, buffer: np.ndarray) -> np.ndarray:
def modulate_data(self, buffer: IQArray) -> IQArray:
"""

:param buffer: Buffer in which the modulated data shall be written, initialized with zeros
Expand Down
8 changes: 7 additions & 1 deletion src/urh/controller/dialogs/ModulatorDialog.py
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,7 @@
from PyQt5.QtWidgets import QDialog, QMessageBox

from urh import constants
from urh.signalprocessing.IQArray import IQArray
from urh.signalprocessing.Modulator import Modulator
from urh.signalprocessing.ProtocolAnalyzer import ProtocolAnalyzer
from urh.ui.ui_modulation import Ui_DialogModulation
Expand Down Expand Up @@ -36,11 +37,16 @@ def __init__(self, modulators, tree_model=None, parent=None):

self.modulators = modulators

for graphic_view in (self.ui.gVCarrier, self.ui.gVData, self.ui.gVModulated):
for graphic_view in (self.ui.gVCarrier, self.ui.gVData):
graphic_view.scene_y_min = -1
graphic_view.scene_y_max = 1
graphic_view.scene_x_zoom_stretch = 1.1

min_max_mod = IQArray.min_max_for_dtype(self.current_modulator.get_dtype())
self.ui.gVModulated.scene_y_min = min_max_mod[0]
self.ui.gVModulated.scene_y_max = min_max_mod[1]
self.ui.gVModulated.scene_x_zoom_stretch = 1.1

self.set_ui_for_current_modulator()

self.ui.cbShowDataBitsOnly.setText(self.tr("Show Only Data Sequence\n"))
Expand Down
Loading