Skip to content

Commit

Permalink
IDT v0.0.2 (#29931)
Browse files Browse the repository at this point in the history 
* Squashed commit of the following:

commit 29d46b1
Author: bozowski <[email protected]>
Date:   Mon Oct 23 21:48:48 2023 -0700

    Nit

commit b432937
Author: bozowski <[email protected]>
Date:   Mon Oct 23 21:11:22 2023 -0700

    README and error reporting and controlelr resturct

commit 4ac762d
Author: bozowski <[email protected]>
Date:   Mon Oct 23 19:15:59 2023 -0700

    Logging

commit 5b6b325
Author: bozowski <[email protected]>
Date:   Mon Oct 23 18:38:25 2023 -0700

    Move logging

commit 9df3206
Author: bozowski <[email protected]>
Date:   Mon Oct 23 18:13:57 2023 -0700

    Prober

commit 4683b34
Author: bozowski <[email protected]>
Date:   Mon Oct 23 17:55:03 2023 -0700

    Make logcat resiliant

commit 12bb816
Author: bozowski <[email protected]>
Date:   Mon Oct 23 17:30:37 2023 -0700

    Warn when file is not growing

commit 1451a02
Author: bozowski <[email protected]>
Date:   Mon Oct 23 16:03:36 2023 -0700

    Real time analysis

commit 8c8ac7f
Author: bozowski <[email protected]>
Date:   Mon Oct 23 12:22:00 2023 -0700

    Fix capabilities, screen

commit 6ccfcc7
Author: bozowski <[email protected]>
Date:   Mon Oct 23 11:27:38 2023 -0700

    Snoop

commit 29d8969
Author: bozowski <[email protected]>
Date:   Mon Oct 23 11:12:26 2023 -0700

    HCI logs

commit a856bf6
Author: bozowski <[email protected]>
Date:   Fri Oct 20 15:04:32 2023 -0700

    Fix pull

commit 1239a90
Author: bozowski <[email protected]>
Date:   Fri Oct 20 14:51:22 2023 -0700

    OK

commit 2b3db4d
Author: bozowski <[email protected]>
Date:   Fri Oct 20 14:09:16 2023 -0700

    Screen recording

commit 5d02aa6
Author: bozowski <[email protected]>
Date:   Fri Oct 20 13:34:05 2023 -0700

    Nits

commit 85517ff
Author: bozowski <[email protected]>
Date:   Fri Oct 20 13:07:40 2023 -0700

    Fix prober, color config

commit c5d668c
Author: bozowski <[email protected]>
Date:   Fri Oct 20 12:45:20 2023 -0700

    Screen

commit 6352581
Author: bozowski <[email protected]>
Date:   Thu Oct 19 21:39:05 2023 -0700

    Nit

commit 1c8a45a
Author: bozowski <[email protected]>
Date:   Thu Oct 19 21:22:14 2023 -0700

    Nit

commit 51c187f
Author: bozowski <[email protected]>
Date:   Thu Oct 19 21:10:07 2023 -0700

    Splash

commit 0091588
Author: bozowski <[email protected]>
Date:   Thu Oct 19 20:47:48 2023 -0700

    Refactor

commit c480a02
Author: bozowski <[email protected]>
Date:   Thu Oct 19 15:41:39 2023 -0700

    Refactor

commit 024eeb6
Author: bozowski <[email protected]>
Date:   Thu Oct 19 15:27:05 2023 -0700

    Bugreport

commit efcc621
Author: bozowski <[email protected]>
Date:   Thu Oct 19 14:51:30 2023 -0700

    Nit

commit d5474b7
Author: bozowski <[email protected]>
Date:   Thu Oct 19 14:33:18 2023 -0700

    Configs

commit 0fcbb10
Author: bozowski <[email protected]>
Date:   Thu Oct 19 04:26:54 2023 -0700

    TODO

commit b1c7a30
Author: bozowski <[email protected]>
Date:   Thu Oct 19 04:20:18 2023 -0700

    Clean BUILD

commit d86778a
Author: bozowski <[email protected]>
Date:   Thu Oct 19 04:16:19 2023 -0700

    Logging bug

commit 6ebf7c0
Author: bozowski <[email protected]>
Date:   Thu Oct 19 03:03:48 2023 -0700

    Branch

* Restyled by prettier-markdown

* Restyled by shellharden

* Restyled by autopep8

* Restyled by isort

* Spelling

* Lints

* Support RPi install

* Restyled by prettier-markdown

* Uneeded sudo

* Readme clarification

* Remove dead comment

* Remove dead comment

* Squashed commit of the following:

commit 36f3ceb
Author: Austin Bozowski <[email protected]>
Date:   Wed Oct 25 20:07:16 2023 -0700

    Multiproc to async and target macOS

* Fix stopping procs on mac

* Replace multiproc with async

* Fix sudo kill issue on macOS

* Note

* Fix stopping pcap issue on macOS

* Cleanup

* BLE scanning on macOS

* Final cleanup for macOS and README

* Squashed commit of the following:

commit 3da0875
Author: Austin Bozowski <[email protected]>
Date:   Fri Nov 3 01:49:45 2023 -0700

    README

commit b7d115d
Author: Austin Bozowski <[email protected]>
Date:   Fri Nov 3 01:44:58 2023 -0700

    Refactor

commit b9dca9d
Author: Austin Bozowski <[email protected]>
Date:   Fri Nov 3 01:16:11 2023 -0700

    Refactor

commit d960b41
Author: Austin Bozowski <[email protected]>
Date:   Fri Nov 3 00:43:21 2023 -0700

    Logging

commit 4f6644c
Author: Austin Bozowski <[email protected]>
Date:   Fri Nov 3 00:18:11 2023 -0700

    Discovery

commit 6900a19
Author: Austin Bozowski <[email protected]>
Date:   Thu Nov 2 23:55:17 2023 -0700

    README

commit f2e9a8c
Author: Austin Bozowski <[email protected]>
Date:   Thu Nov 2 23:44:15 2023 -0700

    README

commit dc98974
Author: Austin Bozowski <[email protected]>
Date:   Thu Nov 2 23:27:02 2023 -0700

    Write dnssd log

commit 1fe3f98
Author: Austin Bozowski <[email protected]>
Date:   Thu Nov 2 22:34:32 2023 -0700

    Cleanup probers

commit 5e48bc0
Author: Austin Bozowski <[email protected]>
Date:   Thu Nov 2 14:55:03 2023 -0700

    Probes

commit c9d14ed
Author: Austin Bozowski <[email protected]>
Date:   Thu Nov 2 13:22:21 2023 -0700

    Probers

commit 12307e4
Author: Austin Bozowski <[email protected]>
Date:   Wed Nov 1 15:43:31 2023 -0700

    Earlier prober

* Nits

* Remove temp tests

* Prevent re run of Bash proc and nits

* Restyled by prettier-markdown

* Restyled by autopep8

* Restyled by isort

* Spelling

* Lint

* Respond comment.

* Comment and rename

* Nits

* Clarify

* Add info on interfaces to README

* Document failure mode of not allowing bt perms in macOS

* Restyled by prettier-markdown

* Restyled by isort

* Verify py version and fix non py dep check

* Move host dependencies back to main

* Fix py version check

* Make task cleanup more graceful

* Nit

* Respond comments

* Ensure which is available in host_platform

* Add mac tcpdump build script thanks to James
Rename timeout config and explain it in comments
Slight change to where timeouts are in controller

* README

* Logging

* Explain all DNS SD content to the user (parse TXT etc.)

* Remove TODO

* TTL and nits

* Logging, limit tracert, nits

* Restyled by prettier-markdown

* Restyled by shellharden

* Restyled by shfmt

* Restyled by autopep8

* Restyled by isort

* Integrate mac android tcp and bump restyle

* Use termcolor for colored logs

* Timeout config documentation

* Remove unnecessary abstractions from controller

* Cleanup py version check

* Cleanup error reporting in controller

* Remove unneeded instance vars in playservices

* Fix logcat file scope issue introd in last commit

* Fix dedent

* Add bison to Dockerfile

* Make screen on check in android explicit

* Restyled by autopep8

* Restyled by isort

---------

Co-authored-by: Restyled.io <[email protected]>
  • Loading branch information
@aBozowski @restyled-commits @pull
2 people authored and pull[bot] committed Dec 7, 2023
1 parent f459299 commit d1ae9da
Show file tree
Hide file tree
Showing 55 changed files with 2,444 additions and 585 deletions.
1 change: 1 addition & 0 deletions src/tools/interop/idt/.gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -7,3 +7,4 @@ __pycache__/
pycache/
venv/
.zip
BUILD
6 changes: 6 additions & 0 deletions src/tools/interop/idt/Dockerfile
View file
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ RUN apt-get update && \
adb \
aircrack-ng \
apt-utils \
bison \
byacc \
dnsutils \
flex \
gcc-aarch64-linux-gnu \
gcc-arm-linux-gnueabi \
git \
glib-2.0 \
kmod \
Expand Down
209 changes: 131 additions & 78 deletions src/tools/interop/idt/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,32 @@ issue uncovered via the manual test. Each ecosystem may implement an analysis
that analyzes capture data, displays info to the user, probes the local
environment and generates additional artifacts.

## Getting started
## Single host installation (no Raspberry Pi)

All features of `idt` are available on macOS and Linux (tested with Debian based
systems).
If you would prefer to execute capture and discovery from a Raspberry Pi, read
the next section instead.

The machine running `idt` should be connected to the same Wi-Fi network used for
testing.
Follow the steps below to execute capture and discovery without a Raspberry Pi:

- From the parent directory of `idt`, run `source idt/scripts/alias.sh`.
- Optionally, run `source idt/scripts/setup_shell.sh` to install aliases
permanently.
- After `idt` aliases are available in your environment, calling any `idt`
command will automatically create a new virtual environment and install
python dependencies.
- If you're missing non-Python dependencies, you'll be prompted to install
them until they're available.
- Bluetooth discovery on macOS will require granting the program where `idt`
is run, e.g. terminal emulator or IDE permission to access bluetooth in
macOS settings.
- Failure to do so may result in any of the following:
- A single `abort` message and no further output in the terminal.
- Failure with a relevant stack trace in the terminal.
- A prompt to allow the application access to bluetooth.

## Raspberry Pi installation

Expand Down Expand Up @@ -137,62 +162,21 @@ idt_clean
```
NOTE the idt artifacts directory is contained in idt, so running this will
delete any artifacts ([TODO] change).
delete any artifacts.
Then from the admin computer:
```
idt_push
```
## Single host installation (no Raspberry Pi)
Follow the steps below to execute capture and discovery without a Raspberry Pi.
### Linux installation
#### Requirements
- This package should work on most Debian (/based) systems.
- `idt` is currently tested on `Python 3.11`.
- `adb` and `tcpdump` are required.
- The machine running `idt` should be connected to the same Wi-Fi network used
for testing.
#### Setup
- From the parent directory of `idt`, run `source idt/scripts/alias.sh`.
- Optionally, run `source idt/scripts/setup_shell.sh` to install aliases
permanently.
> You may use `idt` in a Python virtual environment OR using a container from
> the idt image.
#### Python virtual environment
- After `idt` aliases are available in your environment, calling any `idt`
command will automatically create a new virtual environment and install
dependencies.
#### Docker
- Run `idt_build` and `idt_activate` to enter the `idt` container.
[TODO] Podman
### macOS installation
Most features other than BLE should work on macOS.
Follow the Linux installation steps above, but do not use Docker.
[TODO] macOS BLE support
## User guide
> **_IMPORTANT_**
> `idt_` commands are shell aliases helpful for administrative commands.
> `idt` invokes the `idt` python package.
> `idt` invokes the `idt` python package.
> Output from `idt` will generally be colorized while output from sub processes
> is generally not.
RPi users, as needed:
Expand All @@ -208,45 +192,38 @@ RPi users, as needed:
### Capture
> **_IMPORTANT_**
> Ensure you've made it to the log line "Starting real time analysis, press
> enter to stop!" before launching the app under test.
```
idt capture -h

usage: idt capture [-h] [--platform {Android}]
[--ecosystem {PlayServices,PlayServicesUser,ALL}]
[--pcap {t,f}]
[--interface {wlp0s20f3,docker0,lo}]
[--additional {t,f}]
usage: idt capture [-h] [--platform {Android}] [--ecosystem {PlayServicesUser,PlayServices,ALL}] [--pcap {t,f}] [--interface {wlp0s20f3,lo,docker0,any}]

options:
-h, --help show this help message and exit
--platform {Android}, -p {Android}
Run capture for a particular platform
(default Android)
--ecosystem {PlayServices,PlayServicesUser,ALL}, -e {PlayServices,PlayServicesUser,ALL}
Run capture for a particular ecosystem or ALL
ecosystems (default ALL)
Run capture for a particular platform (default Android)
--ecosystem {PlayServicesUser,PlayServices,ALL}, -e {PlayServicesUser,PlayServices,ALL}
Run capture for a particular ecosystem or ALL ecosystems (default ALL)
--pcap {t,f}, -c {t,f}
Run packet capture (default t)
--interface {wlp0s20f3,docker0,lo}, -i {wlp0s20f3,docker0,lo}
Run packet capture against a specified
interface (default wlp0s20f3)
--additional {t,f}, -a {t,f}
Run ble and mdns scanners in the background
while capturing (default t)
--interface {wlp0s20f3,lo,docker0,any}, -i {wlp0s20f3,lo,docker0,any}
Specify packet capture interface (default any)
```
For packet capture interface (`-i`/`--interface`:
- On macOS, the only available interface is `any`.
- On Linux, `idt` checks available interfaces from `/sys/class/net/` as well
as allowing `any`.
#### Artifacts
Each ecosystem and platform involved in the capture will have their own
subdirectory in the root artifact dir.
To download your artifacts, run these commands from your admin computer:
`idt_fetch_artifacts`
On windows admin computers, you may use `FileZilla` to pull the archive listed
at the end of output.
### Discovery
```
Expand All @@ -260,7 +237,7 @@ options:
Specify the type of discovery to execute
```
#### ble
#### BLE
```
idt discover -t b
Expand All @@ -274,13 +251,90 @@ idt discover -t d
#### Artifacts
There is a per device log for ble scanning in `ble` subdirectory of the root
artifact dir.
There is a per device log in `ble` and `dnssd` subdirectory of the root artifact
dir.
### Probe
```
usage: idt probe [-h]

options:
-h, --help show this help message and exit
```
[TODO] dnssd per device log
Collect contextually relevant networking info from the local environment and
provide artifacts.
## Troubleshooting
- Wireless `adb` may fail to connect indefinitely depending on network
configuration. Use a wired connection if wireless fails repeatedly.
- Change log level from `INFO` to `DEBUG` in root `config.py` for additional
logging.
- Compiling `tcpdump` for android may require additional dependencies.
- If the build script fails for you, try
`idt_go && source idt/scripts/compilers.sh`.
- You may disable colors and splash by setting `enable_color` in `config.py`
to `False`.
- `idt_clean_child` will kill any stray `tcpdump` and `adb` commands.
- `idt_check_child` will look for leftover processes.
- Not expected to be needed outside of development scenarios.
## Project overview
- The entry point is in `idt.py` which contains simple CLI parsing with
`argparse`.
### `capture`
- `base` contains the base classes for ecosystems and platforms.
- `controller` contains the ecosystem and platform producer and controller
- `loader` is a generic class loader that dynamically imports classes matching
a given super class from a given directory.
- `/platform` and `/ecosystem` contain one package for each platform and
ecosystem, which should each contain one implementation of the respective
base class.
### `discovery`
- `matter_ble` provides a simple ble scanner that shows matter devices being
discovered and lost, as well as their VID/PID, RSSI, etc.
- `matter_dnssd` provides a simple DNS-SD browser that searches for matter
devices and thread border routers.
### `probe`
- `probe` contains the base class for (`idt`'s) host platform specific
implementation.
- Reuses the dnssd discovery implementation to build probe targets.
- Calls platform + addr type specific probe methods for each target.
- `linux` and `mac` contain `probe` implementations for each host platform.
### `utils`
- `log` contains logging utilities used by everything in the project.
- `artifact` contains helper functions for managing artifacts.
- `shell` contains a simple helper class for background and foreground Bash
commands.
- `host_platform` contains helper functions for the interacting with the host
running `idt`.
### Conventions
- `config.py` should be used to hold development configs within the directory
where they are needed.
- It may also hold configs for flaky/cumbersome features that might need
to be disabled in an emergency.
- `config.py` **should not** be used for everyday operation.
- When needed, execute builds in a folder called `BUILD` within the source
tree.
- `idt_clean_all` deletes all `BUILD` dirs and `BUILD` is in `.gitignore`.
## Extending functionality
### Capture
Ecosystem and Platform implementations are dynamically loaded.
For each package in `capture/ecosystem`, the ecosystem loader expects a module
Expand All @@ -299,14 +353,13 @@ $ idt capture -h
usage: idt capture [-h] [--platform {Android}] [--ecosystem {DemoExtEcosystem...
```
> **IMPORTANT:** Note the following runtime expectations of ecosystems:
> `analyze_capture()` must not block the async event loop excessively and must
> not interact with standard in
The platform loader functions the same as `capture/ecosystem`.
For each package in `capture/platform`, the platform loader expects a module
name matching the package name.
This module must contain a single class which is a subclass of
`capture.base.PlatformLogStreamer`.
Note the following runtime expectations of platforms:
- Start should be able to be called repeatedly without restarting streaming.
- Stop should not cause an error even if the stream is not running.
2 changes: 2 additions & 0 deletions src/tools/interop/idt/__main__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@
#

from idt import InteropDebuggingTool
from utils.host_platform import verify_py_version

if __name__ == "__main__":
verify_py_version()
InteropDebuggingTool()
6 changes: 2 additions & 4 deletions src/tools/interop/idt/capture/__init__.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -17,16 +17,14 @@

from capture import ecosystem, platform

from .factory import EcosystemCapture, EcosystemController, EcosystemFactory, PlatformFactory, PlatformLogStreamer
from .controller import EcosystemCapture, PlatformLogStreamer
from .pcap import PacketCaptureRunner

__all__ = [
'ecosystem',
'platform',
'controller',
'EcosystemCapture',
'EcosystemController',
'EcosystemFactory',
'PacketCaptureRunner',
'PlatformFactory',
'PlatformLogStreamer',
]
Loading

0 comments on commit d1ae9da

Please sign in to comment.