Technical_Documentation

Technical Documentation

Modules

views.py

Central file to accept and propagate all requests from clients. Those include:

• upload firmware: User either drag and drops file or selects it by hand. For further information see (### file uploading)
  It is passed as FirmwareFile attached to the request which is stored as model in the database and saved packed on the server machine.
• analyse firmware: User fills django form which is send togehter with the request.
  It includes Reference to FirmwareFile, normal emba Options and expert mode Options, which is saved into the DB.
  Both the File and the Metainformation are passed to the BoundedExecuter (see ###)
• service dashboard: Displays currently running Analyzers with status information. Communication is done via asgi and channels. To retrieve the status information the emba log is read asynchronously
• central dashboard: TODO

file uploading

Archiver

The Archiver is basically an extended wrapper around shutils to support all common archive formats.
It supports packing, unpacking and validating of different archive types.
Unittest labeled as test_archiver.

BoundedExecutor

The BoundedExecutor is basically an extended wrapper around pythons ThreadPoolExecutor to support a selectable finite upper bound for running processes.
Internally it uses additional BoundedSemaphore to track the current load, on submit() the semaphore ius decremented on finished process incremented.
On submit_firmware, the Archiver is utilized to unpack the selected firmware to an new directory and also starting the emba.sh process with the blocking subprocess.call.
Unittest labeled as test_boundedExecutor.

inotify_wrap.py

The inotify reader class uses a python wrapper for the linux system call inotify(7). In our implementation, it adds a watch on the emba.log and sends events whenever the file changed. In this way we can trigger the next steps for live reading the emba.log.

asgi

For live data exchange between server and client, the Django framework provides us with websocket communication via asgi, which is also called Django Channels. For server side handling the framework has the consumers.py class in place, whereas for client side handling you just have to open a websocket connection, opening the IP and Port specified in the backend. As a result, multiple clients can connect to the backend. The url routing is declared in the routing.py file, which is the equivalent to the urls.py file for HTTP communication.

Frontend - embaServiceDashboard.js

This script running on client side, takes the messages sent via websocket, to show live information about running emba processes in the backend. Currently, it displays the percentage, the current module as well as the current phase, the emba process is currently in. In the future, it shows a container per process with the previous information extended by the firmware name and some other details about the backend procedures.

Backend - consumers.py

Our current server implementation creates a temporary empty log file and then waits in a blocking loop for new file changes on the emba.log via the inotify wrap. Whenever the log file has changed, the difference between the emba.log file and the temporary log file is calculated via a python difflib script and passed to rxpy method for further processing. After extracting the relevant information a message dictionary is updated and sent to the client. Then the changes are written to the temporary log file so that future diff computations will be correct.

Database

authentication

models

There are currently 2 different models:

• FirmwareFile:
  Used to track zipped firmwares. Featuring FileField as well as DateTimeField.
• Firmware:
  Used to track Analyzed firmwares. Custom fields provide additional attribute for defining expert field: CharFieldExpertMode, BooleanFieldExpertMode.
  Fields include reference to FirmwareFile, basic fields (e.g. version, vendor, architecture) and expert fields (e.g. web-report, multi-threading, dependencies).
  For further information on available flags consult the script like this: emba.sh -h

forms

The current forms are generated from above defined models. The forms additionally take care of html attributes like expert mode and class on instancing.
See https://docs.djangoproject.com/en/3.2/topics/forms/modelforms/ for further information.

Guidelines

testing

The project uses the django testing environment. To write your own unittest you need a python file labeled test_. Within there need to be class extending TestCase. On test execution all methodes in test classes are invoked an run.
For an example on unittests consult test_archiver and test_boundedExecutor.

There is Pipeline checking regression by running the django test environment: python manage.py test.
You are encouraged to run the tests locally beforehand.

logging

For logging use djangos logging environment.
Configuration can be found in embark/settings.py as dict LOGGING. Logs can be inspected at embark/logs/*.log

Usage (this will create the logfile 'web'):

logger = logging.getLogger('web')

[...]

logger.info("my very own log message")

For further reading see how to logging.

codestyle

Every contributor is obliged to the PEP8 coding style on python code. To check your conformity with pep8 locally: pycodestyle .
To get further information about the violation run: pycodestyle --show-source .
For further setting see pycodestyle documentation