Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Documenting the JS Instrument #949

Merged
merged 4 commits into from
Dec 21, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
22 changes: 12 additions & 10 deletions docs/Platform-Architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,29 +21,31 @@ In OpenWPM we have a watchdog thread that tries to ensure two things.
- `memory_watchdog`
- It is part of default manager_params. It is set to false by default which can manually be set to true.
- It is a watchdog that tries to ensure that no Firefox instance takes up too much memory.
- It is mostly useful for long running cloud crawls.
- It is mostly useful for long-running cloud crawls.

### Issuing commands

OpenWPM uses the `CommandSequence` as a fundamental unit of work.
A `CommandSequence` describes as series of steps that will execute in order on a particular browser.
All available Commands are visible by inspecting the `CommandSequence` API.

For example you could wire up a `CommandSequence` to go to a given url and take a screenshot of it by writing:
For example, you could wire up a `CommandSequence` to go to a given url and take a screenshot of it by writing:

```python
command_sequence = CommandSequence(url)
# Start by visiting the page
command_sequence.get(sleep=3, timeout=60)
command_sequence.save_screenshot()
from openwpm.command_sequence import CommandSequence
url = "https://example.com"
command_sequence = CommandSequence(url)
# Start by visiting the page
command_sequence.get(sleep=3, timeout=60)
command_sequence.save_screenshot()
```

But this on its own would do nothing, because `CommandSequence`s are not automatically scheduled.
Instead, you need to submit them to a `TaskManager` by calling:

```python
manager.execute_command_sequence(command_sequence)
manager.close()
manager.execute_command_sequence(command_sequence)
manager.close()
```

Please note that you need to close the manager, because by default `CommandSequence`s are executed in a non-blocking fashion meaning that you might reach the end of your main function/file before the CommandSequence completed running.
Expand Down Expand Up @@ -87,15 +89,15 @@ which provides stability in logging data despite the possibility of individual b
## The WebExtension

All of our data collection happens in the OpenWPM WebExtension, which can be found under [Extension](../Extension).
The Extension makes heavy use of priviliged APIs and can only be installed on unbranded or custom builds of Firefox with add-on security disabled.
The Extension makes heavy use of privileged APIs and can only be installed on unbranded or custom builds of Firefox with add-on security disabled.

The currently supported instruments can be found in [Configuration.md](Configuration.md#Instruments)

## Data Aggregator

### Overview

One of the Data Aggregators, contained in `openwpm/DataAggregator`, gets spawned in a separate process and receives data from the WebExtension and the platform alike. We as previously mentioned we support both local as well as remote data saving.
One of the Data Aggregators, contained in `openwpm/DataAggregator`, gets spawned in a separate process and receives data from the WebExtension and the platform alike. We as previously mentioned we support both local and remote data saving.
The most useful feature of the Data Aggregator is the fact that it is isolated from the other processes through a network socket interface (see `openwpm/SocketInterface.py`).

### Data Logged
Expand Down
4 changes: 4 additions & 0 deletions docs/developers/CurrentFlow.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Loading
Loading