Skip to content
This repository has been archived by the owner on Dec 8, 2021. It is now read-only.

Documentation of LaSuli code and its structure

Jump to bottom
triinuerik edited this page Apr 15, 2019 · 29 revisions

Introduction

In this section, the structure of LaSuli web extension will be documented and explained. Also, some main development principles and notices will be described to give a clearer understanding of the project's inner workings. LaSuli was developed according to the Mozilla Browser Extension standard. The complete documentation for Mozilla Browser Extensions can be found here.

Structure

A web extension is essentially a collection of files that are packaged for easy distribution and installation. The files and directories usually follow a certain structure, which components will be described below.

LaSuli project structure

Here can be seen the general structure of LaSuli, to help better understand the chapters below and to get a visual idea of the building blocks of this extension.

manifest.json

This is the only file that is mandatory for every extension. In this file, the basic metadata of the extension will be described.

In LaSuli's project, this file can be found in the extension directory. The following metadata is described in the manifest.json file of LaSuli's project:

  • Name
  • Description
  • Version
  • Manifest version
  • Permissions

The manifest.json file can also contain pointers to several other types of files. In the case of LaSuli:

  • Applications
  • Default settings for both browser and sidebar (title, icons, etc)
  • Background scripts

Background scripts

Web extensions often need to perform long-term operations, that should not depend on the particular web page or browser window. To help with that, we have background scripts. They are loaded as soon as the web extension is opened and will stay until the extension is disabled/uninstalled. It is also possible to load background pages by specifying a HTML-file. The script has to be specified in the manifest.json file (as mentioned above).

In LaSuli, the background scripts can be found in src/backgroundScripts directory. The files are:

  • background.js
  • colorList.js
  • model.js
  • Resource.js
  • Viewpoint.js

Some more in depth explanations on those can be found in the model.js chapter below.

The specification in the manifest.json file is done in the following manner:

"background": {
    "scripts" : [
      "/dist/background.js"
    ]
  },

Another bonus of background scripts is the ability to use any of the WebExtension APIs in the script, provided you have specified the permissions in the manifest.json file.

In LaSuli, the APIs are accessed using the browser namespace. For example: tabs = browser.tabs;

Sidebar

A web extension can also include various user interface components, whose content is defined as a HTML-document. These can be for example:

  • Sidebars
  • Popups
  • Option pages

In the case of LaSuli, we have a sidebar. A sidebar is a pane on the left-hand side of the webpage. In LaSuli, this is where the user can see the results of their analysis. In the project, the HTML-document can be found in extension/page directory. All of the functionality is defined by files in the src/sidebar directory.

Similarly to background scripts, the sidebar has to be specified in the manifest.json file. It is done in the following manner:

"sidebar_action": {
    "default_panel": "/page/sidebar.html"
  },

All of the aforementioned user interface components are a type of Extension pages. It means that it is also possible to access the WebExtensions API, just like with background scripts. This is done also with the browser namespace. But meanwhile they are in their own tab and they have their own JavaScript event queue, their own globals, etc.

Content scripts

Content scripts are scripts that are loaded into the web page and run within the context of that page (as opposed to background scripts which are part of the extension, or scripts which are part of the web site itself). There is only one global scope per frame per extension, so variables from one content script can directly be accessed by another content script, regardless of how the content script was loaded. Background scripts can't directly access the content of web pages. So if your extension needs to do that, you need content scripts. In the case of LaSuli, we need to highlight words on the web page, so that is why content scripts are useful to this project.

Content scripts are able to use a small subset of the WebExtension APIs and they can exchange messages with the background scripts, which will be more thoroughly described below.

In LaSuli project, we can find the content scripts in src/contentScripts directory.

There are 3 ways to load a content script into a web page:

  1. At install time, into pages that match URL patterns using the content_scripts key in your manifest.json
  2. At runtime, into pages that match URL patterns using the contentScripts API
  3. At runtime, into specific tabs using tabs.executeScript()

An example from LaSuli, where a background script uses the 3rd method to load a content script:

async () => {
	await tabs.executeScript(this.props.tabId, {
		file: '/dist/content.js'
	});
}

This function is asynchronous (notice the async/await keywords), which is described in more detail below.

Web accessible resources

Web accessible resources are images, HTML, CSS and JavaScript that is included in the extension. In the case of LaSuli, there are some buttons and icons that are displayed in the extension. These are located in the extension/button and extension/icons directories.

LaSuli icon LaSuli icon grey

Here can be seen the icons used in LaSuli.

model.js

As mentioned before in the chapter about background scripts, in LaSuli's project they are found in the src/backgroundScripts directory. In the manifest.js specification we can see, that the automatically loaded script is called background.js. But this script doesn't contain a lot of logic by itself.

For this purpose, we utilise the model.js file. This file combines the Resource and Viewpoint classes to provide functionality for the extension. In that file is contained the main functionality, such as:

  • Connecting to and verifying a database
  • Handling the sessions
  • Fetching the highlights and viewpoints

Then, these functionalities are used in the background.js file by importing the model.js file and handling everything in an object-oriented way.

Some important principles

In this section, some more important principles of development in LaSuli will be described. Understanding these concepts and how they work will provide better understanding of this web extension and its mechanisms.

Internationalization and Localization

There are countless different languages and cultures in the world, which means that websites also have to be adapted to that. This is where the concepts of internationalization and localization become important. They focus on the linguistic aspect of development. The widely recognised notions for them are i18n and l10n (counting the letters between the first and last letters). In this chapter, they will be more thoroughly described and there will be proposed one framework to use in LaSuli project.

Life cycle of i18n and l10n

Here can be seen a general example of the lifecycle of an international product and the different steps it contains. This model is based on the one presented by the Localization Industry Standards Association.

i18n

The definitions of internationalization vary, but the W3C standard defines it so: Internationalization is the design and development of a product, application or document content that enables easy localization for target audiences that vary in culture, region, or language. That means, it is a way of writing code in a dynamic way, so that it could easily be adapted to different languages, without hardcoding the language. i18n could be thought as a "mindset" of writing code: designing the project in a way that could be useful to users all over the world.

l10n

The W3C standard defines localization as the adaptation of a product, application or document content to meet the language, cultural and other requirements of a specific target market. In essence, it means adapting an application developed in the i18n-mindset to one or many specific languages and/or cultures. That means making the necessary translations for the i18n-developed application to use. But in addition to just translations, it can mean also adapting other aspects to the language/culture like:

  • Date and time formats
  • Currency
  • Legal requirements
  • Colors and symbols

and many more...

LinguiJS

LinguiJS is an internationalization framework for JavaScript. It's the one used in doing i18n for LaSuli. It was the chosen solution for being:

  • Clean and readable - text and messages are surrounded just by tags, which keepts the code cohesive
  • Universal - LinguiJS can be used anywhere, providing functionality for JavaScript and also React
  • Unopinionated - LinguiJS supports message keys as well as auto generated messages

It also possesses a very powerful tool called CLI. CLI provides the lingui command for extracting, merging and compiling message catalogs. The extracting will generate .json files, with key-value pairs in Message catalogs, which only need to be translated.

There are 3 different ways to provide the key for translations. They will be explained and compared.

1. Source language

This is the default for providing the key value for translations. The key is generated from the content of <Trans> tags, which means it uses source language.

<h1><Trans>Message</Trans></h1>

The generated key will look like this:

{
   "Message": ""
}

It can be seen that in the extracted .json file, the key is automatically taken from the contents of the <Trans> tags.

2. Message ID

It is also possible to assign an id attribute inside the <Trans> tags, which signals to the extracting tool to use this value as the key.

<h1><Trans id="message.title" defaults="Message" /></h1>

Which generates the following key:

{
   "message.title": ""
}

Here the key will be the previously assigned id value. The translation will be automatically provided for the source language of the application.

3. Hybrid

It is also possible to have a hybrid of the previous two. That means, there is both an id attribute for the key and a source language in the <Trans> tags.

<h1><Trans id="message.title">Message</Trans></h1>

The generated key is identical to the previous option:

{
   "message.title": ""
}

Again, the translation will be automatically provided for the source language.

Content negotiation

Content negotiation is a concept that is a part of the HTTP protocol and is closely associated with Internationalization and Localization. It is a way to accept exactly the kind of resources the user has specified, which among other things also includes their preferred language. Using Content negotiation we are able to serve a different representation of a resource at the same URI. This means that the user requests a resource with the general URI, but the server is able to identify the user's preferences and return the most appropriate resources according to them. There are two ways to negotiate with the servers:

  1. Server-driven negotiation (proactive negotiation)
  2. Agent-driven negotiation (reactive negotiation)

The more common mechanism is server-driven negotiation, which will be explained a bit more thoroughly.

Server-driven negotiation

Server-driven negotiation (also called proactive negotiation) is done using special HTTP headers, which allow the client or server pass additional information with the sent request or response. These headers are carrying the data to identify the preference choices the user has made. The server will then analyse the contents of these headers and offer the best representation of the resource possible. A visual representation can be seen on the diagram below (from MDN Web Docs).

Screenshot 2019-04-15 at 11 20 33

There are different kinds of headers to specify the different kinds of preferences the client wishes to receive. Among many other things, the most common ones are:

  • Character encodings
  • Language
  • MIME types of media resources
  • Device type and its screen measurements
  • RAM memory of the device
  • Browser being used

Accept-Language

With Internationalization and Localization being our main concern regarding Content negotiation, the more interesting HTTP header is Accept-Language. This header lets the server know which natural languages the client is able to understand and also the preferred locale. This header is a hint to be used when the server has no way of determining the language via another way, like a specific URL, that is controlled by an explicit user decision.

Here is an example of an Accept-Language header that can be sent with a request.

Accept-Language: en-US,en;q=0.5

Firstly in the header is seen the language tag (also known as locale identifier). It consists of the 2-3 letter base language tag (en in the example) that can optionally be followed by subtags separated by -. Subtags can be used to specify a dialect or a region or an alphabet (en-US in the example).

Optionally, there can also be specified quality values. Quality values are used to describe the order of priority of values, the highest priority being 1 and the lowest 0. They are marked by ;q=

Arrow functions

Arrow functions were introduced with ES6 as a new syntax for writing JavaScript functions, with the aim to save developers time and simplify the syntax of functions. They utilize a new token, =>, that looks like an arrow.

What is ES6?

If you are not familiar with the notation of ES6, a small explanation will be given. ES6 is an update to EcmaScript. EcmaScript is the standardized scripting language that JavaScript implements. For a long time, developers have not been content with the previous update - it lacks a lot of useful features that other languages have. Because JavaScript is so widely used (LaSuli is completely developed in JavaScript and React), a new update will have a big impact on web development. That's why ES6 is such an important improvement.

Back to arrow functions...

Arrow functions allow us to use fewer lines and represent the code more clearly, simplify function scoping (the visibility of variables) and the this keyword. They work much like Lambdas in other languages like C# or Python. By using arrow functions, we avoid having to type the function keyword, return keyword and curly brackets.

Now will be demonstrated some examples of arrow functions from ES6 and a comparison with how it used to be with the previous version, ES5.

It is easily seen that the ES6 version looks more clean and is definitely more readable. When there is only one expression in the body of the functions, curly brackets are not even needed. Also, the function uses only one line:

// ES5
var multiplyES5 = function(x, y) {
  return x * y;
};

// ES6
const multiplyES6 = (x, y) => x * y;

It can be seen that, when there is only on parameter, the parentheses are optional:

//ES5
var phraseSplitterEs5 = function phraseSplitter(phrase) {
  return phrase.split(' ');
};

//ES6
const phraseSplitterEs6 = phrase => phrase.split(" ");

When the function doesn't require any parameters, you can just leave the parentheses empty:

//ES5
var docLogEs5 = function docLog() {
    console.log(document);
};

//ES6
var docLogEs6 = () => { console.log(document); };

This example shows the powerful way that arrow functions allow us to manipulate arrays. It is so much more readable and allows really smart ways to filter/collect data:

// ES5
var prices = smartPhones.map(function(smartPhone) {
  return smartPhone.price;
});

// ES6
const prices = smartPhones.map(smartPhone => smartPhone.price);

There are many more uses for arrow functions, which can be further researched on the many resources available online. The main points given in this chapter are just to give a better understanding of LaSuli's code, as this notion is widely used there.

Asynchronous operations

As it was also mentioned before, the background and content scripts utilise the JavaScript/WebExtension APIs in their work. To do this, we use the browser namespace. Some of these APIs will return a Promise, which is an object that represents the eventual completion (or failure) of an asynchronous operation, and its resulting value. In the example below, the functionality of a Promise will be demonstrated:

var promise1 = new Promise(function(resolve, reject) {
  setTimeout(function() {
    resolve('foo');
  }, 300);
});

promise1.then(function(value) {
  console.log(value);
  // expected output: "foo"
});

console.log(promise1);
// expected output: [object Promise]

As it is seen in the example, the Promise itself is just an object and calling on it will not return any value. But by applying the .then() function on it, we can access the value when the Promise will get fulfilled (or rejected). It is also possible to chain multiple Promises together, which is called chaining. It is very useful in cases when the later Promises depend on the completion of the first ones. It looks like this:

doSomething()
.then(function(result) {
  return doSomethingElse(result);
})
.then(function(newResult) {
  return doThirdThing(newResult);
})
.then(function(finalResult) {
  console.log('Got the final result: ' + finalResult);
})
.catch(failureCallback);

Sometimes these chains can get very long and complicated. This is where the async/await functionality becomes very useful. It helps to simplify and clean your code by helping to write asynchronous code in a way that looks synchronous. Here are the steps to follow for using async/await in React:

  1. Put the async keyword in front of your functions
  2. Use await in the function’s body
  3. Catch the errors

Let's look at some LaSuli code examples where this method is used.

browser.tabs.onActivated.addListener(async (activeInfo) => {
	try {
		let tab = await tabs.get(activeInfo.tabId);
		await updateHighlightNumber(tab.id, tab.url, false);
	} catch (e) {
		errorHandler(e, activeInfo.tabId);
	}
});

As we can see, indeed the code resembles synchronous code, it looks simple and clean. The Promise returned from the API (browser namespace) will be handled nicely.

NB! It has to be remembered that async/await is not supported by all browsers (ex. IE) and behave accordingly.

Messages

In order to communicate with the background scripts, the content scripts utilise a messaging system. There are two basic patterns for communicating between the background scripts and content scripts:

  1. You can send one-off messages, with an optional response
  2. You can set up a longer-lived connection between the two sides, and use that connection to exchange messages.

LaSuli uses the first option, so this one will be described below.

In content script In background script
Send a message browser.runtime.sendMessage() browser.tabs.sendMessage()
Receive a message browser.runtime.onMessage browser.runtime.onMessage

As it is seen, there are 3 different APIs to communicate, depending on the type of script and if it is needed to send or receive a message. The API for receiving a message stays the same for both content and background scripts, but it is important to pay attention to the differences in sending a message.

Below are some examples from LaSuli to illustrate this concept.

Let's say that we have a background script file that wishes to send a message to a content script. It will utilise the browser.tabs.sendMessage API to send a message. Notice the aim attribute of the message.

browser.tabs.sendMessage(this.props.tabId, {
	aim: 'highlight',
	labels,
	fragments
})

This message will be received by the content script using the browser.runtime.onMessage API.

browser.runtime.onMessage.addListener(messageHandler);

This will call the function messageHandler, whose aim is to decipher the message and act accordingly. The cases are recognised by the aim attribute (remember from before).

messageHandler = async (message) => {
	switch (message.aim) {
	case 'highlight':
		erase();
		highlight(message.fragments, message.labels);
		return true;

Here is presented a diagram of the messaging system in LaSuli. The titles over each section are indicating the type of the scripts and which directory they are located in. The sending and receiving scripts are indicated by the way the arrow is pointing. On each connection (arrow) we can see the different types of messages, which can be distinguished by their aim attribute.

Diagram of message system in LaSuli

For example, it can be seen that the script Display.jsx, located in the sidebar directory, is sending messages to the script content.js, located in contentScripts directory. The communication is done using two kinds of messages, whose aims are isLoaded and highlight.

Bibliography

Clone this wiki locally