Skip to content

Rulers: Tokens and Processing

Jump to bottom
Alteras1 edited this page Jun 26, 2020 · 13 revisions

Tokens are the workhorse of the entire extension. Simply put, they determine the HTML output that will conform to the specified method. Unfortunately, we're still not completely aware of what a token can and can't do, but we'll cover everything we know for certain.

But first, let us start with passing tokens, as well as relevant tag components into a function. See Dissection of a Tag for descriptions of tag, tagInfo, and content.

Method Functions

Functions can be made to process the tag, the tag arguments, and even the tag contents, providing far more flexibility than just outputting a simple HTML element + class (see wrap). Here are the two types of functions that can be made.

method: function (token, [tagInfo, content]) { } //single token

method: function (startToken, endToken, [tagInfo, content]) { }  //double token

Each function requires a single token or the combination startToken and endToken, with the optional arguments tagInfo and content. You'll find throughout our code that we regularly switch between using token and state for single token's, we don't know why.

From what we have found, any method can take any function. Also, all that a function needs to do is just build the HTML into the token, without any need to return anything. Though we do have some functions doing a return true for some reason we're not quite sure why.

Although a function can be passed tagInfo and content, it is not necessary for there to be anything in them, and not having them doesn't mean a tag with them will not be processed. It is up to the code to determine how to process these optional arguments.

tagInfo

As briefly touched upon in Dissection of a Tag, a tag is composed of three parts: tag, tagInfo, content. This comes together like so:

[tag tagInfo]content[/tag]

tag and content are self-explanatory strings (Note: content can be manipulated inside the ruler). tagInfo on the other hand is more interesting, as it is completely optional. tagInfo comes in two types, a single argument, and a multi-argument.

Single Argument tagInfo

[tag=value]

Here, the value is stored as a string in the key _default, accessible via tagInfo.attrs. If no value is provided, tagInfo.attrs['_default'] returns NULL. This happens when the tag is [tag] or [tag arg1=value1 ...].

Multi-Argument tagInfo

[tag arg1=value1 arg2=value2 ...]

Any number of arguments can be provided. Similar to the single argument, every value is optional, and if missing, returns a NULL. Each one creates a key/value pair in tagInfo.attrs under their respective names.
{arg1: "value1", arg2: "value2", ...}
It is up to the ruler's code to determine how to process a tag when given its arguments. The best example of this is the [font] tag.

It should be noted that it is illegal to create a combined tag structure [tag=value1 arg=value2], as _default will have the value "value1 arg=value2"

Token Components

With that out of the way, let us talk about tokens. The idea of a token seems to be a long series of defined HTML elements. It's a bit weird, but just think of it as a stack, and the ruler determines what to add to the stack. The method you'll be using the most will be token.push().

token.push()

So, token.push() seems to be a shorthand for pushing the HTML for token.type, token.tag, and token.nesting all at the same time, as described in the API doc. The way the command is used is:

token.push(type, tag, nesting);
// or
token = state.push(type, tag, nesting);

Here are some examples.

token = state.push("div_open", "div", 1);
// or a token can be set manually
startToken.type = "div_open";
startToken.tag = "div";
startToken.nesting = 1;

token.type

token.type is a string of the type of HTML tag (i.e. "paragraph_open", "span_close", "code_inline"). It can also take some special keywords, which will explained later. It should be noted, if the type doesn't exist, expect the BBCode to only be rendered as <>. We think this is because the engine begins to render a HTML element before attempting to check if its even valid.

token.tag

token.tag is a string of the HTML tag itself (i.e. "p", "span", "div"). An empty string is also acceptable (as shown later).

token.nesting

token.nesting is an integer that defines the nature of the tag. 1 for opening, -1 for closing, and 0 for self-closing.

token.attrs

To apply classes, styles, hrefs, and what not that a HTML tag can have, use token.attrs. It takes an array of strings of any size, and will properly render the defined inline attribute.

token.attrs = [[attribute, value], ... ];
// such as
token.attrs = [["class", "bbcode-background"], ["style", "background-color: " + bgOption]];
// or
startToken.attrs = [["class", "bbcode-column column-width-" + columnOption]];

The attribute will automatically be applied to the HTML tag that was just pushed.

token.content

To set the content inside a token, set token.content to a string.

token.content = "content of the token"

A good example of its usage is [ooc]text[/ooc] which outputs <div class="bbcode-ooc"><div>OOC</div>text</div>

  ruler.push("ooc", {
    tag: "ooc",
    replace: function (state, tagInfo, content) {
      let token = state.push("div_open", "div", 1);
      token.attrs = [["class", "bbcode-ooc"]];

      state.push("div_open", "div", 1);

      token = state.push("text", "", 0);
      token.content = "OOC";

      state.push("div_close", "div", -1);

      token = state.push("text", "", 0);
      token.content = content;

      state.push("div_close", "div", -1);

      return true;
    }
  });

While optional, a special token can be passed before the content, that will determine how token.content is rendered. So far, we have found only two of these tokens: "text" and "inline".

token = state.push("text", "", 0)
token = state.push("inline", "", 0)

"text" forces the content to be rendered as plain text, while "inline" makes it so that only inline tags can be rendered. To support nesting block level tags, before and after methods should be used instead, without defining either of these tokens at the end of the before method.

Note: While you can just pass the content of the tag into token.content, this should only be for tags that use the replace method. It can work with other methods, even wrap, but it is ill-advised for tags that intend to support nesting.

Clone this wiki locally