Skip to content

Commit

Permalink
Java script engine switcher and highlight js (#452)
Browse files Browse the repository at this point in the history
* Adds server-side code block formatting

This applies the highlight.js highlighting on the server allowing the client side script to be dropped. It does this using the MsieJavaScriptEngine to execute the script. The script itself is actually the node version, but I applied browserify to it so that it wouldn't require node anymore. Kind of a hack, but it works. I included the steps I took to build the file in regenerating-highlight-all.js so that it could be updated in the future with a newer version.

* Uses ThreadLocal<MsieJsEngine> to reuse JS engine 

MsieJsEngine is certainly not thread safe so I was rebuildingi t from scratch within the loop. No sense in doing that, so switch the code to use a ThreadLocal instance to pool the JS instances between threads.  Not a huge win for small sites, but for a 1000 document test run it went from 15s to 1s on my machine.

* Adds test for auto-highlighting

Adds a test where the code block doesn't have a language and highlightAuto
must be called. This not only covers that scenario but is an important
test case for javascript execution engines. Because highlight.js doesn't
know the language it must iterate through the languages it is configured
with and check them one by one using regular expressions. These have a
wide variety of implementations so it actually provides a good test case
for a javascript languages regex support. It seems anything doing JS -> IL
starts to fall short here due to the differences in supported Regex
escaping and language features. And with regex being a huge part of
linting and text manipulation supporting regex is a vital part of
evaluating a JS engine.

* Adds JavaScriptEngineSwitcher for javascript processing

Adds the following packages to Core

* JavaScriptEngineSwitcher - an abstraction around common JS processing libraries
* JsPool - a pool around the engine switcher. We'll be creating these a lot potentially so good pooling is a must
* Jint - managed implementation of a javascript processor. We'll make this the default.
* JavaScriptEngineSwitcher.Jint - allows Jint to be plugged into the JavaScriptEngineSwitcher

For a module that wants to use the javascript engine it's pretty simple. They just need to call GetJsEngineFromPool off of the context. When they are done then must call ReturnJsEngineToPool, This exposes more or less the same interface as IJsEngine from the JavaScriptEngineSwitcher so it should be easy to find help and examples for devs to work with. The implementation is just a wrapper of the default implementation plus a couple of helpers around ensuring libraries only get loaded once per engine.

Things get tricky in two spots

1. Testing. With this being exposed off of IExecutionContext the TestContext project needed some implementation. Rather than bring in a JS engine here too I made it a Func<> that could be set to a factory for building up the JsEngine by the test. I added a Wyam.Tests.JavaScript with an implementation of an engine that could be used for testing. It's just a copy and paste from Wyam.Core but it does allow test projects to be able to avoid needing to reference Core just to get some JS processing to work
2. Reloading the configuration. Because the configuration could potentially change the JS engine it needed the ability to wipe the singleton that JavaScriptEngineSwitcher uses. Had to add a couple static methods off of Engine and ExecutionContext to get those settings cleared out properly.

* Marks CanHighlightAutoCodeBlocks as Ignored

Test is failing due to Jint's regex implementation, but is still valuable to keep around for testing those fixes in the future

* Removes escaping of @ in code blocks

This was escaping ALL @ signs in the document instead of just the ones in the code blocks. Not good for things like actual javascript and the such. I tried to get AngularSharp to be nice and let me escape it via the InnerHtml but they insisted on changing it back. 

Taking a step back I realized all of this is foolish. This should really only be called after Razor is invoked. Added a test to make sure the highlighting is cool with the escaped html in a code block razor will generate.

* Tweaks highlight.js to have more .net compliant regex

The ADA and Lisp languages used some regex syntax that .Net couldn't handle. By changing this it allows Jint to run successfully.
  • Loading branch information
phil-scott-78 authored and daveaglick committed Feb 17, 2017
1 parent ac03425 commit 6ab96dc
Show file tree
Hide file tree
Showing 29 changed files with 17,064 additions and 1 deletion.
21 changes: 21 additions & 0 deletions Wyam.sln
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,12 @@ Project("{2150E333-8FDC-42A3-9474-1A3956D46DE8}") = "integration", "integration"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Wyam.Docs", "src\extensions\Wyam.Docs\Wyam.Docs.csproj", "{0EB71E59-8D5C-45F3-BF1D-8649BA76D139}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Wyam.Highlight", "src\extensions\Wyam.Highlight\Wyam.Highlight.csproj", "{5FE758A2-1221-48FE-8A3C-8B3B9A32AFAB}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Wyam.Highlight.Tests", "tests\extensions\Wyam.Highlight.Tests\Wyam.Highlight.Tests.csproj", "{A6DE4586-1398-4071-90AB-5CCC1C949720}"
EndProject
Project("{FAE04EC0-301F-11D3-BF4B-00C04F79EFBC}") = "Wyam.Testing.JavaScript", "src\core\Wyam.Testing.JavaScript\Wyam.Testing.JavaScript.csproj", "{95080CB0-DC4E-4617-8F73-6E06A6CA2F5F}"
EndProject
Global
GlobalSection(SolutionConfigurationPlatforms) = preSolution
Debug|Any CPU = Debug|Any CPU
Expand Down Expand Up @@ -294,6 +300,18 @@ Global
{0EB71E59-8D5C-45F3-BF1D-8649BA76D139}.Debug|Any CPU.Build.0 = Debug|Any CPU
{0EB71E59-8D5C-45F3-BF1D-8649BA76D139}.Release|Any CPU.ActiveCfg = Release|Any CPU
{0EB71E59-8D5C-45F3-BF1D-8649BA76D139}.Release|Any CPU.Build.0 = Release|Any CPU
{5FE758A2-1221-48FE-8A3C-8B3B9A32AFAB}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{5FE758A2-1221-48FE-8A3C-8B3B9A32AFAB}.Debug|Any CPU.Build.0 = Debug|Any CPU
{5FE758A2-1221-48FE-8A3C-8B3B9A32AFAB}.Release|Any CPU.ActiveCfg = Release|Any CPU
{5FE758A2-1221-48FE-8A3C-8B3B9A32AFAB}.Release|Any CPU.Build.0 = Release|Any CPU
{A6DE4586-1398-4071-90AB-5CCC1C949720}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{A6DE4586-1398-4071-90AB-5CCC1C949720}.Debug|Any CPU.Build.0 = Debug|Any CPU
{A6DE4586-1398-4071-90AB-5CCC1C949720}.Release|Any CPU.ActiveCfg = Release|Any CPU
{A6DE4586-1398-4071-90AB-5CCC1C949720}.Release|Any CPU.Build.0 = Release|Any CPU
{95080CB0-DC4E-4617-8F73-6E06A6CA2F5F}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
{95080CB0-DC4E-4617-8F73-6E06A6CA2F5F}.Debug|Any CPU.Build.0 = Debug|Any CPU
{95080CB0-DC4E-4617-8F73-6E06A6CA2F5F}.Release|Any CPU.ActiveCfg = Release|Any CPU
{95080CB0-DC4E-4617-8F73-6E06A6CA2F5F}.Release|Any CPU.Build.0 = Release|Any CPU
EndGlobalSection
GlobalSection(SolutionProperties) = preSolution
HideSolutionNode = FALSE
Expand Down Expand Up @@ -350,5 +368,8 @@ Global
{2E5842F3-4797-4BB5-9A55-F3A005B434F8} = {596452BD-AE34-4989-A36F-88FF72B2DFD8}
{C5AED397-4463-4DE7-8A7E-74A4C949E417} = {596452BD-AE34-4989-A36F-88FF72B2DFD8}
{0EB71E59-8D5C-45F3-BF1D-8649BA76D139} = {5A431149-2B88-40C3-9717-CA6CCF214317}
{5FE758A2-1221-48FE-8A3C-8B3B9A32AFAB} = {5A431149-2B88-40C3-9717-CA6CCF214317}
{A6DE4586-1398-4071-90AB-5CCC1C949720} = {2E5842F3-4797-4BB5-9A55-F3A005B434F8}
{95080CB0-DC4E-4617-8F73-6E06A6CA2F5F} = {4813B17D-EFCA-401D-B0E6-7B24B3946A70}
EndGlobalSection
EndGlobal
3 changes: 3 additions & 0 deletions src/clients/Wyam/EngineManager.cs
Original file line number Diff line number Diff line change
Expand Up @@ -104,6 +104,9 @@ public bool Configure()
{
try
{
// make sure we clear out anything in the JavaScriptEngineSwitcher instance
Engine.ResetJsEngines();

// Make sure the root path exists
if (!Engine.FileSystem.GetRootDirectory().Exists)
{
Expand Down
26 changes: 26 additions & 0 deletions src/core/Wyam.Common/Execution/IExecutionContext.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
using Wyam.Common.Configuration;
using Wyam.Common.Documents;
using Wyam.Common.IO;
using Wyam.Common.JavaScript;
using Wyam.Common.Meta;
using Wyam.Common.Modules;
using Wyam.Common.Tracing;
Expand Down Expand Up @@ -298,6 +299,31 @@ public interface IExecutionContext : IMetadata
/// <returns>The cloned or new document.</returns>
IDocument GetDocument(IDocument sourceDocument, IEnumerable<KeyValuePair<string, object>> items);

/// <summary>
/// Gets an engine from the pool. This engine should be returned to the pool via
/// <see cref="ReturnJsEngineToPool"/> when you are finished with it.
/// If an engine is free, this method returns immediately with the engine.
/// If no engines are available creates a new engine. If MaxEngines has been reached, blocks until an engine is
/// avaiable again.
/// </summary>
/// <param name="timeout">
/// Maximum time to wait for a free engine.
/// </param>
/// <returns>A JavaScript engine</returns>
IJsEngine GetJsEngineFromPool(TimeSpan? timeout = null);


/// <summary>
/// Returns an engine to the pool so it can be reused
/// </summary>
/// <param name="engine">Engine to return</param>
void ReturnJsEngineToPool(IJsEngine engine);

/// <summary>
/// Disposes all engines in this pool, and creates new engines in their place.
/// </summary>
void RecycleJsEngines();

// This provides access to the same enhanced type conversion used to convert metadata types
bool TryConvert<T>(object value, out T result);

Expand Down
160 changes: 160 additions & 0 deletions src/core/Wyam.Common/JavaScript/IJsEngine.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
using System;
using System.Reflection;
using System.Text;

namespace Wyam.Common.JavaScript
{
/// <summary>
/// Defines a interface of JS engine
/// </summary>
public interface IJsEngine : IDisposable
{
/// <summary>
/// Gets the name of JS engine
/// </summary>
string Name
{
get;
}

/// <summary>
/// Gets the version of original JS engine
/// </summary>
string Version
{
get;
}

/// <summary>
/// Evaluates an expression
/// </summary>
/// <param name="expression">JS-expression</param>
/// <returns>Result of the expression</returns>
object Evaluate(string expression);

/// <summary>
/// Evaluates an expression
/// </summary>
/// <typeparam name="T">Type of result</typeparam>
/// <param name="expression">JS-expression</param>
/// <returns>Result of the expression</returns>
T Evaluate<T>(string expression);

/// <summary>
/// Executes code
/// </summary>
/// <param name="code">Code</param>
void Execute(string code);

/// <summary>
/// Executes code from JS-file. If the file should only be loaded once per lifetime of the
/// JsEngine, see <see cref="RequireFile"/>
/// </summary>
/// <param name="path">Path to the JS-file</param>
/// <param name="encoding">Text encoding</param>
void ExecuteFile(string path, Encoding encoding = null);

/// <summary>
/// Executes code from embedded JS-resource. If the resource should only be loaded once per lifetime of the
/// JsEngine, see <see cref="RequireResource"/>
/// </summary>
/// <param name="resourceName">The case-sensitive resource name without the namespace of the specified type</param>
/// <param name="type">The type, that determines the assembly and whose namespace is used to scope
/// the resource name</param>
void ExecuteResource(string resourceName, Type type);

/// <summary>
/// Executes code from embedded JS-resource. If the resource should only be loaded once per lifetime of the
/// JsEngine, see <see cref="RequireResource"/>
/// </summary>
/// <param name="resourceName">The case-sensitive resource name</param>
/// <param name="assembly">The assembly, which contains the embedded resource</param>
void ExecuteResource(string resourceName, Assembly assembly);

/// <summary>
/// Calls a function
/// </summary>
/// <param name="functionName">Function name</param>
/// <param name="args">Function arguments</param>
/// <returns>Result of the function execution</returns>
object CallFunction(string functionName, params object[] args);

/// <summary>
/// Calls a function
/// </summary>
/// <typeparam name="T">Type of function result</typeparam>
/// <param name="functionName">Function name</param>
/// <param name="args">Function arguments</param>
/// <returns>Result of the function execution</returns>
T CallFunction<T>(string functionName, params object[] args);

/// <summary>
/// Сhecks for the existence of a variable
/// </summary>
/// <param name="variableName">Variable name</param>
/// <returns>Result of check (true - exists; false - not exists</returns>
bool HasVariable(string variableName);

/// <summary>
/// Gets the value of variable
/// </summary>
/// <param name="variableName">Variable name</param>
/// <returns>Value of variable</returns>
object GetVariableValue(string variableName);

/// <summary>
/// Gets the value of variable
/// </summary>
/// <typeparam name="T">Type of variable</typeparam>
/// <param name="variableName">Variable name</param>
/// <returns>Value of variable</returns>
T GetVariableValue<T>(string variableName);

/// <summary>
/// Sets the value of variable
/// </summary>
/// <param name="variableName">Variable name</param>
/// <param name="value">Value of variable</param>
void SetVariableValue(string variableName, object value);

/// <summary>
/// Removes a variable
/// </summary>
/// <param name="variableName">Variable name</param>
void RemoveVariable(string variableName);

/// <summary>
/// Embeds a host object to script code
/// </summary>
/// <param name="itemName">The name for the new global variable or function that will represent the object</param>
/// <param name="value">The object to expose</param>
/// <remarks>Allows to embed instances of simple classes (or structures) and delegates.</remarks>
void EmbedHostObject(string itemName, object value);

/// <summary>
/// Embeds a host type to script code
/// </summary>
/// <param name="itemName">The name for the new global variable that will represent the type</param>
/// <param name="type">The type to expose</param>
/// <remarks>
/// Host types are exposed to script code in the form of objects whose properties and
/// methods are bound to the type's static members.
/// </remarks>
void EmbedHostType(string itemName, Type type);

/// <summary>
/// Ensures a resource is loaded for this engine. If it is already
/// loaded then nothing is executed.
/// </summary>
/// <param name="resourceName">Name of the resource.</param>
/// <param name="type">The type.</param>
void RequireResource(string resourceName, Type type);

/// <summary>
/// Ensures a resource is loaded for this engine. If it is already
/// loaded then nothing is executed.
/// </summary>
/// <param name="path">The path is load.</param>
void RequireFile(string path);
}
}
1 change: 1 addition & 0 deletions src/core/Wyam.Common/Wyam.Common.csproj
Original file line number Diff line number Diff line change
Expand Up @@ -59,6 +59,7 @@
<Compile Include="Modules\ModuleList.cs" />
<Compile Include="Modules\ModuleExtensions.cs" />
<Compile Include="Modules\NamedModule.cs" />
<Compile Include="JavaScript\IJsEngine.cs" />
<Compile Include="Util\LinkGenerator.cs" />
<Compile Include="IO\DirectoryEqualityComparer.cs" />
<Compile Include="IO\DirectoryPath.cs" />
Expand Down
1 change: 1 addition & 0 deletions src/core/Wyam.Configuration/KnownExtensionGenerated.cs
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ public partial class KnownExtension
public static readonly KnownExtension Feeds = new KnownExtension("Wyam.Feeds");
public static readonly KnownExtension Git = new KnownExtension("Wyam.Git");
public static readonly KnownExtension GitHub = new KnownExtension("Wyam.GitHub");
public static readonly KnownExtension Highlight = new KnownExtension("Wyam.Highlight");
public static readonly KnownExtension Html = new KnownExtension("Wyam.Html");
public static readonly KnownExtension Images = new KnownExtension("Wyam.Images");
public static readonly KnownExtension Json = new KnownExtension("Wyam.Json");
Expand Down
13 changes: 13 additions & 0 deletions src/core/Wyam.Core/Execution/Engine.cs
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
using System.Collections.Generic;
using System.Linq;
using System.Reflection;
using JavaScriptEngineSwitcher.Core;
using Wyam.Common.Configuration;
using Wyam.Common.Documents;
using Wyam.Common.Execution;
Expand Down Expand Up @@ -150,6 +151,18 @@ public void CleanOutputPath()
}
}

/// <summary>
/// Resets the JavaScript Engine pool and clears the JavaScript Engine Switcher
/// to an empty list of engine factories and default engine. Useful on configuration
/// change where we might have a new configuration.
/// </summary>
public static void ResetJsEngines()
{
ExecutionContext.ResetJsPool();
JsEngineSwitcher.Instance.EngineFactories.Clear();
JsEngineSwitcher.Instance.DefaultEngineName = string.Empty;
}

public void Execute()
{
CheckDisposed();
Expand Down
58 changes: 57 additions & 1 deletion src/core/Wyam.Core/Execution/ExecutionContext.cs
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,9 @@
using System.Linq;
using System.Reflection;
using System.Threading.Tasks;
using JavaScriptEngineSwitcher.Core;
using JavaScriptEngineSwitcher.Jint;
using JSPool;
using Wyam.Common.Caching;
using Wyam.Common.Configuration;
using Wyam.Common.Documents;
Expand All @@ -15,7 +18,9 @@
using Wyam.Common.Execution;
using Wyam.Common.Tracing;
using Wyam.Common.Util;
using Wyam.Core.JavaScript;
using Wyam.Core.Meta;
using IJsEngine = Wyam.Common.JavaScript.IJsEngine;

namespace Wyam.Core.Execution
{
Expand All @@ -24,7 +29,9 @@ internal class ExecutionContext : IExecutionContext, IDisposable
private readonly Pipeline _pipeline;

private bool _disposed;


private static Lazy<JsPool<IJsEngine>> JsPool = new Lazy<JsPool<IJsEngine>>(JsPoolFactory);

public Engine Engine { get; }

public IReadOnlyCollection<byte[]> DynamicAssemblies => Engine.DynamicAssemblies;
Expand Down Expand Up @@ -281,5 +288,54 @@ private IReadOnlyList<IDocument> Execute(IEnumerable<IModule> modules, IEnumerab
public IReadOnlyList<IDocument> DocumentList(string key) => Settings.DocumentList(key);

public dynamic Dynamic(string key, object defaultValue = null) => Settings.Dynamic(key, defaultValue);

public IJsEngine GetJsEngineFromPool(TimeSpan? timeout = null)
{
return JsPool.Value.GetEngine(timeout);
}

public void ReturnJsEngineToPool(IJsEngine engine)
{
JsPool.Value.ReturnEngineToPool(engine);
}

public void RecycleJsEngines()
{
JsPool.Value.Recycle();
}

/// <summary>
/// Destroys the existing pool and rebuilds it. Useful for configuration
/// changes which might adjust the configuration of the javascript engine
/// using the JavaScriptEngineSwitcher's Instance singleton
/// </summary>
public static void ResetJsPool()
{
JsPool.Value.Dispose();
JsPool = new Lazy<JsPool<IJsEngine>>(JsPoolFactory);
}

private static JsPool<IJsEngine> JsPoolFactory()
{
// builds the default js pool.
// first we need to check if the JsEngineSwitcher has been configured. We'll do this
// by checking the DefaultEngineName being set. If that's there we can safely assume
// its been configured somehow (maybe via config.wyam). If not we'll wire up
// Jint as the default engine
if (string.IsNullOrWhiteSpace(JsEngineSwitcher.Instance.DefaultEngineName))
{
JsEngineSwitcher.Instance.EngineFactories.Add(new JintJsEngineFactory());
JsEngineSwitcher.Instance.DefaultEngineName = JintJsEngine.EngineName;
}

Trace.Information($"Using {JsEngineSwitcher.Instance.DefaultEngineName} for JavaScript engine");

var pool = new JsPool<IJsEngine>(new JsPoolConfig<IJsEngine>()
{
EngineFactory = () => new JsEngine(JsEngineSwitcher.Instance.CreateDefaultEngine()),
});

return pool;
}
}
}
Loading

0 comments on commit 6ab96dc

Please sign in to comment.