Skip to content

Commit

Permalink
Handle OOM in Java language server
Browse files Browse the repository at this point in the history
Add parameters to the java server to make lemminx crash
when it runs out of memory.
Detect when the java server shuts down due to running out of memory,
display a message to the user that this happened,
and don't attempt to restart the server.

Closes redhat-developer#527

Signed-off-by: David Thompson <[email protected]>
  • Loading branch information
datho7561 committed Jul 19, 2021
1 parent 475372e commit fa7d432
Show file tree
Hide file tree
Showing 8 changed files with 125 additions and 13 deletions.
19 changes: 19 additions & 0 deletions docs/Troubleshooting.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,3 +21,22 @@ You can kill the process by:

* on Windows OS: run `taskkill /F /PID ...` all instances
* on other OS: run `kill -9 ...` all instances

### The Language Server Crashes Due to an Out Of Memory Error

If you are working with large XML files or referencing large schema files,
this may lead to the language server running out of memory.
The Java language server is more likely to run out memory than the binary language server.
Switching to the binary language server
or increasing the memory available to the Java language server could resolve this issue.

If you get an Out of Memory Error, but aren't working with large XML files,
then there may be a memory leak in the language server.
Please [file a issue](https://github.com/redhat-developer/vscode-xml/issues) with a description of what you were doing if this is the case.

#### How to increase the amount of memory available to the Java Language Server

1. Go to settings
2. Navigate to the setting `xml.server.vmargs`
3. Add `-Xmx 1.5G` to the setting string. This allows the the language server to use at most 1.5 gigabytes of memory.
4. If the problem persists, you can increase the `1.5G` to `2G` or higher
18 changes: 17 additions & 1 deletion package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions package.json
Original file line number Diff line number Diff line change
Expand Up @@ -68,6 +68,7 @@
],
"devDependencies": {
"@types/fs-extra": "^8.0.0",
"@types/glob": "^7.1.4",
"@types/node": "^10.14.16",
"@types/vscode": "^1.37.0",
"@types/yauzl": "^2.9.1",
Expand Down
47 changes: 44 additions & 3 deletions src/client/clientErrorHandler.ts
Original file line number Diff line number Diff line change
@@ -1,20 +1,26 @@
import { window } from "vscode";
import * as fs from "fs-extra";
import { commands, ExtensionContext, window } from "vscode";
import { CloseAction, ErrorAction, ErrorHandler, Message } from "vscode-languageclient";
import { ClientCommandConstants } from "../commands/commandConstants";
import glob = require("glob");

/**
* An error handler that restarts the language server,
* unless it has been restarted 5 times in the last 3 minutes
* unless it has been restarted 5 times in the last 3 minutes,
* or if it crashed due to an Out Of Memory Error
*
* Adapted from [vscode-java](https://github.com/redhat-developer/vscode-java)
*/
export class ClientErrorHandler implements ErrorHandler {

private restarts: number[];
private name: string;
private context: ExtensionContext;

constructor(name: string) {
constructor(name: string, context: ExtensionContext) {
this.name = name;
this.restarts = [];
this.context = context;
}

error(_error: Error, _message: Message, _count: number): ErrorAction {
Expand All @@ -23,6 +29,12 @@ export class ClientErrorHandler implements ErrorHandler {

closed(): CloseAction {
this.restarts.push(Date.now());
const heapProfileGlob = new glob.GlobSync(`${this.context.globalStorageUri.path}/java_*.hprof`);
if (heapProfileGlob.found.length) {
cleanUpHeapDumps(this.context);
showOOMMessage();
return CloseAction.DoNotRestart;
}
if (this.restarts.length < 5) {
return CloseAction.Restart;
} else {
Expand All @@ -37,3 +49,32 @@ export class ClientErrorHandler implements ErrorHandler {
}

}

/**
* Deletes all the heap dumps generated by Out Of Memory errors
*
* @returns when the heap dumps have been deleted
*/
export async function cleanUpHeapDumps(context: ExtensionContext): Promise<void> {
const heapProfileGlob = new glob.GlobSync(`${context.globalStorageUri.path}/java_*.hprof`);
for (let heapProfile of heapProfileGlob.found) {
await fs.remove(heapProfile);
}
}

/**
* Shows a message about the server crashing due to an out of memory issue
*/
async function showOOMMessage(): Promise<void> {
const DOCS = 'More info...';
const result = await window.showErrorMessage('The XML Language Server crashed due to an Out Of Memory Error, and will not be restarted. ', //
DOCS);
if (result === DOCS) {
await commands.executeCommand(ClientCommandConstants.OPEN_DOCS,
{
page: 'Troubleshooting',
section: 'the-language-server-crashes-due-to-an-out-of-memory-error'
}
);
}
}
12 changes: 8 additions & 4 deletions src/client/xmlClient.ts
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { commands, ExtensionContext, extensions, Position, TextDocument, TextEdi
import { Command, ConfigurationParams, ConfigurationRequest, DidChangeConfigurationNotification, ExecuteCommandParams, LanguageClientOptions, MessageType, NotificationType, RequestType, RevealOutputChannelOn, TextDocumentPositionParams } from "vscode-languageclient";
import { Executable, LanguageClient } from 'vscode-languageclient/node';
import { XMLFileAssociation } from '../api/xmlExtensionApi';
import { ClientCommandConstants, ServerCommandConstants } from '../commands/commandConstants';
import { ServerCommandConstants } from '../commands/commandConstants';
import { registerClientServerCommands } from '../commands/registerCommands';
import { onExtensionChange } from '../plugin';
import { RequirementsData } from "../server/requirements";
Expand Down Expand Up @@ -37,7 +37,7 @@ let languageClient: LanguageClient;

export async function startLanguageClient(context: ExtensionContext, executable: Executable, logfile: string, externalXmlSettings: ExternalXmlSettings, requirementsData: RequirementsData): Promise<LanguageClient> {

const languageClientOptions: LanguageClientOptions = getLanguageClientOptions(logfile, externalXmlSettings, requirementsData);
const languageClientOptions: LanguageClientOptions = getLanguageClientOptions(logfile, externalXmlSettings, requirementsData, context);
languageClient = new LanguageClient('xml', 'XML Support', executable, languageClientOptions);

languageClient.onTelemetry(async (e: TelemetryEvent) => {
Expand Down Expand Up @@ -106,7 +106,11 @@ export async function startLanguageClient(context: ExtensionContext, executable:
return languageClient;
}

function getLanguageClientOptions(logfile: string, externalXmlSettings: ExternalXmlSettings, requirementsData: RequirementsData): LanguageClientOptions {
function getLanguageClientOptions(
logfile: string,
externalXmlSettings: ExternalXmlSettings,
requirementsData: RequirementsData,
context: ExtensionContext): LanguageClientOptions {
return {
// Register the server for xml and xsl
documentSelector: [
Expand All @@ -133,7 +137,7 @@ function getLanguageClientOptions(logfile: string, externalXmlSettings: External
shouldLanguageServerExitOnShutdown: true
}
},
errorHandler: new ClientErrorHandler('XML'),
errorHandler: new ClientErrorHandler('XML', context),
synchronize: {
//preferences starting with these will trigger didChangeConfiguration
configurationSection: ['xml', '[xml]', 'files.trimFinalNewlines', 'files.trimTrailingWhitespace', 'files.insertFinalNewline']
Expand Down
6 changes: 5 additions & 1 deletion src/extension.ts
Original file line number Diff line number Diff line change
Expand Up @@ -10,21 +10,23 @@
* Microsoft Corporation - Auto Closing Tags
*/

import * as fs from 'fs-extra';
import * as os from 'os';
import * as path from 'path';
import { ExtensionContext, extensions, languages } from "vscode";
import { Executable, LanguageClient } from 'vscode-languageclient/node';
import { XMLExtensionApi } from './api/xmlExtensionApi';
import { getXmlExtensionApiImplementation } from './api/xmlExtensionApiImplementation';
import { cleanUpHeapDumps } from './client/clientErrorHandler';
import { getIndentationRules } from './client/indentation';
import { startLanguageClient } from './client/xmlClient';
import { registerClientOnlyCommands } from './commands/registerCommands';
import { collectXmlJavaExtensions } from './plugin';
import * as requirements from './server/requirements';
import { prepareExecutable } from './server/serverStarter';
import { ExternalXmlSettings } from "./settings/externalXmlSettings";
import { getXMLConfiguration } from './settings/settings';
import { Telemetry } from './telemetry';
import { registerClientOnlyCommands } from './commands/registerCommands';

let languageClient: LanguageClient;

Expand Down Expand Up @@ -52,6 +54,8 @@ export async function activate(context: ExtensionContext): Promise<XMLExtensionA
storagePath = os.homedir() + "/.lemminx";
}
const logfile = path.resolve(storagePath + '/lemminx.log');
await fs.ensureDir(context.globalStorageUri.path);
await cleanUpHeapDumps(context);

const externalXmlSettings: ExternalXmlSettings = new ExternalXmlSettings();

Expand Down
4 changes: 2 additions & 2 deletions src/server/binary/binaryServerStarter.ts
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ import { getProxySettings, getProxySettingsAsEnvironmentVariables, ProxySettings
import { getXMLConfiguration } from "../../settings/settings";
import { Telemetry } from '../../telemetry';
import { addTrustedHash, getTrustedHashes } from './binaryHashManager';
const glob = require('glob');
import glob = require('glob');

const HTTPS_PATTERN = /^https:\/\//;
const JAR_ZIP_AND_HASH_REJECTOR = /(?:\.jar)|(?:\.zip)|(?:\.sha256)$/;
Expand Down Expand Up @@ -430,4 +430,4 @@ async function acceptBinaryDownloadResponse(response: http.IncomingMessage): Pro

async function openProxyDocumentation(): Promise<void> {
await commands.executeCommand(ClientCommandConstants.OPEN_DOCS, { page: "Proxy" });
}
}
31 changes: 29 additions & 2 deletions src/server/java/javaServerStarter.ts
Original file line number Diff line number Diff line change
@@ -1,16 +1,31 @@
import * as os from 'os';
import * as path from 'path';
import { ExtensionContext, workspace } from 'vscode';
import { ExtensionContext, window, workspace } from 'vscode';
import { Executable } from 'vscode-languageclient/node';
import { getProxySettings, getProxySettingsAsJVMArgs, jvmArgsContainsProxySettings, ProxySettings } from '../../settings/proxySettings';
import { getJavaagentFlag, getKey, getXMLConfiguration, IS_WORKSPACE_VMARGS_XML_ALLOWED, xmlServerVmargs } from '../../settings/settings';
import { RequirementsData } from '../requirements';
const glob = require('glob');
import glob = require('glob');

declare var v8debug;

const DEBUG = (typeof v8debug === 'object') || startedInDebugMode();

/**
* Argument that tells the program to crash when an OutOfMemoryError is raised
*/
const CRASH_ON_OOM = '-XX:+ExitOnOutOfMemoryError';

/**
* Argument that tells the program to generate a heap dump file when an OutOfMemoryError is raised
*/
const HEAP_DUMP = '-XX:+HeapDumpOnOutOfMemoryError';

/**
* Argument that tells the program where to generate the heap dump that is created when an OutOfMemoryError is raised and `HEAP_DUMP` has been passed
*/
const HEAP_DUMP_LOCATION = '-XX:HeapDumpPath=';

export async function prepareJavaExecutable(
context: ExtensionContext,
requirements: RequirementsData,
Expand Down Expand Up @@ -63,6 +78,18 @@ function prepareParams(requirements: RequirementsData, xmlJavaExtensions: string
params.push(watchParentProcess + 'false');
}
}
if (vmargs.indexOf(CRASH_ON_OOM) < 0) {
params.push(CRASH_ON_OOM);
}
if (vmargs.indexOf(HEAP_DUMP) < 0) {
params.push(HEAP_DUMP);
}
if (vmargs.indexOf(HEAP_DUMP_LOCATION) < 0) {
params.push(`${HEAP_DUMP_LOCATION}${context.globalStorageUri.path}`);
} else {
window.showWarningMessage('Heap dump location has been modified; if you are experiencing Out Of Memory crashes, vscode-xml won\'t be able to detect them');
}

// "OpenJDK 64-Bit Server VM warning: Options -Xverify:none and -noverify
// were deprecated in JDK 13 and will likely be removed in a future release."
// so only add -noverify for older versions
Expand Down

0 comments on commit fa7d432

Please sign in to comment.