Skip to content

Commit

Permalink
feat: use fs.watch
Browse files Browse the repository at this point in the history 
Turbowatch uses [`fs.watch`](https://nodejs.org/api/fs.html#fswatchfilename-options-listener), which is known to have platform-specific caveats. Unfortunately, Watchman cannot be used due to it not supporting symbolic links (issue [#105](facebook/watchman#105 (comment))) and Chokidar cannot be used due to it failing to detect file changes (issue [#1240](paulmillr/chokidar#1240)). This is not an issue if you are using MacOS, though it may have undersirable side-effects on other platforms. Please raise an issue if you discover a platform-specific issue.
BREAKING CHANGE: Potentially breaking changes for non-MacOS platforms.
  • Loading branch information
@gajus
gajus committed Mar 20, 2023
1 parent 0b0b4a8 commit b2f97e7
Show file tree
Hide file tree
Showing 6 changed files with 57 additions and 110 deletions.
4 changes: 3 additions & 1 deletion README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -58,6 +58,8 @@ Refer to recipes:
<sup><sup>1</sup> Undocumented</sup><br>
<sup><sup>2</sup> Nodemon only provides the ability to [send a custom signal](https://github.com/remy/nodemon#gracefully-reloading-down-your-script) to the worker.</sup><br>

> **Warning** Turbowatch uses [`fs.watch`](https://nodejs.org/api/fs.html#fswatchfilename-options-listener), which is known to have platform-specific caveats. Unfortunately, Watchman cannot be used due to it not supporting symbolic links (issue [#105](https://github.com/facebook/watchman/issues/105#issuecomment-1469496330)) and Chokidar cannot be used due to it failing to detect file changes (issue [#1240](https://github.com/paulmillr/chokidar/issues/1240)). This is not an issue if you are using MacOS, though it may have undersirable side-effects on other platforms. Please raise an issue if you discover a platform-specific issue.
## API

Turbowatch [defaults](#recipes) are a good choice for most projects. However, Turbowatch has _many_ options that you should be familiar with for advance use cases.
Expand Down Expand Up @@ -205,7 +207,7 @@ type Expression =
| ['not', Expression];
```

> **Note** Turbowatch expressions are a subset of [Watchman expressions](https://facebook.github.io/watchman/docs/expr/allof.html). Originally, Turbowatch was developed to leverage Watchman as a superior backend for watching a large number of files. However, along the way, we discovered that Watchman does not support symbolic links (issue [#105](https://github.com/facebook/watchman/issues/105#issuecomment-1469496330)). Unfortunately, that makes Watchman unsuitable for projects that utilize linked dependencies (which is the direction in which the ecosystem is moving for dependency management in monorepos). As such, Watchman was replaced with chokidar. We are hoping to provide Watchman as a backend in the future. Therefore, we made Turbowatch expressions syntax compatible with a subset of Watchman expressions.
> **Note** Turbowatch expressions are a subset of [Watchman expressions](https://facebook.github.io/watchman/docs/expr/allof.html). Originally, Turbowatch was developed to leverage Watchman as a superior backend for watching a large number of files. However, along the way, we discovered that Watchman does not support symbolic links (issue [#105](https://github.com/facebook/watchman/issues/105#issuecomment-1469496330)). Unfortunately, that makes Watchman unsuitable for projects that utilize linked dependencies (which is the direction in which the ecosystem is moving for dependency management in monorepos). As such, Watchman was replaced with [`fs.watch`](https://nodejs.org/api/fs.html#fswatchfilename-options-listener). We are hoping to provide Watchman as a backend in the future. Therefore, we made Turbowatch expressions syntax compatible with a subset of Watchman expressions.
## Recipes

Expand Down
8 changes: 3 additions & 5 deletions src/bin/turbowatch.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,10 @@ const main = async () => {
return;
}

const userScript = jiti(__filename)(resolvedPath)
.default as Promise<TurbowatchController>;
const turbowatchController = jiti(__filename)(resolvedPath)
.default as TurbowatchController;

if (typeof userScript?.then !== 'function') {
if (typeof turbowatchController?.shutdown !== 'function') {
console.error(
'Expected user script to export an instance of TurbowatchController',
);
Expand All @@ -63,8 +63,6 @@ const main = async () => {
return;
}

const turbowatchController = await userScript;

process.once('SIGINT', () => {
log.warn('received SIGINT; gracefully terminating');

Expand Down
12 changes: 6 additions & 6 deletions src/subscribe.test.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -76,16 +76,16 @@ it('removes duplicates', async () => {

subscription.trigger([
{
event: 'add',
path: '/foo',
event: 'change',
filename: '/foo',
},
{
event: 'add',
path: '/foo',
event: 'change',
filename: '/foo',
},
{
event: 'add',
path: '/bar',
event: 'change',
filename: '/bar',
},
]);

Expand Down
16 changes: 8 additions & 8 deletions src/subscribe.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@ import { generateShortId } from './generateShortId';
import { Logger } from './Logger';
import {
type ActiveTask,
type ChokidarEvent,
type FSWatcherEvent,
type Subscription,
type Trigger,
} from './types';
Expand All @@ -18,7 +18,7 @@ export const subscribe = (trigger: Trigger): Subscription => {

let first = true;

let eventQueue: ChokidarEvent[] = [];
let eventQueue: FSWatcherEvent[] = [];

const handleSubscriptionEvent = async () => {
if (trigger.abortSignal?.aborted) {
Expand Down Expand Up @@ -93,17 +93,17 @@ export const subscribe = (trigger: Trigger): Subscription => {

const event = {
files: eventQueue
.filter(({ path }) => {
if (affectedPaths.includes(path)) {
.filter(({ filename }) => {
if (affectedPaths.includes(filename)) {
return false;
}

affectedPaths.push(path);
affectedPaths.push(filename);
return true;
})
.map(({ path }) => {
.map(({ filename }) => {
return {
name: path,
name: filename,
};
}),
};
Expand Down Expand Up @@ -205,7 +205,7 @@ export const subscribe = (trigger: Trigger): Subscription => {
});
}
},
trigger: async (events: readonly ChokidarEvent[]) => {
trigger: async (events: readonly FSWatcherEvent[]) => {
eventQueue.push(...events);

await handleSubscriptionEvent();
Expand Down
8 changes: 4 additions & 4 deletions src/types.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -140,9 +140,9 @@ export type Configuration = {
readonly triggers: readonly TriggerInput[];
};

export type ChokidarEvent = {
event: 'add' | 'addDir' | 'change' | 'unlink' | 'unlinkDir';
path: string;
export type FSWatcherEvent = {
event: 'rename' | 'change';
filename: string;
};

/**
Expand All @@ -160,7 +160,7 @@ export type Subscription = {
expression: Expression;
initialRun: boolean;
teardown: () => Promise<void>;
trigger: (events: readonly ChokidarEvent[]) => Promise<void>;
trigger: (events: readonly FSWatcherEvent[]) => Promise<void>;
};

export type TurbowatchController = {
Expand Down
119 changes: 33 additions & 86 deletions src/watch.ts
View file
Original file line number Diff line number Diff line change
Expand Up @@ -3,14 +3,14 @@ import { Logger } from './Logger';
import { subscribe } from './subscribe';
import { testExpression } from './testExpression';
import {
type ChokidarEvent,
type Configuration,
type ConfigurationInput,
type FSWatcherEvent,
type JsonObject,
type Subscription,
type TurbowatchController,
} from './types';
import * as chokidar from 'chokidar';
import fs from 'node:fs';
import { serializeError } from 'serialize-error';
import { debounce } from 'throttle-debounce';

Expand All @@ -20,7 +20,7 @@ const log = Logger.child({

export const watch = (
configurationInput: ConfigurationInput,
): Promise<TurbowatchController> => {
): TurbowatchController => {
const {
project,
triggers,
Expand Down Expand Up @@ -51,7 +51,25 @@ export const watch = (

const subscriptions: Subscription[] = [];

const watcher = chokidar.watch(project);
const watcher = fs.watch(project, {recursive: true}, (event, filename) => {
queuedEvents.push({
event,
filename,
});

evaluateSubscribers();
});

watcher.on('error', (error) => {
log.error(
{
error: serializeError(error) as unknown as JsonObject,
},
'could not watch project',
);

shutdown();
});

const shutdown = async () => {
clearInterval(indexingIntervalId);
Expand Down Expand Up @@ -102,19 +120,19 @@ export const watch = (
);
}

let queuedChokidarEvents: ChokidarEvent[] = [];
let queuedEvents: FSWatcherEvent[] = [];

const evaluateSubscribers = debounce(
userDebounce.wait,
() => {
const currentChokidarEvents =
queuedChokidarEvents as readonly ChokidarEvent[];
queuedEvents as readonly FSWatcherEvent[];

queuedChokidarEvents = [];
queuedEvents = [];

for (const subscription of subscriptions) {
const relevantEvents = currentChokidarEvents.filter((chokidarEvent) => {
return testExpression(subscription.expression, chokidarEvent.path);
return testExpression(subscription.expression, chokidarEvent.filename);
});

if (relevantEvents.length) {
Expand All @@ -127,84 +145,13 @@ export const watch = (
},
);

let ready = false;

const discoveredFiles: string[] = [];

watcher.on('all', (event, path) => {
if (ready) {
queuedChokidarEvents.push({
event,
path,
});

evaluateSubscribers();
} else {
if (discoveredFiles.length < 10) {
discoveredFiles.push(path);
}

discoveredFileCount++;
for (const subscription of subscriptions) {
if (subscription.initialRun) {
void subscription.trigger([]);
}
});

return new Promise((resolve, reject) => {
watcher.on('error', (error) => {
log.error(
{
error: serializeError(error) as unknown as JsonObject,
},
'could not watch project',
);

if (ready) {
shutdown();
} else {
reject(error);
}
});

watcher.on('ready', () => {
ready = true;

clearInterval(indexingIntervalId);

if (discoveredFiles.length > 10) {
log.trace(
{
files: discoveredFiles.slice(0, 10).map((file) => {
return file;
}),
},
'discovered %d files in %s; showing first 10',
discoveredFileCount,
project,
);
} else {
log.trace(
{
files: discoveredFiles.map((file) => {
return file;
}),
},
'discovered %d %s in %s',
discoveredFileCount,
discoveredFiles.length === 1 ? 'file' : 'files',
project,
);
}

log.info('Initial scan complete. Ready for changes');

for (const subscription of subscriptions) {
if (subscription.initialRun) {
void subscription.trigger([]);
}
}
}

resolve({
shutdown,
});
});
});
return {
shutdown,
};
};

0 comments on commit b2f97e7

Please sign in to comment.