Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

webui: Integrate eslint-config-yscope and refactor accordingly. #351

Merged
merged 81 commits into from
Apr 16, 2024
Merged
Show file tree
Hide file tree
Changes from 51 commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
793e4d6
Add `eslint-config-yscope`; configure `@babel/plugin-transform-react-…
junhaoliao Apr 8, 2024
2a1f8b2
Clean up `main.html`.
junhaoliao Apr 8, 2024
e93415a
Reformat/Refactor `ingestion/server/publications.js`.
junhaoliao Apr 8, 2024
bddf2b5
Reformat/Refactor `ingestion/server/StatsDbManager.js`.
junhaoliao Apr 8, 2024
33b464b
Reformat/Refactor `ingestion/collections.js`.
junhaoliao Apr 9, 2024
086f9d1
Reformat/Refactor `search/server/collections.js` and `search/server/m…
junhaoliao Apr 9, 2024
096eac7
Reformat/Refactor `/imports/api`.
junhaoliao Apr 9, 2024
afa0bc6
Reformat/Refactor IngestView.
junhaoliao Apr 9, 2024
ba8d412
Update `eslint-config-yscope` to v0.0.16 in package.json.
junhaoliao Apr 9, 2024
7bf0fd2
Replace relative paths more than two level up in imports with `/impor…
junhaoliao Apr 9, 2024
8db3ef0
Fix incorrect destructured param docs.
junhaoliao Apr 9, 2024
d5d5590
In destructured params docs, use name `props` instead of `param` for …
junhaoliao Apr 9, 2024
f4c6457
Replace React functional component return types with `React.ReactElem…
junhaoliao Apr 9, 2024
fef7be7
Refactor Bootstrap panels into a separate component.
junhaoliao Apr 9, 2024
fc5ea9b
Remove debug print.
junhaoliao Apr 9, 2024
c7a0fde
Refactor SearchControls.
junhaoliao Apr 9, 2024
7ba9746
Refactor SearchResults.
junhaoliao Apr 9, 2024
2569b81
Remove ".jsx" extension in imports.
junhaoliao Apr 9, 2024
d988075
Reformat/Refactor SearchFilterControlsDrawer.
junhaoliao Apr 9, 2024
91a024f
Import actually used components from "react-bootstrap" than the whole…
junhaoliao Apr 9, 2024
7407837
Reformat/Refactor SearchResultsTable.
junhaoliao Apr 9, 2024
5ae2c5c
Reformat/Refactor SearchView.
junhaoliao Apr 9, 2024
67d9856
Remove component SCSS imports from `App.scss`.
junhaoliao Apr 9, 2024
45f0462
Correct LOCAL_STORAGE_KEYS JSDoc enum type.
junhaoliao Apr 9, 2024
b7880bd
Add missing `SearchStatus.scss`.
junhaoliao Apr 9, 2024
eacd863
Create background patterns.
junhaoliao Apr 9, 2024
24ffce2
Reformat/Refactor utils.
junhaoliao Apr 9, 2024
bcf3885
Reformat/Refactor server/main.js.
junhaoliao Apr 9, 2024
d7b7ac2
Reformat/Refactor tests.
junhaoliao Apr 9, 2024
2169eec
Reformat/Refactor launcher.
junhaoliao Apr 9, 2024
19af7cc
Update eslint-config-yscope version.
junhaoliao Apr 9, 2024
65bcb12
Disable rule `require-atomic-updates` for `dbConnPool` init.
junhaoliao Apr 9, 2024
3dcc96d
Rename setSidebarStateCollapsed to setIsSidebarCollapsed.
junhaoliao Apr 10, 2024
b7f8ddd
Suppress rules `sort-keys` and `no-warning-comments` in datetime.js.
junhaoliao Apr 10, 2024
d8c5dc8
Upgrade "eslint-config-yscope" version.
junhaoliao Apr 10, 2024
6602f16
Add `lint`, `lint:check`, `lint:fix` scripts into package.json.
junhaoliao Apr 10, 2024
ea90951
Add linting instructions.
junhaoliao Apr 10, 2024
2687200
Fix typo.
junhaoliao Apr 10, 2024
8eb1fd0
Remove redundant newline in README.md.
junhaoliao Apr 10, 2024
ef03400
Explicitly add `@babel/core` into dependency list.
junhaoliao Apr 10, 2024
b640c34
`Object.freeze()` all enums.
junhaoliao Apr 10, 2024
91fe2d3
Add info about `eslint-config-yscope` "uninstalled peer dependencies"…
junhaoliao Apr 10, 2024
927e2c2
Update ESLint installation instructions to avoid a package-lock.json …
junhaoliao Apr 10, 2024
7082dc6
Update package-lock.json.
junhaoliao Apr 10, 2024
88073dd
Fix README.md styling.
junhaoliao Apr 10, 2024
6c6b3a2
Add `meteortesting:mocha` to `.meteor/packages` to avoid related depe…
junhaoliao Apr 10, 2024
f5b3b20
Rename `Source Sans Pro` with `Source Sans 3` and upgrade the fetchin…
junhaoliao Apr 10, 2024
0808ea7
Apply suggestions from code review.
junhaoliao Apr 12, 2024
862845c
Alphabetize "dependencies" in `package.json`.
junhaoliao Apr 12, 2024
3efced1
Rename all `_MS` postfixes to `_MILLIS` for clearly indicating millis…
junhaoliao Apr 12, 2024
4760538
Remove extra line in timeline options.plugins.tooltip object.
junhaoliao Apr 12, 2024
20d3450
Apply suggestions from code review
junhaoliao Apr 12, 2024
b0b7b43
Apply suggestions from code review
junhaoliao Apr 13, 2024
83ec359
Add "mt-1" class back to SearchControlsCaseSensitivityCheck to align …
junhaoliao Apr 13, 2024
4d7e1dc
Use relative units for widths in SearchResultsTable.
junhaoliao Apr 13, 2024
718c054
Re-organize source structures in SearchControls.
junhaoliao Apr 13, 2024
55a8d25
Add Roboto font for timestamps in search results; Implement dynamic s…
junhaoliao Apr 14, 2024
aa82c4f
Rename 'rest' props to 'rootColProps' in Panel component.
junhaoliao Apr 14, 2024
4ef1348
Renamed `BYTES_PER_KILOBYTE` to `BYTES_PER_KIBIBYTE`.
junhaoliao Apr 14, 2024
2d4a67d
Add documentation for various functions and enums.
junhaoliao Apr 14, 2024
de4787d
Update `eslint-config-yscope` to v0.0.21 .
junhaoliao Apr 14, 2024
1bde8c5
Apply suggestions from code review
junhaoliao Apr 14, 2024
2c8b72e
Re-organize IngestView panels.
junhaoliao Apr 14, 2024
4e6d78c
Move Details.jsx into a dedicated folder.
junhaoliao Apr 14, 2024
d16996a
Rename `Details/Details.jsx` to `Details/index.js`.
junhaoliao Apr 14, 2024
5b71129
Refactor DetailsRow into a separate component.
junhaoliao Apr 14, 2024
9bb8f90
Rename source files whose have same name as its parent folder to avoi…
junhaoliao Apr 14, 2024
94efefa
Rename source files whose have same name as its parent folder to avoi…
junhaoliao Apr 14, 2024
b0098e9
Use classes instead of IDs for styling components; use CSS variable f…
junhaoliao Apr 14, 2024
09797c9
Move search results table related source files into a separate direct…
junhaoliao Apr 14, 2024
dd87f9a
Move other search results source files into separate directories; Ren…
junhaoliao Apr 14, 2024
ca1c050
Move SearchView types to api folder.
junhaoliao Apr 14, 2024
70dc364
Correctly import types in search/types.js .
junhaoliao Apr 14, 2024
c992125
Move CompressionStats typedef into a both client and server accessibl…
junhaoliao Apr 14, 2024
91ce0ad
Remove redundant "id" from timeline-chart component.
junhaoliao Apr 15, 2024
4716bad
Use "className" instead of "id" to style search results load sensor.
junhaoliao Apr 15, 2024
53b9276
Use absolute path for importing datetime util.
junhaoliao Apr 15, 2024
ddff74d
Apply suggestions from code review
junhaoliao Apr 15, 2024
e7f1cea
Update components/webui/README.md
junhaoliao Apr 15, 2024
4b19979
Update components/webui/README.md
junhaoliao Apr 15, 2024
b50257f
Remove redundant introduction in README.md subsections.
junhaoliao Apr 16, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions components/webui/.meteor/packages
Original file line number Diff line number Diff line change
Expand Up @@ -20,3 +20,4 @@ [email protected] # Update client in development without reloading t
[email protected] # Define static page content in .html files
react-meteor-data # React higher-order component for reactively tracking Meteor data
fourseven:scss
meteortesting:mocha
83 changes: 81 additions & 2 deletions components/webui/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,8 +2,12 @@

## Requirements

* [Node.js 14](https://nodejs.org/download/release/v14.21.3/) (Meteor.js only
[supports](https://docs.meteor.com/install#prereqs-node) Node.js versions >= 10 and <= 14)
* Node.js.
To manage different versions of Node.js, we recommend using
[nvm (Node Version Manager)](https://github.com/nvm-sh/nvm).
* **Node.js 14** - For running Meteor. Meteor.js only
[supports](https://docs.meteor.com/install#prereqs-node) Node.js versions >= 10 and <= 14.
* **Node.js 18 or higher** - For linting.
* [Meteor.js](https://docs.meteor.com/install.html#installation)

## Install the dependencies
Expand All @@ -15,6 +19,16 @@ meteor npm install
If you ever add a package manually to `package.json` or `package.json` changes
for some other reason, you should rerun this command.

> ℹ️ When running this command, you might see warnings related to uninstalled `eslint-config-yscope` peer dependencies,
> like the ones below:
> ```
> npm WARN [email protected] requires a peer of eslint@^8.57.0 but
> none is installed. You must install peer dependencies yourself.
> ```
> **These `eslint-config-yscope` warnings can be safely ignored.** They occur because the default npm version in
> Node.js 14 does not automatically install peer dependencies. If needed, peer dependencies are automatically installed
> when switching to Node.js version 18 or higher for linting purposes, as outlined in the [Linting](#linting) section.

# Running in development

The full functionality of the webui depends on other components in the CLP
Expand Down Expand Up @@ -44,3 +58,68 @@ package:
```
5. The Web UI should now be available at `http://<webui.host>:<webui.port>`
(e.g., http://localhost:4000).

# Linting

We enforce code quality and consistency across our project using ESLint. You can use the following
npm scripts defined in `package.json` to check and fix linting issues.

## Preparing Your Environment for Linting

Due to specific dependencies, linting this project requires Node.js version 18 or higher. Follow
these steps to set up your environment for linting:
1. **Switch to Node.js Version 18 or Higher**

If you have not installed Node.js 18 or higher, use nvm to install it first:

```shell
nvm install 18
```

Switch to Node.js 18:

```shell
nvm use 18
```

2. **Re-install Dependencies**

After switching to the correct Node.js version, re-install the project dependencies with the
following command to prevent `npm` from checking the `package-lock.json` file version:

```shell
npm install --no-package-lock
```

## Checking for Linting Errors

To check for linting errors across the project, run the following command:

```shell
npm run lint
```

This command will run ESLint on the source code, reporting any linting errors without fixing them.

## Automatically Fixing Linting Errors

If you want to automatically fix linting errors that can be resolved by ESLint, use the following
command:

```shell
npm run lint:fix
```

This command attempts to automatically fix any linting issues found in the same
directories.

## Linting Specific Files

While the above commands lint multiple directories, you might sometimes need to lint specific files.
You can do so by running ESLint directly with a custom file path, for example:

```shell
eslint path/to/src.js
```

Replace `path/to/src.js` with the path to the file you want to lint.
6 changes: 3 additions & 3 deletions components/webui/client/main.html
Original file line number Diff line number Diff line change
Expand Up @@ -4,10 +4,10 @@
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
<link href="https://fonts.googleapis.com/css2?family=Roboto+Mono:[email protected]&display=swap"
rel="stylesheet">
<link href='https://fonts.googleapis.com/css?family=Source+Sans+Pro' rel='stylesheet'
type='text/css'>
<link href="https://fonts.googleapis.com/css2?family=Source+Sans+3:ital,wght@0,200..900;1,200..900&display=swap"
rel="stylesheet">
</head>

<body>
<div id="root"></div>
<div id="root"></div>
</body>
25 changes: 14 additions & 11 deletions components/webui/client/main.jsx
Original file line number Diff line number Diff line change
@@ -1,18 +1,21 @@
import React from "react";
import {Meteor} from "meteor/meteor";
import {StrictMode} from "react";
import ReactDOM from "react-dom/client";
import {Router} from "react-router";

import {createBrowserHistory} from "history";
import {Meteor} from "meteor/meteor";
import {render} from "react-dom";
import {Router, Switch} from "react-router";

import {App} from "/imports/ui/App.jsx";
import {App} from "/imports/ui/App";


Meteor.startup(() => {
const routes = (
<Router history={createBrowserHistory()}>
<App/>
</Router>
);
const root = ReactDOM.createRoot(document.getElementById("root"));

render(routes, document.getElementById("root"));
root.render(
<StrictMode>
<Router history={createBrowserHistory()}>
<App/>
</Router>
</StrictMode>
);
});
5 changes: 3 additions & 2 deletions components/webui/imports/api/constants.js
Original file line number Diff line number Diff line change
@@ -1,11 +1,12 @@
/**
* Enum for the CLP package's possible storage engines. These must match the values in
* Enum of the CLP package's possible storage engines. These must match the values in
* clp_py_utils.clp_config.StorageEngine.
*
* @enum {string}
*/
const CLP_STORAGE_ENGINES = Object.freeze({
CLP: "clp",
CLP_S: "clp-s"
CLP_S: "clp-s",
});

export {CLP_STORAGE_ENGINES};
16 changes: 13 additions & 3 deletions components/webui/imports/api/ingestion/collections.js
Original file line number Diff line number Diff line change
@@ -1,8 +1,18 @@
import {Mongo} from "meteor/mongo";


const StatsCollection = new Mongo.Collection(Meteor.settings.public.StatsCollectionName);
/**
* Enum representing the statistics collection IDs.
*
* @enum {string}
*/
const STATS_COLLECTION_ID = Object.freeze({
COMPRESSION: "compression",
});

const STATS_COLLECTION_ID_COMPRESSION = "compression_stats";
const StatsCollection = new Mongo.Collection(Meteor.settings.public.StatsCollectionName);

export {StatsCollection, STATS_COLLECTION_ID_COMPRESSION};
export {
STATS_COLLECTION_ID,
StatsCollection,
};
72 changes: 48 additions & 24 deletions components/webui/imports/api/ingestion/server/StatsDbManager.js
Original file line number Diff line number Diff line change
@@ -1,21 +1,33 @@
const CLP_ARCHIVES_TABLE_COLUMN_NAMES = {
/**
* Enum of the column names for the `clp_archives` table.
*
* @enum {string}
*/
const CLP_ARCHIVES_TABLE_COLUMN_NAMES = Object.freeze({
BEGIN_TIMESTAMP: "begin_timestamp",
END_TIMESTAMP: "end_timestamp",
UNCOMPRESSED_SIZE: "uncompressed_size",
SIZE: "size",
};
});

const CLP_FILES_TABLE_COLUMN_NAMES = {
/**
* Enum of the column names for the `clp_files` table.
*
* @enum {string}
*/
const CLP_FILES_TABLE_COLUMN_NAMES = Object.freeze({
ORIG_FILE_ID: "orig_file_id",
NUM_MESSAGES: "num_messages",
};
});

/**
* Class for retrieving compression stats from the database.
*/
class StatsDbManager {
#sqlDbConnPool;

#clpArchivesTableName;

#clpFilesTableName;

/**
Expand All @@ -24,7 +36,7 @@ class StatsDbManager {
* @param {string} tableNames.clpArchivesTableName
* @param {string} tableNames.clpFilesTableName
*/
constructor(sqlDbConnPool, {
constructor (sqlDbConnPool, {
clpArchivesTableName,
clpFilesTableName,
}) {
Expand All @@ -35,7 +47,7 @@ class StatsDbManager {
}

/**
* @typedef {object} Stats
* @typedef {object} CompressionStats
* @property {number|null} begin_timestamp
* @property {number|null} end_timestamp
* @property {number|null} total_uncompressed_size
Expand All @@ -46,26 +58,38 @@ class StatsDbManager {

/**
* Queries compression stats.
* @returns {Promise<Stats>}
*
* @return {Promise<CompressionStats>}
* @throws {Error} on error.
*/
async getCompressionStats() {
const [queryStats] = await this.#sqlDbConnPool.query(
`SELECT a.begin_timestamp AS begin_timestamp,
a.end_timestamp AS end_timestamp,
a.total_uncompressed_size AS total_uncompressed_size,
a.total_compressed_size AS total_compressed_size,
b.num_files AS num_files,
b.num_messages AS num_messages
FROM (SELECT MIN(${CLP_ARCHIVES_TABLE_COLUMN_NAMES.BEGIN_TIMESTAMP}) AS begin_timestamp,
MAX(${CLP_ARCHIVES_TABLE_COLUMN_NAMES.END_TIMESTAMP}) AS end_timestamp,
SUM(${CLP_ARCHIVES_TABLE_COLUMN_NAMES.UNCOMPRESSED_SIZE}) AS total_uncompressed_size,
SUM(${CLP_ARCHIVES_TABLE_COLUMN_NAMES.SIZE}) AS total_compressed_size
FROM ${this.#clpArchivesTableName}) a,
(SELECT NULLIF(COUNT(DISTINCT ${CLP_FILES_TABLE_COLUMN_NAMES.ORIG_FILE_ID}), 0) AS num_files,
SUM(${CLP_FILES_TABLE_COLUMN_NAMES.NUM_MESSAGES}) AS num_messages
FROM ${this.#clpFilesTableName}) b;`,
);
async getCompressionStats () {
/* eslint-disable @stylistic/js/max-len */
const [queryStats] = await this.#sqlDbConnPool.query(`
SELECT
a.begin_timestamp AS begin_timestamp,
a.end_timestamp AS end_timestamp,
a.total_uncompressed_size AS total_uncompressed_size,
a.total_compressed_size AS total_compressed_size,
b.num_files AS num_files,
b.num_messages AS num_messages
FROM
(
SELECT
MIN(${CLP_ARCHIVES_TABLE_COLUMN_NAMES.BEGIN_TIMESTAMP}) AS begin_timestamp,
MAX(${CLP_ARCHIVES_TABLE_COLUMN_NAMES.END_TIMESTAMP}) AS end_timestamp,
SUM(${CLP_ARCHIVES_TABLE_COLUMN_NAMES.UNCOMPRESSED_SIZE}) AS total_uncompressed_size,
SUM(${CLP_ARCHIVES_TABLE_COLUMN_NAMES.SIZE}) AS total_compressed_size
FROM ${this.#clpArchivesTableName}
) a,
(
SELECT
NULLIF(COUNT(DISTINCT ${CLP_FILES_TABLE_COLUMN_NAMES.ORIG_FILE_ID}), 0) AS num_files,
SUM(${CLP_FILES_TABLE_COLUMN_NAMES.NUM_MESSAGES}) AS num_messages
FROM ${this.#clpFilesTableName}
) b;
`);
/* eslint-enable @stylistic/js/max-len */

return queryStats[0];
}
}
Expand Down
39 changes: 25 additions & 14 deletions components/webui/imports/api/ingestion/server/publications.js
Original file line number Diff line number Diff line change
@@ -1,14 +1,15 @@
import {logger} from "/imports/utils/logger";
import {Meteor} from "meteor/meteor";

import {STATS_COLLECTION_ID_COMPRESSION, StatsCollection} from "../collections";
import {logger} from "/imports/utils/logger";

import {
STATS_COLLECTION_ID,
StatsCollection,
} from "../collections";
import StatsDbManager from "./StatsDbManager";


/**
* @type {number}
*/
const STATS_REFRESH_INTERVAL_MS = 5000;
const STATS_REFRESH_INTERVAL_MILLIS = 5000;

/**
* @type {StatsDbManager|null}
Expand All @@ -23,16 +24,16 @@ let refreshMeteorInterval = null;
/**
* Updates the compression statistics in the StatsCollection.
*
* @returns {Promise<void>}
* @return {Promise<void>}
*/
const refreshCompressionStats = async () => {
if (Meteor.server.stream_server.all_sockets().length === 0) {
if (0 === Meteor.server.stream_server.all_sockets().length) {
return;
}

const stats = await statsDbManager.getCompressionStats();
const filter = {
id: STATS_COLLECTION_ID_COMPRESSION,
_id: STATS_COLLECTION_ID.COMPRESSION,
};
const modifier = {
$set: stats,
Expand All @@ -45,6 +46,8 @@ const refreshCompressionStats = async () => {
};

/**
* Initializes the StatsDbManager and sets the refresh interval for compression stats updates.
*
* @param {import("mysql2/promise").Pool} sqlDbConnPool
* @param {object} tableNames
* @param {string} tableNames.clpArchivesTableName
Expand All @@ -60,9 +63,15 @@ const initStatsDbManager = (sqlDbConnPool, {
clpFilesTableName,
});

refreshMeteorInterval = Meteor.setInterval(refreshCompressionStats, STATS_REFRESH_INTERVAL_MS);
refreshMeteorInterval = Meteor.setInterval(
refreshCompressionStats,
STATS_REFRESH_INTERVAL_MILLIS
);
};

/**
* De-initializes the StatsDbManager by clearing the refreshMeteorInterval.
*/
const deinitStatsDbManager = () => {
if (null !== refreshMeteorInterval) {
Meteor.clearInterval(refreshMeteorInterval);
Expand All @@ -74,19 +83,21 @@ const deinitStatsDbManager = () => {
* Updates and publishes compression statistics.
*
* @param {string} publicationName
*
* @returns {Mongo.Cursor}
* @return {Mongo.Cursor}
*/
Meteor.publish(Meteor.settings.public.StatsCollectionName, async () => {
logger.debug(`Subscription '${Meteor.settings.public.StatsCollectionName}'`);

await refreshCompressionStats();

const filter = {
id: STATS_COLLECTION_ID_COMPRESSION,
_id: STATS_COLLECTION_ID.COMPRESSION,
};

return StatsCollection.find(filter);
});

export {initStatsDbManager, deinitStatsDbManager};
export {
deinitStatsDbManager,
initStatsDbManager,
};
Loading
Loading