Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(mojaloop/#3433): nodejs upgrade #964

Merged
merged 60 commits into from
Sep 4, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
60 commits
Select commit Hold shift + click to select a range
c30fc36
chore: update deps
oderayi Aug 29, 2023
378d498
chore: update deps
oderayi Aug 29, 2023
ef9c143
ci: set default for DOCKER_ORG env var
oderayi Aug 29, 2023
13eccf1
ci: update nvm config
oderayi Aug 29, 2023
719ab40
chore: update audit allow list
oderayi Aug 29, 2023
fd004bf
ci: update nvmrc config
oderayi Aug 29, 2023
60270b1
ci: fix bug with docker-imgage local cache
oderayi Aug 29, 2023
db12278
ci: add default machine deps install
oderayi Aug 29, 2023
8598d2c
ci: update ci config
oderayi Aug 29, 2023
935ed58
ci: use main branch from ml-core-testing-harness
oderayi Aug 30, 2023
0bc6a1c
chore: restore packages to main.
oderayi Aug 30, 2023
accac69
chore: restore .nvmrc to main.
oderayi Aug 30, 2023
60dce7b
chore: restore ci config to main.
oderayi Aug 30, 2023
7687f34
ci: apply updated ci config
oderayi Aug 30, 2023
88b39ef
chore: update npm deps
oderayi Aug 30, 2023
93af251
chore: upgrade ml-number
oderayi Aug 30, 2023
1dcfc7f
chore: upgrade ml-number, error-handling and logger
oderayi Aug 30, 2023
7d04550
chore: upgrade ml-number, error-handling and logger
oderayi Aug 30, 2023
38d79e5
chore: upgrade health, metrics, shared, and stream
oderayi Aug 30, 2023
dcb6adf
chore: revert health
oderayi Aug 30, 2023
400a06a
chore: revert stream
oderayi Aug 30, 2023
f33d0e4
chore: upgrade health
oderayi Aug 30, 2023
d3e12ee
chore: upgrade event-sdk
oderayi Aug 30, 2023
b0bc605
chore: upgrade cs-shared
oderayi Aug 30, 2023
61f5721
chore: upgrade nodejs to 18.17.1
oderayi Aug 31, 2023
0bbefd4
chore: remove glob from .ncuru and upgrade it
oderayi Aug 31, 2023
e484738
ci: display runtime versions. chore: update audit allow list
oderayi Aug 31, 2023
3202102
docs: update readme
oderayi Aug 31, 2023
719d2bb
ci: update workflow
oderayi Aug 31, 2023
2fe7668
fix: fix docker local build. update deps
oderayi Sep 1, 2023
6c58720
ci: use main branch of ml-core-test-harness
oderayi Sep 1, 2023
d997bc2
ci: make reports directory
oderayi Sep 1, 2023
5663423
ci: use new funtional test script
oderayi Sep 1, 2023
3b19060
ci: update funtional test script
oderayi Sep 1, 2023
b129b0e
ci: fix functional test script
oderayi Sep 1, 2023
53193ee
ci: update functional test script
oderayi Sep 1, 2023
e17b2f2
fix: use NODE_VERSION env var to build with specified node version
oderayi Sep 1, 2023
eb2b2a3
ci: update ci
oderayi Sep 1, 2023
a5f6708
ci: migrate master to main
oderayi Sep 1, 2023
504fa08
ci: replace hardcoded node vesion with variable
oderayi Sep 1, 2023
2d3ba5e
ci: update nvm config
oderayi Sep 1, 2023
c82e21a
ci: update publish tagging
oderayi Sep 1, 2023
94fa067
removed build, and fixed local tags on license/image scan
mdebarros Sep 1, 2023
ddb3c65
chore(snapshot): 17.0.4-snapshot.0
mdebarros Sep 1, 2023
3d8b284
removed build, and fixed local tags on license/image scan
mdebarros Sep 1, 2023
52a351b
chore(snapshot): 17.0.4-snapshot.1
mdebarros Sep 1, 2023
f9c8d01
chore(snapshot): 17.0.4-snapshot.2
mdebarros Sep 1, 2023
1238a79
cleanup
mdebarros Sep 1, 2023
3d769b7
chore(snapshot): 17.0.4-snapshot.3
mdebarros Sep 1, 2023
6dd0156
upd to readme
mdebarros Sep 1, 2023
8fee2cd
chore(snapshot): 17.0.4-snapshot.4
mdebarros Sep 1, 2023
a8802f4
updated package with docker:build, and updated main readme
mdebarros Sep 1, 2023
939fd24
chore(snapshot): 17.0.4-snapshot.5
mdebarros Sep 1, 2023
8322360
updating the test-functiona.sh cleanup
mdebarros Sep 4, 2023
00e6ad6
aligned docker compose to remove zookeeper dependency, and update kaf…
mdebarros Sep 4, 2023
b4da06e
updated deps, and aligned ci config
mdebarros Sep 4, 2023
ecb630a
chore(snapshot): 17.0.4-snapshot.6
mdebarros Sep 4, 2023
514148e
cleaned up package.json unit and int tests
mdebarros Sep 4, 2023
b034967
moved kafka configs into config folder
mdebarros Sep 4, 2023
1e34832
removed unused kafka configs
mdebarros Sep 4, 2023
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
422 changes: 264 additions & 158 deletions .circleci/config.yml

Large diffs are not rendered by default.

4 changes: 1 addition & 3 deletions .ncurc.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,5 @@ reject: [
# TODO: New versions from 2.2.0 onwards introduce a newer incompatible version of the ILP-Packet that is not compatible with the Mojaloop Specification
"ilp-packet",
# TODO: v6+ (ref: https://github.com/sindresorhus/get-port/releases/tag/v6.0.0) is an ESM library and thus not compatible with CommonJS. Future story needed to resolve.
"get-port",
# TODO: Glob v9.x introducing a dependency Package "[email protected]" which is licensed under "BlueOak-1.0.0" which is not permitted by the Mojaloop License Policy
"glob"
"get-port"
]
2 changes: 1 addition & 1 deletion .nvmrc
Original file line number Diff line number Diff line change
@@ -1 +1 @@
16.15.0
18.17.1
2 changes: 1 addition & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,7 @@ All notable changes to this project will be documented in this file. See [standa

### Features

* **mojaloop/#2880:** simplify liquidity cover check and fix issue with validation (for master) ([#918](https://github.com/mojaloop/central-ledger/issues/918)) ([825faf7](https://github.com/mojaloop/central-ledger/commit/825faf7c1c918b12eddedeb97eea0a2d563457e4))
* **mojaloop/#2880:** simplify liquidity cover check and fix issue with validation (for main) ([#918](https://github.com/mojaloop/central-ledger/issues/918)) ([825faf7](https://github.com/mojaloop/central-ledger/commit/825faf7c1c918b12eddedeb97eea0a2d563457e4))

## [16.2.0](https://github.com/mojaloop/central-ledger/compare/v16.1.0...v16.2.0) (2022-08-15)

Expand Down
34 changes: 24 additions & 10 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -1,23 +1,30 @@
FROM node:16.15.0-alpine as builder
# Arguments
ARG NODE_VERSION=lts-alpine

# NOTE: Ensure you set NODE_VERSION Build Argument as follows...
#
# export NODE_VERSION="$(cat .nvmrc)-alpine" \
# docker build \
# --build-arg NODE_VERSION=$NODE_VERSION \
# -t mojaloop/central-ledger:local \
# . \
#

# Build Image
FROM node:${NODE_VERSION} as builder

WORKDIR /opt/app

RUN apk --no-cache add git
RUN apk add --no-cache -t build-dependencies make gcc g++ python3 libtool libressl-dev openssl-dev autoconf automake \
RUN apk add --no-cache -t build-dependencies make gcc g++ python3 libtool openssl-dev autoconf automake bash \
&& cd $(npm root -g)/npm \
&& npm config set unsafe-perm true \
&& npm install -g node-gyp

COPY package.json package-lock.json* /opt/app/

RUN npm ci

COPY src /opt/app/src
COPY config /opt/app/config
COPY migrations /opt/app/migrations
COPY seeds /opt/app/seeds
COPY test /opt/app/test

FROM node:16.15.0-alpine
FROM node:${NODE_VERSION}
WORKDIR /opt/app

# Create empty log file & link stdout to the application log file
Expand All @@ -29,6 +36,13 @@ RUN adduser -D ml-user
USER ml-user

COPY --chown=ml-user --from=builder /opt/app .

COPY src /opt/app/src
COPY config /opt/app/config
COPY migrations /opt/app/migrations
COPY seeds /opt/app/seeds
COPY test /opt/app/test

RUN npm prune --production

EXPOSE 3001
Expand Down
10 changes: 5 additions & 5 deletions Onboarding.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# Onboarding

>*Note:* Before completing this guide, make sure you have completed the _general_ onboarding guide in the [base mojaloop repository](https://github.com/mojaloop/mojaloop/blob/master/onboarding.md#mojaloop-onboarding).
>*Note:* Before completing this guide, make sure you have completed the _general_ onboarding guide in the [base mojaloop repository](https://github.com/mojaloop/mojaloop/blob/main/onboarding.md#mojaloop-onboarding).

## Contents

Expand All @@ -21,7 +21,7 @@

# 1. <a name='Prerequisites'></a>Prerequisites

If you have followed the [general onboarding guide](https://github.com/mojaloop/mojaloop/blob/master/onboarding.md#mojaloop-onboarding), you should already have the following cli tools installed:
If you have followed the [general onboarding guide](https://github.com/mojaloop/mojaloop/blob/main/onboarding.md#mojaloop-onboarding), you should already have the following cli tools installed:

* `brew` (macOS), [todo: windows package manager]
* `curl`, `wget`
Expand Down Expand Up @@ -179,7 +179,7 @@ docker-compose down -v

## 6. (Optional) Connecting MySQLWorkbench to MySQL inside Docker

If you installed MySQLWorkbench from the [general onboarding guide](https://github.com/mojaloop/mojaloop/blob/master/onboarding.md#5-mysqlworkbench-optional), follow these instructions to get MySQLWorkbench connected to the `mysql` container running in docker.
If you installed MySQLWorkbench from the [general onboarding guide](https://github.com/mojaloop/mojaloop/blob/main/onboarding.md#5-mysqlworkbench-optional), follow these instructions to get MySQLWorkbench connected to the `mysql` container running in docker.

Please follow the below instructions:

Expand Down Expand Up @@ -237,12 +237,12 @@ npm run test:integration
### 8. Testing the `central-ledger` API with Postman

<!-- TODO: Verify if this link is still useful and applicable.
>Note: Check the [general onboarding guide](https://github.com/mojaloop/mojaloop/blob/master/onboarding.md#2-postman) for additional information.
>Note: Check the [general onboarding guide](https://github.com/mojaloop/mojaloop/blob/main/onboarding.md#2-postman) for additional information.
-->

#### 8.1 Prerequisites:

1. Follow the steps as described in [`5.2. Verifying Mojaloop Deployment` from the Deployment Guide](https://github.com/mojaloop/documentation/tree/master/deployment-guide#52-verifying-mojaloop-deployment).
1. Follow the steps as described in [`5.2. Verifying Mojaloop Deployment` from the Deployment Guide](https://github.com/mojaloop/documentation/tree/main/deployment-guide#52-verifying-mojaloop-deployment).
2. Clone the [Postman Collection repo](https://github.com/mojaloop/postman):
```bash
# Clone Mojaloop Postman repo
Expand Down
94 changes: 67 additions & 27 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# central-ledger

[![Git Commit](https://img.shields.io/github/last-commit/mojaloop/central-ledger.svg?style=flat)](https://github.com/mojaloop/central-ledger/commits/master)
[![Git Commit](https://img.shields.io/github/last-commit/mojaloop/central-ledger.svg?style=flat)](https://github.com/mojaloop/central-ledger/commits/main)
[![Git Releases](https://img.shields.io/github/release/mojaloop/central-ledger.svg?style=flat)](https://github.com/mojaloop/central-ledger/releases)
[![Docker pulls](https://img.shields.io/docker/pulls/mojaloop/central-ledger.svg?style=flat)](https://hub.docker.com/r/mojaloop/central-ledger)
[![Npm Version](https://img.shields.io/npm/v/@mojaloop/central-ledger.svg?style=flat)](https://www.npmjs.com/package/@mojaloop/central-ledger)
Expand Down Expand Up @@ -32,6 +32,35 @@ The following documentation represents the services, APIs and endpoints responsi
- [Automated Releases](#automated-releases)
- [Potential problems](#potential-problems)

## Docker Image

### Official Packaged Release

This package is available as a pre-built docker image on Docker Hub: [https://hub.docker.com/r/mojaloop/central-ledger](https://hub.docker.com/r/mojaloop/central-ledger)

### Build from Source

You can also build it directly from source: [https://github.com/mojaloop/central-ledger](hhttps://github.com/mojaloop/central-ledger)

However, take note of the default argument in the [Dockerfile](./Dockerfile) for `NODE_VERSION`:

```dockerfile
ARG NODE_VERSION=lts-alpine
```

It is recommend that you set the `NODE_VERSION` argument against the version set in the local [.nvmrc](./.nvmrc).

This can be done using the following command: `npm run docker:build`

Or via docker build directly:

```bash
docker build \
--build-arg NODE_VERSION="$(cat .nvmrc)-alpine" \
-t mojaloop/ml-api-adapter:local \
.
```

## Running Locally

Please follow the instruction in [Onboarding Document](Onboarding.md) to setup and run the service locally.
Expand Down Expand Up @@ -140,27 +169,38 @@ If you want to run integration tests in a repetitive manner, you can startup the

If you want to run functional tests locally utilizing the [ml-core-test-harness](https://github.com/mojaloop/ml-core-test-harness), you can run the following commands:

```bash
git clone --depth 1 --branch v0.0.2 https://github.com/mojaloop/ml-core-test-harness.git ./IGNORE/ml-core-test-harness
```
```bash
docker build -t mojaloop/central-ledger:local .
```

```bash
docker build -t mojaloop/central-ledger:local .
```
```bash
npm run test:functional
```

```bash
cd IGNORE/ml-core-test-harness
export CENTRAL_LEDGER_VERSION=local
docker-compose --project-name ttk-func --ansi never --profile all-services --profile ttk-provisioning --profile ttk-tests up -d
```
By default this will clone the [ml-core-test-harness](https://github.com/mojaloop/ml-core-test-harness) into `$ML_CORE_TEST_HARNESS_DIR`.

See default values as specified in the [test-functional.sh](./test/scripts/test-functional.sh) script.

Check test container logs for test results into `$ML_CORE_TEST_HARNESS_DIR` directory.

If you want to not have the [ml-core-test-harness](https://github.com/mojaloop/ml-core-test-harness) shutdown automatically by the script, make sure you set the following env var `export ML_CORE_TEST_SKIP_SHUTDOWN=true`.

By doing so, you are then able access TTK UI using the following URI: <http://localhost:9660>.

Or alternatively, you can monitor the `ttk-func-ttk-tests-1` (See `ML_CORE_TEST_HARNESS_TEST_FUNC_CONT_NAME` in the [test-functional.sh](./test/scripts/test-functional.sh) script) container for test results with the following command:

```bash
docker logs -f ttk-func-ttk-tests-1
```

TTK Test files:

Check test container logs for test results
- **Test Collection**: `$ML_CORE_TEST_HARNESS_DIR/docker/ml-testing-toolkit/test-cases/collections/tests/p2p.json`
- **Env Config**: `$ML_CORE_TEST_HARNESS_DIR//docker/ml-testing-toolkit/test-cases/environments/default-env.json`

Or access TTK UI using the following URI: <http://localhost:9660>
Configuration modifiers:

TTK Test files:
- Test Collection: ./IGNORE/ml-core-test-harness/docker/ml-testing-toolkit/test-cases/collections/tests/p2p.json
- Env Config: ./IGNORE/ml-core-test-harness/docker/ml-testing-toolkit/test-cases/environments/default-env.json
- **central-ledger**: [./docker/config-modifier/configs/central-ledger.js](./docker/config-modifier/configs/central-ledger.js)

## Development environment

Expand Down Expand Up @@ -193,12 +233,12 @@ If you want to run functional tests locally utilizing the [ml-core-test-harness]

## Auditing Dependencies

We use `npm-audit-resolver` along with `npm audit` to check dependencies for node vulnerabilities, and keep track of resolved dependencies with an `audit-resolve.json` file.
We use `audit-ci` along with `npm audit` to check dependencies for node vulnerabilities, and keep track of resolved dependencies with an `audit-ci.jsonc` file.

To start a new resolution process, run:

```bash
npm run audit:resolve
npm run audit:fix
```

You can then check to see if the CI will pass based on the current dependencies with:
Expand All @@ -207,7 +247,7 @@ You can then check to see if the CI will pass based on the current dependencies
npm run audit:check
```

And commit the changed `audit-resolve.json` to ensure that CircleCI will build correctly.
The [audit-ci.jsonc](./audit-ci.jsonc) contains any audit-exceptions that cannot be fixed to ensure that CircleCI will build correctly.

## Container Scans

Expand All @@ -225,26 +265,26 @@ As part of our CI/CD process, we use a combination of CircleCI, standard-version
npm package and github-release CircleCI orb to automatically trigger our releases
and image builds. This process essentially mimics a manual tag and release.

On a merge to master, CircleCI is configured to use the mojaloopci github account
On a merge to main, CircleCI is configured to use the mojaloopci github account
to push the latest generated CHANGELOG and package version number.

Once those changes are pushed, CircleCI will pull the updated master, tag and
Once those changes are pushed, CircleCI will pull the updated main, tag and
push a release triggering another subsequent build that also publishes a docker image.

### Potential problems

- There is a case where the merge to master workflow will resolve successfully, triggering
- There is a case where the merge to main workflow will resolve successfully, triggering
a release. Then that tagged release workflow subsequently failing due to the image scan,
audit check, vulnerability check or other "live" checks.

This will leave master without an associated published build. Fixes that require
This will leave main without an associated published build. Fixes that require
a new merge will essentially cause a skip in version number or require a clean up
of the master branch to the commit before the CHANGELOG and bump.
of the main branch to the commit before the CHANGELOG and bump.

This may be resolved by relying solely on the previous checks of the
merge to master workflow to assume that our tagged release is of sound quality.
merge to main workflow to assume that our tagged release is of sound quality.
We are still mulling over this solution since catching bugs/vulnerabilities/etc earlier
is a boon.

- It is unknown if a race condition might occur with multiple merges with master in
- It is unknown if a race condition might occur with multiple merges with main in
quick succession, but this is a suspected edge case.
26 changes: 10 additions & 16 deletions audit-ci.jsonc
Original file line number Diff line number Diff line change
Expand Up @@ -10,26 +10,20 @@
"GHSA-phwq-j96m-2c2q",
"GHSA-7hx8-2rxv-66xv",
"GHSA-282f-qqgm-c34q",
"GHSA-6vfc-qv3f-vr6c",
"GHSA-wc69-rhjr-hc9g",
"GHSA-f825-f98c-gj3g",
"GHSA-g954-5hwp-pp24",
"GHSA-rjqq-98f6-6j3r",
"GHSA-mjxr-4v3x-q3m4",
"GHSA-g64q-3vg8-8f93",
"GHSA-5854-jvxx-2cg9",
"GHSA-w5p7-h5w8-2hfq",
"GHSA-p9pc-299p-vxgp",
// TODO: Investigate issue with json5 lib as this is due to an underlying dependant lib dependency.
"GHSA-9c47-m6qq-7p4h", // https://github.com/advisories/GHSA-9c47-m6qq-7p4h
"GHSA-8cf7-32gw-wr33",
"GHSA-hjrf-2m68-5959",
// TODO: Upgrade jsonwebtoken in the central-services-shared lib --> https://github.com/mojaloop/project/issues/3097
"GHSA-qwph-4952-7xr6", // https://github.com/advisories/GHSA-qwph-4952-7xr6
"GHSA-hjrf-2m68-5959", // https://github.com/advisories/GHSA-hjrf-2m68-5959
"GHSA-27h2-hvpr-p74q", // https://github.com/advisories/GHSA-27h2-hvpr-p74q
// Knex dependency has been upgraded to v2.4x as advised by this advisory. Not sure why its still reporting it as an issue?
// TODO: Investigate as to why this is still being reported even though Knex was upgraded! :(
"GHSA-4jv9-3563-23j3", // https://github.com/advisories/GHSA-4jv9-3563-23j3
// TODO: To be investigated
"GHSA-rc47-6667-2j5j", // https://github.com/advisories/GHSA-rc47-6667-2j5j
"GHSA-6vfc-qv3f-vr6c",
"GHSA-mjxr-4v3x-q3m4",
"GHSA-rjqq-98f6-6j3r",
"GHSA-g64q-3vg8-8f93",
"GHSA-5854-jvxx-2cg9",
"GHSA-2mvq-xp48-4c77",
"GHSA-w5p7-h5w8-2hfq",
"GHSA-p9pc-299p-vxgp"
]
}
2 changes: 1 addition & 1 deletion config/default.json
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@
"MONGODB": {
"DISABLED": true,
"DEBUG": false,
"HOST": "localhost",
"HOST": "objstore",
"PORT": 27017,
"USER": "",
"PASSWORD": "",
Expand Down
Loading