Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

build: [master] Release 4.34.1 #319

Merged
merged 32 commits into from
Sep 15, 2020
Merged
Changes from 1 commit
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
dac3e89
Merge pull request #245 from opengovsg/release-4.33.0
liangyuanruo Sep 1, 2020
a3dde59
chore(deps-dev): bump @typescript-eslint/parser from 3.3.0 to 3.10.1 …
dependabot[bot] Sep 1, 2020
f6bc8b5
fix(deps): remove ajv as dependency (#248)
dependabot[bot] Sep 1, 2020
c72d433
fix: prevent discriminated models from being created before their bas…
karrui Sep 1, 2020
0c475fe
chore(deps-dev): bump prettier from 2.0.5 to 2.1.1 (#249)
dependabot[bot] Sep 2, 2020
4eb8413
fix(dev): fix Localstack yet again (#252)
mantariksh Sep 2, 2020
c304e26
refactor: typify webhook and migrate from middleware pattern (#251)
arshadali172 Sep 2, 2020
b913625
chore(deps-dev): bump jest from 26.2.2 to 26.4.2 (#257)
dependabot[bot] Sep 2, 2020
c285ea8
chore(deps-dev): bump eslint from 7.7.0 to 7.8.1 (#258)
dependabot[bot] Sep 2, 2020
7a81f71
fix(deps): bump lodash from 4.17.19 to 4.17.20 (#259)
dependabot[bot] Sep 2, 2020
f67732c
feat: upgrade Sentry SDK (#254)
mantariksh Sep 2, 2020
19216a0
refactor: remove unused Nodemailer env vars (#253)
liangyuanruo Sep 2, 2020
a8d9962
chore: remove form_field.isFutureOnly key (#235)
tshuli Sep 2, 2020
3858eb4
refactor: remove redundant feature factory (#261)
mantariksh Sep 2, 2020
1bfb064
fix: revert changes to configureAws (#266)
mantariksh Sep 3, 2020
68a9ea5
refactor: use convict for configuration (#190)
arshadali172 Sep 3, 2020
137c410
chore(deps-dev): bump @babel/preset-env from 7.11.0 to 7.11.5 (#268)
dependabot[bot] Sep 3, 2020
56d8565
chore(deps-dev): bump testcafe from 1.8.6 to 1.9.1 (#271)
dependabot[bot] Sep 3, 2020
e6ff2a7
feat: upgrade localstack version (#275)
mantariksh Sep 3, 2020
b254c9d
chore(deps-dev): bump @babel/core from 7.10.2 to 7.11.5 (#270)
dependabot[bot] Sep 3, 2020
246daa8
fix: fix invocations of logger that does not adhere to expected shape…
karrui Sep 3, 2020
d19a105
feat: add try-catch block to custom logger for js files (#267)
karrui Sep 3, 2020
87df2fa
feat: verified sms modal (#274)
arshadali172 Sep 3, 2020
423b8b5
chore(deps-dev): bump @typescript-eslint/eslint-plugin and @typescrip…
dependabot[bot] Sep 3, 2020
3e61aca
revert(convict): "refactor: use convict for configuration (#190)" (#285)
liangyuanruo Sep 6, 2020
fe64fcf
revert: reintroduce convict (#287)
arshadali172 Sep 7, 2020
4aeb037
fix: upgrade mongoose from 5.9.19 to 5.10.0 (#289)
snyk-bot Sep 7, 2020
beae593
chore(deps-dev): bump stylelint-config-prettier from 8.0.1 to 8.0.2 (…
dependabot[bot] Sep 7, 2020
5cba28a
refactor: migrate /auth endpoint handling to Typescript, Domain Drive…
karrui Sep 7, 2020
830211a
Bump version
Sep 8, 2020
d5bfd5a
build: bump version to 4.34.1
mantariksh Sep 10, 2020
22b8ab1
feat: log all critical bounces (#288)
mantariksh Sep 10, 2020
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
refactor: use convict for configuration (#190)
* Move session secret to config

* Move port to config

* Move otpLifeSpan, bounceLifeSpan and submissionsTopUp to config

* Move appConfig to config

* Move formsgSdkMode to config

* Move awsConfig to config

* Move types out

* Move cspReportUri and chromiumBin to config

* Fix bugs with config loading

* Move banners and custom watch grp to config

* Move dbHost to config

* Remove deprecated env vars

* Move ses to convict and enforce that prod ses are defined

* Remove repeated code to check bucket url

* Reorganise banner env vars under banner key/header

* Add missing env var to documentation and re-organise CSP_REPORT_URI'

* Reorganise based on order in documentation

* Move nodeEnv to convict

* Move bucket urls to own convict that gets validated after

* Move schemas to separate file

* Fix format of nodeEnv

* Edit nodemailer configuration to be based on environment

* Reorganise and add comments

* Remove convict config to Vars for clarity

* Add types to convict schemas

* Fix typing issues

* Add documentation on FORMSG_LOCALSTACK_ENDPT

* Reorganise for better readability

* Fix bug with db host

* Remove test env vars that no longer need to be unset

* Use getProperties so that typing can be enforced

* Remove warn validation on dev environment

* Remove default for session secret

* Add defaults for chromium bin

* Add validation for db host

* Split schemas into option, compulsory and prodOnly

* Fix typing

* Use new schemas and load dbUri based on environment

* Update package

* Remove env vars that don't need to be defined during tests

* Move cspReportUri to sentry feature, enforce type url and only pass to Helmet if defined

* Add missing documentation

* Simplify logic

* Remove otpGenerator

* Define defaults directly in schema

* Validate aws endpoint

* Move defaults to constant file

* Fix imports for constants file

* Update default for bounceLifeSpan

* Mark the relevant env vars as being sensitive

* Add todo to clean up MyInfo env vars

* Reference issue in todo

* Move aws default endpoint to aws endpoint env var and enforce region in production
urls

* Fix rebase issues

* Remove FORMSG_LOCALSTACK_ENDPT

* Remove logger from config

* Require config in logger now that config does not use logger

* Remove aws region default

* Use config.nodeEnv instead of process.env

* Update docs/DEPLOYMENT_SETUP.md

Co-authored-by: Antariksh Mahajan <[email protected]>

* Update docs/DEPLOYMENT_SETUP.md

Co-authored-by: Antariksh Mahajan <[email protected]>

* Add AWS_ENDPOINT to docker file for dev purposes

* Add comments to specify how convict defaults work

* Add a session secret default for dev environment

* Add default for aws endpoint in docker dev file

Co-authored-by: Arshad Ali <[email protected]>
Co-authored-by: Antariksh Mahajan <[email protected]>
3 people authored Sep 3, 2020
commit 68a9ea59ccb8f8112fea9af486dbe514fbec5bc1
2 changes: 1 addition & 1 deletion .template-env
Original file line number Diff line number Diff line change
@@ -25,7 +25,7 @@ AWS_ACCESS_KEY_ID=
FORMSG_SDK_MODE=


#### Optional variables, some have defaults defined here, as well as in `config/defaults`
#### Optional variables, some have defaults defined here, as well as in `config/schema`

## App Config
# APP_NAME=FormSG
2 changes: 2 additions & 0 deletions docker-compose.yml
Original file line number Diff line number Diff line change
@@ -28,6 +28,7 @@ services:
- BOUNCE_LIFE_SPAN=1800000
- AWS_ACCESS_KEY_ID=fakeKey
- AWS_SECRET_ACCESS_KEY=fakeSecret
- SESSION_SECRET=thisisasecret
- GA_TRACKING_ID
- SENTRY_CONFIG_URL
- TWILIO_ACCOUNT_SID
@@ -63,6 +64,7 @@ services:
- IS_SP_MAINTENANCE
- IS_CP_MAINTENANCE
- AGGREGATE_COLLECTION
- AWS_ENDPOINT=http://localhost:4572

mockpass:
build: https://github.com/opengovsg/mockpass.git
39 changes: 26 additions & 13 deletions docs/DEPLOYMENT_SETUP.md
Original file line number Diff line number Diff line change
@@ -96,7 +96,7 @@ The following env variables are set in Travis:

### Core Features

#### App and Database
#### App Config

| Variable | Description |
| :----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
@@ -105,19 +105,27 @@ The following env variables are set in Travis:
| `APP_URL` | Defaults to `'https://form.gov.sg'`. |
| `APP_KEYWORDS` | Defaults to `'forms, formbuilder, nodejs'`. |
| `APP_IMAGES` | Defaults to `'/public/modules/core/img/og/img_metatag.png,/public/modules/core/img/og/logo-vertical-color.png'`. |
| `APP_TWITTER_IMAGE` | ath to Twitter image. Defaults to `'/public/modules/core/img/og/logo-vertical-color.png'`. |
| `APP_TWITTER_IMAGE` | Path to Twitter image. Defaults to `'/public/modules/core/img/og/logo-vertical-color.png'`. |

#### App and Database

| Variable | Description |
| :----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `DB_HOST` | A MongoDB URI. |
| `OTP_LIFE_SPAN` | Time in milliseconds that admin login OTP is valid for. Defaults to 900000ms or 15 minutes. |
| `BOUNCE_LIFE_SPAN` | Time in milliseconds that bounces are tracked for each form. Defaults to 1800000ms or 30 minutes. Only relevant if you have set up AWS to send bounce and delivery notifications to the /emailnotifications endpoint. |
| `PORT` | Server port. Defaults to `5000`. |
| `NODE_ENV` | [Express environment mode](https://expressjs.com/en/advanced/best-practice-performance.html#set-node_env-to-production). Defaults to `'development'`. This should always be set to a production environment |
| `SESSION_SECRET` | Secret for `express-session`. Defaults to `'sandcrawler-138577'`. This should always be set in a production environment. |
| `SUBMISSIONS_TOP_UP` | Use this to inflate the number of submissions displayed on the landing page. Defaults to `0`. |

#### Banners

| Variable | Description |
| :----------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `SITE_BANNER_CONTENT` | If set, displays a banner message on both private routes that `ADMIN_BANNER_CONTENT` covers **and** public form routes that `IS_GENERAL_MAINTENANCE` covers. Overrides all other banner environment variables |
| `ADMIN_BANNER_CONTENT` | If set, displays a banner message on private admin routes such as the form list page as well as form builder pages. |
| `IS_LOGIN_BANNER` | If set, displays a banner message on the login page |
| `IS_GENERAL_MAINTENANCE` | If set, displays a banner message on all forms. Overrides `IS_SP_MAINTENANCE` and `IS_CP_MAINTENANCE`. |
| `IS_SP_MAINTENANCE` | If set, displays a banner message on SingPass forms. Overrides `IS_CP_MAINTENANCE`. |
| `IS_CP_MAINTENANCE` | If set, displays a banner message on SingPass forms. |
| `SUBMISSIONS_TOP_UP` | Use this to inflate the number of submissions displayed on the landing page. Defaults to `0`. |

#### AWS services

@@ -126,6 +134,7 @@ The following env variables are set in Travis:
| `AWS_REGION` | AWS region. |
| `AWS_ACCESS_KEY_ID` | AWS IAM access key ID used to access S3. |
| `AWS_SECRET_ACCESS_KEY` | AWS IAM access secret used to access S3. |
| `AWS_ENDPOINT` | AWS S3 bucket endpoint. |
| `IMAGE_S3_BUCKET` | Name of S3 bucket for image field uploads. |
| `LOGO_S3_BUCKET` | Name of S3 bucket for form logo uploads. |
| `LOGO_S3_BUCKET` | Name of S3 bucket for form logo uploads. |
@@ -152,6 +161,7 @@ The following env variables are set in Travis:
| `MAIL_LOGGER` | If set to true then logs to console. If value is not set or is false then nothing is logged. |
| `MAIL_DEBUG` | If set to `true`, then logs SMTP traffic, otherwise logs only transaction events. |
| `CHROMIUM_BIN` | Filepath to chromium binary. Required for email autoreply PDF generation with Puppeteer. |
| `BOUNCE_LIFE_SPAN` | Time in milliseconds that bounces are tracked for each form. Defaults to 10800000ms or 3 hours. Only relevant if you have set up AWS to send bounce and delivery notifications to the /emailnotifications endpoint. |

### Additional Features

@@ -180,7 +190,8 @@ If this feature is enabled, client-side error events will be piped to [sentry.io

| Variable | Description |
| :------------------ | ----------------------------------------------------------------------------------------------------- |
| `SENTRY_CONFIG_URL` | Sentry.io URL for configuring the Sentry SDK. |
| `CSP_REPORT_URI` | Reporting URL for Content Security Policy violdations. Can be configured to use a Sentry.io endpoint. |
| `SENTRY_CONFIG_URL` | Sentry.io URL for configuring the Raven SDK. |
| `CSP_REPORT_URI` | Reporting URL for Content Security Policy violdations. Can be configured to use a Sentry.io endpoint. |

#### Examples page Using Pre-Computed Results
@@ -235,6 +246,8 @@ Note that MyInfo is currently not supported for storage mode forms and enabling
| `MYINFO_CLIENT_CONFIG` | Configures [MyInfoGovClient](https://github.com/opengovsg/myinfo-gov-client). Set this to either`stg` or `prod` to fetch MyInfo data from the corresponding endpoints. |
| `MYINFO_FORMSG_KEY_PATH` | Filepath to MyInfo private key, which is used to decrypt returned responses. |
| `MYINFO_APP_KEY` | (deprecated) Directly specify contents of the MyInfo FormSG private key. Only works if `NODE_ENV` is set to `development`. |
| `IS_SP_MAINTENANCE` | If set, displays a banner message on SingPass forms. Overrides `IS_CP_MAINTENANCE`. |
| `IS_CP_MAINTENANCE` | If set, displays a banner message on CorpPass forms. |

#### Verified Emails/SMSes

@@ -258,9 +271,9 @@ If this feature is enabled, storage mode forms will also support authentication

### Tests

| Variable | Description |
| :------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `MONGO_BINARY_VERSION` | Version of the Mongo binary used. Defaults to `'latest'` according to [MongoMemoryServer](https://github.com/nodkz/mongodb-memory-server) docs. |
| `PWD` | Path of working directory. |
| `MOCK_WEBHOOK_CONFIG_FILE` | Path of configuration file for mock webhook server |
| `MOCK_WEBHOOK_PORT` | Port of mock webhook server |
| Variable | Description |
| :--------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- |
| `MONGO_BINARY_VERSION` | Version of the Mongo binary used. Defaults to `'latest'` according to [MongoMemoryServer](https://github.com/nodkz/mongodb-memory-server) docs. |
| `PWD` | Path of working directory. |
| `MOCK_WEBHOOK_CONFIG_FILE` | Path of configuration file for mock webhook server |
| `MOCK_WEBHOOK_PORT` | Port of mock webhook server |
8 changes: 4 additions & 4 deletions init-localstack.sh
Original file line number Diff line number Diff line change
@@ -1,10 +1,10 @@
#!/bin/bash
set -x
until $(curl --output /dev/null --silent --head --fail http://localhost:4572); do
until $(curl --output /dev/null --silent --head --fail $AWS_ENDPOINT); do
printf 'Waiting for Localstack to be ready...'
sleep 5
done
awslocal --endpoint-url=http://localhost:4572 s3 mb s3://$IMAGE_S3_BUCKET
awslocal --endpoint-url=http://localhost:4572 s3 mb s3://$LOGO_S3_BUCKET
awslocal --endpoint-url=http://localhost:4572 s3 mb s3://$ATTACHMENT_S3_BUCKET
awslocal --endpoint-url=$AWS_ENDPOINT s3 mb s3://$IMAGE_S3_BUCKET
awslocal --endpoint-url=$AWS_ENDPOINT s3 mb s3://$LOGO_S3_BUCKET
awslocal --endpoint-url=$AWS_ENDPOINT s3 mb s3://$ATTACHMENT_S3_BUCKET
set +x
10 changes: 10 additions & 0 deletions package-lock.json

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

2 changes: 2 additions & 0 deletions package.json
Original file line number Diff line number Diff line change
@@ -87,6 +87,7 @@
"angular-ui-router": "~1.0.22",
"async": "~1.5.2",
"await-to-js": "^2.1.1",
"aws-info": "^1.1.0",
"aws-sdk": "^2.734.0",
"axios": "^0.20.0",
"bcrypt": "^5.0.0",
@@ -131,6 +132,7 @@
"lodash": "^4.17.20",
"mobile-detect": "^1.4.2",
"moment-timezone": "0.5.31",
"mongodb-uri": "^0.9.7",
"mongoose": "^5.9.10",
"multiparty": ">=4.1.3",
"ng-infinite-scroll": "^1.3.0",
8 changes: 4 additions & 4 deletions src/app/controllers/authentication.server.controller.js
Original file line number Diff line number Diff line change
@@ -16,7 +16,7 @@ const validator = require('validator')
const { StatusCodes } = require('http-status-codes')

const config = require('../../config/config')
const defaults = require('../../config/defaults').default
const { LINKS } = require('../../shared/constants')
const PERMISSIONS = require('../utils/permission-levels').default
const { getRequestIp } = require('../utils/request')
const logger = require('../../config/logger').createLoggerWithLabel(module)
@@ -55,7 +55,7 @@ exports.validateDomain = function (req, res, next) {
return res
.status(StatusCodes.INTERNAL_SERVER_ERROR)
.send(
`Unable to validate email domain. If this issue persists, please submit a Support Form (${defaults.links.supportFormLink}).`,
`Unable to validate email domain. If this issue persists, please submit a Support Form (${LINKS.supportFormLink}).`,
)
}
// Agency not found
@@ -286,7 +286,7 @@ exports.verifyOtp = function (req, res, next) {
return res
.status(StatusCodes.INTERNAL_SERVER_ERROR)
.send(
`Unable to login at this time. Please submit a Support Form (${defaults.links.supportFormLink}).`,
`Unable to login at this time. Please submit a Support Form (${LINKS.supportFormLink}).`,
)
}
if (!updatedRecord) {
@@ -412,7 +412,7 @@ exports.signIn = function (req, res) {
return res
.status(StatusCodes.INTERNAL_SERVER_ERROR)
.send(
`User signin failed. Please try again later and if the problem persists, submit our Support Form (${defaults.links.supportFormLink}).`,
`User signin failed. Please try again later and if the problem persists, submit our Support Form (${LINKS.supportFormLink}).`,
)
}
let userObj = {
2 changes: 2 additions & 0 deletions src/app/factories/spcp-myinfo.factory.js
Original file line number Diff line number Diff line change
@@ -56,6 +56,8 @@ const spcpFactory = ({ isEnabled, props }) => {
singpassEserviceId: props.spEsrvcId,
}
let myInfoGovClient
// TODO: These env vars should move to spcp-myinfo.config and be validated
// as part of convict (Issue #255)
if (config.nodeEnv === 'production') {
let myInfoPrefix =
process.env.MYINFO_CLIENT_CONFIG === 'stg' ? 'STG2-' : 'PROD2-'
429 changes: 121 additions & 308 deletions src/config/config.ts

Large diffs are not rendered by default.

63 changes: 0 additions & 63 deletions src/config/defaults.ts

This file was deleted.

6 changes: 6 additions & 0 deletions src/config/feature-manager/sentry.config.ts
Original file line number Diff line number Diff line change
@@ -9,6 +9,12 @@ const sentryFeature: RegisterableFeature<FeatureNames.Sentry> = {
default: null,
env: 'SENTRY_CONFIG_URL',
},
cspReportUri: {
doc: 'Endpoint for content security policy reporting',
format: 'url',
default: null,
env: 'CSP_REPORT_URI',
},
},
}

1 change: 1 addition & 0 deletions src/config/feature-manager/types.ts
Original file line number Diff line number Diff line change
@@ -26,6 +26,7 @@ export interface IGoogleAnalytics {

export interface ISentry {
sentryConfigUrl: string
cspReportUri: string
}

export interface ISms {
14 changes: 5 additions & 9 deletions src/config/logger.ts
Original file line number Diff line number Diff line change
@@ -8,7 +8,9 @@ import { v4 as uuidv4 } from 'uuid'
import { format, Logger, LoggerOptions, loggers, transports } from 'winston'
import WinstonCloudWatch from 'winston-cloudwatch'

import defaults from './defaults'
import { Environment } from '../types'

import { aws, customCloudWatchGroup, isDev, nodeEnv } from './config'

// Params to enforce the logging format.
type CustomLoggerParams = {
@@ -20,12 +22,6 @@ type CustomLoggerParams = {
error?: Error
}

// Cannot use config due to logger being instantiated first, and
// having circular dependencies.
const isDev = ['development', 'test'].includes(process.env.NODE_ENV)
const customCloudWatchGroup = process.env.CUSTOM_CLOUDWATCH_LOG_GROUP
const awsRegion = process.env.AWS_REGION || defaults.aws.region

// A variety of helper functions to make winston logging like console logging,
// allowing multiple arguments.
// Retrieved from
@@ -167,7 +163,7 @@ const createLoggerOptions = (label: string): LoggerOptions => {
),
transports: [
new transports.Console({
silent: process.env.NODE_ENV === 'test',
silent: nodeEnv === Environment.Test,
}),
],
exitOnError: false,
@@ -238,7 +234,7 @@ export const createCloudWatchLogger = (label: string) => {
// not share sequence tokens. Hence generate a unique ID for each instance
// of the logger.
logStreamName: uuidv4(),
awsRegion,
awsRegion: aws.region,
jsonMessage: true,
}),
]
363 changes: 363 additions & 0 deletions src/config/schema.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,363 @@
import { PackageMode } from '@opengovsg/formsg-sdk/dist/types'
import awsInfo from 'aws-info'
import convict, { Schema } from 'convict'
import { email, url } from 'convict-format-with-validator'
import { isNil } from 'lodash'
import mongodbUri from 'mongodb-uri'
import validator from 'validator'

import {
Environment,
IBucketUrlSchema,
ICompulsoryVarsSchema,
IOptionalVarsSchema,
IProdOnlyVarsSchema,
} from '../types'

convict.addFormat(url)
convict.addFormat(email)
convict.addFormat({
name: 'string[]',
validate: (val: string[]) => {
if (!Array.isArray(val)) {
throw new Error('must be of type Array')
}
if (val.some((i) => typeof i !== 'string')) {
throw new Error('Elements must be of type string')
}
},
coerce: (val: string): string[] => {
return val.split(',')
},
})

/**
* Verifies that S3 bucket url is a valid url with or without trailing slash
*/
const validateBucketUrl = (
val: string,
{
isDev,
hasTrailingSlash,
region,
}: { isDev: boolean; hasTrailingSlash: boolean; region: string },
) => {
if (!validator.isURL(val, { require_tld: !isDev })) {
throw new Error('must be a url')
}
if (hasTrailingSlash) {
if (!/[/]$/.test(val)) {
throw new Error('must end with a slash')
}
} else {
if (/[/]$/.test(val)) {
throw new Error('must not end with a slash')
}
}
// Region should be specified correctly in production
const isRegionCorrect = new RegExp(`^https://s3.${region}.amazonaws.com`, 'i')
if (!isDev && !isRegionCorrect.test(val)) {
throw new Error(`region should be ${region}`)
}
}

// If the default value does not match the format specified, the configuration built from this schema
// will throw an error upon validation (i.e. All env vars with default null have to be specified)
export const compulsoryVarsSchema: Schema<ICompulsoryVarsSchema> = {
awsConfig: {
imageS3Bucket: {
doc: 'S3 Bucket to upload images to',
format: String,
default: null,
env: 'IMAGE_S3_BUCKET',
},
logoS3Bucket: {
doc: 'S3 Bucket to upload logos to',
format: String,
default: null,
env: 'LOGO_S3_BUCKET',
},
attachmentS3Bucket: {
doc: 'S3 Bucket to upload encrypted attachments to',
format: String,
default: null,
env: 'ATTACHMENT_S3_BUCKET',
},
},
core: {
sessionSecret: {
doc: 'Session Secret',
format: String,
default: null,
env: 'SESSION_SECRET',
sensitive: true,
},
},
}

// If the following environment variables are not specified, we will fall back to the defaults provided
export const optionalVarsSchema: Schema<IOptionalVarsSchema> = {
appConfig: {
title: {
doc: 'Application name in window title',
format: String,
default: 'FormSG',
env: 'APP_NAME',
},
description: {
doc: 'Application description in meta tag',
format: String,
default: 'Form Manager for Government',
env: 'APP_DESC',
},
appUrl: {
doc: 'Application url in meta tag',
format: 'url',
default: 'https://form.gov.sg',
env: 'APP_URL',
},
keywords: {
doc: 'Application keywords in meta tag',
format: String,
default: 'forms, formbuilder, nodejs',
env: 'APP_KEYWORDS',
},
twitterImage: {
doc: 'Application image in twitter meta tag',
format: String,
default: '/public/modules/core/img/og/logo-vertical-color.png',
env: 'APP_TWITTER_IMAGE',
},
images: {
doc: 'Application images in meta tag',
format: 'string[]',
default: [
'/public/modules/core/img/og/img_metatag.png',
'/public/modules/core/img/og/logo-vertical-color.png',
],
env: 'APP_IMAGES',
},
},
banner: {
isGeneralMaintenance: {
doc: 'Load env variable with General Maintenance banner text.',
format: String,
default: '',
env: 'IS_GENERAL_MAINTENANCE',
},
isLoginBanner: {
doc: 'The banner message on login page. Allows for HTML.',
format: String,
default: '',
env: 'IS_LOGIN_BANNER',
},
siteBannerContent: {
doc:
'The banner message to show on all pages. Allows for HTML. Will supersede all other banner content if it exists.',
format: String,
default: '',
env: 'SITE_BANNER_CONTENT',
},
adminBannerContent: {
doc: 'The banner message to show on on admin pages. Allows for HTML.',
format: String,
default: '',
env: 'ADMIN_BANNER_CONTENT',
},
},
formsgSdkMode: {
doc:
'Inform SDK which public keys are to be used to sign, encrypt, or decrypt data that is passed to it',
format: ['staging', 'production', 'development', 'test'],
default: 'production' as PackageMode,
env: 'FORMSG_SDK_MODE',
},
mail: {
from: {
doc: 'Sender email address',
format: 'email',
default: 'donotreply@mail.form.gov.sg',
env: 'MAIL_FROM',
},
logger: {
doc: 'If set to true then logs to console',
format: 'Boolean',
default: false,
env: 'MAIL_LOGGER',
},
debug: {
doc:
'If set to true, then logs SMTP traffic, otherwise logs only transaction events.',
format: 'Boolean',
default: false,
env: 'MAIL_DEBUG',
},
bounceLifeSpan: {
doc: 'TTL of bounce documents in milliseconds',
format: 'int',
default: 10800000,
env: 'BOUNCE_LIFE_SPAN',
},
chromiumBin: {
doc: 'Path to chromium executable for PDF generation',
format: String,
default: '/usr/bin/chromium-browser',
env: 'CHROMIUM_BIN',
},
maxMessages: {
doc:
'Nodemailer config to help to keep the connection up-to-date for long-running messaging',
format: 'int',
default: 100,
env: 'SES_MAX_MESSAGES',
},
maxConnections: {
doc: 'Connection pool to send email in parallel to the SMTP server',
format: 'int',
default: 38,
env: 'SES_POOL',
},
socketTimeout: {
doc: 'Milliseconds of inactivity to allow before killing a connection',
format: 'int',
default: 600000,
env: 'MAIL_SOCKET_TIMEOUT',
},
},
awsConfig: {
region: {
doc: 'Region that S3 bucket is located in',
format: Object.keys(awsInfo.data.regions),
default: 'ap-southeast-1',
env: 'AWS_REGION',
},
customCloudWatchGroup: {
doc:
'Name of CloudWatch log group to store short-term logs. Log streams are separated by date.',
format: String,
default: '',
env: 'CUSTOM_CLOUDWATCH_LOG_GROUP',
},
},
core: {
port: {
doc: 'Application Port',
format: 'port',
default: 5000,
env: 'PORT',
},
otpLifeSpan: {
doc:
'OTP Life Span for Login. (Should be in miliseconds, e.g. 1000 * 60 * 15 = 15 mins)',
format: 'int',
default: 900000,
env: 'OTP_LIFE_SPAN',
},
submissionsTopUp: {
doc: 'Number of submissions to top up submissions statistic by',
format: 'int',
default: 0,
env: 'SUBMISSIONS_TOP_UP',
},
nodeEnv: {
doc: 'Express environment mode',
format: [Environment.Prod, Environment.Dev, Environment.Test],
default: Environment.Prod,
env: 'NODE_ENV',
},
},
}

export const prodOnlyVarsSchema: Schema<IProdOnlyVarsSchema> = {
port: {
doc: 'SMTP port number',
format: 'port',
default: null,
env: 'SES_PORT',
},
host: {
doc: 'SMTP hostname',
format: String,
default: null,
env: 'SES_HOST',
},
user: {
doc: 'SMTP username',
format: String,
default: null,
env: 'SES_USER',
},
pass: {
doc: 'SMTP password',
format: String,
default: null,
env: 'SES_PASS',
sensitive: true,
},
dbHost: {
doc: 'Database URI',
format: (val) => {
// Will throw error if scheme and hosts are not present
const uriObject = mongodbUri.parse(val)
/*
e.g. mongodb://database:27017/formsg will be parsed into:
{
scheme: 'mongodb',
database: 'formsg',
hosts: [ { host: 'database', port: 27017 } ]
}
e.g. https://form.gov.sg will be parsed into:
{
scheme: 'https',
hosts: [ { host: 'form.gov.sg' } ]
}
*/
if (uriObject.scheme !== 'mongodb') {
throw new Error('Scheme must be mongodb')
}
if (isNil(uriObject.database)) {
throw new Error('Database must be specified')
}
},
default: null,
env: 'DB_HOST',
sensitive: true,
},
}

export const loadS3BucketUrlSchema = ({
isDev,
region,
}: {
isDev: boolean
region: string
}): Schema<IBucketUrlSchema> => {
return {
endPoint: {
doc: 'Endpoint for S3 buckets',
format: (val) =>
validateBucketUrl(val, { isDev, hasTrailingSlash: false, region }),
default: 'https://s3.ap-southeast-1.amazonaws.com', // NOTE NO TRAILING / AT THE END OF THIS URL!
env: 'AWS_ENDPOINT',
},
attachmentBucketUrl: {
doc:
'Url of attachment S3 bucket derived from S3 endpoint and bucket name',
format: (val) =>
validateBucketUrl(val, { isDev, hasTrailingSlash: true, region }),
default: null,
},
logoBucketUrl: {
doc: 'Url of logo S3 bucket derived from S3 endpoint and bucket name',
format: (val) =>
validateBucketUrl(val, { isDev, hasTrailingSlash: false, region }),
default: null,
},
imageBucketUrl: {
doc: 'Url of images S3 bucket derived from S3 endpoint and bucket name',
format: (val) =>
validateBucketUrl(val, { isDev, hasTrailingSlash: false, region }),
default: null,
},
}
}
112 changes: 63 additions & 49 deletions src/loaders/express/helmet.ts
Original file line number Diff line number Diff line change
@@ -1,7 +1,10 @@
import { RequestHandler } from 'express'
import helmet from 'helmet'
import { get } from 'lodash'

import config from '../../config/config'
import featureManager from '../../config/feature-manager'
import { FeatureNames } from '../../config/feature-manager/types'

const helmetMiddlewares = () => {
// Only add the "Strict-Transport-Security" header if request is https.
@@ -24,57 +27,68 @@ const helmetMiddlewares = () => {
policy: 'strict-origin-when-cross-origin',
})

const cspCoreDirectives = {
defaultSrc: ["'self'"],
imgSrc: [
"'self'",
'data:',
'https://www.googletagmanager.com/',
'https://www.google-analytics.com/',
`https://s3-${config.aws.region}.amazonaws.com/agency.form.sg/`, // Agency logos
config.aws.imageBucketUrl, // Image field
config.aws.logoBucketUrl, // Form logo
'*', // TODO: Remove when we host our own images for Image field and Form Logo
],
fontSrc: ["'self'", 'data:', 'https://fonts.gstatic.com/'],
scriptSrc: [
"'self'",
'https://www.googletagmanager.com/',
'https://ssl.google-analytics.com/',
'https://www.google-analytics.com/',
'https://www.tagmanager.google.com/',
'https://www.google.com/recaptcha/',
'https://www.recaptcha.net/recaptcha/',
'https://www.gstatic.com/recaptcha/',
'https://www.gstatic.cn/',
'https://www.google-analytics.com/',
],
connectSrc: [
"'self'",
'https://www.google-analytics.com/',
'https://ssl.google-analytics.com/',
'https://sentry.io/api/',
config.aws.attachmentBucketUrl, // Attachment downloads
config.aws.imageBucketUrl, // Image field
config.aws.logoBucketUrl, // Form logo
],
frameSrc: [
"'self'",
'https://www.google.com/recaptcha/',
'https://www.recaptcha.net/recaptcha/',
],
objectSrc: ["'none'"],
styleSrc: [
"'self'",
'https://www.google.com/recaptcha/',
'https://www.recaptcha.net/recaptcha/',
'https://www.gstatic.com/recaptcha/',
'https://www.gstatic.cn/',
],
formAction: ["'self'"],
upgradeInsecureRequests: !config.isDev,
}

const reportUri = get(
featureManager.props(FeatureNames.Sentry),
'cspReportUri',
undefined,
)
const cspOptionalDirectives = reportUri ? { reportUri } : {}

const contentSecurityPolicyMiddleware = helmet.contentSecurityPolicy({
directives: {
defaultSrc: ["'self'"],
imgSrc: [
"'self'",
'data:',
'https://www.googletagmanager.com/',
'https://www.google-analytics.com/',
`https://s3-${config.aws.region}.amazonaws.com/agency.form.sg/`, // Agency logos
config.aws.imageBucketUrl, // Image field
config.aws.logoBucketUrl, // Form logo
'*', // TODO: Remove when we host our own images for Image field and Form Logo
],
fontSrc: ["'self'", 'data:', 'https://fonts.gstatic.com/'],
scriptSrc: [
"'self'",
'https://www.googletagmanager.com/',
'https://ssl.google-analytics.com/',
'https://www.google-analytics.com/',
'https://www.tagmanager.google.com/',
'https://www.google.com/recaptcha/',
'https://www.recaptcha.net/recaptcha/',
'https://www.gstatic.com/recaptcha/',
'https://www.gstatic.cn/',
'https://www.google-analytics.com/',
],
connectSrc: [
"'self'",
'https://www.google-analytics.com/',
'https://ssl.google-analytics.com/',
'https://sentry.io/api/',
config.aws.attachmentBucketUrl, // Attachment downloads
config.aws.imageBucketUrl, // Image field
config.aws.logoBucketUrl, // Form logo
],
frameSrc: [
"'self'",
'https://www.google.com/recaptcha/',
'https://www.recaptcha.net/recaptcha/',
],
objectSrc: ["'none'"],
styleSrc: [
"'self'",
'https://www.google.com/recaptcha/',
'https://www.recaptcha.net/recaptcha/',
'https://www.gstatic.com/recaptcha/',
'https://www.gstatic.cn/',
],
formAction: ["'self'"],
upgradeInsecureRequests: !config.isDev,
reportUri: config.cspReportUri,
...cspCoreDirectives,
...cspOptionalDirectives,
},
})

4 changes: 4 additions & 0 deletions src/shared/constants.ts
Original file line number Diff line number Diff line change
@@ -27,3 +27,7 @@ export const FORM_DUPLICATE_KEYS = [
'authType',
'inactiveMessage',
]

export const LINKS = {
supportFormLink: 'https://go.gov.sg/formsg-support',
}
132 changes: 132 additions & 0 deletions src/types/config.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
import { PackageMode } from '@opengovsg/formsg-sdk/dist/types'
import aws from 'aws-sdk'
import { SessionOptions } from 'express-session'
import { ConnectionOptions } from 'mongoose'
import Mail from 'nodemailer/lib/mailer'

// Enums
export enum Environment {
Dev = 'development',
Prod = 'production',
Test = 'test',
}

// Typings
export type AppConfig = {
title: string
description: string
appUrl: string
keywords: string
images: string[]
twitterImage: string
}

export type DbConfig = {
uri: string
options: ConnectionOptions
}

export type AwsConfig = {
imageS3Bucket: string
logoS3Bucket: string
attachmentS3Bucket: string
region: string
logoBucketUrl: string
imageBucketUrl: string
attachmentBucketUrl: string
s3: aws.S3
endPoint: string
}

export type MailConfig = {
mailFrom: string
mailer: {
from: string
}
transporter: Mail
}

export type Config = {
app: AppConfig
db: DbConfig
aws: AwsConfig
mail: MailConfig

cookieSettings: SessionOptions['cookie']
// Consts
isDev: boolean
nodeEnv: Environment
port: number
sessionSecret: string
chromiumBin: string
otpLifeSpan: number
bounceLifeSpan: number
formsgSdkMode: PackageMode
submissionsTopUp: number
customCloudWatchGroup: string
isGeneralMaintenance: string
isLoginBanner: string
siteBannerContent: string
adminBannerContent: string

// Functions
configureAws: () => Promise<void>
}

// Interface
export interface IProdOnlyVarsSchema {
port: number
host: string
user: string
pass: string
dbHost: string
}

export interface ICompulsoryVarsSchema {
core: {
sessionSecret: string
}
awsConfig: {
imageS3Bucket: string
logoS3Bucket: string
attachmentS3Bucket: string
}
}

export interface IOptionalVarsSchema {
appConfig: AppConfig
formsgSdkMode: PackageMode
core: {
port: number
otpLifeSpan: number
submissionsTopUp: number
nodeEnv: Environment
}
banner: {
isGeneralMaintenance: string
isLoginBanner: string
siteBannerContent: string
adminBannerContent: string
}
awsConfig: {
region: string
customCloudWatchGroup: string
}
mail: {
from: string
logger: boolean
debug: boolean
bounceLifeSpan: number
chromiumBin: string
maxMessages: number
maxConnections: number
socketTimeout: number
}
}

export interface IBucketUrlSchema {
attachmentBucketUrl: string
logoBucketUrl: string
imageBucketUrl: string
endPoint: string
}
1 change: 1 addition & 0 deletions src/types/index.ts
Original file line number Diff line number Diff line change
@@ -16,3 +16,4 @@ export * from './token'
export * from './user'
export * from './verification'
export * from './admin_verification'
export * from './config'
13 changes: 3 additions & 10 deletions tests/.test-basic-env
Original file line number Diff line number Diff line change
@@ -1,17 +1,7 @@
APP_NAME=FormSG

SES_PORT=1025
SES_HOST=
SES_USER=
SES_PASS=
DB_HOST=
PORT=

OTP_LIFE_SPAN=900000
FORMSG_LOCALSTACK_ENDPT=http://localhost:4572
SERVICES=s3

CHROMIUM_BIN=/usr/bin/chromium-browser
MONGO_BINARY_VERSION=3.6.12
MOCK_WEBHOOK_CONFIG_FILE=webhook-server-config.csv
MOCK_WEBHOOK_PORT=4000
@@ -24,3 +14,6 @@ FORMSG_SDK_MODE=test

AWS_ACCESS_KEY_ID=fakeAccessKeyId
AWS_SECRET_ACCESS_KEY=fakeSecretAccessKey
SESSION_SECRET=sandcrawler-138577

AWS_ENDPOINT=http://localhost:4572
14 changes: 3 additions & 11 deletions tests/.test-full-env
Original file line number Diff line number Diff line change
@@ -45,20 +45,12 @@ TWILIO_MESSAGING_SERVICE_SID=MGXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
VERIFICATION_SECRET_KEY=zLnXIV0cGjODell5w1usEHcGOJ/xsQDuDOw2BPcPEQOKV4Ojfbw/9QCkE8A25L/E7o/gG4dKA+eNFEGJ+BBi+w==
SIGNING_SECRET_KEY=/u+LP57Ib9y5Ytpud56FzuitSC9O6lJ4EOLOFHpsHlYpRjVdPfRqv5et5WOxLXD9zcSkOzagBJsXobd6+9pQkw==

APP_NAME=FormSG
SESSION_SECRET=sandcrawler-138577

SES_PORT=1025
SES_HOST=
SES_USER=
SES_PASS=
DB_HOST=
PORT=

OTP_LIFE_SPAN=900000
FORMSG_LOCALSTACK_ENDPT=http://localhost:4572

SERVICES=s3

CHROMIUM_BIN=/usr/bin/chromium-browser
IMAGE_S3_BUCKET=local-image-bucket
LOGO_S3_BUCKET=local-logo-bucket
ATTACHMENT_S3_BUCKET=local-attachment-bucket
@@ -72,4 +64,4 @@ MOCK_WEBHOOK_PORT=4000
AWS_ACCESS_KEY_ID=fakeAccessKeyId
AWS_SECRET_ACCESS_KEY=fakeSecretAccessKey

BOUNCE_LIFE_SPAN=1800000
AWS_ENDPOINT=http://localhost:4572
2 changes: 1 addition & 1 deletion tests/end-to-end/encrypt-submission.e2e.js
Original file line number Diff line number Diff line change
@@ -52,7 +52,7 @@ fixture('Storage mode submissions')

// Create s3 bucket for attachments
const s3 = new aws.S3({
endpoint: process.env.FORMSG_LOCALSTACK_ENDPT,
endpoint: process.env.AWS_ENDPOINT,
s3ForcePathStyle: true,
})
await s3