MVP: Schemas for Roofing Tiles/Slate + Prodcom

.eslintrc.json

@@ -0,0 +1,54 @@
+{
+  "env": {
+    "node": true,
+    "es6": true
+  },
+  "extends": ["standard", "eslint:recommended"],
+  "parserOptions": {
+    "ecmaVersion": 12,
+    "ecmaFeatures": {
+      "modules": true
+    }
+  },
+  "plugins": ["json"],
+  "rules": {
+    "no-multiple-empty-lines": [1, { "max": 1 }],
+    "consistent-return": 1,
+    "indent": [
+      2,
+      2,
+      {
+        "SwitchCase": 1
+      }
+    ],
+    "comma-dangle": 0,
+    "new-cap": 2,
+    "no-alert": 1,
+    "no-console": 0,
+    "no-dupe-class-members": 0,
+    "no-unused-expressions": 0,
+    "no-var": 2,
+    "prefer-arrow-callback": [
+      2,
+      {
+        "allowNamedFunctions": false
+      }
+    ],
+    "quotes": [
+      2,
+      "double",
+      {
+        "avoidEscape": true,
+        "allowTemplateLiterals": true
+      }
+    ],
+    "semi": [2, "always"],
+    "space-before-function-paren": 0,
+    "padded-blocks": [
+      "error",
+      {
+        "blocks": "never"
+      }
+    ]
+  }
+}

.github/CODEOWNERS

@@ -0,0 +1,3 @@
+schemas/* @ONSdigital/eq-runner @ONSdigital/eq-author @ONSdigital/sdx @ONSdigital/sds
+examples/* @ONSdigital/eq-runner @ONSdigital/eq-author @ONSdigital/sdx @ONSdigital/sds
+docs/* @ONSdigital/eq-runner @ONSdigital/eq-author @ONSdigital/sdx @ONSdigital/sds

.github/PULL_REQUEST_TEMPLATE.md

@@ -0,0 +1,11 @@
+### What is the context of this PR?
+<!--- Why is this change required? What problem does it solve? -->
+
+### Links
+<!--- Add any links to issues (trello, github issues) -->
+<!--- Links to any documentation -->
+
+### Checklist
+
+* [ ] Changes to the spec have been added to example schemas in [examples/](/examples/)
+* [ ] JSON Schema definitions have been updated

.github/workflows/pull_request.yml

@@ -0,0 +1,24 @@
+name: PR
+
+'on':
+  pull_request:
+    branches:
+      - main
+
+concurrency:
+  group: '${{ github.head_ref }}'
+  cancel-in-progress: true
+
+jobs:
+  lint-and-validate:
+    runs-on: ubuntu-22.04
+    steps:
+      - uses: actions/checkout@v2
+      - name: Install Node dependencies
+        run: make build
+      - name: Lint JS/JSON files
+        run: make lint
+      - name: Validate JSON schemas
+        if: always()
+        run: make validate-schemas
+

.gitignore

@@ -0,0 +1,73 @@
+# Byte-compiled / optimized / DLL files
+__pycache__/
+*.py[cod]
+*$py.class
+
+# C extensions
+*.so
+
+# Distribution / packaging
+.Python
+env/
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# PyInstaller
+#  Usually these files are written by a python script from a template
+#  before PyInstaller builds the exe, so as to inject date/other infos into it.
+*.manifest
+*.spec
+
+# Installer logs
+pip-log.txt
+pip-delete-this-directory.txt
+
+# Unit test / coverage reports
+htmlcov/
+.tox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*,cover
+.hypothesis/
+
+# Translations
+*.mo
+*.pot
+
+# Django stuff:
+*.log
+
+# Sphinx documentation
+docs/_build/
+
+# PyBuilder
+target/
+
+#Ipython Notebook
+.ipynb_checkpoints
+
+# IDE/OS
+.DS_Store
+.idea
+.vscode
+
+# Node modules
+node_modules/
+
+# hidden files
+_*

.nvmrc

@@ -0,0 +1 @@
+v18

Makefile

@@ -0,0 +1,11 @@
+build:
+	npm install
+
+format:
+	npm run format
+
+lint:
+	npm run lint
+
+validate-schemas:
+	./scripts/validateSchemas.js

README.md

@@ -1 +1,87 @@
 # sds-schema-definitions
+
+This project holds the documentation and JSON schema definitions for the surveys that require supplementary data.
+
+## Proposing Changes
+
+1. Open a PR with proposed API changes
+2. Add reviewers to the PR this change may affect
+3. Changes should be agreed upon and approved between all affected teams
+4. Any work that will come out of the proposal should be put on the teams backlogs
+
+## Supported Surveys
+
+| Survey Name   | Survey ID |
+|---------------|-----------|
+| Prodcom       | 014       |
+| PPI           | 132       |
+| EPI           | 133       |
+| IPI           | 156       |
+| ASHE          | 141       |
+| BRES          | 221       |
+| BRS           | 241       |
+| Roofing Tiles | 068       |
+| Slate         | 071       |
+
+## Docs
+
+Documentation can be found in [docs/](./docs).
+
+- [Main README](docs/README.md)
+- [Prodcom README](docs/prodcom.md)
+- [Roofing Tiles and Slate README](docs/roofing_tiles_and_slate.md)
+- [JSON Schema Definitions](schemas)
+- [JSON Examples (Unit Data)](examples)
+
+## JSON Schema Validation
+
+The supplementary data JSON schemas can be validated using JSON Schema definitions. The JSON schemas are defined using [Draft 2020-12](https://json-schema.org/specification-links.html#2020-12) and are validated via [AJV](https://ajv.js.org/).
+
+*Note: Each JSON schema is self-sufficient and has no external or internal references. Although this does create duplication in the schema, it avoids requiring SDS from needing to resolve the schema references which has a greater overhead than some duplication. This can be revisited in future whereby, the publisher resolves it before it gets to SDS or SDS resolves it before serving it back to Author.*
+
+### Prerequisites
+- Node installed matching the version specified in `.nvmrc`. It is recommended that you use [nvm](https://github.com/nvm-sh/nvm) to manage your Node versions.
+- [NPM](https://docs.npmjs.com/downloading-and-installing-node-js-and-npm) (JS package and dependency manager)
+
+**Install dependencies:**
+
+```bash
+make build
+```
+
+**Validate all examples schemas**
+
+```bash
+./scripts/validateSchemas.js
+```
+
+**Validate a single file or folder**
+
+```bash
+./scripts/validateSchemas.js <json-schema-file> <json-file-or-folder-to-validate>
+```
+
+For example:
+```bash
+./scripts/validateSchemas.js schemas/prodcom.json examples/prodcom/v1.json
+```
+
+Help:
+```bash
+./scripts/validateSchemas.js --help
+```
+
+
+## Development
+
+Format JSON/JS files
+```bash
+make format
+```
+
+Lint JSON/JS files
+```bash
+make lint
+```
+
+

docs/README.md

@@ -0,0 +1,55 @@
+# SDS Unit Data Schema
+
+This document describes the general structure that all supplementary data sets should follow.
+
+## Schema
+
+v1 Template: [template_v1.json](/schemas/template_v1.json)
+
+| Path                           | Description                                                                                                                                    | Mandatory |
+|--------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
+| `schema_version`               | A constant string indicating the version of the schema specification being used.                                                               | Yes       |
+| `key_field`                    | A constant string indicating the name of the key field for the top level data. The string must exist as a top level property.                  | Yes       |
+| `items`                        | An object used to represent repeating items. For example, list of employees, companies, products etc. See: [Repeating Items](#repeating-items) | No        |
+
+At the top level, additional supplementary data can be added. However, currently only simple key value strings, objects, arrays are supported. Nested arrays and deeply nested objected are not supported at this top level.
+
+### Repeating Items
+
+Repeating items are any bits of data that have the same semantic structure but contain different data. For example, a list of employees, companies, products etc.
+This is used to support repeating/looping features in eQ Runner. `items` are used as `lists` in runner, therefore `items` allow you to use any `list` feature in eQ Runner such as Looping and Repeating sections.
+
+Each top-level property of `items` is considered a `list`. Each property will represent a list in eQ Runner. For example, `items.employees` will be a list of employees. 
+
+Items must follow this spec:
+
+```json
+{
+  "type": "object",
+  "properties": {
+    "<some-name>": {
+      "type": "object",
+      "properties": {
+        "key_field": {
+          "const": "identifier",
+          "description": "The name to key field for the items"
+        },
+        "values": {
+          "type": "array",
+          "items": {
+            "type": "object",
+            "properties": {},
+          }
+        }
+      },
+      "additionalProperties": false,
+      "required": ["key_field", "values"]
+    }
+  },
+  "additionalProperties": false,
+  "required": ["<some-name>"]
+}
+```
+
+Data about each item is represented in `items.<some-name>.values`. Each item must have a property which match `items.<some-name>.values.key_field` property.
+

docs/prodcom.md

@@ -0,0 +1,40 @@
+# Prodcom Survey SDS Schema
+
+This document describes the schema for the Prodcom survey.
+
+## Schema
+
+Schema: [prodcom.json](/schemas/prodcom.json)
+
+**The table below only describes data that is survey specific. The generic structure of supplementary data is documented in [README.md](/docs/README.md)**
+
+| Path                                               | Description                                                              | Mandatory                       |
+|----------------------------------------------------|--------------------------------------------------------------------------|---------------------------------|
+| `note`                                             | A note that is used at the questionnaire or product level.               | No                              |
+| `note.title`                                       | The title of the note. This note may be used at survey or product level. | Yes                             |
+| `note.description`                                 | The description of the note                                              | Yes                             |
+| `items.products.key_field`                         | The field used as the key identifier for products                        | Yes                             |
+| `items.products.values[].identifier`               | Identifier for a product                                                 | Yes                             |
+| `items.products.values[].name`                     | Name of the product                                                      | Yes                             |
+| `items.products.values[].cn_codes`                 | Customs and Excise Codes for the product                                 | Yes                             |
+| `items.products.values[].guidance_include`         | Guidance to used on the "Include" panel                                  | No                              |
+| `items.products.values[].guidance_include.title`   | Title for the guidance on including certain items                        | Yes                             |
+| `items.products.values[].guidance_include.list`    | List of items to be included                                             | Yes                             |
+| `items.products.values[].guidance_exclude`         | Title for the guidance on excluding certain items                        | No                              |
+| `items.products.values[].guidance_exclude.title`   | Title for the guidance on excluding certain items                        | Yes                             |
+| `items.products.values[].guidance_exclude.list`    | List of items to be excluded                                             | Yes                             |
+| `items.products.values[].value_sales`              | Data for the "Value of sales" answer                                     | Yes                             |
+| `items.products.values[].value_sales.answer_code`  | Answer code for the value of sales of a product                          | Yes                             |
+| `items.products.values[].value_sales.label`        | Label for the value of sales of a product                                | Yes                             |
+| `items.products.values[].volume_sales`             | Data for the "Volume of sales" answer                                    | No                              |
+| `items.products.values[].volume_sales.answer_code` | Answer code for the volume of sales of a product                         | Yes                             |
+| `items.products.values[].volume_sales.label`       | Label for the volume of sales of a product                               | Yes                             |
+| `items.products.values[].volume_sales.unit_label`  | Unit label for the volume of sales of a product                          | Yes                             |
+| `items.products.values[].total_volume`             | Data for the "Total volume produced" answer                              | No (Yes if volume_sales exists) |
+| `items.products.values[].total_volume.answer_code` | Answer code for the total volume produced of a product                   | Yes                             |
+| `items.products.values[].total_volume.label`       | Label for the total volume produced of a product                         | Yes                             |
+| `items.products.values[].total_volume.unit_label`  | Unit label for the total volume produced of a product                    | Yes                             |
+
+## Examples
+
+Examples can be found at [examples/prodcom](/examples/prodcom).

docs/roofing_tiles_and_slate.md

@@ -0,0 +1,21 @@
+# Roofing Tiles + Slate Survey SDS Schema
+
+This document describes the schema for the Roofing Tiles + Slate survey.
+
+## Schema
+
+Schema: [roofing_tiles_and_stale.json](/schemas/roofing_tiles_and_slate.json)
+
+**The table below only describes data that is survey specific. The generic structure of supplementary data is documented in [README.md](/docs/README.md)**
+
+| Path                                    | Description                                                                           | Mandatory |
+|-----------------------------------------|---------------------------------------------------------------------------------------|-----------|
+| `items.local_units`                     | Data about the local units                                                            | Yes       |
+| `items.local_units.key_field`           | A constant string indicating the name of the key field for the local units.           | Yes       |
+| `items.local_units.values[].lu_ref`     | A string representing the local unit reference.                                       | Yes       |
+| `items.local_units.values[].lu_name`    | A string representing the name of the local unit.                                     | Yes       |
+| `items.local_units.values[].lu_address` | An array containing fields of the address for the local unit.                         | Yes       |
+
+## Examples
+
+Examples can be found at [examples/roofing_tiles_and_slate](/examples/roofing_tiles_and_slate).