Merge pull request #17 from bbc/jamessa-modified

Expand `modified_by` and `created_by` metadata

.github/workflows/ci-workflow.yml

@@ -30,9 +30,11 @@ jobs:
         with:
           fetch-depth: 0 # check out the entire repo history for SHA verification in lastSuccessful step
 
+      - name: Lint Markdown docs
+        run: make lint-markdown
+
       - name: Lint specification
-        run: make lint
-        working-directory: ./api
+        run: make lint-apispec
 
       - name: Render API docs
         run: make render

.markdownlint-cli2.yaml

@@ -0,0 +1,22 @@
+---
+globs:
+  - '**/*.md'
+
+ignores:
+  - 'LICENCE.md'
+  - 'ICLA.md'
+  - '.github/'
+
+default: true
+
+config:
+  default: true
+
+  # Allow arbitrary line length
+  MD013: false
+
+  # Allow duplicate headings
+  MD024: false
+
+customRules:
+  - markdownlint-rule-max-one-sentence-per-line  # Enforce one-sentence-per-line using custom rule

CONTRIBUTING.md

@@ -1,19 +1,27 @@
 # Contributing
-Thank you for your interest in our work. If you're more generally interested in our Time Addressable Media Stores work, the implementations we've built internally or opportunities for collaboration, please reach out to us at <[email protected]>
+
+Thank you for your interest in our work.
+If you're more generally interested in our Time Addressable Media Stores work, the implementations we've built internally or opportunities for collaboration, please reach out to us at <[email protected]>
 
 ## API Improvements
-We welcome suggestions for changes and improvements to the API. Feel free to reach out using the email address above, or alternatively open a GitHub Issue.
+
+We welcome suggestions for changes and improvements to the API.
+Feel free to reach out using the email address above, or alternatively open a GitHub Issue.
 
 ## Submitting Changes
-If you've fixed a problem or thought of an improvement, feel free to fork the repository and submit a Pull Request in the usual way. The process is:
+
+If you've fixed a problem or thought of an improvement, feel free to fork the repository and submit a Pull Request in the usual way.
+The process is:
 
 1. Fork the repository
 2. For more significant changes, consider starting with writing an ADR document: see [the ADR readme](./docs/adr/decisions/README.md) for more
 3. Make, commit and push changes to a branch on your fork (make sure the commit message contains your email address)
 4. If you haven't already done so, sign the CLA (see below)
-5. Open a Pull Request with your changes. Don't worry about leaving empty spaces on the PR template (it's aimed at internal contributions), just fill in the description
+5. Open a Pull Request with your changes.
+   Don't worry about leaving empty spaces on the PR template (it's aimed at internal contributions), just fill in the description
 
 ## Contributor Licence Agreement (CLA)
+
 We desire that contributors of pull requests have signed, and submitted by email, an [Individual Contributor Licence Agreement (ICLA)](./ICLA.md), which is based on the Apache CLA.
 
 The purpose of this agreement is to clearly define the terms under which intellectual property has been contributed to the BBC and thereby allow us to defend the project should there be a legal dispute regarding the software at some future time.

Makefile

@@ -1,3 +1,5 @@
+NUM_OF_PARENT:=$(shell echo $$(( $(words $(MAKEFILE_LIST)))) )
+topdir:=$(realpath $(dir $(word $(NUM_OF_PARENT),$(MAKEFILE_LIST))))
 
 all: help
 
@@ -10,10 +12,19 @@ mock-server-up:
 mock-server-down:
 	$(MAKE) -C api/ $@
 
+lint-markdown:
+	docker run -v ${topdir}:/workdir davidanson/markdownlint-cli2-rules:v0.12.1
+
+lint-apispec:
+	$(MAKE) -C api/ lint
+
+lint: lint-markdown lint-apispec
+
 help:
 	@echo "tams-api"
 	@echo "make render           - Generate HTML rendered version of OpenAPI document"
 	@echo "make mock-server-up   - Start a mock API server on http://localhost:4010"
 	@echo "make mock-server-down - Stop the mock API server"
+	@echo "make lint             - Lint API specification document and Markdown files"
 
 .PHONY: help render mock-server-up mock-server-down

README.md

@@ -1,4 +1,5 @@
 # Time-addressable Media Store API
+
 This repository contains API definitions for a Time-addressable Media Store (TAMS) server, which can be used to store, query and access segmented media - distinct from files and streams, but sharing characteristics of both.
 
 BBC R&D have demonstrated use of the TAMS approach as part of composable, software-defined workflows which can run in the cloud, on-premise or in a hybrid environment.
@@ -26,33 +27,48 @@ Media can be moved between locations and storage tiers as needed.
 Users of TAMS are insulated from the details of the underlying storage.
 
 ## Documentation
+
 - OpenAPI Specification: [TimeAddressableMediaStore.yaml](./api/TimeAddressableMediaStore.yaml)
 - Rendered Documentation: [https://bbc.github.io/tams](https://bbc.github.io/tams)
 
-This repo contains some automation to run a mock version of the API using [Stoplight Prism](https://stoplight.io/open-source/prism). To run the mock server using Docker, try something like the command below (or run `make mock-server-up`):
-```
+This repo contains some automation to run a mock version of the API using [Stoplight Prism](https://stoplight.io/open-source/prism).
+To run the mock server using Docker, try something like the command below (or run `make mock-server-up`):
+
+```shell
 docker run --rm --init --name mock-tams -v "$(pwd)":/data:ro -p 4010:4010 stoplight/prism mock /data/TimeAddressableMediaStore.yaml -h 0.0.0.0
 ```
 
 A mock API server will start at <http://localhost:4010>
 
 ## Design
-The store handles Flows which exist on an infinite timeline, are immutable, and can be grouped by Sources (based on the Flows and Sources in the [AMWA NMOS MS-04 model](https://specs.amwa.tv/ms-04/releases/v1.0.0/docs/2.1._Summary_and_Definitions.html)). A flow ID and timerange refers to a sequence of grains (_e.g._ frames of video or set of audio samples) and any point in a Flow can be uniquely addressed by a `<flow_id, timestamp>` tuple. This unique address is guaranteed to refer to a specific frame, or set of audio samples, so it can be safely passed around other tools or programs. At any time the unique address can be exchanged for the media data by an API call. But if that is not needed, media work can be done purely by reference.
 
-Grains are grouped into Flow Segments, containing for example one second of content, wrapped in a container format such as MPEG-TS. The store provides a mechanism to upload and register new segments, and an interface to request all the segments covering a particular timerange and their download URLs; an approach inspired by chunked streaming protocols like HTTP Live Streaming.
+The store handles Flows which exist on an infinite timeline, are immutable, and can be grouped by Sources (based on the Flows and Sources in the [AMWA NMOS MS-04 model](https://specs.amwa.tv/ms-04/releases/v1.0.0/docs/2.1._Summary_and_Definitions.html)).
+A flow ID and timerange refers to a sequence of grains (_e.g._ frames of video or set of audio samples) and any point in a Flow can be uniquely addressed by a `<flow_id, timestamp>` tuple.
+This unique address is guaranteed to refer to a specific frame, or set of audio samples, so it can be safely passed around other tools or programs.
+At any time the unique address can be exchanged for the media data by an API call.
+But if that is not needed, media work can be done purely by reference.
+
+Grains are grouped into Flow Segments, containing for example one second of content, wrapped in a container format such as MPEG-TS.
+The store provides a mechanism to upload and register new segments, and an interface to request all the segments covering a particular timerange and their download URLs; an approach inspired by chunked streaming protocols like HTTP Live Streaming.
 
-Segments may be stored separately from the metadata linking them to a position on the timeline, separating the metadata and data planes. For example our implementation uses an object store (_e.g._ AWS S3) for the segments, passing S3 URLs to the client to upload directly and taking advantage of the scalability of cloud object storage. Segments are also de-coupled from their point in the timeline by a link between their `<flow_id, timestamp>` tuple and the underlying object ID, so a single segment can appear at multiple points in multiple flows. This allows for copy-on-write semantics: immutability means a new Flow must be created to make changes to existing parts of the timeline, but for unmodified portions of the timeline the new `<flow_id, timestamp>` tuple points to the existing object ID.
+Segments may be stored separately from the metadata linking them to a position on the timeline, separating the metadata and data planes.
+For example our implementation uses an object store (_e.g._ AWS S3) for the segments, passing S3 URLs to the client to upload directly and taking advantage of the scalability of cloud object storage.
+Segments are also de-coupled from their point in the timeline by a link between their `<flow_id, timestamp>` tuple and the underlying object ID, so a single segment can appear at multiple points in multiple flows.
+This allows for copy-on-write semantics: immutability means a new Flow must be created to make changes to existing parts of the timeline, but for unmodified portions of the timeline the new `<flow_id, timestamp>` tuple points to the existing object ID.
 
 The Flow model is aligned with the principles and schemas of [AMWA NMOS IS-04](https://specs.amwa.tv/is-04/releases/v1.3.2/APIs/schemas/) to facilitate easy integration of NMOS-compliant media devices.
 
 ### Reading and Writing in the Store
+
 The process of reading from the store is:
+
 1. Client identifies the Flow ID and timerange of interest
 2. Client makes a request to [`GET flows/<flow_id>/segments?timerange=<timerange>`](https://bbc.github.io/tams/#/operations/GET_flows-flowId-segments) and receives a list of segments, timeranges and download URLs
 3. Client downloads each URL, concatenates the segments together and unwraps the grains within
 4. The first and last Flow Segment may contain more grains than requested, so the client should skip any received not in the requested timerange
 
 The process of writing to the store is:
+
 1. Client creates a Flow if necessary by making a request to [`PUT flows/<flow_id>`](https://bbc.github.io/tams/#/operations/PUT_flows-flowId)
 2. Client makes a request to [`POST flows/<flow_id>/storage`](https://bbc.github.io/tams/#/operations/POST_flows-flowId-storage) with the timerange to be written
 3. Store responds with a list of segment timeranges and URLs to PUT segments to, along with an optional `pre` URL to call before writing
@@ -61,16 +77,27 @@ The process of writing to the store is:
 6. Client makes requests to [`POST flows/<flow_id>/segments`](https://bbc.github.io/tams/#/operations/POST_flows-flowId-segments) with details of each new segment created, to register them on the timeline
 
 ### Flows, Sources and Mutation
-Flows in the store are assumed to be immutable: once a grain has been written to a point on the timeline on a specific Flow, it cannot be changed. However Flows can always be extended, with empty spaces on the timeline filled in, and areas of the timeline can be permanently erased.
 
-When it becomes necessary to mutate content, for example reversioning content or performing production operations, the Flow ID (and potentially Source ID) will change. Various scenarios are explored in the [Practical Guidance for Media](https://specs.amwa.tv/ms-04/releases/v1.0.0/docs/3.0._Practical_Guidance_for_Media.html) section of AMWA MS-04.
+Flows in the store are assumed to be immutable: once a grain has been written to a point on the timeline on a specific Flow, it cannot be changed.
+However Flows can always be extended, with empty spaces on the timeline filled in, and areas of the timeline can be permanently erased.
+
+When it becomes necessary to mutate content, for example reversioning content or performing production operations, the Flow ID (and potentially Source ID) will change.
+Various scenarios are explored in the [Practical Guidance for Media](https://specs.amwa.tv/ms-04/releases/v1.0.0/docs/3.0._Practical_Guidance_for_Media.html) section of AMWA MS-04.
 
 ### API Versioning
-The API is versioned using a major and minor version number. A breaking change results in a major version increment and the minor version is reset to 0. Features such new endpoints or new (optional) data properties result in a minor version increment. Other changes such as bug fixes and documentation changes do not result in version updates. Note that the version may change frequently whilst the API is still under development!
+
+The API is versioned using a major and minor version number.
+A breaking change results in a major version increment and the minor version is reset to 0.
+Features such new endpoints or new (optional) data properties result in a minor version increment.
+Other changes such as bug fixes and documentation changes do not result in version updates.
+Note that the version may change frequently whilst the API is still under development!
 
 ## Proposals, Decisions and Architecture Changes
-This repository uses [(M)ADR documents](https://adr.github.io/madr/) to propose significant changes, facilitate discussions and decision making, and to store a record of options that were considered. 
+
+This repository uses [(M)ADR documents](https://adr.github.io/madr/) to propose significant changes, facilitate discussions and decision making, and to store a record of options that were considered.
 These documents may be found in the [docs/adr/decisions](./docs/adr/decisions/) directory, and are managed as described by the [ADR Readme](./docs/adr/decisions/README.md).
 
 ## Get in touch
-If you'd like to know more about our work on TAMS, talk about our implementations or start a collaboration, contact us on <[email protected]>. Also see [CONTRIBUTING.md](./CONTRIBUTING.md) for more about how to make contributions.
+
+If you'd like to know more about our work on TAMS, talk about our implementations or start a collaboration, contact us on <[email protected]>.
+Also see [CONTRIBUTING.md](./CONTRIBUTING.md) for more about how to make contributions.

api/examples/deletion-request-get-200.json

@@ -7,5 +7,6 @@
     "created": "2018-12-10T15:40:08.339376+00:00",
     "updated": "2018-12-10T15:40:09.339393+00:00",
     "expiry": "2018-12-11T15:40:09.339393+00:00",
+    "created_by": "tams-dev",
     "status": "started"
 }

api/examples/deletion-requests-get-200.json

@@ -8,6 +8,7 @@
         "created": "2018-12-10T15:40:08.339376+00:00",
         "updated": "2018-12-10T15:40:09.339393+00:00",
         "expiry": "2018-12-11T15:40:09.339393+00:00",
+        "created_by": "tams-dev",
         "status": "started"
     },
     {
@@ -19,6 +20,7 @@
         "created": "2018-12-10T15:40:08.339376+00:00",
         "updated": "2018-12-10T15:40:09.339393+00:00",
         "expiry": "2018-12-11T15:40:09.339393+00:00",
+        "created_by": "tams-dev",
         "status": "started"
     },
     {
@@ -30,6 +32,7 @@
         "created": "2018-12-10T15:40:08.339376+00:00",
         "updated": "2018-12-10T15:18:52.426792+00:00",
         "expiry": "2018-12-11T15:40:09.339393+00:00",
+        "created_by": "tams-dev",
         "status": "error",
         "error": {
             "type": "TAMSError",

api/examples/flow-get-200-audio-aac.json

@@ -9,6 +9,7 @@
     "label": "capture_1",
     "format": "urn:x-nmos:format:audio",
     "created_by": "tams-dev",
+    "modified_by": "tams-dev",
     "tags": {
         "quality": "web",
         "_tams_segmentation_rate": "375/8"

api/examples/flow-get-200-video-h264.json

@@ -9,6 +9,7 @@
   "label": "bbb",
   "format": "urn:x-nmos:format:video",
   "created_by": "tams-dev",
+  "modified_by": "tams-dev",
   "tags": {
     "quality": "full",
     "_tams_segmentation_rate": "10"

api/examples/flow-get-200-video-raw.json

@@ -9,6 +9,7 @@
     "label": "capture_1",
     "format": "urn:x-nmos:format:video",
     "created_by": "tams-dev",
+    "modified_by": "tams-dev",
     "tags": {
         "quality": "full",
         "_tams_segmentation_rate": "25"

api/examples/flows-get-200.json

@@ -10,6 +10,7 @@
     "label": "bbb",
     "format": "urn:x-nmos:format:video",
     "created_by": "tams-dev",
+    "modified_by": "tams-dev",
     "tags": {
       "quality": "full",
       "_tams_segmentation_rate": "10"
@@ -53,6 +54,7 @@
     "label": "capture_1",
     "format": "urn:x-nmos:format:audio",
     "created_by": "tams-dev",
+    "modified_by": "tams-dev",
     "tags": {
       "quality": "web",
       "_tams_segmentation_rate": "375/8"
@@ -77,6 +79,7 @@
     "label": "capture_1",
     "format": "urn:x-nmos:format:video",
     "created_by": "tams-dev",
+    "modified_by": "tams-dev",
     "tags": {
       "quality": "full",
       "_tams_segmentation_rate": "25"

api/schemas/deletion-request.json

@@ -37,6 +37,10 @@
             "type": "string",
             "format": "date-time"
         },
+        "created_by": {
+            "description": "A string identifier for the entity that created the deletion request. Implementations SHOULD set suitable default values for `created_by` based on the principal accessing the system, and MAY permit clients to edit the value, subject to suitable permissions-based limitations.",
+            "type": "string"
+        },
         "updated": {
             "description": "Date/Time when this deletion request was updated",
             "type": "string",

api/schemas/flow-core.json

@@ -28,7 +28,11 @@
       "type": "string"
     },
     "created_by": {
-      "description": "A string identifier for the entity that created the flow",
+      "description": "A string identifier for the entity that created the flow. Implementations SHOULD set suitable default values for `created_by` based on the principal accessing the system, and MAY permit clients to edit the value, subject to suitable permissions-based limitations.",
+      "type": "string"
+    },
+    "modified_by": {
+      "description": "A string identifier for the entity that modified the flow metadata most recently. Implementations SHOULD set suitable default values for `modified_by` based on the principal accessing the system, and MAY permit clients to edit the value, subject to suitable permissions-based limitations.",
       "type": "string"
     },
     "tags": {

docs/adr/decisions/0000-use-markdown-adrs-to-record-design-decisions.md

@@ -25,7 +25,8 @@ It's also beneficial to anyone involved in the collaborative development of the
 ## Decision Outcome
 
 Chosen option: "Use (M)ADR documents, one per decision", because
-Meets all of the requirements. Could be supplemented with use of Issues and/or Discussions if/when the need arises, but these serve slightly different purposes.
+Meets all of the requirements.
+Could be supplemented with use of Issues and/or Discussions if/when the need arises, but these serve slightly different purposes.
 
 ### Consequences
 
@@ -35,7 +36,7 @@ Meets all of the requirements. Could be supplemented with use of Issues and/or D
 
 ### Implementation
 
-Implemented by https://github.com/bbc/tams/pull/15/
+Implemented by <https://github.com/bbc/tams/pull/15/>
 
 ## Pros and Cons of the Alternative Options
 
@@ -50,4 +51,3 @@ It was felt that GitHub Discussions, whilst a potentially useful adjunct to ADRs
 ### Use some combination of the considered options
 
 This will be kept under review, and other mechanisms may be introduced to support the use of ADRs
-