From e248b2f0d547b5c2982461d4ab1fbcacf518ff83 Mon Sep 17 00:00:00 2001 From: Andrey Meshkov Date: Fri, 10 Nov 2023 10:12:06 +0300 Subject: [PATCH 1/9] Replace Diff-Name with resource name See the discussion here: https://github.com/AdguardTeam/FiltersCompiler/issues/192#issuecomment-1792291460 Updated the spec according to the discussion: 1. Diff-Name is replaced with a resource name specified as a part of Diff-Path 2. Optional timestamp added to the diff directive --- README.md | 40 ++++++++++---------- examples/02_validation/README.md | 6 ++- examples/02_validation/patches/v1.0.0.patch | 2 +- examples/02_validation/patches/v1.0.1.patch | 2 +- examples/03_batch/README.md | 12 ++++-- examples/03_batch/list1/list1.txt | 3 +- examples/03_batch/list1/list1_v1.0.0.txt | 3 +- examples/03_batch/list1/list1_v1.0.1.txt | 3 +- examples/03_batch/list2/list2.txt | 3 +- examples/03_batch/list2/list2_v1.0.0.txt | 3 +- examples/03_batch/list2/list2_v1.0.1.txt | 3 +- examples/03_batch/patches/batch_v1.0.0.patch | 8 ++-- examples/03_batch/patches/batch_v1.0.1.patch | 20 ++++------ examples/03_batch/patches/list1_v1.0.0.patch | 4 +- examples/03_batch/patches/list1_v1.0.1.patch | 10 ++--- examples/03_batch/patches/list2_v1.0.0.patch | 4 +- examples/03_batch/patches/list2_v1.0.1.patch | 10 ++--- 17 files changed, 65 insertions(+), 71 deletions(-) diff --git a/README.md b/README.md index afd2c86..e2ec8fc 100644 --- a/README.md +++ b/README.md @@ -26,15 +26,18 @@ In order to use the differential update mechanism, we propose several new metada This field will provide the relative path where the differential file (diff) for the filter list can be found. This differential file will take the user from their current version of the filter list to the next version. Crucially, within this differential update, the `Diff-Path` field will be updated to point to the subsequent version's diff. This ensures that the ad blocker knows where to find the next differential update. -* `Diff-Path` must be a relative path to the filter list file. +* `Diff-Path` must be a relative path to the filter list file, i.e. `/list.patch` or `../list.patch`. * `Diff-Path` is a mandatory field for enabling the differential updates mechanism. -#### `! Diff-Name:` +#### Resource name -This field is only mandatory for filter lists that support batch differential updates. It specifies the name of the resource to be patched. See the [Batch Updates](#batch-updates) section for more details. +If a list supports batch updates, the `Diff-Path` can also have a "hash" part, i.e. `/path.patch#resourceName`. This "hash" is the name of the resource to be patched. In this case, the ad blocker will only download the diff file once and then apply it to all lists that are specified in the diff file. See the [Batch Updates](#batch-updates) section for more details. -* `Diff-Name` must be a string of length 1-64. Validation regex: `[a-zA-Z0-9-_ ]{1,64}`. -* `Diff-Name` is only mandatory when the filter list supports batch differential updates. In all other cases it is ignored. +Later in the document it will be referred as "resource name". + +* This part is only mandatory when the filter list supports batch differential updates. +* The "hash" part of the URL must be a string of length 1-64. Validation regex: `[a-zA-Z0-9-_ ]{1,64}`. +* When specified, `diff name` directive in the diff file MUST match the resource name, see [Diff Files Format](#diff-files-format) for more details. #### `! Diff-Expires:` @@ -52,11 +55,12 @@ We propose using the [RCS format](https://www.gnu.org/software/diffutils/manual/ In order to support batch updates and be able to validate patch result, the standard format is extended with the `diff` directive: -`diff name:[name] checksum:[checksum] lines:[lines]` +`diff name:[name] checksum:[checksum] lines:[lines] timestamp:[timestamp]` -* `name` - name of a corresponding filter list (see `Diff-Name`). +* `name` - name of a corresponding filter list. It is only mandatory when [resource name](#resource-name) is specified in the list. * `checksum` - the expected SHA1 checksum of the file after the patch is applied. This is used to validate the patch. * `lines` - the number of lines that follow that make up the RCS diff block. Note, that `lines` are counted using the same algorithm as used by `wc -l`, i.e. it basically counts `\n`. +* `timestamp` - the timestamp of the patch. A number of milliseconds since the Unix epoch. `diff` directive is optional. If it is not specified, the patch is applied without validation. @@ -88,7 +92,7 @@ Any unexpected error during the update process should be treated as a fatal erro ### Batch Updates -The mechanism allows having a single diff file for multiple filter lists. In order to achieve this, the `Diff-Name` field MUST be specified for each filter list that supports batch differential updates. The `Diff-Name` field is then used to match a filter list with its corresponding patch in the diff file. This is achieved by using the `diff name:` directive in the diff file which links a patch to a filter list. +The mechanism allows having a single diff file for multiple filter lists. In order to achieve this, the [resource name](#resource-name) MUST be specified for each filter list that supports batch differential updates. The [resource name](#resource-name) is then used to match a filter list with its corresponding patch in the diff file. This is achieved by using the `diff name:` directive in the diff file which links a patch to a filter list. * The list that is getting patched MUST have this exact patch file specified in `Diff-Path`. * If a filter list specified inside the batch patch is not installed in the ad blocker, the patch for this file SHOULD be ignored. @@ -104,12 +108,11 @@ Let's take an example: ```adblock ! Title: List 1 - ! Diff-Path: ../patches/batch.patch - ! Diff-Name: list1 + ! Diff-Path: ../patches/batch.patch#list1 ``` * `Diff-Path` is relative to `list1.txt` location so the final URL of the diff file will be `https://example.com/patches/batch.patch`. - * `Diff-Name` is mandatory for lists that support batch differential updates. + * [Resource name](#resource-name) is set to `list1` here. It is mandatory for lists that support batch differential updates. * List 2 @@ -117,26 +120,25 @@ Let's take an example: ```adblock ! Title: List 2 - ! Diff-Path: ../patches/batch.patch - ! Diff-Name: list2 + ! Diff-Path: ../patches/batch.patch#list2 ``` * `Diff-Path` is relative to `list2.txt` location so the final URL of the diff file will be `https://example.com/patches/batch.patch`. - * `Diff-Name` is mandatory for lists that support batch differential updates. + * [Resource name](#resource-name) is set to `list2` here. It is mandatory for lists that support batch differential updates. * `batch.patch` A file that contains patches for both `list1.txt` and `list2.txt`. It uses the `diff name:` directive to point at which patch should be applied to which list. - ```diff - diff name:list1 checksum:e3c9c883378dc2a3aec9f71578c849891243bc2c lines:3 + ```diff + diff name:list1 checksum:e3c9c883378dc2a3aec9f71578c849891243bc2c lines:3 timestamp:1699599870000 d2 1 a2 1 - ! Diff-Path: patches/batch_new.patch - diff name:list2 checksum:be09384422b8d7f20da517d1245360125868f0b9 lines:3 + ! Diff-Path: patches/batch_new.patch#list1 + diff name:list2 checksum:be09384422b8d7f20da517d1245360125868f0b9 lines:3 timestamp:1699599870000 d2 1 a2 1 - ! Diff-Path: patches/batch_new.patch + ! Diff-Path: patches/batch_new.patch#list2 ``` ### Examples diff --git a/examples/02_validation/README.md b/examples/02_validation/README.md index 7fa1a44..32ac6a2 100644 --- a/examples/02_validation/README.md +++ b/examples/02_validation/README.md @@ -19,9 +19,10 @@ diff -n filter_v1.0.0.txt filter_v1.0.1.txt > patches/v1.0.0.patch # Calc the SHA1 sum of filter_v1.0.1.txt and append it to the patch file. FILENAME="filter_v1.0.1.txt" && \ PATCHFILE="patches/v1.0.0.patch" && \ +TIMESTAMP="$(date +%s)000" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ + echo "diff checksum:$SHASUM lines:$NUMLINES timestamp:$TIMESTAMP" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Calculating the RFC diff for filter.txt. @@ -30,8 +31,9 @@ diff -n filter_v1.0.1.txt filter.txt > patches/v1.0.1.patch # Calc the SHA1 sum of filter.txt and append it to the patch file. FILENAME="filter.txt" && \ PATCHFILE="patches/v1.0.1.patch" && \ +TIMESTAMP="$(date +%s)000" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l $PATCHFILE | awk '{print $1}') && \ - echo "diff checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ + echo "diff checksum:$SHASUM lines:$NUMLINES timestamp:$TIMESTAMP" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE ``` diff --git a/examples/02_validation/patches/v1.0.0.patch b/examples/02_validation/patches/v1.0.0.patch index 6471282..70b8396 100644 --- a/examples/02_validation/patches/v1.0.0.patch +++ b/examples/02_validation/patches/v1.0.0.patch @@ -1,4 +1,4 @@ -diff checksum:be09384422b8d7f20da517d1245360125868f0b9 lines:4 +diff checksum:be09384422b8d7f20da517d1245360125868f0b9 lines:4 timestamp:1699600190000 d2 3 a4 3 ! Version: v1.0.1 diff --git a/examples/02_validation/patches/v1.0.1.patch b/examples/02_validation/patches/v1.0.1.patch index c68ce2f..fc56305 100644 --- a/examples/02_validation/patches/v1.0.1.patch +++ b/examples/02_validation/patches/v1.0.1.patch @@ -1,4 +1,4 @@ -diff checksum:e3c9c883378dc2a3aec9f71578c849891243bc2c lines:5 +diff checksum:e3c9c883378dc2a3aec9f71578c849891243bc2c lines:5 timestamp:1699600198000 d2 3 a4 3 ! Version: v1.0.2 diff --git a/examples/03_batch/README.md b/examples/03_batch/README.md index 41c5a52..6f93de9 100644 --- a/examples/03_batch/README.md +++ b/examples/03_batch/README.md @@ -18,9 +18,10 @@ diff -n list1/list1_v1.0.0.txt list1/list1_v1.0.1.txt > patches/list1_v1.0.0.pat # Calc the SHA1 sum of list1_v1.0.1.txt and prepend it to the patch file. FILENAME="list1/list1_v1.0.1.txt" && \ PATCHFILE="patches/list1_v1.0.0.patch" && \ +TIMESTAMP="$(date +%s)000" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff name:list1 checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ + echo "diff name:list1 checksum:$SHASUM lines:$NUMLINES timestamp:${TIMESTAMP}" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Calculating the RFC diff for list2_v1.0.1.txt. @@ -29,9 +30,10 @@ diff -n list2/list2_v1.0.0.txt list2/list2_v1.0.1.txt > patches/list2_v1.0.0.pat # Calc the SHA1 sum of list1_v1.0.1.txt and prepend it to the patch file. FILENAME="list2/list2_v1.0.1.txt" && \ PATCHFILE="patches/list2_v1.0.0.patch" && \ +TIMESTAMP="$(date +%s)000" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff name:list2 checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ + echo "diff name:list2 checksum:$SHASUM lines:$NUMLINES timestamp:${TIMESTAMP}" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Concatenate files into a batch diff file. @@ -47,9 +49,10 @@ diff -n list1/list1_v1.0.1.txt list1/list1.txt > patches/list1_v1.0.1.patch # Calc the SHA1 sum of list1_v1.0.1.txt and prepend it to the patch file. FILENAME="list1/list1.txt" && \ PATCHFILE="patches/list1_v1.0.1.patch" && \ +TIMESTAMP="$(date +%s)000" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff name:list1 checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ + echo "diff name:list1 checksum:$SHASUM lines:$NUMLINES timestamp:${TIMESTAMP}" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Calculating the RFC diff for list2_v1.0.1.txt. @@ -58,9 +61,10 @@ diff -n list2/list2_v1.0.1.txt list2/list2.txt > patches/list2_v1.0.1.patch # Calc the SHA1 sum of list1_v1.0.1.txt and prepend it to the patch file. FILENAME="list2/list2.txt" && \ PATCHFILE="patches/list2_v1.0.1.patch" && \ +TIMESTAMP="$(date +%s)000" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff name:list2 checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ + echo "diff name:list2 checksum:$SHASUM lines:$NUMLINES timestamp:${TIMESTAMP}" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Concatenate files into a batch diff file. diff --git a/examples/03_batch/list1/list1.txt b/examples/03_batch/list1/list1.txt index 4311ad7..eee6f75 100644 --- a/examples/03_batch/list1/list1.txt +++ b/examples/03_batch/list1/list1.txt @@ -1,4 +1,3 @@ ! Title: Batch-Updatable List 1 -! Diff-Path: ../patches/batch_v1.0.2.patch -! Diff-Name: list1 +! Diff-Path: ../patches/batch_v1.0.2.patch#list1 ||example.net^ diff --git a/examples/03_batch/list1/list1_v1.0.0.txt b/examples/03_batch/list1/list1_v1.0.0.txt index 9ed2259..04d9d50 100644 --- a/examples/03_batch/list1/list1_v1.0.0.txt +++ b/examples/03_batch/list1/list1_v1.0.0.txt @@ -1,4 +1,3 @@ ! Title: Batch-Updatable List 1 -! Diff-Path: ../patches/batch_v1.0.0.patch -! Diff-Name: list1 +! Diff-Path: ../patches/batch_v1.0.0.patch#list1 ||example.org^ diff --git a/examples/03_batch/list1/list1_v1.0.1.txt b/examples/03_batch/list1/list1_v1.0.1.txt index b899856..1cff435 100644 --- a/examples/03_batch/list1/list1_v1.0.1.txt +++ b/examples/03_batch/list1/list1_v1.0.1.txt @@ -1,4 +1,3 @@ ! Title: Batch-Updatable List 1 -! Diff-Path: ../patches/batch_v1.0.1.patch -! Diff-Name: list1 +! Diff-Path: ../patches/batch_v1.0.1.patch#list1 ||example.com^ diff --git a/examples/03_batch/list2/list2.txt b/examples/03_batch/list2/list2.txt index d661f71..5dd736f 100644 --- a/examples/03_batch/list2/list2.txt +++ b/examples/03_batch/list2/list2.txt @@ -1,4 +1,3 @@ ! Title: Batch-Updatable List 2 -! Diff-Path: ../patches/batch_v1.0.2.patch -! Diff-Name: list1 +! Diff-Path: ../patches/batch_v1.0.2.patch#list2 ||test.net^ diff --git a/examples/03_batch/list2/list2_v1.0.0.txt b/examples/03_batch/list2/list2_v1.0.0.txt index 3cd41fe..50ecd9d 100644 --- a/examples/03_batch/list2/list2_v1.0.0.txt +++ b/examples/03_batch/list2/list2_v1.0.0.txt @@ -1,4 +1,3 @@ ! Title: Batch-Updatable List 2 -! Diff-Path: ../patches/batch_v1.0.0.patch -! Diff-Name: list1 +! Diff-Path: ../patches/batch_v1.0.0.patch#list2 ||test.org^ diff --git a/examples/03_batch/list2/list2_v1.0.1.txt b/examples/03_batch/list2/list2_v1.0.1.txt index e8a9c69..77cc67f 100644 --- a/examples/03_batch/list2/list2_v1.0.1.txt +++ b/examples/03_batch/list2/list2_v1.0.1.txt @@ -1,4 +1,3 @@ ! Title: Batch-Updatable List 2 -! Diff-Path: ../patches/batch_v1.0.1.patch -! Diff-Name: list1 +! Diff-Path: ../patches/batch_v1.0.1.patch#list2 ||test.com^ diff --git a/examples/03_batch/patches/batch_v1.0.0.patch b/examples/03_batch/patches/batch_v1.0.0.patch index 9620258..fe1e222 100644 --- a/examples/03_batch/patches/batch_v1.0.0.patch +++ b/examples/03_batch/patches/batch_v1.0.0.patch @@ -1,10 +1,10 @@ -f ../list1/list1.txt 76ad224737a93b407dca319c46587fe394f1c557 4 +diff name:list1 checksum:5b8abcb6763972f65970a036ee3551818c43499d lines:4 timestamp:1699599862000 d2 2 a3 2 -! Diff-Path: ../patches/batch_v1.0.1.patch +! Diff-Path: ../patches/batch_v1.0.1.patch#list1 ||example.com^ -f ../list2/list2.txt c97a5b68e5ee0349872d420e55e616b0bb979aee 4 +diff name:list2 checksum:a089ee91e339e4a5374dfbed8728e17e768d7c9b lines:4 timestamp:1699599870000 d2 2 a3 2 -! Diff-Path: ../patches/batch_v1.0.1.patch +! Diff-Path: ../patches/batch_v1.0.1.patch#list2 ||test.com^ diff --git a/examples/03_batch/patches/batch_v1.0.1.patch b/examples/03_batch/patches/batch_v1.0.1.patch index 2e92713..684f4d4 100644 --- a/examples/03_batch/patches/batch_v1.0.1.patch +++ b/examples/03_batch/patches/batch_v1.0.1.patch @@ -1,14 +1,10 @@ -diff name:list1 checksum:646d4e41c48aeeadc7194dad416a8842add4fb34 lines:6 -d2 1 -a2 1 -! Diff-Path: ../patches/batch_v1.0.2.patch -d4 1 -a4 1 +diff name:list1 checksum:11edb61b350b1c15b49c14ce7a6eba7ff1e058a8 lines:4 timestamp:1699599881000 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.2.patch#list1 ||example.net^ -diff name:list2 checksum:ad03f3c6ea989c336d1d2fa4f4f19aa397c57375 lines:6 -d2 1 -a2 1 -! Diff-Path: ../patches/batch_v1.0.2.patch -d4 1 -a4 1 +diff name:list2 checksum:e4b57a4da7fbe9072dada875695a16762b05eab2 lines:4 timestamp:1699599890000 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.2.patch#list2 ||test.net^ diff --git a/examples/03_batch/patches/list1_v1.0.0.patch b/examples/03_batch/patches/list1_v1.0.0.patch index d594d74..d2d19cc 100644 --- a/examples/03_batch/patches/list1_v1.0.0.patch +++ b/examples/03_batch/patches/list1_v1.0.0.patch @@ -1,5 +1,5 @@ -f ../list1/list1.txt 76ad224737a93b407dca319c46587fe394f1c557 4 +diff name:list1 checksum:5b8abcb6763972f65970a036ee3551818c43499d lines:4 timestamp:1699599862000 d2 2 a3 2 -! Diff-Path: ../patches/batch_v1.0.1.patch +! Diff-Path: ../patches/batch_v1.0.1.patch#list1 ||example.com^ diff --git a/examples/03_batch/patches/list1_v1.0.1.patch b/examples/03_batch/patches/list1_v1.0.1.patch index cabccef..022c14c 100644 --- a/examples/03_batch/patches/list1_v1.0.1.patch +++ b/examples/03_batch/patches/list1_v1.0.1.patch @@ -1,7 +1,5 @@ -diff name:list1 checksum:646d4e41c48aeeadc7194dad416a8842add4fb34 lines:6 -d2 1 -a2 1 -! Diff-Path: ../patches/batch_v1.0.2.patch -d4 1 -a4 1 +diff name:list1 checksum:11edb61b350b1c15b49c14ce7a6eba7ff1e058a8 lines:4 timestamp:1699599881000 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.2.patch#list1 ||example.net^ diff --git a/examples/03_batch/patches/list2_v1.0.0.patch b/examples/03_batch/patches/list2_v1.0.0.patch index 54e13af..1b5af1d 100644 --- a/examples/03_batch/patches/list2_v1.0.0.patch +++ b/examples/03_batch/patches/list2_v1.0.0.patch @@ -1,5 +1,5 @@ -f ../list2/list2.txt c97a5b68e5ee0349872d420e55e616b0bb979aee 4 +diff name:list2 checksum:a089ee91e339e4a5374dfbed8728e17e768d7c9b lines:4 timestamp:1699599870000 d2 2 a3 2 -! Diff-Path: ../patches/batch_v1.0.1.patch +! Diff-Path: ../patches/batch_v1.0.1.patch#list2 ||test.com^ diff --git a/examples/03_batch/patches/list2_v1.0.1.patch b/examples/03_batch/patches/list2_v1.0.1.patch index 45fd5ce..b87cdec 100644 --- a/examples/03_batch/patches/list2_v1.0.1.patch +++ b/examples/03_batch/patches/list2_v1.0.1.patch @@ -1,7 +1,5 @@ -diff name:list2 checksum:ad03f3c6ea989c336d1d2fa4f4f19aa397c57375 lines:6 -d2 1 -a2 1 -! Diff-Path: ../patches/batch_v1.0.2.patch -d4 1 -a4 1 +diff name:list2 checksum:e4b57a4da7fbe9072dada875695a16762b05eab2 lines:4 timestamp:1699599890000 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.2.patch#list2 ||test.net^ From 7ec1f53069295a9c955a629ed8ff703b9936a827 Mon Sep 17 00:00:00 2001 From: Andrey Meshkov Date: Fri, 10 Nov 2023 10:18:04 +0300 Subject: [PATCH 2/9] Minor changes in the text --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index e2ec8fc..0dc4ffa 100644 --- a/README.md +++ b/README.md @@ -26,12 +26,12 @@ In order to use the differential update mechanism, we propose several new metada This field will provide the relative path where the differential file (diff) for the filter list can be found. This differential file will take the user from their current version of the filter list to the next version. Crucially, within this differential update, the `Diff-Path` field will be updated to point to the subsequent version's diff. This ensures that the ad blocker knows where to find the next differential update. -* `Diff-Path` must be a relative path to the filter list file, i.e. `/list.patch` or `../list.patch`. +* `Diff-Path` must be a relative path to the filter list file, i.e. `/list.patch` or `../list.patch` or similar. * `Diff-Path` is a mandatory field for enabling the differential updates mechanism. #### Resource name -If a list supports batch updates, the `Diff-Path` can also have a "hash" part, i.e. `/path.patch#resourceName`. This "hash" is the name of the resource to be patched. In this case, the ad blocker will only download the diff file once and then apply it to all lists that are specified in the diff file. See the [Batch Updates](#batch-updates) section for more details. +If a list supports batch updates, the `Diff-Path` MUST also have a "hash" part, i.e. `/path.patch#resourceName`. This "hash" is the name of the resource to be patched. In this case, the ad blocker will only download the diff file once and then apply it to all lists that are specified in the diff file. See the [Batch Updates](#batch-updates) section for more details. Later in the document it will be referred as "resource name". From c9b4c30cead07bfc0d8f3ba2cdf5fa87cd8aaea7 Mon Sep 17 00:00:00 2001 From: Andrey Meshkov Date: Wed, 15 Nov 2023 14:03:03 +0300 Subject: [PATCH 3/9] Encoding additional information in Diff-Path --- README.md | 85 +++++++++++++------ examples/01_simple/README.md | 10 ++- examples/01_simple/filter.txt | 2 +- examples/01_simple/filter_v1.0.0.txt | 2 +- examples/01_simple/filter_v1.0.1.txt | 2 +- .../01_simple/patches/v1.0.0-472234-1.patch | 5 ++ examples/01_simple/patches/v1.0.0.patch | 5 -- .../{v1.0.1.patch => v1.0.1-472235-1.patch} | 2 +- examples/02_validation/README.md | 25 +++--- examples/02_validation/filter.txt | 2 +- examples/02_validation/filter_v1.0.0.txt | 2 +- examples/02_validation/filter_v1.0.1.txt | 2 +- .../patches/v1.0.0-m-28334060-60.patch | 6 ++ examples/02_validation/patches/v1.0.0.patch | 6 -- .../patches/v1.0.1-m-28334120-60.patch | 6 ++ examples/02_validation/patches/v1.0.1.patch | 6 -- .../patches/v1.0.2-m-28334180-60.patch | 0 examples/03_batch/README.md | 41 +++++---- examples/03_batch/list1/list1.txt | 2 +- examples/03_batch/list1/list1_v1.0.0.txt | 2 +- examples/03_batch/list1/list1_v1.0.1.txt | 2 +- examples/03_batch/list2/list2.txt | 2 +- examples/03_batch/list2/list2_v1.0.0.txt | 2 +- examples/03_batch/list2/list2_v1.0.1.txt | 2 +- .../batch_v1.0.0-s-1700045842-3600.patch | 10 +++ examples/03_batch/patches/batch_v1.0.0.patch | 10 --- .../batch_v1.0.1-s-1700049442-3600.patch | 10 +++ examples/03_batch/patches/batch_v1.0.1.patch | 10 --- .../batch_v1.0.2-s-1700053042-3600.patch | 0 .../list1_v1.0.0-s-1700045842-3600.patch | 5 ++ examples/03_batch/patches/list1_v1.0.0.patch | 5 -- .../list1_v1.0.1-s-1700049442-3600.patch | 5 ++ examples/03_batch/patches/list1_v1.0.1.patch | 5 -- .../list2_v1.0.0-s-1700045842-3600.patch | 5 ++ examples/03_batch/patches/list2_v1.0.0.patch | 5 -- .../list2_v1.0.1-s-1700049442-3600.patch | 5 ++ examples/03_batch/patches/list2_v1.0.1.patch | 5 -- 37 files changed, 175 insertions(+), 126 deletions(-) create mode 100644 examples/01_simple/patches/v1.0.0-472234-1.patch delete mode 100644 examples/01_simple/patches/v1.0.0.patch rename examples/01_simple/patches/{v1.0.1.patch => v1.0.1-472235-1.patch} (50%) create mode 100644 examples/02_validation/patches/v1.0.0-m-28334060-60.patch delete mode 100644 examples/02_validation/patches/v1.0.0.patch create mode 100644 examples/02_validation/patches/v1.0.1-m-28334120-60.patch delete mode 100644 examples/02_validation/patches/v1.0.1.patch create mode 100644 examples/02_validation/patches/v1.0.2-m-28334180-60.patch create mode 100644 examples/03_batch/patches/batch_v1.0.0-s-1700045842-3600.patch delete mode 100644 examples/03_batch/patches/batch_v1.0.0.patch create mode 100644 examples/03_batch/patches/batch_v1.0.1-s-1700049442-3600.patch delete mode 100644 examples/03_batch/patches/batch_v1.0.1.patch create mode 100644 examples/03_batch/patches/batch_v1.0.2-s-1700053042-3600.patch create mode 100644 examples/03_batch/patches/list1_v1.0.0-s-1700045842-3600.patch delete mode 100644 examples/03_batch/patches/list1_v1.0.0.patch create mode 100644 examples/03_batch/patches/list1_v1.0.1-s-1700049442-3600.patch delete mode 100644 examples/03_batch/patches/list1_v1.0.1.patch create mode 100644 examples/03_batch/patches/list2_v1.0.0-s-1700045842-3600.patch delete mode 100644 examples/03_batch/patches/list2_v1.0.0.patch create mode 100644 examples/03_batch/patches/list2_v1.0.1-s-1700049442-3600.patch delete mode 100644 examples/03_batch/patches/list2_v1.0.1.patch diff --git a/README.md b/README.md index 0dc4ffa..c38a0c5 100644 --- a/README.md +++ b/README.md @@ -10,7 +10,7 @@ The core issue revolves around the mechanism by which these filter lists are upd * Bandwidth Consumption: Continuously downloading the entire list, especially for minor updates, consumes unnecessary bandwidth. * Latency: Each full download requires more time than if only the changes were fetched. -* Server Load: Hosting servers experience an unnecessary load +* Server Load: Hosting servers experience an unnecessary load. ## Solution @@ -20,14 +20,48 @@ This approach significantly reduces bandwidth consumption, minimizes latency, an ### Changes To Filter Lists Metadata -In order to use the differential update mechanism, we propose several new metadata fields. +#### `! Diff-Path` -#### `! Diff-Path:` +In order to use the differential update mechanism we propose to add one new field to the filter list metadata: `Diff-Path`. This field will provide the relative path where the differential file (diff) for the filter list can be found. This differential file will take the user from their current version of the filter list to the next version. Crucially, within this differential update, the `Diff-Path` field will be updated to point to the subsequent version's diff. This ensures that the ad blocker knows where to find the next differential update. -* `Diff-Path` must be a relative path to the filter list file, i.e. `/list.patch` or `../list.patch` or similar. +`Diff-Path` also encodes additional information in the file name: + +```adblock +${patchName}[-${resolution}]-${epochTimestampMin}-${expirationPeriodMin}.patch#${resourceName} +``` + +* `patchName` - name of the patch file, an arbitrary string to identify the patch. +* `epochTimestamp` - epoch timestamp when the patch was generated (the unit of that timestamp depends on `resolution`, see below). +* `expirationPeriod` - expiration time for the diff update (the unit depends on `resolution`, see below). +* `resolution` - is an optional field, that specifies the resolution for both `expirationPeriod` and `epochTimestamp`. It can be either `h` (hours), `m` (minutes) or `s` (seconds). If `resolution` is not specified, it is assumed to be `h`. +* `resourceName` - a name of the resource that is being patched. This is used to support batch updates, see [Batch Updates](#batch-updates) section for more details. + +The following limitations are imposed on the `Diff-Path`: + +* `Diff-Path` MUST be a relative path to the filter list file, i.e. `/list-28332968-60.patch` or `../list-28332968-60.patch` or similar. * `Diff-Path` is a mandatory field for enabling the differential updates mechanism. +* `Diff-Path` MUST point to a file name with the name format conforming to the format described above. If the file name is different, the field is considered invalid and the differential update mechanism is disabled for the filter list. +* `patchName` MUST be a string of length 1-64 with no spaces or other special characters. Validation regex: `[a-zA-Z0-9_.]{1,64}`. +* `expirationPeriodMin` MUST be a positive integer, greater than 0, lesser than 43200 (30 days). +* `epochTimestampMin` MUST be a valid epoch timestamp in minutes. +* `resourceName` is an optional part, it's explained in the [Resource name](#resource-name) section. + +#### Examples + +* `list1_v1.0.0-m-28334180-60.patch#list1` + * Patch name is `list1_v1.0.0`. + * Resolution is set to `m` (minutes). + * Timestamp is `28334180` minutes from epoch, i.e. `Wed, 15 Nov 2023 12:20:00 GMT`. + * Expiration period is `60` minutes. + * Resource name is set to `list1`. +* `list1_v1.0.0-472236-1.patch` + * Patch name is `list1_v1.0.0`. + * Resolution is not specified so it is assumed to be `h` (hours). + * Timestamp is `472236` hours from epoch, i.e. `Wed, 15 Nov 2023 12:00:00 GMT`. + * Expiration period is `1` hour. + * Resource name is not specified, i.e. the patch does not support batch updates. #### Resource name @@ -39,15 +73,9 @@ Later in the document it will be referred as "resource name". * The "hash" part of the URL must be a string of length 1-64. Validation regex: `[a-zA-Z0-9-_ ]{1,64}`. * When specified, `diff name` directive in the diff file MUST match the resource name, see [Diff Files Format](#diff-files-format) for more details. -#### `! Diff-Expires:` - -This is essentially a time-to-live (TTL) for the differential update. It dictates the frequency with which the ad blocker should attempt to fetch the differential update. For instance, if the `Diff-Expires` is set to `1 hour`, it means the ad blocker should attempt to download the differential update once every hour. - -`! Diff-Expires:` is an optional field. If it is not set, the ad blocker may fallback to `! Expires:` or to some pre-defined default value. It is recommended to have it specified to avoid inconsistency between different ad blockers. - #### `! Expires:` -`Expires` continues to work as it was working before, i.e. once in a while AdGuard will do the so-called **"full sync"**. When differential updates are available it is recommended to increase the value of `Expires` to a large value, e.g. `10 days`. This will ensure that the ad blocker will not do the full sync too often. +`Expires` continues to work as it was working before, i.e. once in a while the ad blocker will do the so-called **"full sync"**. When differential updates are available it is recommended to increase the value of `Expires` to a large value, e.g. `10 days`. This will ensure that the ad blocker will not do the full sync too often. #### Diff Files Format @@ -55,15 +83,16 @@ We propose using the [RCS format](https://www.gnu.org/software/diffutils/manual/ In order to support batch updates and be able to validate patch result, the standard format is extended with the `diff` directive: -`diff name:[name] checksum:[checksum] lines:[lines] timestamp:[timestamp]` +`diff name:[name] checksum:[checksum] lines:[lines]` * `name` - name of a corresponding filter list. It is only mandatory when [resource name](#resource-name) is specified in the list. * `checksum` - the expected SHA1 checksum of the file after the patch is applied. This is used to validate the patch. * `lines` - the number of lines that follow that make up the RCS diff block. Note, that `lines` are counted using the same algorithm as used by `wc -l`, i.e. it basically counts `\n`. -* `timestamp` - the timestamp of the patch. A number of milliseconds since the Unix epoch. `diff` directive is optional. If it is not specified, the patch is applied without validation. +Note, that it is possible to extend the `diff` directive with additional fields not specified in the spec. The implementation should be able to ignore unknown fields. + > It is recommended to use the `diff checksum:` directive to validate the patching result. This will ensure that the patch is applied correctly and the resulting file is not corrupted. ### Algorithm @@ -72,23 +101,29 @@ In order to support batch updates and be able to validate patch result, the stan 1. Refer to the `Diff-Path` to see if a differential update is available. * If there are several lists with the same `Diff-Path`, download the diff file only once. Refer to the [Batch Updates](#batch-updates) section for the details on how batch patches are applied. + * Calculate the patch expiration date: `${epochTimestampMin} + ${expirationPeriodMin}`. If the expiration date in the past, the patch is considered expired and the ad blocker SHOULD attempt to download the update. 1. If the differential update is available, download and apply it to the current filter list. - * Once the differential update is applied, the `Diff-Path` within the list will be updated to point to the next differential update. - * At this point the ad blocker may decide either to wait for the `Diff-Expires` period and then try again or to immediately try to fetch the next differential update. + * If the differential update is not empty and applied, the `Diff-Path` within the list MUST be updated to point to the next differential update. + * At this point the ad blocker may decide to wait for a while before checking for the next differential update (see [2. Set Update Timer](#2-set-update-timer)) to ensure that the server is not overloaded. 1. If the differential update is not available the server may signal about that by returning one of the following responses: * `404 Not Found` * `200 OK` with empty content (content length is 0) * `204 No Content` - In this case the ad blocker SHOULD wait for the `Diff-Expires` period and then try again. + In this case the ad blocker SHOULD wait for a while and then try again, see [2. Set Update Timer](#2-set-update-timer). #### 2. Set Update Timer -Using the `Diff-Expires` value, set a timer for the next update check. For example, if it's set to `1 hour`, the ad blocker will wait for that duration before checking the `Diff-Path` again. +The update timer depends on the previous update check result. + +1. If the differential update was not empty and applied successfully, the ad blocker SHOULD check the new `Diff-Path` file expiration time. + * If the expiration time is in the future, the ad blocker SHOULD wait until that time and then check for the update again. + * If the expiration time is in the past, the ad blocker SHOULD try to download the new patch and apply it. +1. If the differential update was empty and the list's `Diff-Path` stayed the same, the ad blocker SHOULD delay the next update for at least 30 minutes to avoid overloading the server. #### 3. Fallback Mechanism -Any unexpected error during the update process should be treated as a fatal error and the ad blocker should wait until it is time for the full sync. Note, that it should respect the `Expires` value set by the filter list. +Any unexpected error during the update process SHOULD be treated as a fatal error and the ad blocker should wait until it is time for the full sync. Note, that it should respect the `Expires` value set by the filter list. ### Batch Updates @@ -108,11 +143,12 @@ Let's take an example: ```adblock ! Title: List 1 - ! Diff-Path: ../patches/batch.patch#list1 + ! Diff-Path: ../patches/batch-28334120-60.patch#list1 ``` * `Diff-Path` is relative to `list1.txt` location so the final URL of the diff file will be `https://example.com/patches/batch.patch`. * [Resource name](#resource-name) is set to `list1` here. It is mandatory for lists that support batch differential updates. + * Patch expiration period is set to `60` minutes, creation time is set to `28334120` (Unix timestamp in minutes). * List 2 @@ -120,22 +156,23 @@ Let's take an example: ```adblock ! Title: List 2 - ! Diff-Path: ../patches/batch.patch#list2 + ! Diff-Path: ../patches/batch-28334120-60.patch#list2 ``` * `Diff-Path` is relative to `list2.txt` location so the final URL of the diff file will be `https://example.com/patches/batch.patch`. * [Resource name](#resource-name) is set to `list2` here. It is mandatory for lists that support batch differential updates. - -* `batch.patch` + * Patch expiration period is set to `60` minutes, creation time is set to `28334120` (Unix timestamp in minutes). + +* `batch-28334120-60.patch` A file that contains patches for both `list1.txt` and `list2.txt`. It uses the `diff name:` directive to point at which patch should be applied to which list. ```diff - diff name:list1 checksum:e3c9c883378dc2a3aec9f71578c849891243bc2c lines:3 timestamp:1699599870000 + diff name:list1 checksum:e3c9c883378dc2a3aec9f71578c849891243bc2c lines:3 d2 1 a2 1 ! Diff-Path: patches/batch_new.patch#list1 - diff name:list2 checksum:be09384422b8d7f20da517d1245360125868f0b9 lines:3 timestamp:1699599870000 + diff name:list2 checksum:be09384422b8d7f20da517d1245360125868f0b9 lines:3 d2 1 a2 1 ! Diff-Path: patches/batch_new.patch#list2 diff --git a/examples/01_simple/README.md b/examples/01_simple/README.md index e262ab6..cc51580 100644 --- a/examples/01_simple/README.md +++ b/examples/01_simple/README.md @@ -3,9 +3,11 @@ A filter list with differential updates support. * [filter_v1.0.0.txt](./filter_v1.0.0.txt) - the oldest version of the list. -* [patches/v1.0.0.patch](./patches/v1.0.0.patch) - a patch that provides differential update from `v1.0.0` to `v1.0.1`. +* [patches/v1.0.0-472234-1.patch](./patches/v1.0.0-472234-1.patch) - a patch that provides differential update from `v1.0.0` to `v1.0.1`. + Expiration period is set to `60` minutes, creation time is set to `472234` (Unix timestamp in hours, i.e. `Wed, 15 Nov 2023 10:00:00 GMT`). * [filter_v1.0.1.txt](./filter_v1.0.1.txt) - the next version of the list. -* [patches/v1.0.1.patch](./patches/v1.0.0.patch) - a patch that provides differential update from `v1.0.1` to `v1.0.2`. +* [patches/v1.0.1-472235-1.patch](./patches/v1.0.0-472234-1.patch) - a patch that provides differential update from `v1.0.1` to `v1.0.2`. + Expiration period is set to `60` minutes, creation time is set to `472235` (Unix timestamp in minutes, i.e. `Wed, 15 Nov 2023 11:00:00 GMT`). * [filter.txt](./filter.txt) - the final version of the list. After all differential updates are applied you should get this version. ## How patch files were prepared @@ -13,6 +15,6 @@ A filter list with differential updates support. The patches are created using the `diff` utility: ```shell -diff -n filter_v1.0.0.txt filter_v1.0.1.txt > patches/v1.0.0.patch -diff -n filter_v1.0.1.txt filter.txt > patches/v1.0.1.patch +diff -n filter_v1.0.0.txt filter_v1.0.1.txt > patches/v1.0.0-472234-1.patch +diff -n filter_v1.0.1.txt filter.txt > patches/v1.0.1-472235-1.patch ``` diff --git a/examples/01_simple/filter.txt b/examples/01_simple/filter.txt index 58165d5..f1bb9dd 100644 --- a/examples/01_simple/filter.txt +++ b/examples/01_simple/filter.txt @@ -1,4 +1,4 @@ ! Title: Diff Updates Simple Example List ! Version: v1.0.2 -! Diff-Path: patches/v1.0.2.patch +! Diff-Path: patches/v1.0.2-472236-1.patch ||example.net^ diff --git a/examples/01_simple/filter_v1.0.0.txt b/examples/01_simple/filter_v1.0.0.txt index 538db2a..f908cca 100644 --- a/examples/01_simple/filter_v1.0.0.txt +++ b/examples/01_simple/filter_v1.0.0.txt @@ -1,4 +1,4 @@ ! Title: Diff Updates Simple Example List ! Version: v1.0.0 -! Diff-Path: patches/v1.0.0.patch +! Diff-Path: patches/v1.0.0-472234-1.patch ||example.org^ \ No newline at end of file diff --git a/examples/01_simple/filter_v1.0.1.txt b/examples/01_simple/filter_v1.0.1.txt index 3e134de..d279705 100644 --- a/examples/01_simple/filter_v1.0.1.txt +++ b/examples/01_simple/filter_v1.0.1.txt @@ -1,4 +1,4 @@ ! Title: Diff Updates Simple Example List ! Version: v1.0.1 -! Diff-Path: patches/v1.0.1.patch +! Diff-Path: patches/v1.0.1-472235-1.patch ||example.com^ \ No newline at end of file diff --git a/examples/01_simple/patches/v1.0.0-472234-1.patch b/examples/01_simple/patches/v1.0.0-472234-1.patch new file mode 100644 index 0000000..1ab9420 --- /dev/null +++ b/examples/01_simple/patches/v1.0.0-472234-1.patch @@ -0,0 +1,5 @@ +d2 3 +a4 3 +! Version: v1.0.1 +! Diff-Path: patches/v1.0.1-472235-1.patch +||example.com^ \ No newline at end of file diff --git a/examples/01_simple/patches/v1.0.0.patch b/examples/01_simple/patches/v1.0.0.patch deleted file mode 100644 index 88521b4..0000000 --- a/examples/01_simple/patches/v1.0.0.patch +++ /dev/null @@ -1,5 +0,0 @@ -d2 3 -a4 3 -! Version: v1.0.1 -! Diff-Path: patches/v1.0.1.patch -||example.com^ \ No newline at end of file diff --git a/examples/01_simple/patches/v1.0.1.patch b/examples/01_simple/patches/v1.0.1-472235-1.patch similarity index 50% rename from examples/01_simple/patches/v1.0.1.patch rename to examples/01_simple/patches/v1.0.1-472235-1.patch index 594ccd9..13050e1 100644 --- a/examples/01_simple/patches/v1.0.1.patch +++ b/examples/01_simple/patches/v1.0.1-472235-1.patch @@ -1,5 +1,5 @@ d2 3 a4 3 ! Version: v1.0.2 -! Diff-Path: patches/v1.0.2.patch +! Diff-Path: patches/v1.0.2-472236-1.patch ||example.net^ diff --git a/examples/02_validation/README.md b/examples/02_validation/README.md index 32ac6a2..5ecd4f5 100644 --- a/examples/02_validation/README.md +++ b/examples/02_validation/README.md @@ -3,10 +3,13 @@ Example of a filter list with differential updates support where the patch files also contain the `diff` directive that can be used to validate the patching result. * [filter_v1.0.0.txt](./filter_v1.0.0.txt) - the oldest version of the list. -* [patches/v1.0.0.patch](./patches/v1.0.0.patch) - a patch that provides differential update from `v1.0.0` to `v1.0.1`. +* [patches/v1.0.0-m-28334060-60.patch](./patches/v1.0.0-m-28334060-60.patch) - a patch that provides differential update from `v1.0.0` to `v1.0.1`. * [filter_v1.0.1.txt](./filter_v1.0.1.txt) - the next version of the list. -* [patches/v1.0.1.patch](./patches/v1.0.0.patch) - a patch that provides differential update from `v1.0.1` to `v1.0.2`. +* [patches/v1.0.1-m-28334120-60.patch](./patches/v1.0.1-m-28334120-60.patch) - a patch that provides differential update from `v1.0.1` to `v1.0.2`. * [filter.txt](./filter.txt) - the final version of the list. After all differential updates are applied you should get this version. +* [patches/v1.0.2-m-28334180-60.patch](./patches/v1.0.2-m-28334180-60.patch) - empty patch that signals that there's no patch available for the next update yet. + +Note, that resolution is specified in the patch names (`m` for minutes). ## How patch files were prepared @@ -14,26 +17,28 @@ The patches are created using the `diff` utility: ```shell # Calculating the RFC diff for filter_v1.0.1.txt. -diff -n filter_v1.0.0.txt filter_v1.0.1.txt > patches/v1.0.0.patch +diff -n filter_v1.0.0.txt filter_v1.0.1.txt > patches/v1.0.0-m-28334060-60.patch # Calc the SHA1 sum of filter_v1.0.1.txt and append it to the patch file. FILENAME="filter_v1.0.1.txt" && \ -PATCHFILE="patches/v1.0.0.patch" && \ -TIMESTAMP="$(date +%s)000" && \ +PATCHFILE="patches/v1.0.0-m-28334060-60.patch" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff checksum:$SHASUM lines:$NUMLINES timestamp:$TIMESTAMP" | cat - $PATCHFILE > temp.patch && \ + echo "diff checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Calculating the RFC diff for filter.txt. -diff -n filter_v1.0.1.txt filter.txt > patches/v1.0.1.patch +diff -n filter_v1.0.1.txt filter.txt > patches/v1.0.1-m-28334120-60.patch # Calc the SHA1 sum of filter.txt and append it to the patch file. FILENAME="filter.txt" && \ -PATCHFILE="patches/v1.0.1.patch" && \ -TIMESTAMP="$(date +%s)000" && \ +PATCHFILE="patches/v1.0.1-m-28334120-60.patch" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l $PATCHFILE | awk '{print $1}') && \ - echo "diff checksum:$SHASUM lines:$NUMLINES timestamp:$TIMESTAMP" | cat - $PATCHFILE > temp.patch && \ + echo "diff checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE + +# Make an empty patch file to signal that there's no patch available for the +# next update yet. +touch patches/v1.0.2-m-28334180-60.patch ``` diff --git a/examples/02_validation/filter.txt b/examples/02_validation/filter.txt index 58165d5..ca15afe 100644 --- a/examples/02_validation/filter.txt +++ b/examples/02_validation/filter.txt @@ -1,4 +1,4 @@ ! Title: Diff Updates Simple Example List ! Version: v1.0.2 -! Diff-Path: patches/v1.0.2.patch +! Diff-Path: patches/v1.0.2-m-28334180-60.patch ||example.net^ diff --git a/examples/02_validation/filter_v1.0.0.txt b/examples/02_validation/filter_v1.0.0.txt index 538db2a..3715791 100644 --- a/examples/02_validation/filter_v1.0.0.txt +++ b/examples/02_validation/filter_v1.0.0.txt @@ -1,4 +1,4 @@ ! Title: Diff Updates Simple Example List ! Version: v1.0.0 -! Diff-Path: patches/v1.0.0.patch +! Diff-Path: patches/v1.0.0-m-28334060-60.patch ||example.org^ \ No newline at end of file diff --git a/examples/02_validation/filter_v1.0.1.txt b/examples/02_validation/filter_v1.0.1.txt index 3e134de..d0bc497 100644 --- a/examples/02_validation/filter_v1.0.1.txt +++ b/examples/02_validation/filter_v1.0.1.txt @@ -1,4 +1,4 @@ ! Title: Diff Updates Simple Example List ! Version: v1.0.1 -! Diff-Path: patches/v1.0.1.patch +! Diff-Path: patches/v1.0.1-m-28334120-60.patch ||example.com^ \ No newline at end of file diff --git a/examples/02_validation/patches/v1.0.0-m-28334060-60.patch b/examples/02_validation/patches/v1.0.0-m-28334060-60.patch new file mode 100644 index 0000000..e382f49 --- /dev/null +++ b/examples/02_validation/patches/v1.0.0-m-28334060-60.patch @@ -0,0 +1,6 @@ +diff checksum:1ce52b527d56a245f32138e014b1571c19cfb659 lines:4 +d2 3 +a4 3 +! Version: v1.0.1 +! Diff-Path: patches/v1.0.1-m-28334120-60.patch +||example.com^ \ No newline at end of file diff --git a/examples/02_validation/patches/v1.0.0.patch b/examples/02_validation/patches/v1.0.0.patch deleted file mode 100644 index 70b8396..0000000 --- a/examples/02_validation/patches/v1.0.0.patch +++ /dev/null @@ -1,6 +0,0 @@ -diff checksum:be09384422b8d7f20da517d1245360125868f0b9 lines:4 timestamp:1699600190000 -d2 3 -a4 3 -! Version: v1.0.1 -! Diff-Path: patches/v1.0.1.patch -||example.com^ \ No newline at end of file diff --git a/examples/02_validation/patches/v1.0.1-m-28334120-60.patch b/examples/02_validation/patches/v1.0.1-m-28334120-60.patch new file mode 100644 index 0000000..3065561 --- /dev/null +++ b/examples/02_validation/patches/v1.0.1-m-28334120-60.patch @@ -0,0 +1,6 @@ +diff checksum:bc43fd3b69b5ad82fdc1524a1a419029a2dd4eae lines:5 +d2 3 +a4 3 +! Version: v1.0.2 +! Diff-Path: patches/v1.0.2-m-28334180-60.patch +||example.net^ diff --git a/examples/02_validation/patches/v1.0.1.patch b/examples/02_validation/patches/v1.0.1.patch deleted file mode 100644 index fc56305..0000000 --- a/examples/02_validation/patches/v1.0.1.patch +++ /dev/null @@ -1,6 +0,0 @@ -diff checksum:e3c9c883378dc2a3aec9f71578c849891243bc2c lines:5 timestamp:1699600198000 -d2 3 -a4 3 -! Version: v1.0.2 -! Diff-Path: patches/v1.0.2.patch -||example.net^ diff --git a/examples/02_validation/patches/v1.0.2-m-28334180-60.patch b/examples/02_validation/patches/v1.0.2-m-28334180-60.patch new file mode 100644 index 0000000..e69de29 diff --git a/examples/03_batch/README.md b/examples/03_batch/README.md index 6f93de9..0c8bd9c 100644 --- a/examples/03_batch/README.md +++ b/examples/03_batch/README.md @@ -13,60 +13,65 @@ First, for lists `v1.0.0` -> `v1.0.1`. ```shell # Calculating the RFC diff for list1_v1.0.1.txt. -diff -n list1/list1_v1.0.0.txt list1/list1_v1.0.1.txt > patches/list1_v1.0.0.patch +diff -n list1/list1_v1.0.0.txt list1/list1_v1.0.1.txt > patches/list1_v1.0.0-s-1700045842-3600.patch # Calc the SHA1 sum of list1_v1.0.1.txt and prepend it to the patch file. FILENAME="list1/list1_v1.0.1.txt" && \ -PATCHFILE="patches/list1_v1.0.0.patch" && \ -TIMESTAMP="$(date +%s)000" && \ +PATCHFILE="patches/list1_v1.0.0-s-1700045842-3600.patch" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff name:list1 checksum:$SHASUM lines:$NUMLINES timestamp:${TIMESTAMP}" | cat - $PATCHFILE > temp.patch && \ + echo "diff name:list1 checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Calculating the RFC diff for list2_v1.0.1.txt. -diff -n list2/list2_v1.0.0.txt list2/list2_v1.0.1.txt > patches/list2_v1.0.0.patch +diff -n list2/list2_v1.0.0.txt list2/list2_v1.0.1.txt > patches/list2_v1.0.0-s-1700045842-3600.patch # Calc the SHA1 sum of list1_v1.0.1.txt and prepend it to the patch file. FILENAME="list2/list2_v1.0.1.txt" && \ -PATCHFILE="patches/list2_v1.0.0.patch" && \ -TIMESTAMP="$(date +%s)000" && \ +PATCHFILE="patches/list2_v1.0.0-s-1700045842-3600.patch" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff name:list2 checksum:$SHASUM lines:$NUMLINES timestamp:${TIMESTAMP}" | cat - $PATCHFILE > temp.patch && \ + echo "diff name:list2 checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Concatenate files into a batch diff file. -cat patches/list1_v1.0.0.patch patches/list2_v1.0.0.patch > patches/batch_v1.0.0.patch +cat patches/list1_v1.0.0-s-1700045842-3600.patch patches/list2_v1.0.0-s-1700045842-3600.patch > patches/batch_v1.0.0-s-1700045842-3600.patch ``` Second, for lists `v1.0.1` -> `v1.0.2`. ```shell # Calculating the RFC diff for list1.txt. -diff -n list1/list1_v1.0.1.txt list1/list1.txt > patches/list1_v1.0.1.patch +diff -n list1/list1_v1.0.1.txt list1/list1.txt > patches/list1_v1.0.1-s-1700049442-3600.patch # Calc the SHA1 sum of list1_v1.0.1.txt and prepend it to the patch file. FILENAME="list1/list1.txt" && \ -PATCHFILE="patches/list1_v1.0.1.patch" && \ -TIMESTAMP="$(date +%s)000" && \ +PATCHFILE="patches/list1_v1.0.1-s-1700049442-3600.patch" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff name:list1 checksum:$SHASUM lines:$NUMLINES timestamp:${TIMESTAMP}" | cat - $PATCHFILE > temp.patch && \ + echo "diff name:list1 checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Calculating the RFC diff for list2_v1.0.1.txt. -diff -n list2/list2_v1.0.1.txt list2/list2.txt > patches/list2_v1.0.1.patch +diff -n list2/list2_v1.0.1.txt list2/list2.txt > patches/list2_v1.0.1-s-1700049442-3600.patch # Calc the SHA1 sum of list1_v1.0.1.txt and prepend it to the patch file. FILENAME="list2/list2.txt" && \ -PATCHFILE="patches/list2_v1.0.1.patch" && \ -TIMESTAMP="$(date +%s)000" && \ +PATCHFILE="patches/list2_v1.0.1-s-1700049442-3600.patch" && \ SHASUM=$(shasum -a 1 $FILENAME | awk '{print $1}') && \ NUMLINES=$(wc -l < $PATCHFILE | awk '{print $1}') && \ - echo "diff name:list2 checksum:$SHASUM lines:$NUMLINES timestamp:${TIMESTAMP}" | cat - $PATCHFILE > temp.patch && \ + echo "diff name:list2 checksum:$SHASUM lines:$NUMLINES" | cat - $PATCHFILE > temp.patch && \ mv temp.patch $PATCHFILE # Concatenate files into a batch diff file. -cat patches/list1_v1.0.1.patch patches/list2_v1.0.1.patch > patches/batch_v1.0.1.patch +cat patches/list1_v1.0.1-s-1700049442-3600.patch patches/list2_v1.0.1-s-1700049442-3600.patch > patches/batch_v1.0.1-s-1700049442-3600.patch +``` + +Last step, create an empty patch file to signal that there's no patch available for the +next update yet. + +```shell +# Make an empty patch file to signal that there's no patch available for the +# next update yet. +touch patches/batch_v1.0.2-s-1700053042-3600.patch ``` diff --git a/examples/03_batch/list1/list1.txt b/examples/03_batch/list1/list1.txt index eee6f75..32ef9fd 100644 --- a/examples/03_batch/list1/list1.txt +++ b/examples/03_batch/list1/list1.txt @@ -1,3 +1,3 @@ ! Title: Batch-Updatable List 1 -! Diff-Path: ../patches/batch_v1.0.2.patch#list1 +! Diff-Path: ../patches/batch_v1.0.2-s-1700053042-3600.patch#list1 ||example.net^ diff --git a/examples/03_batch/list1/list1_v1.0.0.txt b/examples/03_batch/list1/list1_v1.0.0.txt index 04d9d50..b5d9895 100644 --- a/examples/03_batch/list1/list1_v1.0.0.txt +++ b/examples/03_batch/list1/list1_v1.0.0.txt @@ -1,3 +1,3 @@ ! Title: Batch-Updatable List 1 -! Diff-Path: ../patches/batch_v1.0.0.patch#list1 +! Diff-Path: ../patches/batch_v1.0.0-s-1700045842-3600.patch#list1 ||example.org^ diff --git a/examples/03_batch/list1/list1_v1.0.1.txt b/examples/03_batch/list1/list1_v1.0.1.txt index 1cff435..6b00005 100644 --- a/examples/03_batch/list1/list1_v1.0.1.txt +++ b/examples/03_batch/list1/list1_v1.0.1.txt @@ -1,3 +1,3 @@ ! Title: Batch-Updatable List 1 -! Diff-Path: ../patches/batch_v1.0.1.patch#list1 +! Diff-Path: ../patches/batch_v1.0.1-s-1700049442-3600.patch#list1 ||example.com^ diff --git a/examples/03_batch/list2/list2.txt b/examples/03_batch/list2/list2.txt index 5dd736f..efcf006 100644 --- a/examples/03_batch/list2/list2.txt +++ b/examples/03_batch/list2/list2.txt @@ -1,3 +1,3 @@ ! Title: Batch-Updatable List 2 -! Diff-Path: ../patches/batch_v1.0.2.patch#list2 +! Diff-Path: ../patches/batch_v1.0.2-s-1700053042-3600.patch#list2 ||test.net^ diff --git a/examples/03_batch/list2/list2_v1.0.0.txt b/examples/03_batch/list2/list2_v1.0.0.txt index 50ecd9d..14c4b46 100644 --- a/examples/03_batch/list2/list2_v1.0.0.txt +++ b/examples/03_batch/list2/list2_v1.0.0.txt @@ -1,3 +1,3 @@ ! Title: Batch-Updatable List 2 -! Diff-Path: ../patches/batch_v1.0.0.patch#list2 +! Diff-Path: ../patches/batch_v1.0.0-s-1700045842-3600.patch#list2 ||test.org^ diff --git a/examples/03_batch/list2/list2_v1.0.1.txt b/examples/03_batch/list2/list2_v1.0.1.txt index 77cc67f..1aaf08a 100644 --- a/examples/03_batch/list2/list2_v1.0.1.txt +++ b/examples/03_batch/list2/list2_v1.0.1.txt @@ -1,3 +1,3 @@ ! Title: Batch-Updatable List 2 -! Diff-Path: ../patches/batch_v1.0.1.patch#list2 +! Diff-Path: ../patches/batch_v1.0.1-s-1700049442-3600.patch#list2 ||test.com^ diff --git a/examples/03_batch/patches/batch_v1.0.0-s-1700045842-3600.patch b/examples/03_batch/patches/batch_v1.0.0-s-1700045842-3600.patch new file mode 100644 index 0000000..f41ed0a --- /dev/null +++ b/examples/03_batch/patches/batch_v1.0.0-s-1700045842-3600.patch @@ -0,0 +1,10 @@ +diff name:list1 checksum:f0ecb30059277cbae9736e2bf4fcdfa4a7cac751 lines:4 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.1-s-1700049442-3600.patch#list1 +||example.com^ +diff name:list2 checksum:9db9474484edf99f9112d3654a00d1a0d20e92eb lines:4 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.1-s-1700049442-3600.patch#list2 +||test.com^ diff --git a/examples/03_batch/patches/batch_v1.0.0.patch b/examples/03_batch/patches/batch_v1.0.0.patch deleted file mode 100644 index fe1e222..0000000 --- a/examples/03_batch/patches/batch_v1.0.0.patch +++ /dev/null @@ -1,10 +0,0 @@ -diff name:list1 checksum:5b8abcb6763972f65970a036ee3551818c43499d lines:4 timestamp:1699599862000 -d2 2 -a3 2 -! Diff-Path: ../patches/batch_v1.0.1.patch#list1 -||example.com^ -diff name:list2 checksum:a089ee91e339e4a5374dfbed8728e17e768d7c9b lines:4 timestamp:1699599870000 -d2 2 -a3 2 -! Diff-Path: ../patches/batch_v1.0.1.patch#list2 -||test.com^ diff --git a/examples/03_batch/patches/batch_v1.0.1-s-1700049442-3600.patch b/examples/03_batch/patches/batch_v1.0.1-s-1700049442-3600.patch new file mode 100644 index 0000000..135e146 --- /dev/null +++ b/examples/03_batch/patches/batch_v1.0.1-s-1700049442-3600.patch @@ -0,0 +1,10 @@ +diff name:list1 checksum:b8ea7b480f0423706a21a66cc2b203f495407049 lines:4 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.2-s-1700053042-3600.patch#list1 +||example.net^ +diff name:list2 checksum:2160f8ccf21038667143c512e7c4491a83fba07a lines:4 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.2-s-1700053042-3600.patch#list2 +||test.net^ diff --git a/examples/03_batch/patches/batch_v1.0.1.patch b/examples/03_batch/patches/batch_v1.0.1.patch deleted file mode 100644 index 684f4d4..0000000 --- a/examples/03_batch/patches/batch_v1.0.1.patch +++ /dev/null @@ -1,10 +0,0 @@ -diff name:list1 checksum:11edb61b350b1c15b49c14ce7a6eba7ff1e058a8 lines:4 timestamp:1699599881000 -d2 2 -a3 2 -! Diff-Path: ../patches/batch_v1.0.2.patch#list1 -||example.net^ -diff name:list2 checksum:e4b57a4da7fbe9072dada875695a16762b05eab2 lines:4 timestamp:1699599890000 -d2 2 -a3 2 -! Diff-Path: ../patches/batch_v1.0.2.patch#list2 -||test.net^ diff --git a/examples/03_batch/patches/batch_v1.0.2-s-1700053042-3600.patch b/examples/03_batch/patches/batch_v1.0.2-s-1700053042-3600.patch new file mode 100644 index 0000000..e69de29 diff --git a/examples/03_batch/patches/list1_v1.0.0-s-1700045842-3600.patch b/examples/03_batch/patches/list1_v1.0.0-s-1700045842-3600.patch new file mode 100644 index 0000000..978b827 --- /dev/null +++ b/examples/03_batch/patches/list1_v1.0.0-s-1700045842-3600.patch @@ -0,0 +1,5 @@ +diff name:list1 checksum:f0ecb30059277cbae9736e2bf4fcdfa4a7cac751 lines:4 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.1-s-1700049442-3600.patch#list1 +||example.com^ diff --git a/examples/03_batch/patches/list1_v1.0.0.patch b/examples/03_batch/patches/list1_v1.0.0.patch deleted file mode 100644 index d2d19cc..0000000 --- a/examples/03_batch/patches/list1_v1.0.0.patch +++ /dev/null @@ -1,5 +0,0 @@ -diff name:list1 checksum:5b8abcb6763972f65970a036ee3551818c43499d lines:4 timestamp:1699599862000 -d2 2 -a3 2 -! Diff-Path: ../patches/batch_v1.0.1.patch#list1 -||example.com^ diff --git a/examples/03_batch/patches/list1_v1.0.1-s-1700049442-3600.patch b/examples/03_batch/patches/list1_v1.0.1-s-1700049442-3600.patch new file mode 100644 index 0000000..d2cad7a --- /dev/null +++ b/examples/03_batch/patches/list1_v1.0.1-s-1700049442-3600.patch @@ -0,0 +1,5 @@ +diff name:list1 checksum:b8ea7b480f0423706a21a66cc2b203f495407049 lines:4 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.2-s-1700053042-3600.patch#list1 +||example.net^ diff --git a/examples/03_batch/patches/list1_v1.0.1.patch b/examples/03_batch/patches/list1_v1.0.1.patch deleted file mode 100644 index 022c14c..0000000 --- a/examples/03_batch/patches/list1_v1.0.1.patch +++ /dev/null @@ -1,5 +0,0 @@ -diff name:list1 checksum:11edb61b350b1c15b49c14ce7a6eba7ff1e058a8 lines:4 timestamp:1699599881000 -d2 2 -a3 2 -! Diff-Path: ../patches/batch_v1.0.2.patch#list1 -||example.net^ diff --git a/examples/03_batch/patches/list2_v1.0.0-s-1700045842-3600.patch b/examples/03_batch/patches/list2_v1.0.0-s-1700045842-3600.patch new file mode 100644 index 0000000..ae3f6ed --- /dev/null +++ b/examples/03_batch/patches/list2_v1.0.0-s-1700045842-3600.patch @@ -0,0 +1,5 @@ +diff name:list2 checksum:9db9474484edf99f9112d3654a00d1a0d20e92eb lines:4 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.1-s-1700049442-3600.patch#list2 +||test.com^ diff --git a/examples/03_batch/patches/list2_v1.0.0.patch b/examples/03_batch/patches/list2_v1.0.0.patch deleted file mode 100644 index 1b5af1d..0000000 --- a/examples/03_batch/patches/list2_v1.0.0.patch +++ /dev/null @@ -1,5 +0,0 @@ -diff name:list2 checksum:a089ee91e339e4a5374dfbed8728e17e768d7c9b lines:4 timestamp:1699599870000 -d2 2 -a3 2 -! Diff-Path: ../patches/batch_v1.0.1.patch#list2 -||test.com^ diff --git a/examples/03_batch/patches/list2_v1.0.1-s-1700049442-3600.patch b/examples/03_batch/patches/list2_v1.0.1-s-1700049442-3600.patch new file mode 100644 index 0000000..48263bb --- /dev/null +++ b/examples/03_batch/patches/list2_v1.0.1-s-1700049442-3600.patch @@ -0,0 +1,5 @@ +diff name:list2 checksum:2160f8ccf21038667143c512e7c4491a83fba07a lines:4 +d2 2 +a3 2 +! Diff-Path: ../patches/batch_v1.0.2-s-1700053042-3600.patch#list2 +||test.net^ diff --git a/examples/03_batch/patches/list2_v1.0.1.patch b/examples/03_batch/patches/list2_v1.0.1.patch deleted file mode 100644 index b87cdec..0000000 --- a/examples/03_batch/patches/list2_v1.0.1.patch +++ /dev/null @@ -1,5 +0,0 @@ -diff name:list2 checksum:e4b57a4da7fbe9072dada875695a16762b05eab2 lines:4 timestamp:1699599890000 -d2 2 -a3 2 -! Diff-Path: ../patches/batch_v1.0.2.patch#list2 -||test.net^ From 2dfe3ed82e0ded790efee50e8f1bfe717e05ed96 Mon Sep 17 00:00:00 2001 From: Andrey Meshkov Date: Wed, 15 Nov 2023 14:07:23 +0300 Subject: [PATCH 4/9] correct resource name regex --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c38a0c5..4a89c28 100644 --- a/README.md +++ b/README.md @@ -70,7 +70,7 @@ If a list supports batch updates, the `Diff-Path` MUST also have a "hash" part, Later in the document it will be referred as "resource name". * This part is only mandatory when the filter list supports batch differential updates. -* The "hash" part of the URL must be a string of length 1-64. Validation regex: `[a-zA-Z0-9-_ ]{1,64}`. +* The "hash" part of the URL must be a string of length 1-64. Validation regex: `[a-zA-Z0-9-_]{1,64}`. * When specified, `diff name` directive in the diff file MUST match the resource name, see [Diff Files Format](#diff-files-format) for more details. #### `! Expires:` From 99d72e19053dd0de4c51801b409b7396e1990a28 Mon Sep 17 00:00:00 2001 From: Andrey Meshkov Date: Wed, 15 Nov 2023 14:12:02 +0300 Subject: [PATCH 5/9] correct mistakes --- README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/README.md b/README.md index 4a89c28..60845a0 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ This field will provide the relative path where the differential file (diff) for `Diff-Path` also encodes additional information in the file name: ```adblock -${patchName}[-${resolution}]-${epochTimestampMin}-${expirationPeriodMin}.patch#${resourceName} +${patchName}[-${resolution}]-${epochTimestamp}-${expirationPeriod}.patch#${resourceName} ``` * `patchName` - name of the patch file, an arbitrary string to identify the patch. @@ -44,8 +44,8 @@ The following limitations are imposed on the `Diff-Path`: * `Diff-Path` is a mandatory field for enabling the differential updates mechanism. * `Diff-Path` MUST point to a file name with the name format conforming to the format described above. If the file name is different, the field is considered invalid and the differential update mechanism is disabled for the filter list. * `patchName` MUST be a string of length 1-64 with no spaces or other special characters. Validation regex: `[a-zA-Z0-9_.]{1,64}`. -* `expirationPeriodMin` MUST be a positive integer, greater than 0, lesser than 43200 (30 days). -* `epochTimestampMin` MUST be a valid epoch timestamp in minutes. +* `epochTimestamp` MUST be a valid epoch timestamp (considering the unit specified in the `resolution` field). +* `expirationPeriod` MUST be a positive integer. * `resourceName` is an optional part, it's explained in the [Resource name](#resource-name) section. #### Examples @@ -101,7 +101,7 @@ Note, that it is possible to extend the `diff` directive with additional fields 1. Refer to the `Diff-Path` to see if a differential update is available. * If there are several lists with the same `Diff-Path`, download the diff file only once. Refer to the [Batch Updates](#batch-updates) section for the details on how batch patches are applied. - * Calculate the patch expiration date: `${epochTimestampMin} + ${expirationPeriodMin}`. If the expiration date in the past, the patch is considered expired and the ad blocker SHOULD attempt to download the update. + * Calculate the patch expiration date: `(${epochTimestamp} + ${expirationPeriod}) * ${resolution}`. If the expiration date in the past, the patch is considered expired and the ad blocker SHOULD attempt to download the update. 1. If the differential update is available, download and apply it to the current filter list. * If the differential update is not empty and applied, the `Diff-Path` within the list MUST be updated to point to the next differential update. * At this point the ad blocker may decide to wait for a while before checking for the next differential update (see [2. Set Update Timer](#2-set-update-timer)) to ensure that the server is not overloaded. From 73decb6907f7c5e11497f95cc443fa1e45380350 Mon Sep 17 00:00:00 2001 From: Andrey Meshkov Date: Wed, 15 Nov 2023 14:15:08 +0300 Subject: [PATCH 6/9] upd examples --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 60845a0..75a4fb3 100644 --- a/README.md +++ b/README.md @@ -40,7 +40,7 @@ ${patchName}[-${resolution}]-${epochTimestamp}-${expirationPeriod}.patch#${resou The following limitations are imposed on the `Diff-Path`: -* `Diff-Path` MUST be a relative path to the filter list file, i.e. `/list-28332968-60.patch` or `../list-28332968-60.patch` or similar. +* `Diff-Path` MUST be a relative path to the filter list file, i.e. `/list-472234-1.patch` or `../list-472234-1.patch` or similar. * `Diff-Path` is a mandatory field for enabling the differential updates mechanism. * `Diff-Path` MUST point to a file name with the name format conforming to the format described above. If the file name is different, the field is considered invalid and the differential update mechanism is disabled for the filter list. * `patchName` MUST be a string of length 1-64 with no spaces or other special characters. Validation regex: `[a-zA-Z0-9_.]{1,64}`. From 8556f25602553318cd233d97cec0f72c2d7cb3a4 Mon Sep 17 00:00:00 2001 From: Andrey Meshkov Date: Thu, 23 Nov 2023 18:39:44 +0300 Subject: [PATCH 7/9] Update README.md Co-authored-by: Dmitry Seregin <105th@users.noreply.github.com> --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index 75a4fb3..acb0c89 100644 --- a/README.md +++ b/README.md @@ -29,7 +29,7 @@ This field will provide the relative path where the differential file (diff) for `Diff-Path` also encodes additional information in the file name: ```adblock -${patchName}[-${resolution}]-${epochTimestamp}-${expirationPeriod}.patch#${resourceName} +${patchName}[-${resolution}]-${epochTimestamp}-${expirationPeriod}.patch[#${resourceName}] ``` * `patchName` - name of the patch file, an arbitrary string to identify the patch. From 01d6cab419379e162b4e8d7afc2f461315a4ca7f Mon Sep 17 00:00:00 2001 From: Andrey Meshkov Date: Mon, 27 Nov 2023 20:04:52 +0300 Subject: [PATCH 8/9] Update examples/01_simple/README.md Co-authored-by: Dmitry Seregin <105th@users.noreply.github.com> --- examples/01_simple/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/01_simple/README.md b/examples/01_simple/README.md index cc51580..247da05 100644 --- a/examples/01_simple/README.md +++ b/examples/01_simple/README.md @@ -7,7 +7,7 @@ A filter list with differential updates support. Expiration period is set to `60` minutes, creation time is set to `472234` (Unix timestamp in hours, i.e. `Wed, 15 Nov 2023 10:00:00 GMT`). * [filter_v1.0.1.txt](./filter_v1.0.1.txt) - the next version of the list. * [patches/v1.0.1-472235-1.patch](./patches/v1.0.0-472234-1.patch) - a patch that provides differential update from `v1.0.1` to `v1.0.2`. - Expiration period is set to `60` minutes, creation time is set to `472235` (Unix timestamp in minutes, i.e. `Wed, 15 Nov 2023 11:00:00 GMT`). + Expiration period is set to `1` hour, creation time is set to `472235` (Unix timestamp in hours, i.e. `Wed, 15 Nov 2023 11:00:00 GMT`). * [filter.txt](./filter.txt) - the final version of the list. After all differential updates are applied you should get this version. ## How patch files were prepared From d7caba628e34b5c577b830a9c0c71187cf3e40cd Mon Sep 17 00:00:00 2001 From: Andrey Meshkov Date: Mon, 27 Nov 2023 20:05:03 +0300 Subject: [PATCH 9/9] Update examples/01_simple/README.md Co-authored-by: Dmitry Seregin <105th@users.noreply.github.com> --- examples/01_simple/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/01_simple/README.md b/examples/01_simple/README.md index 247da05..6af2995 100644 --- a/examples/01_simple/README.md +++ b/examples/01_simple/README.md @@ -4,7 +4,7 @@ A filter list with differential updates support. * [filter_v1.0.0.txt](./filter_v1.0.0.txt) - the oldest version of the list. * [patches/v1.0.0-472234-1.patch](./patches/v1.0.0-472234-1.patch) - a patch that provides differential update from `v1.0.0` to `v1.0.1`. - Expiration period is set to `60` minutes, creation time is set to `472234` (Unix timestamp in hours, i.e. `Wed, 15 Nov 2023 10:00:00 GMT`). + Expiration period is set to `1` hour, creation time is set to `472234` (Unix timestamp in hours, i.e. `Wed, 15 Nov 2023 10:00:00 GMT`). * [filter_v1.0.1.txt](./filter_v1.0.1.txt) - the next version of the list. * [patches/v1.0.1-472235-1.patch](./patches/v1.0.0-472234-1.patch) - a patch that provides differential update from `v1.0.1` to `v1.0.2`. Expiration period is set to `1` hour, creation time is set to `472235` (Unix timestamp in hours, i.e. `Wed, 15 Nov 2023 11:00:00 GMT`).