From 7fbd5f562beb39320af1b3b435ed7463bf92334e Mon Sep 17 00:00:00 2001 From: predictiple Date: Mon, 25 Nov 2024 23:34:10 +0200 Subject: [PATCH] Improvements to VQL reference documentation and minor fixes --- content/vql_reference/_index.md | 2 +- content/vql_reference/basic/_index.md | 2 +- content/vql_reference/basic/format/_index.md | 4 +-- content/vql_reference/basic/log/_index.md | 34 ++++++++++++++++---- content/vql_reference/basic/now/_index.md | 2 +- content/vql_reference/misc/alert/_index.md | 7 ++++ content/vql_reference/misc/link_to/_index.md | 2 +- content/vql_reference/misc/typeof/_index.md | 8 ++--- static/reference/data.json | 2 +- 9 files changed, 46 insertions(+), 17 deletions(-) diff --git a/content/vql_reference/_index.md b/content/vql_reference/_index.md index cb672c2ef74..7b427fe5436 100644 --- a/content/vql_reference/_index.md +++ b/content/vql_reference/_index.md @@ -12,6 +12,6 @@ pre: head:
--- -Search for documentation on VQL plugins or functions. +Search for documentation on VQL plugins, functions and accessors. {{% reference %}} diff --git a/content/vql_reference/basic/_index.md b/content/vql_reference/basic/_index.md index 2b8a169379e..e502c6372c9 100644 --- a/content/vql_reference/basic/_index.md +++ b/content/vql_reference/basic/_index.md @@ -50,7 +50,7 @@ or in condition clauses (i.e. after the `WHERE` keyword). |[join](join)|Function|Join all the args on a separator| |[killkillkill](killkillkill)|Function|Kills the client and forces a restart - this is very aggressive!| |[len](len)|Function|Returns the length of an object| -|[log](log)|Function|Log the message and return TRUE| +|[log](log)|Function|Log a message to the query log stream| |[lowcase](lowcase)|Function|Returns the lowercase version of a string| |[max](max)|Function|Finds the largest item in the aggregate| |[memoize](memoize)|Function|Memoize a query into memory| diff --git a/content/vql_reference/basic/format/_index.md b/content/vql_reference/basic/format/_index.md index 6b7d0b559b0..4a454dd8760 100644 --- a/content/vql_reference/basic/format/_index.md +++ b/content/vql_reference/basic/format/_index.md @@ -43,7 +43,7 @@ of `args=[my_var]`. ### Examples -```sql +```vql LET csv <= '''John,ate,banana Mary,had,little lamb''' LET my_words <= SELECT * FROM parse_csv(accessor="data", filename=csv, auto_headers=True) @@ -53,7 +53,7 @@ returns: `Sentence: John ate a banana.` `Sentence: Mary had a little lamb.` -```sql +```vql LET T <= timestamp(epoch="2024-02-02T04:42:00Z") SELECT format(format="%d-%02d-%02dT%02d:%02d:%06.3fZ", args=[ T.Year, T.Month, T.Day, T.Hour, T.Minute, T.Nanosecond / 1000000000 ]) diff --git a/content/vql_reference/basic/log/_index.md b/content/vql_reference/basic/log/_index.md index bf7653e790a..512f09694f3 100644 --- a/content/vql_reference/basic/log/_index.md +++ b/content/vql_reference/basic/log/_index.md @@ -26,15 +26,14 @@ level|Level to log at (DEFAULT, WARN, ERROR, INFO, DEBUG).|string ### Description -Log the message and return TRUE. +Log a message to the query log stream. Always returns TRUE. -The message will be logged into the query log stream (Viewable in -the Logs tab). The `message` parameter represents a format string +The `message` parameter represents a format string that will be expanded using the `args` parameter list if needed. Since `log()` always returns TRUE it is easy to use in a WHERE -clause as a form of debugging (It is basically equivalent to the -print statement of other languages). +clause as a form of debugging. It is basically equivalent to the +print statement of other languages. ```vql SELECT * FROM glob(...) @@ -44,7 +43,7 @@ WHERE log(message="Value of OSPath is %v", args=OSPath) ### Deduplication Log messages will be deduped according to the `dedup` -parameter - each distinct format string will not be emitted more +parameter - each **distinct format string** will not be emitted more frequently than the `dedup` parameter (by default 60 seconds). This makes it safe to use `log()` frequently without overflowing @@ -66,4 +65,27 @@ SELECT * FROM glob(...) WHERE log(message="Processing file %v", args=OSPath) ``` +### Example + +In this more complex example the query will produce 10 rows, at a rate of +1 row every 5 seconds. However the log messages will be limited to 1 every +15 seconds. + +```vql +SELECT count() AS Count, String AS EventTime FROM clock(period=5) +WHERE log(message="Logging #%v at %v", args=[Count, EventTime], level="INFO", dedup=15) +LIMIT 10 +``` + +Thus the log message will be emitted for the 1st, 4th, 7th, and 10th rows. +To observe the deduplication behaviour in real time you can run this query +in a notebook cell and tweak the arguments to understand their impacts. + +### See also + +- [format](/vql_reference/basic/format/): a function that uses the same + string formatting syntax. +- [alert](/vql_reference/misc/alert/): alerts are a special type of log + message that are added to a server alerts queue, which can be monitored. + diff --git a/content/vql_reference/basic/now/_index.md b/content/vql_reference/basic/now/_index.md index b7357f2266a..b8740733c20 100644 --- a/content/vql_reference/basic/now/_index.md +++ b/content/vql_reference/basic/now/_index.md @@ -35,7 +35,7 @@ Creating a timestamp representing an hour ago: Using `now()` in timestamp arithmetic: -```sql +```vql SELECT * FROM flows(client_id="C.8cfee3cef5dc6915") WHERE state =~ "FINISHED" AND timestamp(epoch=active_time) > now() - 60 * 60 * 24 ``` diff --git a/content/vql_reference/misc/alert/_index.md b/content/vql_reference/misc/alert/_index.md index 1c4602dfbbf..c9431db3800 100644 --- a/content/vql_reference/misc/alert/_index.md +++ b/content/vql_reference/misc/alert/_index.md @@ -27,3 +27,10 @@ condition|If specified we ignore the alert unless the condition is true|Any Generate an alert message. +### See also + +- [log](/vql_reference/basic/log/): alerts and log messages are similar in + concept and use the same deduplication mechanism which is explained with + examples for the `log()` function. + + diff --git a/content/vql_reference/misc/link_to/_index.md b/content/vql_reference/misc/link_to/_index.md index 47f6e33fce5..862c7b26ed1 100644 --- a/content/vql_reference/misc/link_to/_index.md +++ b/content/vql_reference/misc/link_to/_index.md @@ -63,7 +63,7 @@ to a real client, flow id to a real flow, etc. ### Example -```sql +```vql // Setting this in a notebook will tell the GUI to treat this // column as URL. LET ColumnTypes <= dict(HuntLink="url_internal") diff --git a/content/vql_reference/misc/typeof/_index.md b/content/vql_reference/misc/typeof/_index.md index b9efc42a8f5..7e464e8c663 100644 --- a/content/vql_reference/misc/typeof/_index.md +++ b/content/vql_reference/misc/typeof/_index.md @@ -25,12 +25,12 @@ Only the first argument provided will be evaluated and returned. ### Examples -```sql +```vql SELECT typeof(x=1) AS Type FROM scope() ``` returns: `Type: int64` -```sql +```vql LET my_time <= timestamp(epoch="2024-03-26T06:53:37Z") SELECT typeof(thing=my_time) AS Type FROM scope() ``` @@ -41,14 +41,14 @@ returns: `Type: time.Time` The `typeof` function is a more concise alternative to using the more flexible and more powerful `format` function: -```sql +```vql SELECT format(format="%T", args=x) AS Type FROM scope() ``` The `typeof` function is often used to inspect the data type of returned values when writing and testing VQL. -It is also useful as a row filter by including it in the `WHERE` clause of +It is also useful as a row filter by including it in the WHERE clause of a query to ensure that a specific column does not contain values of an unexpected data type. diff --git a/static/reference/data.json b/static/reference/data.json index 4003b2863d1..b5e83d022bf 100644 --- a/static/reference/data.json +++ b/static/reference/data.json @@ -4252,7 +4252,7 @@ }, { "name": "log", - "description": "Log the message and return TRUE.", + "description": "Log a message to the query log stream. Always returns TRUE.", "type": "Function", "version": 2, "args": [