From bac211dcf7863fcaa55ccd5e26b0a383c02da254 Mon Sep 17 00:00:00 2001 From: Lalit Maganti Date: Mon, 18 Sep 2023 11:18:08 +0100 Subject: [PATCH] docs: add PerfettoSQL syntax page and reorg others This CL adds a page for PerfettoSQL syntax and reorganizes other pages and cleanup unnecessary documentation while we're here. Change-Id: Idad2faffaae7f0dc0be70c8076a18c6dc14937b3 --- docs/analysis/builtin.md | 2 +- docs/analysis/common-queries.md | 2 +- docs/analysis/perfetto-sql-syntax.md | 103 ++++++++++++++ docs/analysis/trace-processor.md | 35 ----- docs/toc.md | 23 +-- .../src/gen_sql_tables_reference.js | 3 +- infra/perfetto.dev/src/gen_stdlib_docs_md.py | 133 +++++++++++------- 7 files changed, 203 insertions(+), 98 deletions(-) create mode 100644 docs/analysis/perfetto-sql-syntax.md diff --git a/docs/analysis/builtin.md b/docs/analysis/builtin.md index 29dd1575d7..84deaab354 100644 --- a/docs/analysis/builtin.md +++ b/docs/analysis/builtin.md @@ -1,4 +1,4 @@ -# Built-in Functions +# PerfettoSQL Built-ins These are functions built into C++ which reduce the amount of boilerplate which needs to be written in SQL. diff --git a/docs/analysis/common-queries.md b/docs/analysis/common-queries.md index 018ab947cf..adae686d30 100644 --- a/docs/analysis/common-queries.md +++ b/docs/analysis/common-queries.md @@ -1,4 +1,4 @@ -# Common queries +# PerfettoSQL Common Queries This page acts as a reference guide for queries which often appear when performing ad-hoc analysis. diff --git a/docs/analysis/perfetto-sql-syntax.md b/docs/analysis/perfetto-sql-syntax.md new file mode 100644 index 0000000000..6b0cb2c930 --- /dev/null +++ b/docs/analysis/perfetto-sql-syntax.md @@ -0,0 +1,103 @@ +# PerfettoSQL Syntax +*This page documents the syntax of PerfettoSQL, a dialect of SQL used in trace +processor and other Perfetto analysis tools to query traces.* + +PerfettoSQL is a direct descendent of the +[dialect of SQL implemented by SQLite](https://www.sqlite.org/lang.html). +Specifically, any SQL valid in SQLite is also valid in PerfettoSQL. + +Unfortunately, the SQLite syntax alone is not sufficient for two reasons: +1. It is quite basic e.g. it does not support creating functions or macros +2. It cannot be used to access features which are only available in Perfetto +tooling e.g. it cannot be used to create efficient analytic tables, import +modules from the PerfettoSQL standard library etc. + +For this reason, PerfettoSQL adds new pieces of syntax which make the experience +of writing SQL queries better. All such additons include the keyword `PERFETTO` +to make it clear that they are PerfettoSQL-only. + + + +## Including PerfettoSQL modules +`INCLUDE PERFETTO MODULE` is used to import all tables/views/functions/macros +defined in a PerfettoSQL module (e.g. from the +[PerfettoSQL standard library](/docs/analysis/stdlib-docs.autogen)). + +Note that this statement acts more similar to `#include` statements in C++ +rather than `import` statements from Java/Python. Specifically, all objects +in the module become available in the global namespace without being qualified +by the module name. + +Example: +```sql +-- Include all tables/views/functions from the android.startup.startups module +-- in the standard library. +INCLUDE PERFETTO MODULE android.startup.startups; + +-- Use the android_startups table defined in the android.startup.startups +-- module. +SELECT * +FROM android_startups; +``` + +## Defining functions +`CREATE PEFETTO FUNCTION` allows functions to be defined in SQL. The syntax is +similar to the syntax in PostgreSQL or GoogleSQL. + + + +Example: +```sql +-- Create a scalar function with no arguments. +CREATE PERFETTO FUNCTION constant_fn() RETURNS INT AS SELECT 1; + +-- Create a scalar function taking two arguments. +CREATE PERFETTO FUNCTION add(x INT, y INT) RETURNS INT AS SELECT $x + $y; + +-- Create a table function with no arguments +CREATE PERFETTO FUNCTION constant_tab_fn() +RETURNS TABLE(ts LONG, dur LONG) AS +SELECT column1 as ts, column2 as dur +FROM ( + VALUES + (100, 10), + (200, 20) +); + +-- Create a table function with one argument +CREATE PERFETTO FUNCTION sched_by_utid(utid INT) +RETURNS TABLE(ts LONG, dur LONG, utid INT) AS +SELECT ts, dur, utid +FROM sched +WHERE utid = $utid; +``` + +## Creating efficient tables +`CREATE PERFETTO TABLE` allows defining tables optimized for analytic queries +on traces. These tables are both more performant and more memory efficient than +SQLite native tables created with `CREATE TABLE`. + +Note however the full feature set of `CREATE TABLE` is not supported: +1. Perfetto tables cannot be inserted into and are read-only after creation +2. Perfetto tables must be defined and populated using a `SELECT` statement. + They cannot be defined by column names and types. + +Example: +```sql +-- Create a Perfetto table with constant values. +CREATE PERFETTO TABLE constant_table AS +SELECT column1 as ts, column2 as dur +FROM ( + VALUES + (100, 10), + (200, 20) +); + +-- Create a Perfetto table with a query on another table. +CREATE PERFETTO TABLE slice_sub_table AS +SELECT * +FROM slice +WHERE name = 'foo'; +``` diff --git a/docs/analysis/trace-processor.md b/docs/analysis/trace-processor.md index f1ef185f2b..8b5fdc2327 100644 --- a/docs/analysis/trace-processor.md +++ b/docs/analysis/trace-processor.md @@ -509,24 +509,6 @@ TIP: To see how to add to add a new metric to trace processor, see the checklist The metrics subsystem is a significant part of trace processor and thus is documented on its own [page](/docs/analysis/metrics.md). -## Annotations - -TIP: To see how to add to add a new annotation to trace processor, see the -checklist [here](/docs/contributing/common-tasks.md#new-annotation). - -Annotations attach a human-readable description to a slice in the trace. This -can include information like the source of a slice, why a slice is important and -links to documentation where the viewer can learn more about the slice. -In essence, descriptions act as if an expert was telling the user what the slice -means. - -For example, consider the `inflate` slice which occurs during view inflation in -Android. We can add the following description and link: - -**Description**: Constructing a View hierarchy from pre-processed XML via -LayoutInflater#layout. This includes constructing all of the View objects in the -hierarchy, and applying styled attributes. - ## Creating derived events TIP: To see how to add to add a new annotation to trace processor, see the @@ -557,23 +539,6 @@ creates the exact `launching` slice we want to display in the UI. The other benefit of aligning the two is that changes in metrics are automatically kept in sync with what the user sees in the UI. -## Alerts - -Alerts are used to draw the attention of the user to interesting parts of the -trace; this are usually warnings or errors about anomalies which occurred in the -trace. - -Currently, alerts are not implemented in the trace processor but the API to -create derived events was designed with them in mind. We plan on adding another -column `alert_type` (name to be finalized) to the annotations table which can -have the value `warning`, `error` or `null`. Depending on this value, the -Perfetto UI will flag these events to the user. - -NOTE: we do not plan on supporting case where alerts need to be added to - existing events. Instead, new events should be created using annotations - and alerts added on these instead; this is because the trace processor - storage is monotonic-append-only. - ## Python API The trace processor Python API is built on the existing HTTP interface of `trace processor` diff --git a/docs/toc.md b/docs/toc.md index b0d8fc1ae6..0ed6a886fc 100644 --- a/docs/toc.md +++ b/docs/toc.md @@ -1,6 +1,8 @@ -* [Overview](README.md) +* [Introduction](README.md) -* [Tracing 101](tracing-101.md) +* [Overview](#) + * [Tracing 101](tracing-101.md) + * [FAQ](faq.md) * [Quickstart](#) * [Record traces on Android](quickstart/android-tracing.md) @@ -11,10 +13,8 @@ * [Heap profiling](quickstart/heap-profiling.md) * [Callstack sampling on Android](quickstart/callstack-sampling.md) -* [FAQ](faq.md) - * [Case studies](#) - * [Android boot tracing](case-studies/android-boot-tracing.md) + * [Tracing Android boot](case-studies/android-boot-tracing.md) * [Debugging memory usage](case-studies/memory.md) * [Data sources](#) @@ -40,13 +40,14 @@ * [Interceptors](instrumentation/interceptors.md) * [Trace analysis](#) - * [Trace Processor (SQL)](analysis/trace-processor.md) - * [Batch Trace Processor](analysis/batch-trace-processor.md) - * [Standard library](analysis/stdlib-docs.autogen) - * [Built-in Functions](analysis/builtin.md) + * [Trace Processor](analysis/trace-processor.md) + * [PerfettoSQL Syntax](analysis/perfetto-sql-syntax.md) + * [PerfettoSQL Standard Library](analysis/stdlib-docs.autogen) + * [PerfettoSQL Tables](analysis/sql-tables.autogen) + * [PerfettoSQL Built-ins](analysis/builtin.md) + * [PerfettoSQL Common Queries](analysis/common-queries.md) * [Trace-based metrics](analysis/metrics.md) - * [Common queries](analysis/common-queries.md) - * [SQL tables](analysis/sql-tables.autogen) + * [Batch Trace Processor](analysis/batch-trace-processor.md) * [Stats table](analysis/sql-stats.autogen) * [Pivot tables](analysis/pivot-tables.md) diff --git a/infra/perfetto.dev/src/gen_sql_tables_reference.js b/infra/perfetto.dev/src/gen_sql_tables_reference.js index 2f77cea00e..72800b58a6 100644 --- a/infra/perfetto.dev/src/gen_sql_tables_reference.js +++ b/infra/perfetto.dev/src/gen_sql_tables_reference.js @@ -325,7 +325,8 @@ function main() { graph += '\n```\n'; } - let md = graph; + let title = '# PerfettoSQL Tables\n' + let md = title + graph; for (const tableGroup of tableGroups) { md += `## ${tableGroup}\n` for (const table of tablesByGroup[tableGroup]) { diff --git a/infra/perfetto.dev/src/gen_stdlib_docs_md.py b/infra/perfetto.dev/src/gen_stdlib_docs_md.py index fe61806c74..7959053d76 100644 --- a/infra/perfetto.dev/src/gen_stdlib_docs_md.py +++ b/infra/perfetto.dev/src/gen_stdlib_docs_md.py @@ -32,11 +32,11 @@ def __init__(self, module_name: str, module_files: List[Dict[str, self.files_md = [ FileMd(module_name, file_dict) for file_dict in module_files ] - self.summary_objs = "\n".join( + self.summary_objs = '\n'.join( file.summary_objs for file in self.files_md if file.summary_objs) - self.summary_funs = "\n".join( + self.summary_funs = '\n'.join( file.summary_funs for file in self.files_md if file.summary_funs) - self.summary_view_funs = "\n".join(file.summary_view_funs + self.summary_view_funs = '\n'.join(file.summary_view_funs for file in self.files_md if file.summary_view_funs) @@ -48,13 +48,13 @@ def print_description(self): long_s.append(f'### {file.import_key}') if file.objs: long_s.append('#### Views/Tables') - long_s.append("\n".join(file.objs)) + long_s.append('\n'.join(file.objs)) if file.funs: long_s.append('#### Functions') - long_s.append("\n".join(file.funs)) + long_s.append('\n'.join(file.funs)) if file.view_funs: - long_s.append('#### View Functions') - long_s.append("\n".join(file.view_funs)) + long_s.append('#### Table Functions') + long_s.append('\n'.join(file.view_funs)) return '\n'.join(long_s) @@ -70,75 +70,76 @@ def __init__(self, module_name, file_dict): # Add imports if in file. for data in file_dict['imports']: # Anchor - anchor = f'obj/{module_name}/{data["name"]}' + anchor = f'''obj/{module_name}/{data['name']}''' # Add summary of imported view/table - desc = data["desc"].split(".")[0] - summary_objs_list.append(f'[{data["name"]}](#{anchor})|' - f'{file_dict["import_key"]}|' - f'{desc}') + desc = data['desc'].split('.')[0] + summary_objs_list.append(f'''[{data['name']}](#{anchor})|''' + f'''{file_dict['import_key']}|''' + f'''{desc}''') - self.objs.append(f'\n\n' - f'**{data["name"]}**, {data["type"]}\n\n' - f'{data["desc"]}\n') + self.objs.append(f'''\n\n''' + f'''**{data['name']}**, {data['type']}\n\n''' + f'''{data['desc']}\n''') self.objs.append('Column | Description\n------ | -----------') for name, desc in data['cols'].items(): self.objs.append(f'{name} | {desc}') - self.objs.append("\n\n") + self.objs.append('\n\n') # Add functions if in file for data in file_dict['functions']: # Anchor - anchor = f'fun/{module_name}/{data["name"]}' + anchor = f'''fun/{module_name}/{data['name']}''' # Add summary of imported function - summary_funs_list.append(f'[{data["name"]}](#{anchor})|' - f'{file_dict["import_key"]}|' - f'{data["return_type"]}|' - f'{data["desc"].split(".")[0]}') + summary_funs_list.append(f'''[{data['name']}](#{anchor})|''' + f'''{file_dict['import_key']}|''' + f'''{data['return_type']}|''' + f'''{data['desc'].split('.')[0]}''') self.funs.append( - f'\n\n' - f'**{data["name"]}**\n' - f'{data["desc"]}\n\n' - f'Returns: {data["return_type"]}, {data["return_desc"]}\n\n') + f'''\n\n''' + f'''**{data['name']}**\n''' + f'''{data['desc']}\n\n''' + f'''Returns: {data['return_type']}, {data['return_desc']}\n\n''') if data['args']: self.funs.append('Argument | Type | Description\n' '-------- | ---- | -----------') for name, arg_dict in data['args'].items(): - self.funs.append(f'{name} | {arg_dict["type"]} | {arg_dict["desc"]}') + self.funs.append( + f'''{name} | {arg_dict['type']} | {arg_dict['desc']}''') - self.funs.append("\n\n") + self.funs.append('\n\n') - # Add view functions if in file + # Add table functions if in file for data in file_dict['view_functions']: # Anchor - anchor = rf'view_fun/{module_name}/{data["name"]}' + anchor = rf'''view_fun/{module_name}/{data['name']}''' # Add summary of imported view function - summary_view_funs_list.append(f'[{data["name"]}](#{anchor})|' - f'{file_dict["import_key"]}|' - f'{data["desc"].split(".")[0]}') + summary_view_funs_list.append(f'''[{data['name']}](#{anchor})|''' + f'''{file_dict['import_key']}|''' + f'''{data['desc'].split('.')[0]}''') - self.view_funs.append(f'\n\n' - f'**{data["name"]}**\n' - f'{data["desc"]}\n\n') + self.view_funs.append(f'''\n\n''' + f'''**{data['name']}**\n''' + f'''{data['desc']}\n\n''') if data['args']: self.view_funs.append('Argument | Type | Description\n' '-------- | ---- | -----------') for name, arg_dict in data['args'].items(): self.view_funs.append( - f'{name} | {arg_dict["type"]} | {arg_dict["desc"]}') + f'''{name} | {arg_dict['type']} | {arg_dict['desc']}''') self.view_funs.append('\n') self.view_funs.append('Column | Description\n' '------ | -----------') for name, desc in data['cols'].items(): self.view_funs.append(f'{name} | {desc}') - self.view_funs.append("\n\n") + self.view_funs.append('\n\n') - self.summary_objs = "\n".join(summary_objs_list) - self.summary_funs = "\n".join(summary_funs_list) - self.summary_view_funs = "\n".join(summary_view_funs_list) + self.summary_objs = '\n'.join(summary_objs_list) + self.summary_funs = '\n'.join(summary_funs_list) + self.summary_view_funs = '\n'.join(summary_view_funs_list) def main(): @@ -158,10 +159,44 @@ def main(): common_module = modules_dict.pop('common') with open(args.output, 'w') as f: - f.write("# SQL standard library\n" - "To import any function, view_function, view or table simply run " - "`INCLUDE PERFETTO MODULE {import key};` in your SQL query.\n" - "## Summary\n") + f.write(''' +# PerfettoSQL standard library +*This page documents the PerfettoSQL standard library.* + +## Introduction +The PerfettoSQL standard library is a repository of tables, views, functions +and macros, contributed by domain experts, which make querying traces easier +Its design is heavily inspired by standard libraries in languages like Python, +C++ and Java. + +Some of the purposes of the standard library include: +1) Acting as a way of sharing and commonly written queries without needing +to copy/paste large amounts of SQL. +2) Raising the abstraction level when exposing data in the trace. Many +modules in the standard library convert low-level trace concepts +e.g. slices, tracks and into concepts developers may be more familar with +e.g. for Android developers: app startups, binder transactions etc. + +Standard library modules can be included as follows: +``` +-- Include all tables/views/functions from the android.startup.startups +-- module in the standard library. +INCLUDE PERFETTO MODULE android.startup.startups; + +-- Use the android_startups table defined in the android.startup.startups +-- module. +SELECT * +FROM android_startups; +``` + +More information on importing modules is available in the +[syntax documentation](/docs/analysis/perfetto-sql-syntax#including-perfettosql-modules) +for the `INCLUDE PERFETTO MODULE` statement. + + + +## Summary +''') summary_objs = [common_module.summary_objs ] if common_module.summary_objs else [] @@ -191,21 +226,21 @@ def main(): 'Name | Import | Description\n' '---- | ------ | -----------\n') f.write('\n'.join(summary_objs)) - f.write("\n") + f.write('\n') if summary_funs: - f.write("### Functions\n\n" + f.write('### Functions\n\n' 'Name | Import | Return type | Description\n' '---- | ------ | ----------- | -----------\n') f.write('\n'.join(summary_funs)) - f.write("\n") + f.write('\n') if summary_view_funs: - f.write("### View Functions\n\n" + f.write('### Table Functions\n\n' 'Name | Import | Description\n' '---- | ------ | -----------\n') f.write('\n'.join(summary_view_funs)) - f.write("\n") + f.write('\n') f.write('\n\n') f.write(common_module.print_description())