docs: add PerfettoSQL syntax page and reorg others

This CL adds a page for PerfettoSQL syntax and reorganizes other
pages and cleanup unnecessary documentation while we're here.

Change-Id: Idad2faffaae7f0dc0be70c8076a18c6dc14937b3

docs/analysis/builtin.md

@@ -1,4 +1,4 @@
-# Built-in Functions
+# PerfettoSQL Built-ins
 
 These are functions built into C++ which reduce the amount of boilerplate which
 needs to be written in SQL.

docs/analysis/common-queries.md

@@ -1,4 +1,4 @@
-# Common queries
+# PerfettoSQL Common Queries
 
 This page acts as a reference guide for queries which often appear when
 performing ad-hoc analysis.

docs/analysis/perfetto-sql-syntax.md

@@ -0,0 +1,103 @@
+# PerfettoSQL Syntax
+*This page documents the syntax of PerfettoSQL, a dialect of SQL used in trace
+processor and other Perfetto analysis tools to query traces.*
+
+PerfettoSQL is a direct descendent of the
+[dialect of SQL implemented by SQLite](https://www.sqlite.org/lang.html).
+Specifically, any SQL valid in SQLite is also valid in PerfettoSQL.
+
+Unfortunately, the SQLite syntax alone is not sufficient for two reasons:
+1. It is quite basic e.g. it does not support creating functions or macros
+2. It cannot be used to access features which are only available in Perfetto
+tooling e.g. it cannot be used to create efficient analytic tables, import
+modules from the PerfettoSQL standard library etc.
+
+For this reason, PerfettoSQL adds new pieces of syntax which make the experience
+of writing SQL queries better. All such additons include the keyword `PERFETTO`
+to make it clear that they are PerfettoSQL-only.
+
+<!-- TODO(b/290185551): we should really talk about our "recommendations" (e.g.
+using CREATE PERFETTO TABLE instead of CREATE TABLE) somewhere and reference it
+here. -->
+
+## Including PerfettoSQL modules
+`INCLUDE PERFETTO MODULE` is used to import all tables/views/functions/macros
+defined in a PerfettoSQL module (e.g. from the
+[PerfettoSQL standard library](/docs/analysis/stdlib-docs.autogen)).
+
+Note that this statement acts more similar to `#include` statements in C++
+rather than `import` statements from Java/Python. Specifically, all objects
+in the module become available in the global namespace without being qualified
+by the module name.
+
+Example:
+```sql
+-- Include all tables/views/functions from the android.startup.startups module
+-- in the standard library.
+INCLUDE PERFETTO MODULE android.startup.startups;
+
+-- Use the android_startups table defined in the android.startup.startups
+-- module.
+SELECT *
+FROM android_startups;
+```
+
+## Defining functions
+`CREATE PEFETTO FUNCTION` allows functions to be defined in SQL. The syntax is
+similar to the syntax in PostgreSQL or GoogleSQL.
+
+<!-- TODO(b/290185551): talk about different possible argument/return types. -->
+
+Example:
+```sql
+-- Create a scalar function with no arguments.
+CREATE PERFETTO FUNCTION constant_fn() RETURNS INT AS SELECT 1;
+
+-- Create a scalar function taking two arguments.
+CREATE PERFETTO FUNCTION add(x INT, y INT) RETURNS INT AS SELECT $x + $y;
+
+-- Create a table function with no arguments
+CREATE PERFETTO FUNCTION constant_tab_fn()
+RETURNS TABLE(ts LONG, dur LONG) AS
+SELECT column1 as ts, column2 as dur
+FROM (
+  VALUES
+  (100, 10),
+  (200, 20)
+);
+
+-- Create a table function with one argument
+CREATE PERFETTO FUNCTION sched_by_utid(utid INT)
+RETURNS TABLE(ts LONG, dur LONG, utid INT) AS
+SELECT ts, dur, utid
+FROM sched
+WHERE utid = $utid;
+```
+
+## Creating efficient tables
+`CREATE PERFETTO TABLE` allows defining tables optimized for analytic queries
+on traces. These tables are both more performant and more memory efficient than
+SQLite native tables created with `CREATE TABLE`.
+
+Note however the full feature set of `CREATE TABLE` is not supported:
+1. Perfetto tables cannot be inserted into and are read-only after creation
+2. Perfetto tables must be defined and populated using a `SELECT` statement.
+  They cannot be defined by column names and types.
+
+Example:
+```sql
+-- Create a Perfetto table with constant values.
+CREATE PERFETTO TABLE constant_table AS
+SELECT column1 as ts, column2 as dur
+FROM (
+  VALUES
+  (100, 10),
+  (200, 20)
+);
+
+-- Create a Perfetto table with a query on another table.
+CREATE PERFETTO TABLE slice_sub_table AS
+SELECT *
+FROM slice
+WHERE name = 'foo';
+```

docs/analysis/trace-processor.md

@@ -509,24 +509,6 @@ TIP: To see how to add to add a new metric to trace processor, see the checklist
 The metrics subsystem is a significant part of trace processor and thus is
 documented on its own [page](/docs/analysis/metrics.md).
 
-## Annotations
-
-TIP: To see how to add to add a new annotation to trace processor, see the
-checklist [here](/docs/contributing/common-tasks.md#new-annotation).
-
-Annotations attach a human-readable description to a slice in the trace. This
-can include information like the source of a slice, why a slice is important and
-links to documentation where the viewer can learn more about the slice.
-In essence, descriptions act as if an expert was telling the user what the slice
-means.
-
-For example, consider the `inflate` slice which occurs during view inflation in
-Android. We can add the following description and link:
-
-**Description**: Constructing a View hierarchy from pre-processed XML via
-LayoutInflater#layout. This includes constructing all of the View objects in the
-hierarchy, and applying styled attributes.
-
 ## Creating derived events
 
 TIP: To see how to add to add a new annotation to trace processor, see the
@@ -557,23 +539,6 @@ creates the exact `launching` slice we want to display in the UI.
 The other benefit of aligning the two is that changes in metrics are
 automatically kept in sync with what the user sees in the UI.
 
-## Alerts
-
-Alerts are used to draw the attention of the user to interesting parts of the
-trace; this are usually warnings or errors about anomalies which occurred in the
-trace.
-
-Currently, alerts are not implemented in the trace processor but the API to
-create derived events was designed with them in mind. We plan on adding another
-column `alert_type` (name to be finalized) to the annotations table which can
-have the value `warning`, `error` or `null`. Depending on this value, the
-Perfetto UI will flag these events to the user.
-
-NOTE: we do not plan on supporting case where alerts need to be added to
-      existing events. Instead, new events should be created using annotations
-      and alerts added on these instead; this is because the trace processor
-      storage is monotonic-append-only.
-
 ## Python API
 
 The trace processor Python API is built on the existing HTTP interface of `trace processor`

docs/toc.md

@@ -1,6 +1,8 @@
-* [Overview](README.md)
+* [Introduction](README.md)
 
-* [Tracing 101](tracing-101.md)
+* [Overview](#)
+  * [Tracing 101](tracing-101.md)
+  * [FAQ](faq.md)
 
 * [Quickstart](#)
   * [Record traces on Android](quickstart/android-tracing.md)
@@ -11,10 +13,8 @@
   * [Heap profiling](quickstart/heap-profiling.md)
   * [Callstack sampling on Android](quickstart/callstack-sampling.md)
 
-* [FAQ](faq.md)
-
 * [Case studies](#)
-  * [Android boot tracing](case-studies/android-boot-tracing.md)
+  * [Tracing Android boot](case-studies/android-boot-tracing.md)
   * [Debugging memory usage](case-studies/memory.md)
 
 * [Data sources](#)
@@ -40,13 +40,14 @@
   * [Interceptors](instrumentation/interceptors.md)
 
 * [Trace analysis](#)
-  * [Trace Processor (SQL)](analysis/trace-processor.md)
-  * [Batch Trace Processor](analysis/batch-trace-processor.md)
-  * [Standard library](analysis/stdlib-docs.autogen)
-  * [Built-in Functions](analysis/builtin.md)
+  * [Trace Processor](analysis/trace-processor.md)
+  * [PerfettoSQL Syntax](analysis/perfetto-sql-syntax.md)
+  * [PerfettoSQL Standard Library](analysis/stdlib-docs.autogen)
+  * [PerfettoSQL Tables](analysis/sql-tables.autogen)
+  * [PerfettoSQL Built-ins](analysis/builtin.md)
+  * [PerfettoSQL Common Queries](analysis/common-queries.md)
   * [Trace-based metrics](analysis/metrics.md)
-  * [Common queries](analysis/common-queries.md)
-  * [SQL tables](analysis/sql-tables.autogen)
+  * [Batch Trace Processor](analysis/batch-trace-processor.md)
   * [Stats table](analysis/sql-stats.autogen)
   * [Pivot tables](analysis/pivot-tables.md)
 

infra/perfetto.dev/src/gen_sql_tables_reference.js

@@ -325,7 +325,8 @@ function main() {
     graph += '\n```\n';
   }
 
-  let md = graph;
+  let title = '# PerfettoSQL Tables\n'
+  let md = title + graph;
   for (const tableGroup of tableGroups) {
     md += `## ${tableGroup}\n`
     for (const table of tablesByGroup[tableGroup]) {