Code

[Continued] Google Analytics import#1753

Code

Merged

ukutaht merged 148 commits intomasterplausible/analytics:masterfrom pr/1466plausible/analytics:pr/1466Copy head branch name to clipboardMar 10, 2022

+3,648-475

Commits

Commits on Feb 26, 2022

• Add has_imported_stats boolean to Site

  m-col

  committed
  aac8620

  Copy full SHA for aac8620

• Add Google Analytics import panel to general settings

  m-col

  committed
  9bae8c7

  Copy full SHA for 9bae8c7

• Get GA profiles to display in import settings panel

  m-col

  committed
  167f31a

  Copy full SHA for 167f31a

• Add import_from_google method as entrypoint to import data

  m-col

  committed
  1855942

  Copy full SHA for 1855942

• Add imported_visitors table

  m-col

  committed
  9abe8d3

  Copy full SHA for 9abe8d3

• Remove conflicting code from migration

  m-col

  committed
  25061f2

  Copy full SHA for 25061f2

• Import visitors data into clickhouse database

  m-col

  committed
  5a292ca

  Copy full SHA for 5a292ca

• Pass another dataset to main graph for rendering in red

  Show description for 56a5b5e
  m-col

  committed
  56a5b5e

  Copy full SHA for 56a5b5e

• Hook imported GA data to dashboard timeseries plot

  m-col

  committed
  e0ca9bb

  Copy full SHA for e0ca9bb

• Add settings option to forget imported data

  m-col

  committed
  bffb113

  Copy full SHA for bffb113

• Import sources from google analytics

  m-col

  committed
  54703d2

  Copy full SHA for 54703d2

• Merge imported sources when queried

  m-col

  committed
  b5b5cec

  Copy full SHA for b5b5cec

• Merge imported source data native data when querying sources

  m-col

  committed
  e6ab270

  Copy full SHA for e6ab270

• Start converting metrics to atoms so they can be subqueried

  Show description for 0c5c4b7
  m-col

  committed
  0c5c4b7

  Copy full SHA for 0c5c4b7

• Convery GA (direct) source to empty string

  m-col

  committed
  9e1b46d

  Copy full SHA for 9e1b46d

• Import utm campaign and utm medium from GA

  m-col

  committed
  a2c37fb

  Copy full SHA for a2c37fb

• format

  m-col

  committed
  be412a3

  Copy full SHA for be412a3

• Import all data types from GA into new tables

  m-col

  committed
  afe3535

  Copy full SHA for afe3535

• Handle large amounts of more data more safely

  m-col

  committed
  9ea2cd3

  Copy full SHA for 9ea2cd3

• Fix some mistakes in tables

  m-col

  committed
  f5b8f95

  Copy full SHA for f5b8f95

• Make GA requests in chunks of 5 queries

  m-col

  committed
  dd0462e

  Copy full SHA for dd0462e

• Only display imported timeseries when there is no filter

  m-col

  committed
  2b86326

  Copy full SHA for 2b86326

• Correctly show last 30 minutes timeseries when 'realtime'

  m-col

  committed
  8a969ea

  Copy full SHA for 8a969ea

• Add with_imported key to Query struct

  m-col

  committed
  5b09c8c

  Copy full SHA for 5b09c8c

• Account for injected :is_not filter on sources from dashboard

  m-col

  committed
  e2c185a

  Copy full SHA for e2c185a

• Also add tentative imported_utm_sources table

  Show description for 0a62a4d
  m-col

  committed
  0a62a4d

  Copy full SHA for 0a62a4d

• Return imported data to dashboard for rest of Sources panel

  Show description for 5a0a81d
  m-col

  committed
  5a0a81d

  Copy full SHA for 5a0a81d

• Clear imported stats from all tables when requested

  m-col

  committed
  fa65a50

  Copy full SHA for fa65a50

• Merge entry pages and exit pages from imported data into unfiltered dashboard view

  Show description for a8b0b10
  m-col

  committed
  a8b0b10

  Copy full SHA for a8b0b10

• Display imported devices, browsers and OSs on dashboard

  m-col

  committed
  b8ee7df

  Copy full SHA for b8ee7df

• Display imported country data on dashboard

  m-col

  committed
  0e57d58

  Copy full SHA for 0e57d58

• Add more metrics to entries/exits for modals

  m-col

  committed
  aac954d

  Copy full SHA for aac954d

• make sure data is returned via API with correct keys

  m-col

  committed
  f2d7eab

  Copy full SHA for f2d7eab

• Import regions and cities from GA

  m-col

  committed
  3ab7d57

  Copy full SHA for 3ab7d57

• Capitalize device upon import to match native data

  m-col

  committed
  87ba204

  Copy full SHA for 87ba204

• Leave query limits/offsets until after possibly joining with imported data

  m-col

  committed
  c5bc70a

  Copy full SHA for c5bc70a

• Also import timeOnPage and pageviews for pages from GA

  m-col

  committed
  7936e6f

  Copy full SHA for 7936e6f

• imported_countries -> imported_locations

  m-col

  committed
  bca7e24

  Copy full SHA for bca7e24

• Get timeOnPage and pageviews for pages from GA

  Show description for 3f38a3e
  m-col

  committed
  3f38a3e

  Copy full SHA for 3f38a3e

• Add indicator to dashboard when imported data is being used

  m-col

  committed
  a03759e

  Copy full SHA for a03759e

• Don't show imported data as separately line on main graph

  m-col

  committed
  b8b8ccf

  Copy full SHA for b8b8ccf

• "bounce_rate" -> :bounce_rate, so it works in subqueries

  m-col

  committed
  991cf88

  Copy full SHA for 991cf88

• Drop imported browser and OS versions

  Show description for e03e128
  m-col

  committed
  e03e128

  Copy full SHA for e03e128

• Toggle displaying imported data by clicking indicator

  m-col

  committed
  5dbde9e

  Copy full SHA for 5dbde9e

• Parse referrers with RefInspector

  Show description for 71abc96
  m-col

  committed
  71abc96

  Copy full SHA for 71abc96

• Keep imported data indicator on dashboard and strikethrough when hidden

  m-col

  committed
  4bfee86

  Copy full SHA for 4bfee86

• Add unlink google button to import panel

  m-col

  committed
  5ff0412

  Copy full SHA for 5ff0412

• Rename some GA browsers and OSes to plausible versions

  m-col

  committed
  2c6255e

  Copy full SHA for 2c6255e

• Get main top pages and exit pages panels working correctly with imported data

  m-col

  committed
  543f42b

  Copy full SHA for 543f42b

• mix format

  m-col

  committed
  21fcada

  Copy full SHA for 21fcada

• Fetch time_on_pages for imported data when needed

  m-col

  committed
  a7da346

  Copy full SHA for a7da346

• entry pages need to fetch bounces from GA

  m-col

  committed
  6ca72a2

  Copy full SHA for 6ca72a2

• "sample_percent" -> :sample_percent as only atoms can be used in subqueries

  m-col

  committed
  93ddbab

  Copy full SHA for 93ddbab

• Calculate bounce_rate for joined native and imported data for top pages modal

  m-col

  committed
  67fd47a

  Copy full SHA for 67fd47a

• Flip some query bindings around to be less misleading

  m-col

  committed
  2158098

  Copy full SHA for 2158098

• Fixup entry page modal visit durations

  m-col

  committed
  8083261

  Copy full SHA for 8083261

• mix format

  m-col

  committed
  2881946

  Copy full SHA for 2881946

• Fetch bounces and visit_duration for sources from GA

  m-col

  committed
  d3d9a5c

  Copy full SHA for d3d9a5c

• add more source metrics used for data in modals

  m-col

  committed
  46fb187

  Copy full SHA for 46fb187

• Make sources modals display correct values

  m-col

  committed
  e95e92b

  Copy full SHA for e95e92b

• imported_visitors: bounce_rate -> bounces, avg_visit_duration -> visit_duration

  m-col

  committed
  e78c2e5

  Copy full SHA for e78c2e5

• Merge imported data into aggregate stats

  m-col

  committed
  9e3963a

  Copy full SHA for 9e3963a

• Reformat top graph side icons

  m-col

  committed
  429ec25

  Copy full SHA for 429ec25

• Ensure sample_percent is yielded from aggregate data

  m-col

  committed
  393ad27

  Copy full SHA for 393ad27

• filter event_props should be strings

  m-col

  committed
  d446486

  Copy full SHA for d446486

• Hide imported data from frontend when using filter

  m-col

  committed
  adb3f22

  Copy full SHA for adb3f22

• Fix existing tests

  m-col

  committed
  f737aa7

  Copy full SHA for f737aa7

• fix tests

  m-col

  committed
  c6b314f

  Copy full SHA for c6b314f

• Fix imported indicator appearing when filtering

  m-col

  committed
  ae84c7e

  Copy full SHA for ae84c7e

• comma needed, lost when rebasing

  m-col

  committed
  ecfccd4

  Copy full SHA for ecfccd4

• Import utm_terms and utm_content from GA

  m-col

  committed
  c48b3b5

  Copy full SHA for c48b3b5

• Merge imported utm_term and utm_content

  m-col

  committed
  bb6b085

  Copy full SHA for bb6b085

• Rename imported Countries data as Locations

  m-col

  committed
  866097e

  Copy full SHA for 866097e

• Set imported city schema field to int

  m-col

  committed
  98f27f0

  Copy full SHA for 98f27f0

• Remove utm_terms and utm_content when clearing imported

  m-col

  committed
  ca5f1e5

  Copy full SHA for ca5f1e5

• Clean locations import from Google Analytics

  Show description for 944d07a
  m-col

  committed
  944d07a

  Copy full SHA for 944d07a

• Display imported region and city in dashboard

  m-col

  committed
  c14ded5

  Copy full SHA for c14ded5

• os -> operating_system in some parts of code

  Show description for 787ee44
  m-col

  committed
  787ee44

  Copy full SHA for 787ee44

• to_atom -> to_existing_atom

  m-col

  committed
  e0a06f0

  Copy full SHA for e0a06f0

• format

  m-col

  committed
  decc32f

  Copy full SHA for decc32f

• "events" metric -> :events

  m-col

  committed
  7a1695f

  Copy full SHA for 7a1695f

• ignore imported data when "events" in metrics

  m-col

  committed
  8f35671

  Copy full SHA for 8f35671

• update "bounce_rate"

  m-col

  committed
  1d85271

  Copy full SHA for 1d85271

• atomise some more metrics from new city and region api

  m-col

  committed
  0c50991

  Copy full SHA for 0c50991

• atomise some more metrics for email handlers

  m-col

  committed
  afdd562

  Copy full SHA for afdd562

• "conversion_rate" -> :conversion_rate during csv export

  m-col

  committed
  9aeca6f

  Copy full SHA for 9aeca6f

• Move imported data stats code to own module

  m-col

  committed
  173df31

  Copy full SHA for 173df31

• Move imported timeseries function to Stats.Imported

  m-col

  committed
  f09a99d

  Copy full SHA for f09a99d

• Use Timex.parse to import dates from GA

  m-col

  committed
  8acc085

  Copy full SHA for 8acc085

• has_imported_stats -> imported_source

  m-col

  committed
  95e38a4

  Copy full SHA for 95e38a4

• "time_on_page" -> :time_on_page

  m-col

  committed
  88d925d

  Copy full SHA for 88d925d

• Convert imported GA data to UTC

  m-col

  committed
  b337e72

  Copy full SHA for b337e72

• Clean up GA request code a bit

  Show description for 0e48b25
  m-col

  committed
  0e48b25

  Copy full SHA for 0e48b25

• Fail sooner if GA timezone can't be identified

  m-col

  committed
  11821fd

  Copy full SHA for 11821fd

• Link imported tables to site by id

  m-col

  committed
  ca8531b

  Copy full SHA for ca8531b

• imported_utm_content -> imported_utm_contents

  m-col

  committed
  b534f62

  Copy full SHA for b534f62

• Imported GA from all of time

  m-col

  committed
  a0e2167

  Copy full SHA for a0e2167

• Reorganise GA data fetch logic

  Show description for 1505e7b
  m-col

  committed
  1505e7b

  Copy full SHA for 1505e7b

• Clarify removal of "visits" data when it isn't in metrics

  m-col

  committed
  de3220d

  Copy full SHA for de3220d

• Apply location filters from API

  Show description for 5b57143
  m-col

  committed
  5b57143

  Copy full SHA for 5b57143

• Do not use changeset for setting site.imported_source

  m-col

  committed
  734a520

  Copy full SHA for 734a520

• Add all metrics to all dimensions

  m-col

  committed
  1a6ab5b

  Copy full SHA for 1a6ab5b

• Run GA import in the background

  m-col

  committed
  bbbc625

  Copy full SHA for bbbc625

• Send email when GA import completes

  m-col

  committed
  2ba78d3

  Copy full SHA for 2ba78d3

• Add handler to insert imported data into tests and imported_browsers_factory

  m-col

  committed
  8eba626

  Copy full SHA for 8eba626

• Add remaining import data test factories

  m-col

  committed
  680708f

  Copy full SHA for 680708f

• Add imported location data to test

  m-col

  committed
  4cb02e5

  Copy full SHA for 4cb02e5

• Test main graph with imported data

  m-col

  committed
  6309562

  Copy full SHA for 6309562

• Add imported data to operating systems tests

  m-col

  committed
  516c84d

  Copy full SHA for 516c84d

• Add imported data to pages tests

  m-col

  committed
  4132f64

  Copy full SHA for 4132f64

• Add imported data to entry pages tests

  m-col

  committed
  51ca030

  Copy full SHA for 51ca030

• Add imported data to exit pages tests

  m-col

  committed
  2e13a60

  Copy full SHA for 2e13a60

• Add imported data to devices tests

  m-col

  committed
  d369ba3

  Copy full SHA for d369ba3

• Add imported data to sources tests

  m-col

  committed
  16a9d01

  Copy full SHA for 16a9d01

• Add imported data to UTM tests

  m-col

  committed
  67fba86

  Copy full SHA for 67fba86

• Add new test module for the data import step

  m-col

  committed
  5ac5ba6

  Copy full SHA for 5ac5ba6

• Test import of sources GA data

  m-col

  committed
  cb9ce03

  Copy full SHA for cb9ce03

• Test import of utm_mediums GA data

  m-col

  committed
  de821d6

  Copy full SHA for de821d6

• Test import of utm_campaigns GA data

  m-col

  committed
  d646757

  Copy full SHA for d646757

• Add tests for UTM terms

  m-col

  committed
  cd17fe5

  Copy full SHA for cd17fe5

• Add tests for UTM contents

  m-col

  committed
  72303d8

  Copy full SHA for 72303d8

• Add test for importing pages and entry pages data from GA

  m-col

  committed
  8b90896

  Copy full SHA for 8b90896

• Add test for importing exit page data

  m-col

  committed
  380f9aa

  Copy full SHA for 380f9aa

• Fix module file name typo

  m-col

  committed
  a29a7c1

  Copy full SHA for a29a7c1

• Add test for importing location data from GA

  m-col

  committed
  99e3c15

  Copy full SHA for 99e3c15

• Add test for importing devices data from GA

  m-col

  committed
  82e50cb

  Copy full SHA for 82e50cb

• Add test for importing browsers data from GA

  m-col

  committed
  f5603f9

  Copy full SHA for f5603f9

• Add test for importing OS data from GA

  m-col

  committed
  b37e814

  Copy full SHA for b37e814

• Paginate GA requests to download all data

  m-col

  committed
  55da66a

  Copy full SHA for 55da66a

• Bump clickhouse_ecto version

  m-col

  committed
  2ff7606

  Copy full SHA for 2ff7606

• Move RefInspector wrapper function into module

  m-col

  committed
  ba28622

  Copy full SHA for ba28622

• Drop timezone transform on import

  m-col

  committed
  df6d1bc

  Copy full SHA for df6d1bc

• Order imported by side_id then date

  m-col

  committed
  92ffae1

  Copy full SHA for 92ffae1

• More strings -> atoms

  Show description for 0e9c045
  m-col

  committed
  0e9c045

  Copy full SHA for 0e9c045

• Remove parallelisation of data import

  m-col

  committed
  fdf4d00

  Copy full SHA for fdf4d00

• Split sources and UTM sources from fetched GA data

  Show description for 0cb66d6
  m-col

  committed
  0cb66d6

  Copy full SHA for 0cb66d6

• Keep prop names in queries as strings

  m-col

  committed
  175025b

  Copy full SHA for 175025b

• fix typo

  m-col

  committed
  c574abe

  Copy full SHA for c574abe

Commits on Mar 2, 2022

• Fix import

  ukutaht

  committed
  9213b29

  Copy full SHA for 9213b29

Commits on Mar 3, 2022

• Insert data to clickhouse in batches

  ukutaht

  committed
  5073b39

  Copy full SHA for 5073b39

• Fix link when removing imported data

  ukutaht

  committed
  0f99441

  Copy full SHA for 0f99441

Commits on Mar 4, 2022

• Merge source tables

  ukutaht

  committed
  16c8cb1

  Copy full SHA for 16c8cb1

• Import hostname as well as pathname

  ukutaht

  committed
  c324395

  Copy full SHA for c324395

Commits on Mar 7, 2022

• Record start and end time of imported data

  ukutaht

  committed
  d4b554b

  Copy full SHA for d4b554b

• Track import progress

  ukutaht

  committed
  8a0b532

  Copy full SHA for 8a0b532

Commits on Mar 8, 2022

• Fix month interval with imported data

  ukutaht

  committed
  a7bc60f

  Copy full SHA for a7bc60f

• Do not JOIN when imported date range has no overlap

  ukutaht

  committed
  8ddbfb4

  Copy full SHA for 8ddbfb4

Commits on Mar 10, 2022

• Fix time on page using exits

  ukutaht

  committed
  bd7e40b

  Copy full SHA for bd7e40b