blink_features.usage with ranks and partitioned

[max-ostapenko]

1. Added rank column.
   Resolves blink_features.usage has null rank column #25

2. Removed the blink_features.features table, as it's a duplicate of a pages.features column.
   And now loading usage data directly to blink_features.usage partitioned table.

Table migration script:

CREATE TABLE `blink_features.usage_partitioned` (
  date DATE,
  client STRING,
  rank INT64,
  id STRING,
  feature STRING,
  type STRING,
  num_urls INT64,
  total_urls INT64,
  pct_urls FLOAT64,
  sample_urls ARRAY<STRING>
)
PARTITION BY date
CLUSTER BY client, rank, feature;

INSERT INTO `blink_features.usage_partitioned`
SELECT
  PARSE_DATE('%Y%m%d', yyyymmdd) AS date,
  client,
  NULL AS rank,
  id,
  feature,
  type,
  num_urls,
  total_urls,
  pct_urls,
  sample_urls
FROM `blink_features.usage`;

ALTER TABLE `blink_features.usage` RENAME TO `blink_features.usage_backup`;

CREATE TABLE `blink_features.usage`
COPY `blink_features.usage_partitioned`;

[max-ostapenko]

@tunetheweb do you have edit access to the Looker report using this data?

[tunetheweb]

@tunetheweb do you have edit access to the Looker report using this data?

Yes I do. You can see it here if you wanna clone it to try it against any changed data source.

[max-ostapenko]

@tunetheweb do you have edit access to the Looker report using this data?
Yes I do. You can see it here if you wanna clone it to try it against any changed data source.

Data sources are not accessible - can't clone.

Do you want to migrate the table and adjust the report? Or I can do it if I have edit permission for the data sources and the dashboard.

[max-ostapenko]

@tunetheweb, need your assistance to continue.
How do we update the dashboard?

[tunetheweb]

Yeah on my todo list. Trying to get the reports finished first and then this is next thing.

[tunetheweb]

This works but I wasted a lot of time figuring out why it wouldn't import into Looker Studio and it was due to the Partition filter: Required setting. You can't set this on a Looker Studio connection so I had to change it to a Custom SQL rather than a direct table.

I think the Partition filter: Required setting makes sense for the larger tables, where we want to avoid users running up large bills, but I don't think it's necessary here.

Can we recreate the table without that setting?

[max-ostapenko]

Updated the code.
But to clarify, I thought we'll continue using httparchive.blink_features.usage just with updated schema.

Let me know when I can replace it, or run a replacement script yourself:

ALTER TABLE `blink_features.usage` RENAME TO `blink_features.usage_backup`;

CREATE TABLE `blink_features.usage`
COPY `blink_features.usage_partitioned`;

[tunetheweb]

But to clarify, I thought we'll continue using httparchive.blink_features.usage just with updated schema.

Agreed.

Let me know when I can replace it, or run a replacement script yourself:

Will run this when I'm back rather than change something and leave for holidays! Let's leave this PR open as a reminder and merge after crawl is finished and I'm back.