Koios v1.1.0rc (#238)

## Description
<!--- Describe your changes -->
[Re-created from #227 post renaming base-branch]

- [x] pool_list: Return all fields from pool_info_cache (latest pool entry)
- [x] Remove filter for returning only 'registered' pools from pool_list, pool_relays and pool_metadata endpoints
- [x] Few more linting polishing (remove trailing spaces, move single column/table references onto same line)
- [x] Update pool_info to return same metadata as pool_list and pool_metadata (latest w/o fallback)
- [x] Update all tx_in joins to use the new tx_out.consumed_by_tx_in_id
- [x] Dont include epoch_params in epoch_info_cache, update references accordingly
- [x] Update `*_txs` as per #186 (point 4)
- [x] Update `*_utxos` as per #186 (point 3)
  - [x] credential_utxos
  - [x] address_utxos
  - [x] account_utxos
  - [x] utxo_info
  - [x] asset_utxos
  - [x] script_utxos
- [x] Add script_info and align all endpoints for scripts
- [x] Add temporary cron for fixing epoch count mismatch
- [x] Add pool_registrations/pool_deregistrations endpoint
- [x] Add retired txs to `pool_updates` (other fields for such transactions will return `null`)
- [x] Add reward_withdrawals/treasury_withdrawals endpoint
- [x] Simplify address_assets and account_assets heirarchy to return flat table (helps do horizontal filtering as desired)
- [x] API Spec updates:
  - [x] Add section for Authentication
  - [x] Bump Koios version
  - [x] pool_list
  - [x] pool_metadata
  - [x] pool_relay
  - [x] `*_txs`
  - [x] `*_utxos`
  - [x] `address_assets`
  - [x] `account_assets`
  - [x] `epoch_params`
  - [x] `pool_registrations`/`pool_deregistrations`
  - [x] `reward_withdrawals`/`treasury_withdrawals`
  - [x] All script-related endpoints
  - [x] Update examples
  - [x] Check re-usability
  - [x] Check schemathesis
    - [x] guild
    - [x] preview
    - [x] preprod
    - [x] mainnet
- [x] Add v0 vs v1 to monitoring
  - [x] guild
  - [ ] preview (post merge)
  - [ ] preprod (post merge)
  - [ ] mainnet (post 1.1.0 - instead of 1.1.0rc) release

- [x] Ogmios
  - [x] Add integration for tx Evaluation and submission using ogmios path
  - [x] Add health check based on /health (`version` and `networkSynchronization`)
- [x] Update CHANGELOG
- [x] Bump final specs version number to v1 

## Which issue it fixes?
<!--- Link to issue: Closes #issue-number -->
- [x] Closes #208
- [x] Closes #218
- [x] Closes #186 
- [x] Closes #221
- [x] Closes #224
- [x] Closes #191
- [x] Closes #232
- [x] Closes #240 #241

---------

Co-authored-by: KoT_B_KocMoce <[email protected]>
Co-authored-by: Ola [AHLNET] <[email protected]>

README.md

@@ -1,26 +1,30 @@
 # Koios Artifacts Repository
 
-Various Artifacts related to [Koios project](https://www.koios.rest) management and assets that are used for website, monitoring services as well as public-facing topology files.
+Various Artifacts related to [Koios project](https://www.koios.rest) management and assets that are used for website - look at repository map below for further info.
+Provisioning scripts to run an instance are part of [guild-operators repo](https://cardano-community.github.io/guild-operators/Build/grest/) alongwith Koios SPO tools
 
 ## Repository Map
+
 ```
 .
 ├── grafana-dashboards/ # Grafana dashboards used for monitoring Koios nodes
 ├── html/               # HTML Page(s) used for https://api.koios.rest website
-├── images/             # Images (logos, Design diagram, etc) used for Koios
+├── images/             # Images used for website (incl. projects building on Koios)
 ├── specs/              # Files used for creation of API Specs with examples
 ├── tests/              # Test suites for Koios nodes
 ├── topology/           # Topology files of public Koios nodes
     ├── topology-guild.json
     ├── topology-mainnet.json
-    └── topology-testnet.json
-├── LICENSE                 # License for use of artifacts within this repository
-├── README.md               # This file
-└── projects.json           # List of projects (in JSON format) that are using Koios
+    └── topology-preprod.json
+    └── topology-preview.json
+├── LICENSE             # License for use of artifacts within this repository
+├── README.md           # This file
+└── projects.json       # List of projects (in JSON format) that are using Koios
 ```
+
 ## Project Management
 
-Koios team operates gRest layer in a transparent manner and progress/millestones can be accessed at any time (accessible [here](https://github.com/orgs/cardano-community/projects/1))
+Koios team operates gRest layer in a transparent manner and progress/millestones can be accessed at any time (accessible [here](https://github.com/orgs/cardano-community/projects/1/views/2))
 
 ## API Specs
 
@@ -35,4 +39,4 @@ The specs can be browsed for each network using below:
 
 ## Further discussions
 
-You can connect and discuss with Koios teams on [Telegram](https://t.me/+zE4Lce_QUepiY2U1), or feel free to contribute to any of the repositories.
+You can connect and discuss with Koios teams on [Telegram](https://t.me/CardanoKoios/1), or feel free to contribute to any of the repositories.

files/grest/cron/jobs/epoch-info-cache-update.sh

@@ -1,12 +1,12 @@
 #!/bin/bash
 DB_NAME=cexplorer
 
-tip=$(psql ${DB_NAME} -qbt -c "select extract(epoch from time)::integer from block order by id desc limit 1;" | xargs)
+tip=$(psql ${DB_NAME} -qbt -c "SELECT EXTRACT(EPOCH FROM time)::integer FROM block ORDER BY id DESC LIMIT 1;" | xargs)
 
 if [[ $(( $(date +%s) - tip )) -gt 300 ]]; then
   echo "$(date +%F_%H:%M:%S) Skipping as database has not received a new block in past 300 seconds!" && exit 1
 fi
 
 echo "$(date +%F_%H:%M:%S) Running epoch info cache update..."
-psql ${DB_NAME} -qbt -c "SELECT GREST.EPOCH_INFO_CACHE_UPDATE();" 1>/dev/null 2>&1
+psql ${DB_NAME} -qbt -c "SELECT grest.epoch_info_cache_update();" 1>/dev/null 2>&1
 echo "$(date +%F_%H:%M:%S) Job done!"

files/grest/cron/jobs/epoch-summary-corrections-update.sh

@@ -0,0 +1,12 @@
+#!/bin/bash
+DB_NAME=cexplorer
+
+tip=$(psql ${DB_NAME} -qbt -c "SELECT EXTRACT(EPOCH FROM time)::integer FROM block ORDER BY id DESC LIMIT 1;" | xargs)
+
+if [[ $(( $(date +%s) - tip )) -gt 300 ]]; then
+  echo "$(date +%F_%H:%M:%S) Skipping as database has not received a new block in past 300 seconds!" && exit 1
+fi
+
+echo "$(date +%F_%H:%M:%S) Running epoch summary corrections update..."
+psql ${DB_NAME} -qbt -c "SELECT GREST.EPOCH_SUMMARY_CORRECTIONS_UPDATE();" 1>/dev/null 2>&1
+echo "$(date +%F_%H:%M:%S) Job done!"

files/grest/rpc/00_blockchain/reserve_withdrawals.sql

@@ -0,0 +1,27 @@
+CREATE OR REPLACE FUNCTION grest.reserve_withdrawals()
+RETURNS TABLE (
+  epoch_no word31type,
+  epoch_slot word31type,
+  tx_hash text,
+  block_hash text,
+  block_height word31type,
+  amount text,
+  stake_address text
+)
+LANGUAGE SQL STABLE
+AS $$
+  SELECT
+    b.epoch_no,
+    b.epoch_slot_no,
+    ENCODE(tx.hash,'hex'),
+    ENCODE(b.hash,'hex'),
+    b.block_no,
+    r.amount::text,
+    sa.view
+  FROM reserve AS r
+    LEFT JOIN tx ON r.tx_id = tx.id
+    INNER JOIN block AS b ON tx.block_id = b.id
+    LEFT JOIN stake_address AS sa ON sa.id = r.addr_id;
+$$;
+
+COMMENT ON FUNCTION grest.reserve_withdrawals IS 'A list of withdrawals made from reserves (MIRs)'; --noqa: LT01

files/grest/rpc/00_blockchain/treasury_withdrawals.sql

@@ -0,0 +1,27 @@
+CREATE OR REPLACE FUNCTION grest.treasury_withdrawals()
+RETURNS TABLE (
+  epoch_no word31type,
+  epoch_slot word31type,
+  tx_hash text,
+  block_hash text,
+  block_height word31type,
+  amount text,
+  stake_address text
+)
+LANGUAGE SQL STABLE
+AS $$
+  SELECT
+    b.epoch_no,
+    b.epoch_slot_no,
+    ENCODE(tx.hash,'hex'),
+    ENCODE(b.hash,'hex'),
+    b.block_no,
+    t.amount::text,
+    sa.view
+  FROM treasury AS t
+    LEFT JOIN tx ON t.tx_id = tx.id
+    INNER JOIN block AS b ON tx.block_id = b.id
+    LEFT JOIN stake_address AS sa ON sa.id = t.addr_id;
+$$;
+
+COMMENT ON FUNCTION grest.treasury_withdrawals IS 'A list of withdrawals made from treasury'; --noqa: LT01

files/grest/rpc/01_cached_tables/asset_info_cache.sql

@@ -30,7 +30,7 @@ BEGIN
     WHERE state = 'active'
       AND query ILIKE '%grest.asset_info_cache_update%'
       AND datname = (SELECT current_database())
-  ) THEN 
+  ) THEN
     RAISE EXCEPTION 'Previous asset_info_cache_update query still running but should have completed! Exiting...';
   END IF;
 
@@ -56,7 +56,7 @@ BEGIN
     tx_mint_meta AS (
       SELECT
         mtm.ident,
-        MIN(mtm.tx_id) AS first_mint_tx_id, 
+        MIN(mtm.tx_id) AS first_mint_tx_id,
         MAX(mtm.tx_id) AS last_mint_tx_id
       FROM ma_tx_mint AS mtm
       INNER JOIN tx_metadata AS tm ON tm.tx_id = mtm.tx_id
@@ -74,7 +74,7 @@ BEGIN
     tx_mint_nometa AS (
       SELECT
         mtm.ident,
-        MIN(mtm.tx_id) AS first_mint_tx_id, 
+        MIN(mtm.tx_id) AS first_mint_tx_id,
         MAX(mtm.tx_id) AS last_mint_tx_id
       FROM ma_tx_mint AS mtm
       LEFT JOIN tx_mint_meta ON tx_mint_meta.ident = mtm.ident
@@ -111,7 +111,7 @@ BEGIN
       FROM tx_mint_nometa
     )
 
-  INSERT INTO grest.asset_info_cache  
+  INSERT INTO grest.asset_info_cache
     SELECT
       ma.id,
       MIN(B.time) AS creation_time,

files/grest/rpc/01_cached_tables/asset_registry_cache.sql

@@ -38,7 +38,7 @@ BEGIN
     decimals
   )
   VALUES(
-    _asset_policy, 
+    _asset_policy,
     _asset_name,
     _name,
     _description,

files/grest/rpc/01_cached_tables/epoch_info_cache.sql

@@ -9,37 +9,8 @@ CREATE TABLE IF NOT EXISTS grest.epoch_info_cache (
   i_total_rewards lovelace,
   i_avg_blk_reward lovelace,
   i_last_tx_id bigint,
-  p_min_fee_a word31type,
-  p_min_fee_b word31type,
-  p_max_block_size word31type,
-  p_max_tx_size word31type,
-  p_max_bh_size word31type,
-  p_key_deposit lovelace,
-  p_pool_deposit lovelace,
-  p_max_epoch word31type,
-  p_optimal_pool_count word31type,
-  p_influence double precision,
-  p_monetary_expand_rate double precision,
-  p_treasury_growth_rate double precision,
-  p_decentralisation double precision,
-  p_extra_entropy text,
-  p_protocol_major word31type,
-  p_protocol_minor word31type,
-  p_min_utxo_value lovelace,
-  p_min_pool_cost lovelace,
   p_nonce text,
-  p_block_hash text,
-  p_cost_models character varying,
-  p_price_mem double precision,
-  p_price_step double precision,
-  p_max_tx_ex_mem word64type,
-  p_max_tx_ex_steps word64type,
-  p_max_block_ex_mem word64type,
-  p_max_block_ex_steps word64type,
-  p_max_val_size word64type,
-  p_collateral_percent word31type,
-  p_max_collateral_inputs word31type,
-  p_coins_per_utxo_size lovelace
+  p_block_hash text
 );
 
 COMMENT ON TABLE grest.epoch_info_cache IS 'Contains detailed info for epochs including protocol parameters';
@@ -105,6 +76,20 @@ BEGIN
   DELETE FROM grest.epoch_info_cache
   WHERE epoch_no >= _epoch_no_to_insert_from;
 
+  DROP TABLE IF EXISTS last_tx_id_subset;
+  CREATE TEMP TABLE last_tx_id_subset (
+    epoch_no bigint,
+    tx_id bigint
+  );
+
+  INSERT INTO last_tx_id_subset
+    SELECT b.epoch_no, MAX(tx.id)
+    FROM block AS b
+    INNER JOIN tx ON tx.block_id = b.id
+    WHERE b.block_no IS NOT NULL
+      AND b.tx_count != 0
+    GROUP BY b.epoch_no;
+
   INSERT INTO grest.epoch_info_cache
     SELECT DISTINCT ON (b.time)
       e.no AS epoch_no,
@@ -115,45 +100,20 @@ BEGIN
       EXTRACT(EPOCH FROM e.start_time) AS i_first_block_time,
       EXTRACT(EPOCH FROM e.end_time) AS i_last_block_time,
       CASE -- populated in epoch n + 2
-        WHEN e.no <= _curr_epoch - 2 THEN reward_pot.amount 
+        WHEN e.no <= _curr_epoch - 2 THEN reward_pot.amount
         ELSE NULL
       END AS i_total_rewards,
       CASE -- populated in epoch n + 2
         WHEN e.no <= _curr_epoch THEN ROUND(reward_pot.amount / e.blk_count)
         ELSE NULL
-      END AS i_avg_blk_reward, 
-      last_tx.tx_id AS i_last_tx_id,
-      ep.min_fee_a AS p_min_fee_a,
-      ep.min_fee_b AS p_min_fee_b,
-      ep.max_block_size AS p_max_block_size,
-      ep.max_tx_size AS p_max_tx_size,
-      ep.max_bh_size AS p_max_bh_size,
-      ep.key_deposit AS p_key_deposit,
-      ep.pool_deposit AS p_pool_deposit,
-      ep.max_epoch AS p_max_epoch,
-      ep.optimal_pool_count AS p_optimal_pool_count,
-      ep.influence AS p_influence,
-      ep.monetary_expand_rate AS p_monetary_expand_rate,
-      ep.treasury_growth_rate AS p_treasury_growth_rate,
-      ep.decentralisation AS p_decentralisation,
-      ENCODE(ep.extra_entropy, 'hex') AS p_extra_entropy,
-      ep.protocol_major AS p_protocol_major,
-      ep.protocol_minor AS p_protocol_minor,
-      ep.min_utxo_value AS p_min_utxo_value,
-      ep.min_pool_cost AS p_min_pool_cost,
+      END AS i_avg_blk_reward,
+      (
+        SELECT MAX(tx_id)
+        FROM last_tx_id_subset
+        WHERE epoch_no <= e.no
+      ) AS i_last_tx_id,
       ENCODE(ep.nonce, 'hex') AS p_nonce,
-      ENCODE(b.hash, 'hex') AS p_block_hash,
-      cm.costs AS p_cost_models,
-      ep.price_mem AS p_price_mem,
-      ep.price_step AS p_price_step,
-      ep.max_tx_ex_mem AS p_max_tx_ex_mem,
-      ep.max_tx_ex_steps AS p_max_tx_ex_steps,
-      ep.max_block_ex_mem AS p_max_block_ex_mem,
-      ep.max_block_ex_steps AS p_max_block_ex_steps,
-      ep.max_val_size AS p_max_val_size,
-      ep.collateral_percent AS p_collateral_percent,
-      ep.max_collateral_inputs AS p_max_collateral_inputs,
-      ep.coins_per_utxo_size AS p_coins_per_utxo_size
+      ENCODE(b.hash, 'hex') AS p_block_hash
     FROM epoch AS e
     LEFT JOIN epoch_param AS ep ON ep.epoch_no = e.no
     LEFT JOIN cost_model AS cm ON cm.id = ep.cost_model_id
@@ -167,14 +127,6 @@ BEGIN
         GROUP BY
           e.no
       ) AS reward_pot ON TRUE
-    LEFT JOIN LATERAL (
-        SELECT MAX(tx.id) AS tx_id
-        FROM block AS b
-        INNER JOIN tx ON tx.block_id = b.id
-        WHERE b.epoch_no <= e.no
-          AND b.block_no IS NOT NULL
-          AND b.tx_count != 0
-      ) AS last_tx ON TRUE
     WHERE e.no >= _epoch_no_to_insert_from
     ORDER BY
       b.time ASC,

files/grest/rpc/01_cached_tables/pool_history_cache.sql

@@ -39,9 +39,9 @@ BEGIN
   IF (
     SELECT COUNT(key) != 1
     FROM GREST.CONTROL_TABLE
-    WHERE key = 'epoch_info_cache_last_updated'
+    WHERE key = 'last_active_stake_validated_epoch'
   ) THEN
-    RAISE EXCEPTION 'Epoch Info Cache not yet populated! Exiting...';
+    RAISE EXCEPTION 'Active stake cache not yet populated! Exiting...';
   END IF;
 
   IF _epoch_no_to_insert_from IS NULL THEN
@@ -153,9 +153,9 @@ BEGIN
         ROUND(
           (act.amount / (
             SELECT supply::bigint / (
-                SELECT eic.p_optimal_pool_count
-                FROM grest.epoch_info_cache AS eic
-                WHERE eic.epoch_no = act.epoch_no
+                SELECT ep.optimal_pool_count
+                FROM epoch_param AS ep
+                WHERE ep.epoch_no = act.epoch_no
               )
             FROM grest.totals (act.epoch_no)
             ) * 100

files/grest/rpc/01_cached_tables/pool_info_cache.sql

@@ -55,7 +55,7 @@ BEGIN
   ORDER BY pr.id
   LIMIT 1;
 
-  IF _retiring_epoch IS NULL THEN 
+  IF _retiring_epoch IS NULL THEN
     _pool_status := 'registered';
   ELSIF _retiring_epoch > _current_epoch_no THEN
     _pool_status := 'retiring';
@@ -69,7 +69,7 @@ BEGIN
     tx_hash,
     block_time,
     pool_hash_id,
-    pool_id_bech32, 
+    pool_id_bech32,
     pool_id_hex,
     active_epoch_no,
     vrf_key_hash,
@@ -88,7 +88,7 @@ BEGIN
     SELECT
       _tx_id,
       _update_id,
-      encode(tx.hash::bytea, 'hex'), 
+      encode(tx.hash::bytea, 'hex'),
       EXTRACT(EPOCH FROM b.time),
       _hash_id,
       ph.view,
@@ -174,7 +174,7 @@ BEGIN
   ORDER BY pr.id
   LIMIT 1;
 
-  IF _retiring_epoch IS NULL THEN 
+  IF _retiring_epoch IS NULL THEN
     _pool_status := 'registered';
   ELSIF _retiring_epoch > _current_epoch_no THEN
     _pool_status := 'retiring';
@@ -222,8 +222,8 @@ BEGIN
     END IF;
 
   ELSIF (tg_table_name = 'pool_relay') THEN
-    SELECT pic.id INTO _latest_pool_update_id 
-    FROM grest.pool_info_cache AS pic 
+    SELECT pic.id INTO _latest_pool_update_id
+    FROM grest.pool_info_cache AS pic
     INNER JOIN public.pool_update AS pu ON pu.hash_id = pic.pool_hash_id AND pu.registered_tx_id = pic.tx_id
     WHERE pu.id = new.update_id;
 

files/grest/rpc/01_cached_tables/stake_distribution_cache.sql

@@ -266,7 +266,7 @@ BEGIN
   SELECT (_current_block_height - _last_update_block_height) INTO _last_update_block_diff;
   -- Do nothing until there is a 180 blocks difference in height - 60 minutes theoretical time
   -- 185 in check because last block height considered is 5 blocks behind tip
-  
+
   Raise NOTICE 'Last stake distribution update was % blocks ago...',
     _last_update_block_diff;
     IF (_last_update_block_diff >= 180

files/grest/rpc/02_indexes/13_1_00.sql

@@ -5,7 +5,7 @@ CREATE UNIQUE INDEX IF NOT EXISTS unique_col_txout ON public.collateral_tx_out U
 CREATE UNIQUE INDEX IF NOT EXISTS unique_delegation ON public.delegation USING btree (tx_id, cert_index);
 CREATE UNIQUE INDEX IF NOT EXISTS unique_epoch_param ON public.epoch_param USING btree (epoch_no, block_id);
 CREATE UNIQUE INDEX IF NOT EXISTS unique_ma_tx_mint ON public.ma_tx_mint USING btree (ident, tx_id);
-CREATE UNIQUE INDEX IF NOT EXISTS unique_ma_tx_out ON public.ma_tx_out USING btree (ident, tx_out_id);
+CREATE UNIQUE INDEX IF NOT EXISTS unique_ma_tx_out ON public.ma_tx_out USING btree (ident, tx_out_id DESC);
 CREATE UNIQUE INDEX IF NOT EXISTS unique_param_proposal ON public.param_proposal USING btree (key, registered_tx_id);
 CREATE UNIQUE INDEX IF NOT EXISTS unique_pool_owner ON public.pool_owner USING btree (addr_id, pool_update_id);
 CREATE UNIQUE INDEX IF NOT EXISTS unique_pool_relay ON public.pool_relay USING btree (update_id, ipv4, ipv6, dns_name);