Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add RPC to batch retrieve gateway connection stats #5536

Merged
merged 30 commits into from
Jul 18, 2022

Conversation

KrishnaIyer
Copy link
Member

@KrishnaIyer KrishnaIyer commented Jun 15, 2022

Summary

Closes #5398

Changes

  • Add new BatchGetGatewayConnectionStats.
  • Implement GS registry interface
  • Add CLI support

Testing

  • UT
  • E2E
End to End test with two (simulated) gateways
----------------------------------------------

1. Gateways not created

$./ttn-lw-cli gtw get-connection-stats test-gtw-2 --gateway-ids test-gtw,test-gtw-1
WARN    Using insecure connection to OAuth server
WARN    Using insecure connection to API
error:pkg/identityserver/gormstore:gateway_not_found (gateway `test-gtw-1` not found)
    gateway_id=test-gtw-1
    correlation_id=2905f102c2aa498aaa53223bd22dcf30

2. Gateways created but not connected

$ ./ttn-lw-cli gtw get-connection-stats test-gtw-2 --gateway-ids test-gtw,test-gtw-1
WARN    Using insecure connection to OAuth server
WARN    Using insecure connection to API
{}

3. Gateway 1 connected

$./ttn-lw-cli gtw get-connection-stats test-gtw-2 --gateway-ids test-gtw,test-gtw-1
WARN    Using insecure connection to OAuth server
WARN    Using insecure connection to API
{
  "entries": {
    "test-gtw": {
      "connected_at": "2022-07-13T11:44:14.364400Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    }
  }
}

4. Both gateways connected

$ ./ttn-lw-cli gtw get-connection-stats test-gtw-2 --gateway-ids test-gtw,test-gtw-1
WARN    Using insecure connection to OAuth server
WARN    Using insecure connection to API
{
  "entries": {
    "test-gtw": {
      "connected_at": "2022-07-13T11:44:14.364400Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    },
    "test-gtw-1": {
      "connected_at": "2022-07-13T11:44:34.619971Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    }
  }
}


5. Gateway 2 disconnected

$ ./ttn-lw-cli gtw get-connection-stats test-gtw-2 --gateway-ids test-gtw,test-gtw-1
WARN    Using insecure connection to OAuth server
WARN    Using insecure connection to API
{
  "entries": {
    "test-gtw": {
      "connected_at": "2022-07-13T11:44:14.364400Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    },
    "test-gtw-1": {
      "disconnected_at": "2022-07-13T11:44:45.462092Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    }
  }
}

6. Gateway 1 disconnected

$ ./ttn-lw-cli gtw get-connection-stats test-gtw-2 --gateway-ids test-gtw,test-gtw-1
WARN    Using insecure connection to OAuth server
WARN    Using insecure connection to API
{
  "entries": {
    "test-gtw": {
      "disconnected_at": "2022-07-13T11:44:49.544901Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    },
    "test-gtw-1": {
      "disconnected_at": "2022-07-13T11:44:45.462092Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    }
  }
}


7. Regression

$ ./ttn-lw-cli gtw get-connection-stats test-gtw
WARN    Using insecure connection to OAuth server
WARN    Using insecure connection to API
{
  "disconnected_at": "2022-07-13T11:44:49.544901Z",
  "protocol": "ws",
  "gateway_remote_address": {
    "ip": "127.0.0.1"
  }
}


8. Filter on paths

./ttn-lw-cli gtw get-connection-stats --gateway-ids test-gtw,test-gtw-1 --paths protocol
WARN  Using insecure connection to OAuth server
WARN  Using insecure connection to API
{
  "entries": {
    "test-gtw": {
      "protocol": "ws"
    },
    "test-gtw-1": {
      "protocol": "ws"
    }
  }
}

$ ./ttn-lw-cli gtw get-connection-stats --gateway-ids test-gtw,test-gtw-1 --paths protocol,disconnected_at
WARN  Using insecure connection to OAuth server
WARN  Using insecure connection to API
{
  "entries": {
    "test-gtw": {
      "disconnected_at": "2022-07-13T11:44:49.544901Z",
      "protocol": "ws"
    },
    "test-gtw-1": {
      "disconnected_at": "2022-07-13T11:44:45.462092Z",
      "protocol": "ws"
    }
  }
}


9. Check invalid paths
 
./ttn-lw-cli gtw get-connection-stats --gateway-ids test-gtw,test-gtw-1 --paths protocol,disconnected_at,wrong_path
WARN  Using insecure connection to OAuth server
WARN  Using insecure connection to API
error:pkg/rpcmiddleware/validator:field_mask_paths (forbidden path(s) in field mask)
    forbidden_paths=[wrong_path]
    correlation_id=ceebc010321842b4b35f310f67db2b98
Regressions

Possible regression in get-connection-stats CLI command. Test case 7 covers this.

Checklist

  • Scope: The referenced issue is addressed, there are no unrelated changes.
  • Compatibility: The changes are backwards compatible with existing API, storage, configuration and CLI, according to the compatibility commitments in README.md for the chosen target branch.
  • Documentation: Relevant documentation is added or updated.
  • Changelog: Significant features, behavior changes, deprecations and fixes are added to CHANGELOG.md.
  • Commits: Commit messages follow guidelines in CONTRIBUTING.md, there are no fixup commits left.

@KrishnaIyer KrishnaIyer added this to the v3.20.1 milestone Jun 15, 2022
@KrishnaIyer KrishnaIyer self-assigned this Jun 15, 2022
@github-actions github-actions bot added c/gateway server This is related to the Gateway Server compat/api This could affect API compatibility compat/db This could affect Database compatibility ui/cli This is related to ttn-lw-cli labels Jun 15, 2022
@KrishnaIyer KrishnaIyer modified the milestones: v3.20.1, v3.20.2 Jun 16, 2022
@KrishnaIyer KrishnaIyer force-pushed the feature/5398-batch-retrieval branch 3 times, most recently from 7a5d114 to 415576a Compare July 11, 2022 09:56
@KrishnaIyer KrishnaIyer marked this pull request as ready for review July 11, 2022 11:06
cmd/ttn-lw-cli/commands/gateways.go Outdated Show resolved Hide resolved
pkg/gatewayserver/redis/registry.go Outdated Show resolved Hide resolved
@KrishnaIyer KrishnaIyer force-pushed the feature/5398-batch-retrieval branch from 415576a to f10ca88 Compare July 12, 2022 07:11
@KrishnaIyer KrishnaIyer requested a review from adriansmares July 12, 2022 07:18
Copy link
Contributor

@adriansmares adriansmares left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool, now it is really clean.

pkg/gatewayserver/redis/registry.go Outdated Show resolved Hide resolved
pkg/gatewayserver/redis/registry.go Outdated Show resolved Hide resolved
@KrishnaIyer KrishnaIyer force-pushed the feature/5398-batch-retrieval branch from f2e5577 to 6f852c3 Compare July 12, 2022 09:11
api/gateway.proto Outdated Show resolved Hide resolved
cmd/ttn-lw-cli/commands/gateways.go Outdated Show resolved Hide resolved
@KrishnaIyer KrishnaIyer force-pushed the feature/5398-batch-retrieval branch from 73fe929 to 7624cd0 Compare July 13, 2022 07:53
@KrishnaIyer KrishnaIyer requested a review from htdvisser July 13, 2022 07:53
Copy link
Member

@johanstokking johanstokking left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good but few questions

api/gateway.proto Outdated Show resolved Hide resolved
api/gatewayserver.proto Outdated Show resolved Hide resolved
api/gateway.proto Outdated Show resolved Hide resolved
cmd/ttn-lw-cli/commands/gateways.go Outdated Show resolved Hide resolved
cmd/ttn-lw-cli/commands/gateways.go Outdated Show resolved Hide resolved
pkg/component/grpc_test.go Show resolved Hide resolved
@KrishnaIyer KrishnaIyer force-pushed the feature/5398-batch-retrieval branch from 7624cd0 to abb4508 Compare July 13, 2022 12:52
CHANGELOG.md Outdated Show resolved Hide resolved
@KrishnaIyer KrishnaIyer force-pushed the feature/5398-batch-retrieval branch from ae2133c to f4d2055 Compare July 13, 2022 13:19
cmd/ttn-lw-cli/commands/gateways.go Outdated Show resolved Hide resolved
cmd/ttn-lw-cli/commands/gateways.go Outdated Show resolved Hide resolved
@KrishnaIyer
Copy link
Member Author

Test for CLI Combinations

$ ./ttn-lw-cli gtw get-connection-stats test-gtw
WARN  Using insecure connection to OAuth server
WARN  Using insecure connection to API
{
  "disconnected_at": "2022-07-13T11:44:49.544901Z",
  "protocol": "ws",
  "gateway_remote_address": {
    "ip": "127.0.0.1"
  }
}


$ ./ttn-lw-cli gtw get-connection-stats --gateway-id test-gtw
WARN  Using insecure connection to OAuth server
WARN  Using insecure connection to API
{
  "disconnected_at": "2022-07-13T11:44:49.544901Z",
  "protocol": "ws",
  "gateway_remote_address": {
    "ip": "127.0.0.1"
  }
}

$ ./ttn-lw-cli gtw get-connection-stats --gateway-ids test-gtw
WARN  Using insecure connection to OAuth server
WARN  Using insecure connection to API
{
  "disconnected_at": "2022-07-13T11:44:49.544901Z",
  "protocol": "ws",
  "gateway_remote_address": {
    "ip": "127.0.0.1"
  }
}


$ ./ttn-lw-cli gtw get-connection-stats --gateway-ids test-gtw,test-gtw-1
WARN  Using insecure connection to OAuth server
WARN  Using insecure connection to API
{
  "entries": {
    "test-gtw": {
      "disconnected_at": "2022-07-13T11:44:49.544901Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    },
    "test-gtw-1": {
      "disconnected_at": "2022-07-13T11:44:45.462092Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    }
  }
}

$ ./ttn-lw-cli gtw get-connection-stats test-gtw test-gtw-1
WARN  Using insecure connection to OAuth server
WARN  Using insecure connection to API
{
  "entries": {
    "test-gtw": {
      "disconnected_at": "2022-07-13T11:44:49.544901Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    },
    "test-gtw-1": {
      "disconnected_at": "2022-07-13T11:44:45.462092Z",
      "protocol": "ws",
      "gateway_remote_address": {
        "ip": "127.0.0.1"
      }
    }
  }
}


$ ./ttn-lw-cli gtw get-connection-stats
WARN  Using insecure connection to OAuth server
WARN  Using insecure connection to API
error:cmd/ttn-lw-cli/commands:no_gateway_id (no gateway ID set)
    correlation_id=640a684b72354a5b93db492076374666

@KrishnaIyer KrishnaIyer force-pushed the feature/5398-batch-retrieval branch from b6434fa to 37349ef Compare July 18, 2022 14:50
@KrishnaIyer KrishnaIyer merged commit 8eb17b3 into v3.21 Jul 18, 2022
@KrishnaIyer KrishnaIyer deleted the feature/5398-batch-retrieval branch July 18, 2022 15:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
c/gateway server This is related to the Gateway Server c/identity server This is related to the Identity Server compat/api This could affect API compatibility compat/db This could affect Database compatibility ui/cli This is related to ttn-lw-cli
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Add gateway stats retrieval batch RPC
6 participants