Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

scripts: add convenience script for creating table #348

Merged
merged 5 commits into from
Jul 8, 2019

Conversation

neverchanje
Copy link
Contributor

@neverchanje neverchanje commented Jul 5, 2019

What problem does this PR solve?

Add convenience script for creating tables. Users can specify the table owners and their department, the write throttling, as well as the required attributes like 'partition count' and 'table name'.

If the table is created before, users can still reset the table's environments.

This script is at an early stage.

What is changed and how it works?

I use python for this script, reusing the utilities provided by py_utils/lib.py.

Check List

Tests

  • Manual test (add detailed scripts or steps below)
➜  pegasus git:(create) ✗ ./scripts/create_table.py -t ai_user_info \
                          -d "云平台部-存储平台-KV系统组" \
                          -u "wutao1&qinzuoyan" \
                          -c c3tst-dup2 \
                          -w "2000*delay*100" \
                          -p 16
Confirm to create table:
{
    "cluster": "c3tst-dup2", 
    "depart": "云平台部-存储平台-KV系统组", 
    "partition_count": 16, 
    "table": "ai_user_info", 
    "user": "wutao1&qinzuoyan", 
    "write_throttling": "2000*delay*100"
}
 [y/N]: y
Creating table ai_user_info...
New value of business.info=depart=云平台部-存储平台-KV系统组,user=wutao1&qinzuoyan
New value of replica.write_throttling=2000*delay*100
➜  pegasus git:(create) ✗ ./scripts/create_table.py -t ai_user_info \
                          -d "云平台部-存储平台-KV系统组" \
                          -u "wutao1&qinzuoyan" \
                          -c c3tst-dup2 \
                          -w "2000*delay*100" \
                          -p 16
Confirm to create table:
{
    "cluster": "c3tst-dup2", 
    "depart": "云平台部-存储平台-KV系统组", 
    "partition_count": 16, 
    "table": "ai_user_info", 
    "user": "wutao1&qinzuoyan", 
    "write_throttling": "2000*delay*100"
}
 [y/N]: y
Success: table "ai_user_info" exists
New value of business.info=depart=云平台部-存储平台-KV系统组,user=wutao1&qinzuoyan
Old value of business.info=depart=云平台部-存储平台-KV系统组,user=wutao1&qinzuoyan
Success: business.info keeps unchanged
New value of replica.write_throttling=2000*delay*100
Old value of replica.write_throttling=2000*delay*100
Success: replica.write_throttling keeps unchanged
➜  pegasus git:(create) ✗ ./scripts/create_table.py --help
Usage: create_table.py [OPTIONS]

Options:
  -t, --table TEXT               Name of the table you want to create.
                                 [required]
  -d, --depart TEXT              Department of the table owner. If there are
                                 more than one levels of department, use '-'
                                 to concatenate them.  [required]
  -u, --user TEXT                The table owner. If there are more than one
                                 owners, use '&' to concatenate them.
                                 [required]
  -c, --cluster TEXT             The cluster name. Where you want to place the
                                 table.  [required]
  -p, --partition_count INTEGER  The partition count of the table. Empty means
                                 no create.
  -w, --write_throttling TEXT    {delay_qps_threshold}*delay*{delay_ms},{rejec
                                 t_qps_threshold}*reject*{delay_ms_before_reje
                                 ct}
  --help                         Show this message and exit.

Related changes

  • Need to cherry-pick to the release branch
  • Need to update the documentation
  • Need to be included in the release note

@acelyc111
Copy link
Member

python脚本的一些警告处理下, IDE一般都能提示出来

default="",
callback=validate_param_throttle,
help="{delay_qps_threshold}*delay*{delay_ms},{reject_qps_threshold}*reject*{delay_ms_before_reject}")
def main(table, depart, user, cluster, parts, throttle):
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

参数顺序可以调整下, 从广到细, 从required到optional

@qinzuoyan qinzuoyan merged commit 2887873 into apache:master Jul 8, 2019
@neverchanje neverchanje deleted the create branch July 8, 2019 13:46
@neverchanje neverchanje mentioned this pull request Jul 12, 2019
neverchanje pushed a commit to neverchanje/pegasus that referenced this pull request Jul 13, 2019
Former-commit-id: 747f2646191e94b40aa104b9d6d5f0cdbe9fd006 [formerly 2887873]
Former-commit-id: 49972261a02566cb64e2850de3292dde5167b9f5
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants