Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

支持Oracle数据修改备份SQL功能,支持explain,with查询语法,对Oracle表结构和schema显示进行排序,支持sql tuning advisor,Oracle支持执行计划查看,支持Oracle update/insert/delete/create table/create index的语法SQL审核 #701

Merged
merged 21 commits into from
Apr 25, 2020

Conversation

jan-song
Copy link
Contributor

@jan-song jan-song commented Apr 6, 2020

相关issue #666 #700

@request-info
Copy link

request-info bot commented Apr 6, 2020

你好!感谢你反馈的问题/bug,但是你的描述好像是空的,我们需要你完整的信息,这样才能帮你解决问题 如果不知道怎么写,在新建issue的时候有若干个模板可供选择,祝好!

@jan-song
Copy link
Contributor Author

jan-song commented Apr 6, 2020

1、生成回滚SQL,执行用户需要有grant select any transaction 权限,需要有grant execute on dbms_logmnr 权限
2、数据库需开启最小化附加日志alter database add supplemental log data;
3、需为归档模式;开启附件日志会增加redo日志量,一般不会有多大影响,需评估归档磁盘空间,redo磁盘IO性能
4、sql tuning advisor需要grant advisor权限

sql/engines/oracle.py Outdated Show resolved Hide resolved
sql/engines/oracle.py Outdated Show resolved Hide resolved
sql/engines/oracle.py Outdated Show resolved Hide resolved
@LeoQuote
Copy link
Collaborator

LeoQuote commented Apr 7, 2020

问题不大, 考虑下我的这几个修改意见, 不要重开pr, 直接在你的分支上提交推送就行.

@LeoQuote LeoQuote added area/engine enhancement New feature or request and removed 请提供更多信息 labels Apr 7, 2020
sql/engines/oracle.py Outdated Show resolved Hide resolved
sql/engines/oracle.py Outdated Show resolved Hide resolved
@LeoQuote
Copy link
Collaborator

LeoQuote commented Apr 7, 2020

😅 单元测试也要改一下, 这个比我上面提的要重要一些, 我看还有一些比较简单的引用未声明变量的问题, 一定要修. https://travis-ci.org/github/hhyo/Archery/builds/672116656 看一下这里, 失败的几个都要改到成功, 你本地也可以跑 python manage.py test -v 3

感谢贡献.

@LeoQuote
Copy link
Collaborator

LeoQuote commented Apr 7, 2020

最近 travis-ci 是不是不好使了, 感觉master 上好几次构建都没把状态同步过来, 不行就换 circleci 看看.

@hhyo
Copy link
Owner

hhyo commented Apr 8, 2020

这个pr还需要@ericruan-cn 看下,感觉回滚了他前面的提交

@ericruan-cn
Copy link
Collaborator

ericruan-cn commented Apr 8, 2020 via email

@LeoQuote
Copy link
Collaborator

LeoQuote commented Apr 8, 2020

这个...你解决一下吧,他的都merge进master了

@LeoQuote
Copy link
Collaborator

LeoQuote commented Apr 9, 2020

最好rebase下 master 然后force push,有一些把之前修改回滚的注意一下。

具体可以看这里的教程 : https://wizardforcel.gitbooks.io/atlassian-git-tut/content/5.1-%E4%BB%A3%E7%A0%81%E5%90%88%E5%B9%B6%EF%BC%9AMerge%E3%80%81Rebase-%E7%9A%84%E9%80%89%E6%8B%A9.html

@hhyo hhyo requested a review from ericruan-cn April 13, 2020 04:08
if close_conn:
self.close()
return execute_result

def backup(self,workflow_id,cursor,begin_time,end_time):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

几个建议和问题:

  1. 参数和上面的用法没对上, 这里是必选参数, 上面的是可选参数.
  2. 这里建议直接传 workflow 对象过去
  3. 这里的cursor 是不是必须传的, 如果是必须传, 建议改一个有辨识度的名字, 和 backup 的cursor 区分开来, 如果不是必须传, 可以用self.get_connection get connection 然后再拿到cursor

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

为了获取Oracle当前的会话信息cursor是必须传的 ,这个cursor就是执行的cursor,必须是同一个会话,才能抓取到回滚会话的信息,当然也可以关闭执行SQL会话,将需要的会话信息查询出来做参数传过去,这样备份需要新开Oracle的连接会话,多一次fork process开销,代码也更麻烦

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

参数的问题太专业这边没看懂你指的是什么,一直做 的Oracle运维,没做过研发,抱歉

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

是的,这个要想获得备份语句,需要和已经执行的上线sql在同一会话里面,要不然不方便从v$logmnr_contents 获取到undo sql

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

def backup(self, workflow, current_cursor, begin_time, end_time):

self.backup(workflow, cursor, begin_time, end_time)

# 创建备份库连接
try:
conn = self.get_backup_connection()
cur = conn.cursor()
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这里改成 backup_cursor = conn.cursor() 后面 curcursor 会看花眼

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

本地部署的代码(1.7.8版本)这边加上了Oracle执行计划查看支持(pr已经实现),Oracle数据修改SQL审核(explain plan for 实现),改动的太多了,我按上面要求重新整理测试后再提交一份吧,这个版本上线的Oracle存储过程和本地1.7.8版本的改动不兼容,后续打算把Oracle的sql tuning advisor也集成到SQL优化功能上去,前端代码暂时没看懂,哪位大神能给点思路?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

那块前端代码比较乱,看你需要展示什么信息,如果能做成通用的格式,可以扩充一下现有的engine,具体可以添加一下我的个人微信

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

好吧, 下次你提的时候记得在你的fork里新开一个分支, master 我们没法帮你修改, 一般会有保护.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oracle的sql tuning advisor是Oracle自带的存储过程,执行普通的查询SQL就能获取到SQL优化报告,展示信息其实就是个查询SQL的返回结果。Oracle执行计划获取和这个思路是一样的,执行Oracle特定的存储过程后返回一个clob类型的查询结果。包括Oracle实例性能诊断优化建议报告(ADDM),AWR报告,都是可以通过调用Oracle存储过程的形式获取到,如果要丰富Oracle的功能确实需要做个通用格式,能力有限,完全没思路要怎么弄

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

那实现和返回格式都参考sql优化内的执行计划展示就行

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oracle要有新功能了,好期待。
我准备五一研究一下archery的源代码,希望能看懂。

@LeoQuote LeoQuote changed the base branch from master to oracle April 21, 2020 13:45
@LeoQuote
Copy link
Collaborator

LeoQuote commented Apr 22, 2020

我把这个pr 的base 改成orable 了, 差不多改好后就merge , merge 完了我帮你整理一下再merge 进 master

Oracle执行计划查看支持
oracle 新增SQL tunning advisor
添加Oracle sql tuning advisor
oracle sql tuning advisor支持
添加Oracle SQL类型判断
支持Oracle select、update\delete\insert\create table\create index的语法审核。支持数据修改备份功能。支持执行计划查看。修复查询bug
@jan-song jan-song changed the title 支持Oracle数据修改备份SQL功能,支持explain,with查询语法,对Oracle表结构和schema显示进行排序 支持Oracle数据修改备份SQL功能,支持explain,with查询语法,对Oracle表结构和schema显示进行排序,支持sql tuning advisor Apr 24, 2020
@jan-song jan-song changed the title 支持Oracle数据修改备份SQL功能,支持explain,with查询语法,对Oracle表结构和schema显示进行排序,支持sql tuning advisor 支持Oracle数据修改备份SQL功能,支持explain,with查询语法,对Oracle表结构和schema显示进行排序,支持sql tuning advisor,Oracle支持支持计划查看 Apr 24, 2020
@jan-song jan-song changed the title 支持Oracle数据修改备份SQL功能,支持explain,with查询语法,对Oracle表结构和schema显示进行排序,支持sql tuning advisor,Oracle支持支持计划查看 支持Oracle数据修改备份SQL功能,支持explain,with查询语法,对Oracle表结构和schema显示进行排序,支持sql tuning advisor,Oracle支持执行计划查看 Apr 24, 2020
@jan-song
Copy link
Contributor Author

改好了,大神们一起看看有啥问题没,sql tuning advisor哪位帮忙优化一下显示效果,有些缩进丢失了,Oracle客户端sqlplus和其它官方工具连接没这个问题

@jan-song jan-song changed the title 支持Oracle数据修改备份SQL功能,支持explain,with查询语法,对Oracle表结构和schema显示进行排序,支持sql tuning advisor,Oracle支持执行计划查看 支持Oracle数据修改备份SQL功能,支持explain,with查询语法,对Oracle表结构和schema显示进行排序,支持sql tuning advisor,Oracle支持执行计划查看,支持Oracle update/insert/delete/create table/create index的语法SQL审核 Apr 24, 2020
@czxin788
Copy link
Contributor

好期待,对oracle要有增强功能了。感谢jan-song、hhyo和LeoQuote的无私奉献。

@hhyo
Copy link
Owner

hhyo commented Apr 25, 2020

diff的结果有点奇怪,能rebase一下吗?

@LeoQuote
Copy link
Collaborator

@hhyo 我改了base,直接merge进来自己rebase吧。

@czxin788
Copy link
Contributor

你好,我给执行sql的用户授予dba权限,是可以在archery里面看到生产的回滚语句;
但是我把dba角色去掉,然后授予如下权限,发现在archery里面不能生产回滚sql;

grant connect,resource to czx;
grant select any transaction to czx;
grant execute on dbms_logmnr to czx;
请问,我还需要授予什么角色

@jan-song
Copy link
Contributor Author

@czxin788
权限没细测试过,暂时没环境测试,可以看看archery日志报错,这边是按如下权限授权的
grant execute on dbms_logmnr to deploy;
grant EXECUTE ANY INDEXTYPE to deploy;
grant DROP ANY PROCEDURE to deploy;
grant DROP ANY VIEW to deploy;
grant DROP ANY INDEX to deploy;
grant INSERT ANY TABLE to deploy;
grant UNDER ANY VIEW to deploy;
grant DROP ANY INDEXTYPE to deploy;
grant ALTER ANY INDEXTYPE to deploy;
grant CREATE MATERIALIZED VIEW to deploy;
grant CREATE SESSION to deploy;
grant DEBUG ANY PROCEDURE to deploy;
grant ALTER ANY PROCEDURE to deploy;
grant CREATE ANY TABLE to deploy;
grant FLASHBACK ANY TABLE to deploy;
grant CREATE ANY INDEXTYPE to deploy;
grant CREATE PROCEDURE to deploy;
grant BACKUP ANY TABLE to deploy;
grant CREATE TABLE to deploy;
grant MERGE ANY VIEW to deploy;
grant CREATE ANY MATERIALIZED VIEW to deploy;
grant ALTER ANY INDEX to deploy;
grant COMMENT ANY TABLE to deploy;
grant MANAGE TABLESPACE to deploy;
grant SELECT ANY DICTIONARY to deploy;
grant UNDER ANY TABLE to deploy;
grant CREATE INDEXTYPE to deploy;
grant DROP ANY MATERIALIZED VIEW to deploy;
grant CREATE ANY VIEW to deploy;
grant CREATE VIEW to deploy;
grant DELETE ANY TABLE to deploy;
grant SELECT ANY TABLE to deploy;
grant DROP ANY TABLE to deploy;
grant ALTER TABLESPACE to deploy;
grant CREATE TABLESPACE to deploy;
grant ALTER ANY MATERIALIZED VIEW to deploy;
grant CREATE ANY PROCEDURE to deploy;
grant CREATE ANY INDEX to deploy;
grant UPDATE ANY TABLE to deploy;
grant LOCK ANY TABLE to deploy;
grant ALTER ANY TABLE to deploy;
grant DROP ANY SEQUENCE to deploy;
grant CREATE SEQUENCE to deploy;
grant ALTER ANY SEQUENCE to deploy;
grant SELECT ANY SEQUENCE to deploy;
grant CREATE ANY SEQUENCE to deploy;
grant select any transaction to deploy;
grant advisor to deploy;

@czxin788
Copy link
Contributor

czxin788 commented Nov 21, 2020

感谢,已经验证,想要archery能看到回滚sql,必须开启如下三个权限,缺一不可:
grant select any transaction to deploy;
grant execute on dbms_logmnr to deploy;
grant SELECT ANY DICTIONARY to deploy;

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/engine enhancement New feature or request
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants