Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP]The Release-Note of version 1.2.0-RC1 #2765

Closed
casionone opened this issue Aug 14, 2022 · 1 comment
Closed

[WIP]The Release-Note of version 1.2.0-RC1 #2765

casionone opened this issue Aug 14, 2022 · 1 comment

Comments

@casionone
Copy link
Contributor

casionone commented Aug 14, 2022 •
edited
Loading

Apache Linkis(incubating) 1.2.0 includes all of Project Linkis-1.2.0.

This release mainly supports Presto and ElasticSearch engines for Linkis 1.X architecture, enhances JDBC engine to support configuration multiple data source, supports displaying historical engine information on web, reduced and optimized Linkis modules, improved test cases for some engines, and made a lot of bug fixes and feature improvements.

The main functions are as follows:

  • Added support for Presto engine
  • Added support for ElasticSearch engine
  • Added features to JDBC engine to support data source mode
  • Reduce and optimize Linkis modules
  • Data source module interface optimization

Abbreviations:

  • COMMON: Linkis Common
  • EC: Engineconn
  • ECM: EngineConnManager
  • ECP: EngineConnPlugin
  • DMS: Data Source Manager Service
  • MDS: MetaData Manager Service
  • LM: Linkis Manager
  • PS: Linkis Public Service
  • PE: Linkis Public Enhancement
  • RPC: Linkis Common RPC
  • CG: Linkis Computation Governance

New Feature

  • [ECP-PRESTO][LINKIS-1631] Enhance PrestoEngine, adatps to Linkis1.X architecture
  • [ECP-ES][LINKIS-1632] Enhance ElasticSearchEngine, adatps to Linkis1.X architecture
  • [ECP-JDBC][LINKIS-2092] Linkis jdbc engine supports multiple data source links
  • [COMMON][LINKIS-2191] Add common rest api offline for all services
  • [CG][LINKIS-2222] Single LinkisEntrance task metrics collection
  • [CG][LINKIS-2227] Add ECR history query interface
  • [WEB][LINKIS-2227] Support displaying history EngineConn information
  • [CG][LINKIS-2257] Gateway supports forwarding non-execution related requests of Entrance
  • [CG][LINKIS-2258] Added feature to clean up running tasks when the Entrance process exits normally
  • [PE][LINKIS-2277] In the workspace, files can be moved to another folder
  • [CG][LINKIS-2288] Added query interface for unfinished tasks
  • [CG][PE][LINKIS-2291] Entrance support isolation by routing labels
  • [COMMON][CG][PE][LINKIS-2320] Entrance adds restful interface to support modifying routelabel
  • [CG][PE][LINKIS-2326] Linkis tasks support automatic retry
  • [COMMON][LINKIS-2366][LINKIS-2434] Support knif4j
  • [ECP-JDBC][LINKIS-2392] Jdbc engine support trino
  • [COMMON][LINKIS-2415] Support variable operation

Enhancement

  • [COMMON][LINKIS-1411] Remove sun.misc.BASE64
  • [COMMON][LINKIS-1475] Optimize code style
  • [LM][LINKIS-1763] Add non empty validation to the applicationlist interface
  • [COMMON][LINKIS-1824] Update commons-lang to commons-lang3
  • [COMMON][CG][PE][LINKIS-2077] Optimized to automatically refresh all LinkisManager caches after modifying parameters for the management console
  • [DMS][LINKIS-2082] DataSource Manager module optimization
  • [ECP-JDBC][LINKIS-2140] Consistent adjustment of JDBC connection parameters in the JDBC engine
  • [ECP-JDBC][LINKIS-2141] Change dbcp to druid in JDBC engine
  • [DEPLOY][LINKIS-2193] Add graceful upgrade script
  • [ECP-JDBC][LINKIS-2194] Cancel the supportedDBs in the jdbc engine ConnectionManager.java and add the parameter wds.linkis.jdbc.driver
  • [DMS][LINKIS-2212] Add the default DWSClientConfig constructor for LinkisDataSourceRemoteClient to simplify the client API for internal microservices to call data source services
  • [ECP][LINKIS-2214] In the engine material refresh interface, refeshAll and refesh are changed to refreshAll and refresh
  • [ECP-PYTHON][LINKIS-2216] The python plt show method directly supports display
  • [CG][LINKIS-2217] Added trino engine type
  • [CG][ECP][PE][LINKIS-2264] Module optimization reduces the number of modules
  • [ECP-JDBC][LINKIS-2278] Add test case for jdbc engine
  • [DEPLOY][LINKIS-2293] Add port check in install.sh
  • [COMMON][LINKIS-2299] Add built-in variables run_today_h and run_today_h_std
  • [PE][LINKIS-2344] Optimize metadata sql
  • [PE][LINKIS-2352] Optimize CS module and reduce the number of modules
  • [CG][LINKIS-2362] Move the linkis-engineconn-plugin-framework to linkis-cg module
  • [PE][LINKIS-2368] Automatically create workspace for newly added users
  • [PACKAGE][LINKIS-2374] Optimize of linkis assamble-combined-package module
  • [COMMON][LINKIS-2396] Remove the use of Logging's deprecated method
  • [ECP-SPARK][LINKIS-2405] Support scala multi-version of spark
  • [ECP-SPARK][LINKIS-2419] Remove the config SPARK_SCALA_VERSION and get the scalaVersion from env
  • [COMMON][LINKIS-2441] Knife4j interface document
  • [PE-BML][LINKIS-2450] The new storage path is optimized when the engine material is updated
  • [PE-BML][LINKIS-2475] Fix package name capitalization problem
  • [CG][LINKIS-2477] Optimize build out dependence of linkis-cg-engineconnplugin
  • [CG][ECP][LINKIS-2479] The ECM kill engine needs to be able to complete the kill of the yarn appid
  • [MDS][LINKIS-2481] Linkis-metadat-query-service-hive package name modification
  • [COMMON][CG][GATEWAY][LINKIS-2496] Opt refactor entrance bean conf
  • [COMMON][CG][PE][LINKIS-2508] Feature optimization that supports high concurrency
  • [ECP-PRESTO][LINKIS-2512] Optimize presto engineconn
  • [WEB][LINKIS-2524] Rename web to linkis-web
  • [PE-BML][LINKIS-2531] Update VersionServiceImplTest
  • [COMMON][LINKIS-2549] Optimize the read efficiency of the result set when the result set is output after the script is executed
  • [ECP-SPARK][TEST][LINKIS-2617] Add test case for factory launch extension
  • [COMMON][LINKIS-2618] Optimized module and plugin configuration
  • [ECP-SHELL][TEST][LINKIS-2620] Add test case for shell engine
  • [ECP][LINKIS-2628] EC Yarn app id logs should be printed to a separate log
  • [PE-BML][LINKIS-2633] The rollbackversion function modifies the case
  • [PACKAGE][LINKIS-2635] Add 1.2.0_schema file records to update changes to 1.2.0
  • [PACKAGE][LINKIS-2679] Optimize the default queue and hive default reduces parameters in dml
  • [ECP-JDBC][LINKIS-2741] The connection cache pool key value in the ConnectionManager is adjusted to the data source name and version number
  • [ECP-JDBC][LINKIS-2743] Jdbc data source configuration priority definition.

Bugs Fix

  • [PS-RM][LINKIS-1850] Fix NullPointerException
  • [PE][LINKIS-1879] FileWriter and BufferedWriter are not closed in finally clause
  • [CG][LINKIS-1911] Fix linkis-computation-client failed to submit jobs
  • [CG][LINKIS-2040] Fix HDFSCacheLogWriter getOutPutStream NPE
  • [DMS][LINKIS-2255] The expired field is missing when querying information from a single data source
  • [COMMON][LINKIS-2269] Fix ddl sql bug
  • [COMMON][LINKIS-2314] Fix AbstractDiscovery delayTime calculate error
  • [ECP-HIVE][LINKIS-2321] For hive on tez tasks, the task cannot be ended correctly when canceled
  • [PE][LINKIS-2346] Fix the admin user get tables not return all tables
  • [RPC][LINKIS-2370] Fix linkis-rpc messageUtils.orderIsLast not correct
  • [LM][LINKIS-2372] Fix LM ec history restful bug
  • [PE][LINKIS-2273] Data source edit form supports Chinese and English
  • [PACKAGE][LINKIS-2389] Fixed the bug in the linkis-ps-metadataquery module that was missing after packaging
  • [CG][LINKIS-2412] Fix issue that querying ECM history, permission management fails
  • [CG][LINKIS-2418] Fixed the problem that the task state could not be flipped normally when thread unsafe SimpleDateFormat was used as a global variable
  • [MDS][LINKIS-2435] Fix NPE of metadata mysql query
  • [GATEWAY][LINKIS-2454] When linkis-gateway is debugged locally, knife4j-related class dependencies conflict
  • [COMMON][LINKIS-2456] Fix test bug
  • [ECM][LINKIS-2469] ECM logOperator uses wrong delimiter to get log path
  • [PE][LINKIS-2470] File upload Chinese name is garbled
  • [CG][LINKIS-2471] Orchestrator supports task wait timeout
  • [PE][LINKIS-2472] Throws an exception when the data is empty
  • [ECP-SPARK][LINKIS-2488] Fix the problem that 'CsvRelation' class cannot be serialized
  • [COMMON][LINKIS-2506] Upgrade 1.1.1 ddl,miss engine=innodb default charset=utf8 statement
  • [COMMON][LINKIS-2535] Fix call ExceptionUtils.getStackTrace NPE
  • [PE-BML][LINKIS-2543] Fix ps_bml_resources_version insert new version missing some information
  • [CG][MDS][LINKIS-2547] Fix MetadataQuery sql syntax error and LockManagerMapper method overload bug
  • [COMMON][LINKIS-2559] Fix variable substitution error
  • [ECP-PRESTO][LINKIS-2596] Fix presto missing dependencies when compiling package
  • [ECP-ES][LINKIS-2603] NoSuchMethodError for ES engine
  • [ECP-ES][LINKIS-2604] NumberFormatException for ES engine
  • [PE][LINKIS-2614] Fixed a NPE caused by a client request
  • [COMMON][LINKIS-2631] Fix the dead loop
  • [ECP-SHELL][LINKIS-2654] Fix test case for ShellEngineConnExecutor
  • [LM][LINKIS-2688] Upgrade the default EngineType version of the flink ec
  • [TOOL][LINKIS-2701] Fix github repos page License display with Unknown licenses found

Security related

Dependency changes

Thanks

The release of Apache Linkis(incubating) 1.2.0 is inseparable from the contributors of the Linkis community. Thanks to all the community contributors, including but not limited to the following Contributors (in no particular order): CCweixiao, Dlimeng, QuintinTao, WenxiangFan, aiceflower, barry8023, binbinCheng, casionone, duhanmin, gabeng1996, huangKai-2323, huapan123456, huiyuanjjjjuice, hunter-cloud09, jackxu2011, legendtkl, liangqilang, liuyou2, mindflow94, peacewong, ruY9527, seedscoder, wForget, yyuser5201314

Apache Linkis(incubating) 1.2.0 包括所有 Project Linkis-1.2.0.

Linkis 1.2.0 版本发布主要支持Presto和ElasticSearch引擎；对JDBC引擎新增特性，以支持数据源模式；减少和优化Linkis模块；数据源功能优化；改进部分引擎的测试用例，并进行了大量的bug修复和功能改进。

主要功能如下：

  • 新增对 Presto 引擎的支持
  • 新增对 ElasticSearch 引擎的支持
  • 对JDBC引擎新增特性，以支持数据源模式
  • 减少和优化Linkis模块
  • 数据源模块接口优化

缩写：

  • COMMON: Linkis Common
  • EC: Engineconn
  • ECM: EngineConnManager
  • ECP: EngineConnPlugin
  • DMS: Data Source Manager Service
  • MDS: MetaData Manager Service
  • LM: Linkis Manager
  • PS: Linkis Public Service
  • PE: Linkis Public Enhancement
  • RPC: Linkis Common RPC
  • CG: Linkis Computation Governance

新特性

Enhancement

  • [COMMON][LINKIS-1411] 移除 sun.misc.BASE64
  • [COMMON][LINKIS-1475] 优化代码风格
  • [LM][LINKIS-1763] 将非空验证添加到应用列表接口
  • [COMMON][LINKIS-1824] 将commons-lang更新为commons-lang3
  • [COMMON][CG][PE][LINKIS-2077] 优化为修改管理控制台参数后自动刷新所有LinkisManager缓存
  • [DMS][LINKIS-2082] 数据源管理模块优化
  • [ECP-JDBC][LINKIS-2140] JDBC引擎中JDBC连接参数的一致调整
  • [ECP-JDBC][LINKIS-2141] 在JDBC引擎中将dbcp修改为druid
  • [DEPLOY][LINKIS-2193] 添加优雅升级脚本
  • [ECP-JDBC][LINKIS-2194] 取消JDBC引擎ConnectionManager.java中支持的ddb，并添加参数wds.links.jdbc.driver
  • [DMS][LINKIS-2212] 为LinkisDataSourceRemoteClient添加默认的DWSClientConfig构造函数，以简化内部微服务调用数据源服务的客户端API
  • [ECP][LINKIS-2214] 在引擎材质刷新界面中，refeshAll和refresh被修改为refresshAll和refresh
  • [ECP-PYTHON][LINKIS-2216] python plt show方法直接支持显示
  • [CG][LINKIS-2217] 增加了trino引擎类型
  • [CG][ECP][PE][LINKIS-2264] 模块优化可以减少模块数量
  • [ECP-JDBC][LINKIS-2278] 为JDBC引擎增加测试用例
  • [DEPLOY][LINKIS-2293] 在install.sh中添加端口检查
  • [COMMON][LINKIS-2299] 添加内置变量run_today_h和run_today_h_std
  • [PE][LINKIS-2344] 优化元数据SQL
  • [PE][LINKIS-2352] 优化CS模块，减少模块数量
  • [CG][LINKIS-2362] 将link-engineeconn-plugin-framework移动到link-cg模块
  • [PE][LINKIS-2368] 自动为新添加的用户创建工作区
  • [PACKAGE][LINKIS-2374] 优化Linkis assamble-combined-package模块
  • [COMMON][LINKIS-2396] 删除已弃用的Logging方法
  • [ECP-SPARK][LINKIS-2405] 支持scala spark多版本
  • [ECP-SPARK][LINKIS-2419] 删除配置SPARK_SCALA_VERSION并从env中获取scala版本
  • [COMMON][LINKIS-2441] Knife4j接口文档
  • [PE-BML][LINKIS-2450] 当更新引擎材料时，新的存储路径将得到优化
  • [PE-BML][LINKIS-2475] 修复包名大小写问题
  • [CG][LINKIS-2477] 优化建立link-cg-engineeconnplugin的依赖
  • [CG][ECP][LINKIS-2479] 终止ECM引擎时需要同时终止yarn appid
  • [MDS][LINKIS-2481] Linkis-metadat-query-service-hive包名修改
  • [COMMON][CG][GATEWAY][LINKIS-2496] 优化重构入口bean配置
  • [COMMON][CG][PE][LINKIS-2508] 支持高并发性的特性优化
  • [ECP-PRESTO][LINKIS-2512] 优化presto引擎
  • [WEB][LINKIS-2524] 将web重命名为linkis-web
  • [PE-BML][LINKIS-2531] 更新VersionServiceImplTest
  • [COMMON][LINKIS-2549] 在脚本执行后输出结果集时，优化结果集的读取效率
  • [ECP-SPARK][TEST][LINKIS-2617]为工厂启动扩展添加测试用例
  • [COMMON][LINKIS-2618] 优化模块和插件配置
  • [ECP-SHELL][TEST][LINKIS-2620] 为外壳引擎添加测试用例
  • [ECP][LINKIS-2628] 将EC Yarn应用日志打印到一个单独的日志中
  • [PE-BML][LINKIS-2633] rollbackversion函数修改大写
  • [PACKAGE][LINKIS-2635] 添加1.2.0_schema文件记录以更新对1.2.0的更改
  • [PACKAGE][LINKIS-2679] 优化默认队列，以及减少dml中hive默认参数
  • [ECP-JDBC][LINKIS-2741] ConnectionManager中的连接缓存池键值将调整为数据源名称和版本号
  • [ECP-JDBC][LINKIS-2743] JDBC数据源配置优先级定义

Bugs Fix

  • [PS-RM][LINKIS-1850] 修复NPE问题
  • [PE][LINKIS-1879] FileWriter和BufferedWriter在finally子句中没有关闭
  • [CG][LINKIS-1911] 修复link-computing-client提交作业失败的问题
  • [CG][LINKIS-2040] 修复HDFSCacheLogWriter getOutPutStream NPE问题
  • [DMS][LINKIS-2255] 当从单个数据源查询信息时，将丢失过期字段
  • [COMMON][LINKIS-2269] 修复 ddl sql bug
  • [COMMON][LINKIS-2314] 修复AbstractDiscovery delayTime计算错误
  • [ECP-HIVE][LINKIS-2321] 对于hive on tez任务，取消任务时无法正确结束任务
  • [PE][LINKIS-2346] 修复了管理员用户获取表而不返回所有表的问题
  • [RPC][LINKIS-2370] 修复linkis-rpc messageUtils.orderIsLast不正确
  • [LM][LINKIS-2372] 修复了LM ec历史restful的bug
  • [PE][LINKIS-2273]数据源编辑表单支持中文和英文
  • [PACKAGE][LINKIS-2389] 修复了打包后links-ps-metadataquery模块中缺失的bug
  • [CG][LINKIS-2412] 修复查询ECM历史时，权限管理失败的问题
  • [CG][LINKIS-2418] 修复了将不安全的SimpleDateFormat线程用作全局变量时无法正常翻转任务状态的问题
  • [MDS][LINKIS-2435] 修复了mysql元数据查询的NPE问题
  • [GATEWAY][LINKIS-2454] 当在本地调试link-gateway时，与knife4j相关的类依赖冲突
  • [COMMON][LINKIS-2456] 修复测试bug
  • [ECM][LINKIS-2469] ECM logOperator使用错误的分隔符来获取日志路径
  • [PE][LINKIS-2470] 上传文件中文名称乱码
  • [CG][LINKIS-2471] Orchestrator支持任务等待超时
  • [PE][LINKIS-2472] 当数据为空时抛出异常
  • [ECP-SPARK][LINKIS-2488] 修复CsvRelation类无法序列化的问题
  • [COMMON][LINKIS-2506] 升级1.1.1 ddl 语句缺失 engine=innodb default charset=utf8
  • [COMMON][LINKIS-2535] 修复调用ExceptionUtils.getStackTrace NPE问题
  • [PE-BML][LINKIS-2543] 修复ps_bml_resources_version插入新版本缺失一些信息的问题
  • [CG][MDS][LINKIS-2547] 修复MetadataQuery sql语法错误和LockManagerMapper方法过载的bug
  • [COMMON][LINKIS-2559] 修复变量替换错误
  • [ECP-PRESTO][LINKIS-2596] 修复了编译包时缺少依赖关系的问题
  • [ECP-ES][LINKIS-2603] 修复ES引擎NoSuchMethodError
  • [ECP-ES][LINKIS-2604] 修复ES引擎NumberFormatException
  • [PE][LINKIS-2614] 修复了由客户端请求引起的NPE问题
  • [COMMON][LINKIS-2631] 修复死循环问题
  • [ECP-SHELL][LINKIS-2654] 修复了ShellEngineConnExecutor的测试用例
  • [LM][LINKIS-2688] 升级flink ec的默认EngineType版本
  • [TOOL][LINKIS-2701] 修复github repos页面的许可证显示与未知的许可证发现

Security related

Dependency changes

致谢

Apache Linkis(incubating) 1.2.0的发布离不开Linkis社区的贡献者，感谢所有的社区贡献者，包括但不仅限于以下Contributors（排名不发先后）:
CCweixiao, Dlimeng, QuintinTao, WenxiangFan, aiceflower, barry8023, binbinCheng, casionone,
duhanmin, gabeng1996, huangKai-2323, huapan123456, huiyuanjjjjuice, hunter-cloud09, jackxu2011,
legendtkl, liangqilang, liuyou2, mindflow94, peacewong, ruY9527, seedscoder, wForget, yyuser5201314
@github-actions
Copy link

😊 Welcome to the Apache Linkis (incubating) community!!
We are glad that you are contributing by opening this issue.

Please make sure to include all the relevant context.
We will be here shortly.

If you are interested in contributing to our website project, please let us know!
You can check out our contributing guide on
👉 How to Participate in Project Contribution.

WeChat Group:

image
Mailing Lists:

name description Subscribe Unsubscribe archive
[email protected] community activity information subscribe unsubscribe archive

@aiceflower aiceflower mentioned this issue Aug 15, 2022
Merged
@casionone casionone changed the title [WIP]The Release-Note of version 1.2.0 [WIP]The Release-Note of version 1.2.0-RC1 Aug 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@casionone