feat(backup): add hdfs as an alternative backup provider #647
Conversation
zhangyifan27
force-pushed
the
hdfs_service
branch
from
3e8145e to
b8fcba1
Compare
zhangyifan27
added
the
thirdparty-modified
zhangyifan27
force-pushed
the
hdfs_service
branch
from
df53128 to
1e8d177
Compare
zhangyifan27
added
the
type/config-change
| remove_path_response rem_resp; | ||
|
|
||
| // fisrt clean up all old file in test directory. | ||
| std::cout << "clean up all old files" << std::endl; |
There was a problem hiding this comment.
what about print these logs into log file?
There was a problem hiding this comment.
We could know what stage is the test running if output these message to console, it's useful when upload/download many large files.
| [threadpool.THREAD_POOL_FDS_SERVICE] | ||
| worker_count = 8 | ||
|
|
There was a problem hiding this comment.
These block services will create in total 24 threads on server bootstrap. We should merge them into one shared threadpool. Call it THREAD_POOL_BLOCK_SERVICE then.
We can leave this job to later PR.
| #"GLOB" for non - recursive search | ||
| set(MY_SRC_SEARCH_MODE "GLOB") | ||
|
|
||
| set(MY_PROJ_LIBS "") |
There was a problem hiding this comment.
| set(MY_PROJ_LIBS "") | |
| set(MY_PROJ_LIBS "hdfs ${JAVA_JVM_LIBRARY}") |
|
|
||
| dsn_add_static_library() | ||
|
|
||
| target_link_libraries(${MY_PROJ_NAME} PUBLIC hdfs) |
There was a problem hiding this comment.
| target_link_libraries(${MY_PROJ_NAME} PUBLIC hdfs) |
There was a problem hiding this comment.
target_link_libraries(${MY_PROJ_NAME} PUBLIC hdfs ${JAVA_JVM_LIBRARY}) is required, because libraries(hdfs,jvm) are for both a target and its dependents.
Better to describe the new added configs like: + [security]
+ enable_acl = false
+ super_users =
+ meta_acl_rpc_allow_list = |
| derror_f("Failed to write hdfs file {}, error: {}.", | ||
| file_name(), | ||
| utils::safe_strerror(errno)); | ||
| hdfsCloseFile(_service->get_fs(), writeFile); |
There was a problem hiding this comment.
Should we remove the file when error occurs?
There was a problem hiding this comment.
If writing/uploading file failed, we'll retry to overwrite the same file, so I think removing the file is not necessary.
Use a JNI based C API libhdfs to implement table backup/restore.
New thread pool added:
THREAD_POOL_BLOCK_SERVICE(
THREAD_POOL_FDS_SERVICEandTHREAD_POOL_LOCAL_SERVICEwill be replaced by it later )New configurations added:
When using hdfs as a backup/restore provider, you also need to add a block service configuration like this: