-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
HBASE-27231 FSHLog should retry writing WAL entries when syncs to HDF… #4721
base: master
Are you sure you want to change the base?
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
So here we just reuse most of the logic in AsyncFSWAL, and only leave the sync operation in FSHLog? One of the problem is that, the FSOutputStream may be blocked even if you do not call sync. If the buffer is full, the DFSOutputStream will issue a packet out and if the queue is full, we will block there...
💔 -1 overall
This message was automatically generated. |
@Apache9 ,thank you very much for point this out, yes, but the current |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
💔 -1 overall
This message was automatically generated. |
@Apache9 , after I read the One possible solution for the problem you described I can think of is to wrap @Apache9 , what is your opinion? |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
🎊 +1 overall
This message was automatically generated. |
@Apache9 , would you mind review the comments again and give some suggestions at your convenience ? |
I suggest we just change the code in FSHLog, without touching AsyncFSWAL too much. @comnetwork |
@Apache9 , ok ,I will try to keep the changes within the FSHLog. |
@comnetwork Do you still have interest in implementing this? WHen investigating how to better support sync replication in WAL framework, as well as how could we introduce new WAL implementations, I believe a better way is to also make sync replication work with FSHLog, and also, the new WAL implementation should try to implement AsyncWriter or Writer interface, if they choose any support Writer interface(this is possible if they only implement the hadoop fs APIs), then the new WAL implementation can only work FSHLog, so I think we still need FSHLog in the future. So now I'm OK without touching the code in FSHLog and also in AsyncFSWAL to make them share more code and use similiar mechanism to deal with errors. Let's revive this PR. Thanks. |
I can pick this up if you do not mind @comnetwork Thanks. |
…S failed