-
Notifications
You must be signed in to change notification settings - Fork 262
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Improvement]: Add list, remove, clear, and size operations to shuttle-persist #1066
Changes from 19 commits
40fc370
5357cb4
c58fac0
96f11a6
9e3eea1
9ea36f7
f866b73
ece68b7
f7eee8b
407c95a
45be093
f7186e1
abc1704
5ff0e37
5a8a883
1eba10a
bda6a71
fb9cea5
568e1c2
9c84868
eced7cd
1912a92
107c461
c51f298
d02dcc1
4058a69
c7fe00b
0472444
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,2 @@ | ||
# output files from tests | ||
shuttle_persist/**/*.bin |
Original file line number | Diff line number | Diff line change | ||||
---|---|---|---|---|---|---|
|
@@ -16,6 +16,12 @@ pub enum PersistError { | |||||
Open(std::io::Error), | ||||||
#[error("failed to create folder: {0}")] | ||||||
CreateFolder(std::io::Error), | ||||||
#[error("failed to list contents of folder: {0}")] | ||||||
ListFolder(std::io::Error), | ||||||
#[error("failed to clear folder: {0}")] | ||||||
RemoveFolder(std::io::Error), | ||||||
#[error("failed to remove file: {0}")] | ||||||
RemoveFile(std::io::Error), | ||||||
#[error("failed to serialize data: {0}")] | ||||||
Serialize(BincodeError), | ||||||
#[error("failed to deserialize data: {0}")] | ||||||
|
@@ -41,6 +47,47 @@ impl PersistInstance { | |||||
Ok(serialize_into(&mut writer, &struc).map_err(PersistError::Serialize))? | ||||||
} | ||||||
|
||||||
/// list method returns a vector of strings containing all the keys associated with a PersistInstance | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Nit: should we capitalize these doc comments like we do elsewhere in the codebase? I also don't think we need "list method" as the first part of the comment.
Suggested change
|
||||||
pub fn list(&self) -> Result<Vec<String>, PersistError> { | ||||||
let storage_folder = self.get_storage_folder(); | ||||||
|
||||||
let mut list = Vec::new(); | ||||||
|
||||||
let entries = fs::read_dir(storage_folder).map_err(PersistError::ListFolder)?; | ||||||
for entry in entries { | ||||||
let key = entry.map_err(PersistError::ListFolder)?; | ||||||
let key_name = key | ||||||
.path() | ||||||
.file_stem() | ||||||
.unwrap_or_default() | ||||||
.to_str() | ||||||
.unwrap_or("file name contains non-UTF-8 characters") | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This error should propagate instead of being turned into a key name There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This one is causing a bit of head bashing, but will continue to bash. I'm not seeing how to propagate right in this moment, but will get there. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. You can take a shortcut and map the error to a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This is exactly what I tried but it's not working. The .to_str() method gives off an Option, which I thought .ok_or() would help me convert into a Result<T, E> which I could then .map_err on, but nope. Will get back at it this evening. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Aha, well then don't you already have a There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Hmmmmm...might just have to take my laptop with me to work today and continue to work on this at lunch :) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok, I think I've got this worked out. I figured out how to use ok_or to convert the error related to the potential for invalid Unicode characters. Here's the revised list method: /// list method returns a vector of strings containing all the keys associated with a PersistInstance
pub fn list(&self) -> Result<Vec<String>, PersistError> {
let storage_folder = self.get_storage_folder();
let mut list = Vec::new();
let entries = fs::read_dir(storage_folder).map_err(PersistError::ListFolder)?;
for entry in entries {
let key = entry.map_err(PersistError::ListFolder)?;
let key_name = key
.path()
.file_stem()
.unwrap_or_default()
.to_str()
.ok_or("the file name contains invalid characters").map_err(PersistError::ListName)?
.to_string();
list.push(key_name);
}
Ok(list)
} I did have to introduce a lifetime specifier on our PersistError enum, to satisfy the compiler: #[derive(Error, Debug)]
pub enum PersistError <'a> {
#[error("failed to open file: {0}")]
Open(std::io::Error),
#[error("failed to create folder: {0}")]
CreateFolder(std::io::Error),
#[error("failed to list contents of folder: {0}")]
ListFolder(std::io::Error),
#[error("failed to list the file name: {0}")]
ListName(&'a str),
#[error("failed to clear folder: {0}")]
RemoveFolder(std::io::Error),
#[error("failed to remove file: {0}")]
RemoveFile(std::io::Error),
#[error("failed to serialize data: {0}")]
Serialize(BincodeError),
#[error("failed to deserialize data: {0}")]
Deserialize(BincodeError),
} There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I would say that error does not need a test case, since it should realistically never happen. Since the keys (file names) are |
||||||
.to_string(); | ||||||
list.push(key_name); | ||||||
} | ||||||
Ok(list) | ||||||
} | ||||||
|
||||||
/// clear method removes the keys within the PersistInstance | ||||||
pub fn clear(&self) -> Result<(), PersistError> { | ||||||
let storage_folder = self.get_storage_folder(); | ||||||
fs::remove_dir_all(&storage_folder).map_err(PersistError::RemoveFolder)?; | ||||||
fs::create_dir_all(&storage_folder).map_err(PersistError::CreateFolder)?; | ||||||
Ok(()) | ||||||
} | ||||||
|
||||||
/// size method returns the number of keys in a folder within a PersistInstance | ||||||
pub fn size(&self) -> Result<usize, PersistError> { | ||||||
Ok(self.list()?.len()) | ||||||
} | ||||||
|
||||||
/// remove method deletes a key from the PersistInstance | ||||||
pub fn remove(&self, key: &str) -> Result<(), PersistError> { | ||||||
let file_path = self.get_storage_file(key); | ||||||
fs::remove_file(file_path).map_err(PersistError::RemoveFile)?; | ||||||
Ok(()) | ||||||
} | ||||||
|
||||||
pub fn load<T>(&self, key: &str) -> Result<T, PersistError> | ||||||
where | ||||||
T: DeserializeOwned, | ||||||
|
@@ -111,13 +158,95 @@ mod tests { | |||||
assert_eq!(result, "test"); | ||||||
} | ||||||
|
||||||
#[test] | ||||||
fn test_list_and_size() { | ||||||
let persist = PersistInstance { | ||||||
service_name: ServiceName::from_str("test1").unwrap(), | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The tests are looking really good now! However, there is one case that is not yet covered, but it might simplify the code. If list/load/remove etc are called before any key has been saved, they will fail due to the folder not existing yet. This can be avoided if we ensure the folder exists before any method is called. My idea is to add a method There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. So a new method after all then...got it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm doing a push as a checkpoint to show where I'm at. As a start, I've made a skeleton new method that instantiates a PersistInstance struct, given a ServiceName. I've substituted this in the tests and all works as before. I'm having difficulty now understanding how to get the folder creation incorporated. If I put a &self in as a parameter to new, how do I then instantiate in the tests? Just need a couple of breadcrumbs to set the path forward. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ah! Good point. fn new(...) -> Result<...> {
let instance = Self { ... };
let directory = instance.get_storage_folder();
fs::create_dir_all(...).map_err(...)?;
Ok(instance)
} There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
Having some difficulty now with this piece. I understand what you're saying but am not sure how to make it happen. I will reflect through the day today. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Alright, time to admit it. After sititng staring at the code for a fair bit of time, I have no clue how to convert the errors into a shuttle_service::Error in ResourceBuilder::output. I think I need a hint. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This works // Get the PersistError in, let's say, a match statement, then
return Err(shuttle_service::Error::Custom(
/*The PersistError*/.into(),
)); EDIT: based on the above I would imagine that something like this will work: PersistInstance::new(...).map_err(|e| shuttle_service::Error::Custom(e.into())) There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ok, I feel I've got the new method working. It will convert a PersistError into a shuttle_service::Error. I'm having difficulty with a test though. All I managed to do yesterday evening was test ServiceName, which panics if you pass it something invalid. This doesn't really test the new method erroring out because of an issue with create_dir_all. Also, the cargo-shuttle circleci check is still failing, and I'm not clear why (I have tried to read the log output) or how to resolve it. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yeah, the create_dir_all is hard to test failures of. Perhaps creating a folder in a root-owned directory, but that might not be reproducible on all machines. I would say that we can trust it without a test. The CI fail on cargo-shuttle init is a sporadic error (a hard one to fix :/ ), and not related to your changes. |
||||||
}; | ||||||
|
||||||
persist.save("test", "test").unwrap(); | ||||||
let list_result = persist.list().unwrap().len(); | ||||||
let size_result = persist.size().unwrap(); | ||||||
assert_eq!(list_result, 1); | ||||||
assert_eq!(size_result, 1); | ||||||
} | ||||||
|
||||||
#[test] | ||||||
fn test_list_error() { | ||||||
let persist = PersistInstance { | ||||||
service_name: ServiceName::from_str("test2").unwrap(), | ||||||
}; | ||||||
|
||||||
// unwrap error | ||||||
let result = persist.list().unwrap_err(); | ||||||
assert_eq!( | ||||||
result.to_string(), | ||||||
"failed to list contents of folder: No such file or directory (os error 2)" | ||||||
); | ||||||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Maybe it is unintuitive that calling list() on an empty persist store returns an error. If you also call There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Now just removing the files instead of the entire folder, test adjusted accordingly. |
||||||
} | ||||||
|
||||||
#[test] | ||||||
fn test_remove() { | ||||||
let persist = PersistInstance { | ||||||
service_name: ServiceName::from_str("test3").unwrap(), | ||||||
}; | ||||||
|
||||||
persist.save("test", "test").unwrap(); | ||||||
persist.save("test2", "test2").unwrap(); | ||||||
let list = persist.list().unwrap(); | ||||||
let key = list[0].as_str(); | ||||||
persist.remove(key).unwrap(); | ||||||
let result = persist.list().unwrap(); | ||||||
assert_eq!(result.len(), 1); | ||||||
} | ||||||
|
||||||
#[test] | ||||||
fn test_remove_error() { | ||||||
let persist = PersistInstance { | ||||||
service_name: ServiceName::from_str("test4").unwrap(), | ||||||
}; | ||||||
|
||||||
// unwrap error | ||||||
let result = persist.remove("test4").unwrap_err(); | ||||||
assert_eq!( | ||||||
result.to_string(), | ||||||
"failed to remove file: No such file or directory (os error 2)" | ||||||
); | ||||||
} | ||||||
|
||||||
#[test] | ||||||
fn test_clear() { | ||||||
let persist = PersistInstance { | ||||||
service_name: ServiceName::from_str("test5").unwrap(), | ||||||
}; | ||||||
|
||||||
persist.save("test5", "test5").unwrap(); | ||||||
persist.clear().unwrap(); | ||||||
let result = persist.list().unwrap(); | ||||||
assert_eq!(result.len(), 0); | ||||||
} | ||||||
|
||||||
#[test] | ||||||
fn test_clear_error() { | ||||||
let persist = PersistInstance { | ||||||
service_name: ServiceName::from_str("test6").unwrap(), | ||||||
}; | ||||||
|
||||||
// unwrap error | ||||||
let result = persist.clear().unwrap_err(); | ||||||
assert_eq!( | ||||||
result.to_string(), | ||||||
"failed to clear folder: No such file or directory (os error 2)" | ||||||
); | ||||||
} | ||||||
|
||||||
#[test] | ||||||
fn test_load_error() { | ||||||
let persist = PersistInstance { | ||||||
service_name: ServiceName::from_str("test").unwrap(), | ||||||
}; | ||||||
|
||||||
// unwrapp error | ||||||
// unwrap error | ||||||
let result = persist.load::<String>("error").unwrap_err(); | ||||||
assert_eq!( | ||||||
result.to_string(), | ||||||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Don't need this anymore :)