Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Dealing with NaN values in data #23

Open
Templar129 opened this issue Sep 15, 2023 · 0 comments
Open

Dealing with NaN values in data #23

Templar129 opened this issue Sep 15, 2023 · 0 comments

Comments

@Templar129
Copy link

Templar129 commented Sep 15, 2023

Our topography data inevitably has a lot of NaN values. Some of them are clustering so it would be hard for us to use regular interpolation methods to fill the NaN.
Andrew talked about a package that could be very useful, GStatSim. It uses a few other pakages from scipy and sklearn to fill NaN values especially in Geological data.
Here is the Github: https://gatorglaciology.github.io/gstatsimbook/3_Simple_kriging_and_ordinary_kriging.html

I have choose one of the image we have and tried the package, and I think it would be pretty useful. Right now it works on a 512 x 512 pixel image, but we can apply it to the larger original image.
After we fill all the NaN values, we can use these data in the VAE. The VAE works much better without the NaN, and it also help us decrease the pattern it learns from the NaN which is not what we wanted.

Here is the example of how this package help to fill the NaN.

Screenshot 2023-09-15 at 2 24 21 PM Screenshot 2023-09-15 at 2 24 12 PM
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant