GSdx: Texture Dumping and Replacing #4199
Conversation
|
Okay there's a few things wrong with this.
This is just a few things I've noticed from looking at it for 5 minutes. |
I do realize this. I, however, do not know CMake to make this possible. However, a few people reached out to me, offering help.
Oh, I did not realize. My deepest apologies. Will do.
*glares at Visual Studio* Will fix.
*glares at myself* Will fix. Thanks for your feedback! I will get to it right away! |
|
Regarding the UI, it would fit better in Advanced Settings. |
Ah, I was afraid this would happen because of simply how late I got. I rebased this and ported all the code about it around 5 days or so ago, too. |
|
Note: If you separate pieces out into separate, nicely labeled commits it makes things way easier to review, and much more obvious when you accidentally undo existing commits so that you can fix it before you even open the PR. Things that could have been separate in the current PR: Addition of new libraries, addition of texture dumping, addition of ProcessSTD, addition of ProcessSPC Having both |
Yea, I realized I messed up a bit in that part. However, I have a good reason for this, which I will keep private. I can ensure this won't happen again.
Noted. Will do. Thanks for the feedback ^^ |
|
Hey, that's a neat feature, glad to see interest on this! Unfortunately, you're chiming in at a very bad timing, the project codebase is going to heavily change in the upcoming months and will probably take a bunch of work to rebase. I am not going to tell you to stop fixing it now because of this though, but you'll have to keep this in mind. Feature wise there are probably concerns regarding cache coherency and other things, but I'll let others talk about this more in detail. |
|
Heya Govan! Long time no see!
If this has a chance to get merged, I am willing to rebase this until the grand overhaul. If not, I am willing to do a rewrite at the worst possible case scenario. I know it has some problems I need to address but hey, it's a start, isn't it? This feature actually existing and working in PCSX2 is remarkable enough in my opinion. And I am willing to keep it going.
You know that I was never a Linux nor a macOS guy myself. However, I have been talking with some people over this, and I think it will be possible to fix this (To be honest, I do believe they will be fixed once I use the internally existing classes and libraries). The formatting I will also try to fix to the best of my ability.
This is the main concern I did have, and why I and Vlad were insisting on a LOT of testing. This was finished in around January 1st and was being tested and fixed until today. We were able to spot some problems with memory, but I was able to address them. I do believe, anyways. More testing is necessary, of course it is. But I was not able to get it to behave weirdly or badly even with monumental effort to do so. However, I did hit a hard limit of 4096x4096 textures. That may be cache or memory, but that is only for a single texture it may seem. As loading several 2048x2048 textures worked fine. I do appreciate your feedback. Really, I do. And I will take everything you said into consideration whilst I am fixing all of the error and problems with this feature. And who knows, perhaps I will be able to make it all before the great overhaul. |
|
A thing I noticed on the PR: You are using mIni for loading settings. Don't. And obviously this has a chance to be merged, PRs are always welcome, but this will definitely require a lot more work than what you are proposing! |
I was told. I think I will wait on the YAML implementation for porting the settings and what not.
Thanks for your kind words. However, if you or any of the other devs may or may not think otherwise, please do let me know! I want to fix any and all problems that I can. And I do realize this needs work. I will do my best ^^ |
|
Hey, so how do I give this a shot? I'm willing to be a tester. |
|
@Gunlaus There isn't really anything to test for now, this PR isn't ready for testing nor for review. The 64 bit artifact isn't ready either, so only windows 32 bit will work. |
|
A problem I have encountered. I generate a CRC-checksum from the CLUT data to use with replacements. I do this to generate 1 hash per texture. The other things I tried to hash resulted in 5-10-20-etc. hashes per texture. I use CRCpp to do this, which I was told to use the crc32 function within PCSX2 instead (Which is on zlib). However, I can no longer get a single hash to generate. Anyone has a clue on what is causing it? I try to get the hash like so: // Capture the palette.
auto _palette = m_src->m_palette_obj.get();
auto _len = _palette->m_pal * 4;
// Capture the CLUT data as Bytef for zlib/crc32
std::vector<Bytef> _clut(_palette->m_clut, _palette->m_clut + _len);
// Calculate a CRC32 value from the palette CLUT data.
_currentChecksum = crc32(0, _clut.data(), _len);But it causes a lot of hashes per texture, a problem I had when I was first developing this. EDIT: Nevermind, I was a doofus. I fixed this issue. |
|
Brilliant. First try and it's dumping textures as perfectly as when I used to manually rip them with Texmod. |
|
All of the latest changes should address the following:
|
|
I think you can add the xxHash GitHub repo as submodule. |
…, add comments explaining stuff.
|
@GovanifY , and @Topaz-Reality How about considering: https://github.com/google/highwayhash/blob/master/highwayhash/highwayhash.h I think that this might fit the bill; 128bit should be enough to never worry about a collision again. |
The collisions are not because of CRC32, but rather, the method I use to hash texture in a speedy manner. I did not test with xHash or SipHash or HighwayHash. I am still waiting for #4005 to merge before moving further with anything. However, thanks for the suggestion! If the devs are okay with it, and if it's license is GPLv2 compliant, I will be sure to test it. |
|
Nevermind, I think we cannot use HighwayHash. Turns out it's license (Apache v2) is incompatible with PCSX2's license (GPL v2). |
|
Incredible your work! I would like to know how to find out the values of the textures without the hash. I edit a PS2 game and this is essential. |
You cannot. |
|
so I've been trying to test this out on .hack//IMOQF and it doesn't seem to be working, attached is my yaml file (changed to txt for GitHub upload), am i doing something wrong? |
|
Likely you missed a key detail in the description. The DDS files have to be in an RGBA (uncompressed) format. I spent some time writing a little pipeline to take the dumped textures, run them through waifu2X then spit them out into the required DDS format to test to make sure IMOQ was working. I did experience some issues on the main menu, but I can't be certain that wasn't something with the pipeline screwing up a texture or two. If you'd like I can share it out. To write to the appropriate format I adapted the writing code from the PR into an exe: https://gist.github.com/playmer/9ec5cd78f8a599a8481548edbfb18637 So for further testing you can use that. But yeah, all that said, seems to work: While reviewing the code to understand some of what was going wrong here I think I saw at least one nit, so when I wake up I'll take another look to make sure I'm not missing anything. I will say, I do think compressed DDS is worth looking into. These files can get quite big. For laughs I had upscaled them to 16x and some were in the 150MB or so range. |
If you had actually taken the time to read the PR, you would see the following:
|
|
To be honest, until I sat down and read the code in more detail that line was unclear and insufficient, and somewhat of a buried lede despite me looking through to find the specific import format. Does it mean Exporting? Importing? Both? Until I read the code I didn't know. As it is, the class handles importing via CatchDDS. I'd recommend changing that to something like ReadDDS. The export all happens in the individual renderers, probably should unify that into the class. As for the format itself, I've never even heard of such a format of DDS. None of my converters I'm familiar with supported such a format. |
Of course you have not. It's not like you work with DX or anything. The code may be a little unclear on the DDS part, but literally EVERYTHING else is commented in a way that even a non-programmer can understand. I am not sure how you managed to have problems with it. As for the DDS class itself, it is going to be changed anyways. But if you think you can do a better job than me than please do push to this PR in one way or another (I do not know if Github allows it), so I can use it with my work. Also, 'the class only supports X' means both importing and exporting. It is sufficient and clear, I am sorry you could not understand this very simple to understand line for one reason or another. |
|
On that note; I would like to make it clear that any and all help is appreciated and I do appreciate your feedback. |
|
To clarify, I'm generally familiar with DDS, I didn't know it supported uncompressed RGBA inside of it. Then couldn't find a tool that would create a DDS with that internal format, but I probably just missed it. I typically use Universal formats on disk such as crunch or basis and transcode to things like BC7 or BC3, which are the internal formats I'd generally expect to find in a DDS file. I'm interested in extending this PR with such functionality, along with addressing what I can to help it along. I'd also like to put PNG support back in as an option, as the workflow for dealing with the files mean that a texture pack creator is going to have to interchange their texture to something else anyway. You mentioned this in the description as a possibility so I'm happy to do that. I'll open a PR on your repo when I'm further along and we can discuss it there. |
Thank you sooo much! This will help me alot! |
ghost
mentioned this pull request
This is really helpful for non-Photoshop users, but it doesn't seem to support partial transparency. For Photoshop users, you can use NVIDIA's Photoshop plugin which includes partial transparency support (https://developer.nvidia.com/nvidia-texture-tools-exporter) I save with BGRA and it works perfectly. Filesize is, of course, still a problem unfortunately. |
Yeah, that was a naïve early attempt at figuring out what to use although it's a little odd you're seeing partial transparency issues. I've started to use texconv for this as it does support the needed formats (DXGI_FORMAT_R8G8B8A8_UNORM or DXGI_FORMAT_B8G8R8A8_UNORM). Although I can't speak to the transparency support. Something like |
|
A little late to the party, but I'm wondering why you chose DDS particulary? I don't know how the loading works (AFAICT DDS is merely a container) but won't this be DirectX-exclusive then? And why not use more advanced tech like BC7 (same size compressed but better quality)? |
I use OpenGL without issues. |
|
Amazing work, Topaz! Seriously what I have been dreaming of for a very long time. To make the YAML generation easier for people, I would recommend putting this texturesmapper tool by delltalal into your guide: https://github.com/delltalal/TexturesMapper/ Just FYI. Thanks for your excellent work! |
|
@Topaz-Reality has closed this due to a disagreement with another contributor and will be continuing it in his fork of PCSX2. His original closing message has been deleted for targeted harassment Edit: For those viewing this for Reddit, this PR was closed due to comments by a contributor (not PCSX2 member) on a project completely unrelated to this one. This is all just drama sturred up by them to try and cause problems for us, it's kinda sad, a lot of people were looking forward to this. |
What is this?
GSdx, edited to be able to load DDS files from the local storage, and then replace them accordingly. It also has the ability to dump textures with a specific hash to use with the replacement function (The hash in question is generated with CRC32)
Primarily made because I wanted to, but it also closes the issue: #4038.
How does it work?
In PCSX2's executable directory, you have 2 folders. One of them is
txtconfig, the other one beingtextures.The
txtconfigfolder contains the texture loading configuration, a kind of hashlist, to tell the plugin what file should be loaded from where and what texture should it replace. The file should have the name of[GAME_CRC].yaml. In case of KH2FM, it would betxtconfig/F266B00B.yamlThis file would only have 1 table in it.
ProcessTEX. In that table, the texture replacements would be declared in this format:0x[TEXTURE_CRC]: [FILE_PATH][TEXTURE_CRC]being the hash that is generated for the texture on dump.[FILE_PATH]being the path of the file, relevant to thetexturesfolder in PCSX2's executable directory. For example,0x0CF2C584: "@COM/field2d/zz0field.dds"would load the file from say<pcsx2Install>/textures/@COM/field2d/zz0field.ddsBelow, you can see such a file as an example:
Dumping textures dumps every texture that is caught (and once again if the texture gets "completed") to
<pcsx2Install>/textures/@DUMP/[GAME_CRC]/[TEXTURE_CRC].dds, the[TEXTURE_CRC]being the hash generated for the texture, and ""[GAME_CRC]" is the CRC of the loaded game.To mitigate issues in speed, textures are dumped once a second.
The feature to allow dumps and replacements can be enabled from either the GUI, or by pressing the F10 key.
Pressing F11 will switch between dumping and replacing modes.
Why DDS?
Speed. Loading in, parsing, and handling DDS files, although are costly on space, are much faster and overall easier and more efficient. The poor performance of loading in big PNG files was a big complaint on the testing phases, and the switch to DDS was requested. PNG support can be added to co-exist with DDS later on if necessary.
Advantages:
Disadvantages:
Tested Games:
Known Issues:
Details:
Important Details:
Screenshots:
Here is a screenshot of the new GUI, packing these features:
Here are a few comparison screenshots, taken from Kingdom Hearts - Re: Chain of Memories (If any GUI Elements are missing, it's because they are not present in the DDS file. Did not have time to recreate them for this demonstration):
Great thanks to:
@Xeeynamo - Technical assistance, code checking.
@FerocityVine - General assistance, code testing, texture testing.
@Vladabdf - Texture packs for testing this code, advices given.
@Akita23 - A lot of testing.
Conclusion:
I would say this would be an awesome feature to have in PCSX2. It has been wildly requested, and it works great! No issues, overflows, slowdowns, etc. have been found in the testing period. If anything needs changing, altering or fixing PLEASE let me know.
Sincerely:
EDIT: Typo fixes.
EDIT: Changed content to be consistent with the latest additions and changes (3 times).