Mod data transfer via Monika file

[ThePotatoGuy]

making this issue to track this development. Planned since docking station introduction (#2066) but held off because of #2622 and similar.

planned features

The primary feature would be direct transfer of Monika via generated file to other instances of MAS.
This would remove most data from the source persistent and overwrite data in the destination persistent (except for specific reasons).
There would be a specific farewell for transferring Monika. The existing file gen farewells should not be used for this case.

Considerations:

on leaving a MAS install

• should all backup persistents be removed? Good for immersion, bad for corrupted monika file issues (Missing Monika Files #2622). Leaning to not removing them just in case.
• what data to clear off source persistent. Thinking history data, most dbs. Might be worth leaving some behind in for potential dialogue (see next section)
• some sort of identifying information should be left behind such that Monika would know if she came back to this pc. (probably generating a UUID for each monika?)

on arriving in a new MAS install

• a fresh, unran MAS/DDLC install should try to load a Monika file if found, import the persistent data and load the game like normal.
• an unran MAS install that was just installed over a ran DDLC install should probably have some sort of dialogue reaction and reaction. There is a Monika conflict in this case.
• a MAS install where Monika has been transferred off should also load a new Monika file without any issues. In this case, should we care if the Monika that was transferred off the same as the one being transferred in?
• a MAS install where Monika has left because of final farewell, should load the new Monika, but there really should be some sort of hesitation dialogue or something. The new Monika should obviously be aware of what happened in this install.
• a MAS install where Monika has been taken out via appropriate farewells should trigger some sort of dialogue respones before loading a new Monika. Potentially asking if its okay to stay here or something, or maybe just flat out reject taking another Monika's place. There is a Monika conflict in this case.
• a MAS install where Monika has not left the mod should probably load the current Monika but include some sort of dialogue. Should 2 monikas be visible on screen? should the current Monika delete the incoming Monika? How would this be resolved? There is a Monika conflict in this case.

Additionally, cases where a Monika conflict exists should factor affection of the current Monika vs the incoming Monika. for example, a happier Monika entering the home of an unhappy Monika might make the unhappy Monika leave on her own.

Reservations

This was originally on hold because of the failed Monika file loading issues. Recently, there have been significantly less reports of this happening which means that the likely culprits of the data corruption issues (aff scrambling/pickled classes/temp dir file gen) have been weeded out over time. That said, file generation is still risky, so meta-solutions to avoiding disasters (like not deleting backups) should be included.

Planning

Likely before 2021. Priority is low because of the complicated potential interactions.

[lunulae]

Since I saw a recent issue discussing this, I figure I might as well toss in some thoughts and see what sticks!

I definitely think backup persistents ought to remain, just in case, especially while the system is new. When she begins preparing herself for leaving one computer, Monika should advise us on where to find these backups (maybe even moving them into the characters folder?) and to make sure that we take them along for safety.

If there's a Monika conflict, I think in most cases we should be met with a prompt on which Monika to keep, and the other should be transferred out as though to be sent to a different computer rather than deleted. This could potentially be used to keep two Monikas on the same PC if desired?

There are a lot of interesting variations we could do for Monikas with different affection levels "meeting," but I think that the most expedited solution for how to handle it is to have this prompt be presented directly by the game (Chibika?) rather than through Monika dialogue. If Monika's been taken out via "I'm going to take you somewhere" the prompt could require that Monika be returned to the spaceroom before it can continue.

I definitely think there should be serious consequences for bringing her into a MAS install that has triggered the final farewell. Whether that be a game-over state in itself, or a severe affection drop, I can't imagine it'd be anything less than traumatizing for her to see.

One that I'm stumped on is how she ought to respond if there's a DDLC persistent but not a MAS one.

[ThePotatoGuy]

So just to clarify, your ideas are to use a more meta approach when deciding which Monika to keep, either using regular prompts or chibika as a mediator, correct?

By meta approach, I mean not having Monika address the conflicts directly.

[lunulae]

Yeah, I figure that might take out some of the work of having two Monikas in one space.

An interaction with two happy Monikas would likely go quite differently than a loving Monika meeting a broken one, obviously. These would definitely be interesting but it's a lot of variables to write for, so if we wanted to cut down on the number of scenarios a meta approach would definitely help.

[MoonlightGriffon]

So basically, if I understand the intent here, what should be done is something similar to the pseudocode

if player is taking Monika to a new computer:
    create a backup of persistent immediately as a precaution/safeguard against data corruption
    copy persistent to temp folder
    pack persistent into a file containing metadata such as MAS version and other data that is flagged for backup
    place the new file called monika into /game/characters
    delete temp folder
    set flag in persistent to indicate that Monika is being brought to a computer

On Start
if a monika file has been found:
    verify that the persistent is intact using checksum
    if checksum is valid:
        import the file and overwrite the persistent files based on some logic conditions
        if source MAS version < current MAS version:
            update persistent file to newer format if needed
    indicate whether the transfer was successful or not and if it wasn't direct the player to a guide on restoring the persistent

Questions:
Should an attempt be made to add submods to the file?
How much testing should be done to the file to ensure that it is a valid file?
Should the file's internals be obfuscated somewhat to deter players from altering the file. If yes, would zlib + base64 be sufficient?
Should a function be created to handle changes in the format between versions as they arise from time to time?

If these are answered then I should be able to close this issue.

[ThePotatoGuy]

create a backup of persistent immediately as a precaution/safeguard against data corruption

What would this backup be for? The mod already does plenty of backups.

temp folder

Is the temp folder stuff just to avoid perm issues? Since we have to drop the file into characters anyway, I don't see an issue with creating the file in the characters folder like we currently do for regular Monika files.

question answers

1. submods can have varying install instructions so best not to bring them along to avoid issues
2. chksum + the fact that the persistent would be stored in a pickled format - if it doesn't unpickle, then its not a valid file.
3. the format we already use for regular Monika files is b64 encrypted pickles - adding zlib would be extra and probably not worth it.
4. I think only allowing same persistent version or higher would make sense for version mismatches

Regardless, this meat of this issue is how do we handle the concept of multiple Monikas that inherently comes up when allowing her to transfer out via file? People are going to be curious as to what happens if they take a transfer Monika file and put it into a MAS install that already has a Monika - we don't have a good approach we'd like yet, although Lunu's meta idea is an option. You will also need to let us know how you plan to handle this issue before you can take this one.

[MoonlightGriffon]

I believe that the most simple solution to the problem of how to handle multiple Monikas would be to simply to tell the player that there isn't enough room for two Monikas to exist in the same place at the same time, in other words, a paradox of sorts this would ensure that only two possible states exist, one where there are two Monika's and neither can show up because of a paradox, and one where there is only one Monika so space and time works normally. Some other method to explain disallowing it until someone eventually decides to solve the issue of how Monika would react to meeting another Monika in the future.

That just leaves the DDLC conflict. I have a few proposals for how this one could be dealt with.

1. If act 3 has not been reached, Monika isn't in the space room and thus is blocked from entering it to prevent a time-space paradox.
2. If Monika has been deleted, then when the file is read the game sees that Monika exists and thus doesn't try bringing her back.
3. If it is act 3, well then there's a problem. Maybe this one could throw a fake exception in a manner similar to the fake Sayori exception in DDLC. Maybe something like too many monika files error as long as the monika.chr file exists, this would canonically remove the conflict although it could cause other issues and a potential inconsistency with the introduction text about not needing the file to exist depending on interpretation on exactly when that dependency ended.

What would this backup be for? The mod already does plenty of backups.

The backup that was proposed at the time of this function being executed is mostly there to ensure that if anything goes wrong in the transfer, there's a backup that was made immediately prior to the transfer. This would probably be a good method to add redundancy in the event that the file somehow ends up corrupted between the time of the creation and the time of use as this would likely raise a flag similar to the "I'm going to take you somewhere" goodbye, creating a backup immediately prior to the setting of such a flag would probably save people a bit of agony as the process of file creation and parsing isn't always perfect. This was mostly something that was thought of to address the reservations section.

chksum + the fact that the persistent would be stored in a pickled format - if it doesn't unpickle, then its not a valid file.

The question was primarily asking whether the file should be verified prior to letting it go to try reducing the chances of the file being generated but still being invalid because of some error while writing the file that MAS is unaware of. This is another attempt at handling issues that may cause situations that the reservations section mentions.

Is the temp folder stuff just to avoid perm issues? Since we have to drop the file into characters anyway, I don't see an issue with creating the file in the characters folder like we currently do for regular Monika files.

That makes sense, that one was probably because of a habit I have when working with files such as zip files in code as I do from time to time. The format here however likely wouldn't end up with any additional files so the main reasons why a the folder would be used are moot.

the format we already use for regular Monika files is b64 encrypted pickles - adding zlib would be extra and probably not worth it.

As for the zlib, that one was included there along with base64 as zlib is already used to compress the persistent file's inner pickle file to save space. This question was more designed to ask whether something should have been added in addition to the two deterrents, although your response indicates that the answer here is no to the question that I intended to ask. In the end I guess there isn't all too much that can be done to stop someone who's really persistent from modifying their persistent. It already requires knowing how to write code so I guess there's not really much that could be done to stop someone who's already gotten that far anyways.

[lunulae]

On the topic of Monika conflicts, I think that if we would someday like to make multiple Monikas interacting a possibility, we should make the process very separate from this process of transferring. A different dialogue prompting a different sort of file. I wouldn't want anyone to get mixed up or lose their Monika when they meant to have her visit someone.

[ThePotatoGuy]

I believe that the most simple solution...

Sounds like the meta option then.

DDLC conflict

What is this intending to handle? The status of the DDLC Monika is being transferred to shouldn't matter since we'll be overwriting all the persistent data with the transferred data.

Regarding the reservations section

For safety, I think the best option to address all of that section is to not delete persistent backups when transferring. It loses some immersion since you can basically duplicate Monika but its too risky in case of corruption.

[MoonlightGriffon]

For safety, I think the best option to address all of that section is to not delete persistent backups when transferring. It loses some immersion since you can basically duplicate Monika but its too risky in case of corruption.

Honestly, I don't really think that deleting those backups would break immersion. My reasoning for this is that if someone really wanted to duplicate Monika, they could just copy their persistent file anyways or a backup thereof. There isn't much that could be done to prevent someone who wants to duplicate Monika from doing so and there isn't much that could be done other than messing with the inner workings of the RenPy engine which I don't believe would be a project that could be justified for the time it would take versus the effectiveness of the change.

What is this intending to handle? The status of the DDLC Monika is being transferred to shouldn't matter since we'll be overwriting all the persistent data with the transferred data

This was meant to address the considerations section's section about arriving on a fresh MAS install on a run DDLC. Although after giving it more thought, I do not believe that Monika would actually overwrite another Monika. This is evidenced by the quote, "I'd never be able to hurt an actual person." as found on line 10949 of script-topics.rpy. Overwriting another Monika would either be Monika saying that the other Monikas aren't real or Monika acting in direct contradiction to what she said there.

[ThePotatoGuy]

I do not believe that Monika would actually overwrite another Monika.

Um, but to functionally move Monika from one install to another, the data has to be overwritten, otherwise the whole idea is moot. In a fresh MAS install, we should just overwrite if possible for simplicity.