Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Adding csv archiving #20

Merged
merged 9 commits into from
Oct 3, 2023
Merged

Adding csv archiving #20

merged 9 commits into from
Oct 3, 2023

Conversation

edmoss345
Copy link
Collaborator

Changes to pipeline to accommodate archive functionality

@edmoss345 edmoss345 requested a review from istride September 25, 2023 04:37
@edmoss345
Copy link
Collaborator Author

Three PRs are connected as this archive process required an update to the SA repo, the pipeline and the toolkit
#20
IDEMSInternational/parenttext-south-africa#7
IDEMSInternational/rapidpro-flow-toolkit#99

@edmoss345
Copy link
Collaborator Author

@istride I've made some changes as we discussed. Have pulled the 'archive' command into its own file so it is not part of the general pipeline. However, have been running into problems with the archive process, doesn't seem to save the csvs correctly when there is a ":" in the name of the sheet, but can't pin down where exactly it's failing and can't find a fix. Example output can be found in the SA repo PR. One for us to discuss

https://github.com/IDEMSInternational/parenttext-south-africa/pull/7/files#diff-82e569e3d1dd3c813c81a8ee5044604d27b37acf72c960ebe1b7883b32f86e8d

@istride
Copy link
Collaborator

istride commented Oct 2, 2023

I think the problem with the colon character (":") is specific to Windows. I managed to create an archive using the code in this PR, on my Linux machine, with colon characters intact. In the short-term, we can just use a Linux system to create the archive - via Docker, a VM or Github Actions.

There is potentially a bigger problem here because I can't guarantee that there aren't characters that would not work in Linux file names. There a couple of ways we might deal with this:

  • create a 'safe' file name for each sheet and keep a mapping from the safe name to the real sheet name so that we can restore it when it's needed
  • define a set of invalid characters that may not be used in sheet names and make sure that they are not used

@istride istride force-pushed the adding-csv-archiving branch from c92fdff to 89e981d Compare October 3, 2023 21:37
@istride istride merged commit 2041431 into main Oct 3, 2023
1 check passed
@istride istride deleted the adding-csv-archiving branch December 12, 2023 15:38
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants