Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle items with >10⁶ elements in Excel data I/O #292

Closed
khaeru opened this issue Mar 25, 2020 · 1 comment · Fixed by #309
Closed

Handle items with >10⁶ elements in Excel data I/O #292

khaeru opened this issue Mar 25, 2020 · 1 comment · Fixed by #309
Assignees
Labels
enh New features & functionality

Comments

@khaeru
Copy link
Member

khaeru commented Mar 25, 2020

In #289 (comment), @zikolach pointed out that some Scenarios may have parameters with more than 1 million elements, that might exceed the size limit of the Microsoft Excel format or the Python tools used to read/write it.

Scenario.{read,write}_excel() and the associated format should be:

  • Tested for their behaviour with such 'large' data, and
  • Extended to support round-trip write and read of such data.
@khaeru khaeru added the enh New features & functionality label Mar 25, 2020
@khaeru
Copy link
Member Author

khaeru commented Mar 27, 2020

Further suggestion from @zikolach at #289 (review):

In general would be good to allow splitting items data into multiple sheets as we can reach rows limit. We could avoid to have separate sheet with item name-type mapping by embedding type into sheet name, e.g. type|name|n or similar convention, where type is item type, name - item name and n is a number of sheet for "large" items.

That sounds like a good proposal.

Users may have old Excel spreadsheets sitting around and will expect them to continue working. So we would need to add a second code path for these alternate formats; support both for some time; and then deprecate & eventually remove the current one. We would need an explicit decision about when the current format will no longer be supported.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enh New features & functionality
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants