Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Fix CSV render error #17406

Merged
merged 6 commits into from
Oct 24, 2021
Merged

Fix CSV render error #17406

merged 6 commits into from
Oct 24, 2021

Conversation

KN4CK3R
Copy link
Member

@KN4CK3R KN4CK3R commented Oct 22, 2021

closed #17378

Both errors from #17378 were caused by #15175.

Problem 1 (error with added file):
ToUTF8WithFallbackReader creates a MultiReader from a byte[2048] and the remaining reader. CreateReaderAndGuessDelimiter tries to read 10000 bytes from this reader but only gets 2048 because that's the first reader in the MultiReader. Then the if size < 1e4 thinks the input is at EOF and just returns that.

Problem 2 (error with changed file):
The blob reader gets defer closed. That was fine because the old version reads the whole file into memory. Now with the streaming version the close needs to defer after the method.

@richmahn
Copy link
Contributor

richmahn commented Oct 22, 2021 •
edited
Loading

Shall we add a unit test for this in gitdiff/csv_test.go? Easy to generate more than 2048 chars of a CSV with https://www.convertcsv.com/generate-test-data.htm

Or maybe even more than 1e4 bytes to make sure nothing gets closed/lost?

@GiteaBot GiteaBot added the lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. label Oct 22, 2021
zeripath
zeripath reviewed Oct 22, 2021
View reviewed changes
modules/csv/csv.go Outdated
Comment on lines 32 to 33
var buf = make([]byte, 1e4)
n, err := rd.Read(buf)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Are you trying to fill that buffer if you at all can? There's no guarantee even if the size is 16Mb that the buffer will contain more than one byte. You need to use ReadAtLeast or ReadFull.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Will add this in a follow up PR because that needs to be fixed in more places like LFS pointer detection.

@6543 6543 added this to the 1.16.0 milestone Oct 22, 2021
@GiteaBot GiteaBot added lgtm/need 1 This PR needs approval from one additional maintainer to be merged. and removed lgtm/need 2 This PR needs two approvals by maintainers to be considered for merging. labels Oct 22, 2021
@GiteaBot GiteaBot added lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. and removed lgtm/need 1 This PR needs approval from one additional maintainer to be merged. labels Oct 22, 2021
6543
6543 reviewed Oct 24, 2021
View reviewed changes
6543
6543 reviewed Oct 24, 2021
View reviewed changes
6543
6543 approved these changes Oct 24, 2021
View reviewed changes
Copy link
Member

@6543 6543 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2nd

@6543 6543 merged commit 44f2c27 into go-gitea:main Oct 24, 2021
@lunny
Copy link
Member

lunny commented Oct 25, 2021

Please send backport to v1.15

KN4CK3R added a commit to KN4CK3R/gitea that referenced this pull request Oct 25, 2021
@KN4CK3R
Fix CSV render error (go-gitea#17406)
02b5b3f 
closed go-gitea#17378 

Both errors from go-gitea#17378 were caused by  go-gitea#15175.

Problem 1 (error with added file):
`ToUTF8WithFallbackReader` creates a `MultiReader` from a `byte[2048]` and the remaining reader. `CreateReaderAndGuessDelimiter` tries to read 10000 bytes from this reader but only gets 2048 because that's the first reader in the `MultiReader`. Then the `if size < 1e4` thinks the input is at EOF and just returns that.

Problem 2 (error with changed file):
The blob reader gets defer closed. That was fine because the old version reads the whole file into memory. Now with the streaming version the close needs to defer after the method.
@KN4CK3R KN4CK3R mentioned this pull request Oct 25, 2021
Merged
@KN4CK3R
Copy link
Member Author

KN4CK3R commented Oct 25, 2021

I still think data and size are not good. n and buf are used most of the time (28x buf vs. 2x data and 49x n vs. 1x size).

zeripath added a commit that referenced this pull request Oct 25, 2021
@KN4CK3R @zeripath
Fix CSV render error (#17406) (#17431)
1fbdf96 
Backport #17406.

Closes #17378 

Both errors from #17378 were caused by  #15175.

Problem 1 (error with added file):
`ToUTF8WithFallbackReader` creates a `MultiReader` from a `byte[2048]` and the remaining reader. `CreateReaderAndGuessDelimiter` tries to read 10000 bytes from this reader but only gets 2048 because that's the first reader in the `MultiReader`. Then the `if size < 1e4` thinks the input is at EOF and just returns that.

Problem 2 (error with changed file):
The blob reader gets defer closed. That was fine because the old version reads the whole file into memory. Now with the streaming version the close needs to defer after the method.

Co-authored-by: zeripath <[email protected]>
@KN4CK3R KN4CK3R deleted the fix-17378 branch October 26, 2021 14:20
zeripath added a commit to zeripath/gitea that referenced this pull request Oct 27, 2021
@zeripath
Changelog 1.15.6
027f153 
Unforunately go-gitea#17435 is a somewhat critical bug and therefore we should
really release 1.15.6 as soon as possible.

 ## [1.15.6](https://github.com/go-gitea/gitea/releases/tag/v1.15.6) - 2021-10-27

* BUGFIXES
  * Prevent panic in serv.go with Deploy Keys (go-gitea#17434) (go-gitea#17435)
  * Fix CSV render error (go-gitea#17406) (go-gitea#17431)
  * Read expected buffer size (go-gitea#17409) (go-gitea#17430)

Signed-off-by: Andrew Thornton <[email protected]>
@zeripath zeripath mentioned this pull request Oct 27, 2021
Merged
lunny pushed a commit that referenced this pull request Oct 28, 2021
@zeripath @wxiaoguang
Changelog 1.15.6 (#17457)
c7d8181 
* Changelog 1.15.6

Unforunately #17435 is a somewhat critical bug and therefore we should
really release 1.15.6 as soon as possible.

 ## [1.15.6](https://github.com/go-gitea/gitea/releases/tag/v1.15.6) - 2021-10-27

* BUGFIXES
  * Prevent panic in serv.go with Deploy Keys (#17434) (#17435)
  * Fix CSV render error (#17406) (#17431)
  * Read expected buffer size (#17409) (#17430)

Signed-off-by: Andrew Thornton <[email protected]>

* Add 17456 and its backport

Signed-off-by: Andrew Thornton <[email protected]>

* Add 17464

Signed-off-by: Andrew Thornton <[email protected]>

* Add final pr

* Update date

Co-authored-by: wxiaoguang <[email protected]>
zeripath added a commit to zeripath/gitea that referenced this pull request Oct 28, 2021
@zeripath
Changelog 1.15.6 (go-gitea#17457)
1060e23 
Frontport go-gitea#17457

 ## [1.15.6](https://github.com/go-gitea/gitea/releases/tag/v1.15.6) - 2021-10-28

* BUGFIXES
  * Prevent panic in serv.go with Deploy Keys (go-gitea#17434) (go-gitea#17435)
  * Fix CSV render error (go-gitea#17406) (go-gitea#17431)
  * Read expected buffer size (go-gitea#17409) (go-gitea#17430)
  * Ensure that restricted users can access repos for which they are members (go-gitea#17460) (go-gitea#17464)
  * Make commit-statuses popup show correctly (go-gitea#17447) (go-gitea#17466)
* TESTING
  * Add integration tests for private.NoServCommand and private.ServCommand (go-gitea#17456) (go-gitea#17463)
@zeripath zeripath mentioned this pull request Oct 28, 2021
Merged
zeripath added a commit that referenced this pull request Oct 28, 2021
@zeripath
Changelog 1.15.6 (#17457) (#17468)
141d1a2 
Frontport #17457

 ## [1.15.6](https://github.com/go-gitea/gitea/releases/tag/v1.15.6) - 2021-10-28

* BUGFIXES
  * Prevent panic in serv.go with Deploy Keys (#17434) (#17435)
  * Fix CSV render error (#17406) (#17431)
  * Read expected buffer size (#17409) (#17430)
  * Ensure that restricted users can access repos for which they are members (#17460) (#17464)
  * Make commit-statuses popup show correctly (#17447) (#17466)
* TESTING
  * Add integration tests for private.NoServCommand and private.ServCommand (#17456) (#17463)
@zeripath zeripath added the backport/done All backports for this PR have been created label Dec 25, 2021
Chianina pushed a commit to Chianina/gitea that referenced this pull request Mar 28, 2022
@KN4CK3R
Fix CSV render error (go-gitea#17406)
0c29d91 
closed go-gitea#17378 

Both errors from go-gitea#17378 were caused by  go-gitea#15175.

Problem 1 (error with added file):
`ToUTF8WithFallbackReader` creates a `MultiReader` from a `byte[2048]` and the remaining reader. `CreateReaderAndGuessDelimiter` tries to read 10000 bytes from this reader but only gets 2048 because that's the first reader in the `MultiReader`. Then the `if size < 1e4` thinks the input is at EOF and just returns that.

Problem 2 (error with changed file):
The blob reader gets defer closed. That was fine because the old version reads the whole file into memory. Now with the streaming version the close needs to defer after the method.
Chianina pushed a commit to Chianina/gitea that referenced this pull request Mar 28, 2022
@zeripath
Changelog 1.15.6 (go-gitea#17457) (go-gitea#17468)
0a60418 
Frontport go-gitea#17457

 ## [1.15.6](https://github.com/go-gitea/gitea/releases/tag/v1.15.6) - 2021-10-28

* BUGFIXES
  * Prevent panic in serv.go with Deploy Keys (go-gitea#17434) (go-gitea#17435)
  * Fix CSV render error (go-gitea#17406) (go-gitea#17431)
  * Read expected buffer size (go-gitea#17409) (go-gitea#17430)
  * Ensure that restricted users can access repos for which they are members (go-gitea#17460) (go-gitea#17464)
  * Make commit-statuses popup show correctly (go-gitea#17447) (go-gitea#17466)
* TESTING
  * Add integration tests for private.NoServCommand and private.ServCommand (go-gitea#17456) (go-gitea#17463)
@go-gitea go-gitea locked and limited conversation to collaborators Apr 28, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@zeripath zeripath zeripath left review comments

@richmahn richmahn richmahn approved these changes

@6543 6543 6543 approved these changes

Assignees
No one assigned
Labels
backport/done All backports for this PR have been created lgtm/done This PR has enough approvals to get merged. There are no important open reservations anymore. type/bug
Projects
None yet
Milestone
1.16.0
Development

Successfully merging this pull request may close these issues.

Editing CSV after 2048 chars results in "Can't render this file because it has a wrong number of fields" error
6 participants
@KN4CK3R @richmahn @lunny @zeripath @6543 @GiteaBot