Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Poor performance when using rows to scan in data #119

Closed
jimmiebtlr opened this issue Feb 2, 2022 · 3 comments
Closed

Poor performance when using rows to scan in data #119

jimmiebtlr opened this issue Feb 2, 2022 · 3 comments
Labels
bug Something isn't working
Milestone

Comments

@jimmiebtlr
Copy link

jimmiebtlr commented Feb 2, 2022

Describe the bug
When using rows to read row by row, the performance is around 10x worse than using raw sql.

Note, I also gave it a try using raw sql, and using qrm.ScanOneRowToDest, with the same poor performance seen there.

Some additional background

  • Using a struct with 3 not huge jet models (total probably 30 fields, 10 of which are duplicate with an AS clause, roughtly 10 fields used total out of them).
  • Reducing fields to 1 model with 11 fields cuts the time 50%
  • I'm returning 1M rows.
  • Similar code that uses Query or QueryContext instead of Rows is returning in around 25% the time.

My guess is the overhead of reflecting on every row is driving this behavior.

Environment (please complete the following information):

  • OS: macosx
  • Database: mysql
  • Database driver: mysql
  • Jet version 2.5

Code snippet

q, args := query.Sql()
rows, err := db.Query(q, args...)
// handle err and defer close

for rows.Next() {
	rowData := generator()
	err = qrm.ScanOneRowToDest(rows, rowData)
	// handle err
}

Expected behavior
I expect the results to take < 40% current time.

@jimmiebtlr jimmiebtlr added the bug Something isn't working label Feb 2, 2022
@go-jet
Copy link
Owner

go-jet commented Feb 4, 2022

Hi @jimmiebtlr. Managed to reproduce it. New scanContext is unnecessarily created with each Scan call.
I've made a fix on the rows-perf branch. Rows scan should now be several times faster.

@go-jet go-jet added this to the Version 2.7.1 milestone Feb 4, 2022
@jimmiebtlr
Copy link
Author

Tested rows-perf branch and performance is much improved thanks!!

@go-jet go-jet mentioned this issue Feb 14, 2022
@go-jet
Copy link
Owner

go-jet commented Feb 14, 2022

Fixed with Release 2.7.1

@go-jet go-jet closed this as completed Feb 14, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

2 participants