Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Freeze results that come from the database #480

Merged
merged 2 commits into from
Jan 24, 2024
Merged

Freeze results that come from the database #480

merged 2 commits into from
Jan 24, 2024

Conversation

tenderlove
Copy link
Member

@tenderlove tenderlove commented Jan 23, 2024

Freeze strings and row records so that we can more easily share results among Ractors

Silly example, but here's an example of using database results plus Ractors to make a parallel producer-consumer pattern:

require "sqlite3"
require "securerandom"

def fib n
  if n < 2
    n
  else
    fib(n-2) + fib(n-1)
  end
end

N = 10

# Make a Ractor safe queue that supports N workers
queue = Ractor.new do
  while (work = Ractor.receive)
    Ractor.yield work
  end
  # Shutdown the workers
  N.times { Ractor.yield nil }
end

# Spin up workers
workers = N.times.map do
  Ractor.new(queue) do |q|
    while (work = q.take)
      name, i = work
      p "Ractor: #{Ractor.current} #{name}: #{fib(i)}"
    end
  end
end

db = SQLite3::Database.new ":memory:"
db.execute 'CREATE TABLE "fibs" (name string, fib int)'
stmt = db.prepare "INSERT INTO fibs (name, fib) VALUES (?, ?)"
5000.times { stmt.execute SecureRandom.hex, rand(30) }
stmt.close

stmt = db.prepare 'SELECT name, fib FROM fibs'

# Send work from the database to the queue
stmt.each { |row| queue.send row }

stmt.close

# Tell the queue we've sent all the work
queue.send nil

# Wait for workers to finish
workers.map(&:take)

Freeze strings and row records so that we can more easily share results
among Ractors
@tenderlove
Copy link
Member Author

This change breaks one test in Rails, so I changed Rails: rails/rails#50859

@tenderlove tenderlove merged commit 44ab2aa into main Jan 24, 2024
99 checks passed
@tenderlove tenderlove deleted the freeze-result branch January 24, 2024 02:35
@@ -162,13 +162,15 @@ step(VALUE self)
if (internal_encoding) {
val = rb_str_export_to_enc(val, internal_encoding);
}
rb_obj_freeze(val);
Copy link
Member

@flavorjones flavorjones Apr 16, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm, this seems to break Rails's schema cache tests (which use Marshal):

bin/test test/cases/connection_adapters/schema_cache_test.rb -n/test_gzip_dumps_identical/

Without this line, the test passes. With it, the test fails (in isolation).

I'm looking into it.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Context on that test: rails/rails#49166

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See rails/rails#51581 for the exciting conclusion.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants