Why are these changes needed?

Adds a feature flag to run sync user-defined methods in a threadpool by default. This matches the existing behavior when using a FastAPI ingress.

This should address a lot of user confusion and make it easier to write performant code by default. For example, just sticking a torch model call in a sync method will now provide reasonable performance out of the box.

However, there may be some existing user code that is not thread safe, so we need to do a gentle migration. This PR introduces the behavior behind a feature flag and warns users about the upcoming change and how to opt into the new behavior or maintain existing behavior once it does (just adding async def will do it).

I've opted to set the max thread pool size to max_ongoing_requests, which seems like a reasonable policy. If needed we can add a user-facing API for this in the future.

TODO before merging:

• Get it working for sync generators.
• Add warning for default change (people can keep behavior by changing to async def).
• Add/update UserCallableWrapper tests.
• Add/update some integration tests (verify that request context is set correctly!).
• Set maximum thread pool size.

Related issue number

Closes #44354
Closes #44403
Closes #48903

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(