When virtual threads are enabled, auto-configure an AsyncTaskExecutor that uses them

[wilkinsona]

Currently, TaskExecutionAutoConfiguration auto-configures a ThreadPoolTaskExecutor built using TaskExecutorBuilder. We should consider providing an option to auto-configure a SimpleAsyncTaskExecutor that uses virtual threads instead. Following these changes in Framework, SimpleAsyncTaskExecutor can be configured to use virtual threads by calling setVirtualThreads(true).

[rafaelrc7]

Hello, is this issue available for work? If so, I would like to ask if there is a specific way to check for virtual threads availability in spring-boot. Thanks.

[NicklasWallgren]

@wilkinsona
Would it be a viable option to set a ThreadFactory (which supports virtual threads) directly on the ThreadPoolTaskExecutor via a customizer?

The default coreSize (of 8) would still need to be tweaked.

You would then be able to take advantage of other functionality related to the ThreadPoolTaskExecutor, which isn't present in the AsyncTaskExecutor, such as awaitTermination.

POC main...NicklasWallgren:spring-boot:35710-virtual-threads-thread-pool-task-executor

[wilkinsona]

Thanks for suggestion but I don't think that's the right approach. The JEP has a section that explicitly states that virtual threads should not be pooled. As such, I think it would be confusing for Boot's default configuration when virtual threads are enabled to use a ThreadPoolTaskExecutor with a virtual thread factory.

If you believe that awaitTermination would be useful, you may want to suggest it as an enhancement to the Spring Framework team.

[NicklasWallgren]

Alright, seems reasonable. Thanks!

[vladimirfx]

I've tested virtual threads support in 3.2.0-M1 and missing support for TaskScheduler. No issue was found for virtual threads in scheduling either.

Is it planned to use virtual threads for scheduled tasks?

[vladimirfx]

Currently, I've used this bean definition as a workaround for task scheduling:

    bean<ConcurrentTaskScheduler>(name = "taskScheduler", isLazyInit = true) {
        ConcurrentTaskScheduler(Executors.newScheduledThreadPool(0, Thread.ofVirtual().factory()))
    }

[wilkinsona]

As described in the JEP, virtual threads should not be pooled so we don't think it makes sense to use them for task scheduling.

Advice/commentary from the Framework team:

SchedulingConfigurer: a ThreadPoolTaskScheduler configured there could have setThreadFactory with new VirtualThreadTaskExecutor().getVirtualThreadFactory() or taking the ThreadFactory from a shared VirtualThreadTaskExecutor bean. However, it still pools its threads (1 by default), so isn’t idiomatic with virtual threads to begin with. Might be best to leave this with a single fairly scheduled platform thread by default.

[vladimirfx]

Yes, I know - it is just a workaround for now.

I can't find a suitable TaskScheduler implementation without thread pooling (like SimpleAsyncTaskExecutor for async tasks).

So my question:

Is it planned to use virtual threads for scheduled tasks? (virtual threads-compatible TaskScheduler implementation)

Thank you!

[wilkinsona]

Is it planned to use virtual threads for scheduled tasks?

No. As I said above, the Framework team believe that using a single fairly scheduled platform thread is the best option.

If you disagree with that and think that a virtual thread based implementation of TaskScheduler would be useful, please raise a Spring Framework issue.

[vladimirfx]

It seems I missing something... How async and the scheduled task is different in the sense of execution? Why async task is executed on the virtual thread but the scheduled is not?

What I expect is execution of scheduled tasks on virtual threads when spring.threads.virtual.enabled=true. Just like async tasks behaves...

[wilkinsona]

The fundamental difference is that, by default, async execution is performed using multiple threads and scheduled task execution is performed using a single thread. As such, async execution lends itself well to the use of virtual threads – the pool of multiple platform threads can be replaced with unpooled virtual threads – but scheduled task execution does not. There's little point in replacing a single fairly scheduled platform thread with a single virtual thread.

[vladimirfx]

Thank you for such detailed explanation!

So difference in default concurrency of scheduled tasks. Unfortunately I have no projects for 17 years with scheduler concurrency = 1. And no such defaults either (Quartz, JEE, Quarkus, Micronaut etc)

I've try to file issue and provide PR against Framework.

Thanks again

[jhoeller]

@vladimirfx while a ticket in the Spring Framework issue tracker would be appreciated, I don't see us implementing and maintaining a totally custom TaskScheduler for this. We could provide a convenient out-of-the-box option for a virtual threads ThreadFactory setup on ThreadPoolTaskScheduler, is this what you have in mind? Or maybe rather a separate VirtualThreadTaskScheduler class doing the equivalent internally but not sounding so thread-pooly in the class name? ;-)

That said, a common arrangement considered for such scenarios is a ThreadPoolTaskScheduler with a single scheduler thread, and every scheduled task then handing over to a thread of its own, as indicated by this StackOverflow answer:
https://stackoverflow.com/questions/76587253/how-to-use-virtual-threads-with-scheduledexecutorservice
The equivalent with Spring annotations is an @Scheduled @Async method, this would dispatch to a virtual threads executor with the current Boot arrangement already.

In order to avoid such an @Async declaration on every @Scheduled method, we could also bake such an option into ThreadPoolTaskScheduler so that it dispatches every callback to a given separate TaskExecutor which could point to the virtual one in Boot. This would use a single platform thread for scheduling and one virtual thread per task for the callbacks. Do you see this as preferable to an Executors.newScheduledThreadPool(0, Thread.ofVirtual().factory()) style setup, or are you rather relying on the serialized execution of tasks as in a regular scheduled thread pool?

[jhoeller]

As a side note, is Executors.newScheduledThreadPool(0, Thread.ofVirtual().factory()) really helpful here? If there is usually something scheduled at any point, there is always going to be at least 1 thread around which does the scheduling. If that thread is virtual, it does not occupy any significant resources anyway. From that perspective, Executors.newScheduledThreadPool(1, Thread.ofVirtual().factory()) sounds like the more appropriate arrangement. Also, you were indicating that you need a higher concurrency level for the scheduler, so I suppose that 0 was not indicative to begin with?

I'm wondering which configuration options we actually need for a virtual thread based TaskScheduler. Is it largely the same as on ThreadPoolTaskScheduler, or are there options which definitely do not make sense in a virtual thread setup or would always make sense in a virtual thread setup, in which case a distinct VirtualThreadTaskScheduler might be a better representation.

Generally speaking, our preference is still an @Scheduled @Async like execution model with one virtual thread per scheduled callback. In that case, the scheduler will always be based on a single scheduler thread, so this might actually be better off as a distinct TaskScheduler variant (with a hard-coded single scheduler thread and a pre-defined delegate executor) rather than baking that option into ThreadPoolTaskScheduler. Also, the shutdown behavior would be different since the scheduled callbacks run in unmanaged threads then, without the ScheduledExecutorService waiting for them on shutdown. That's the case for VirtualThreadTaskExecutor as well which also has different shutdown behavior than a ThreadPoolTaskExecutor.

[vladimirfx]

Or maybe rather a separate VirtualThreadTaskScheduler class doing the equivalent internally but not sounding so thread-pooly in the class name? ;-)

Exactly this variant I prototyping now. With configurable concurrency level (implemented though semaphore). I even preserve default concurrency = 1 😉

1 virtual thread for scheduling is better that one platform because most of time such a thread is waiting. Waiting virtual thread is way chipper than platform thread.

From that perspective, Executors.newScheduledThreadPool(1, Thread.ofVirtual().factory()) sounds like the more appropriate arrangement

0 indicating that no thread cache is needed. Yes, 1 is more appropriate but changes nothing in semantic.

Thank you for pointing to shutdown semantic differences - I've try to overcome it.

[jhoeller]

On a related note, I'm about to introduce a SimpleAsyncTaskScheduler (spring-projects/spring-framework#30956) which follows up on my 6.1 M2 executor/scheduler revision. This inherits the setVirtualThreads(true) capability from SimpleAsyncTaskExecutor (and also its configurable concurrency limit) and - when configured that way - uses a single virtual thread for scheduling and and a separate individual virtual thread per scheduled task execution. This is effectively the @Scheduled @Async like execution model that I was referring to above. I'll commit this tonight since this is generally a variant that we see as worth having.

Which still leaves room for a more pool-like virtual thread based scheduler. So if you are working on a custom implementation there, how does it effectively differ from a ThreadPoolTaskScheduler with a virtual ThreadFactory setup which is in turn similar to your ConcurrentTaskScheduler(Executors.newScheduledThreadPool(0, Thread.ofVirtual().factory())) setup from above? As a side note, ThreadPoolTaskScheduler also comes with pause/resume/shutdown lifecycle integration which we need for CRaC.

Last but not least, there is the question of what's a good default scheduler setup for Boot's virtual threads property. There is always the option of providing a custom executor/scheduler instead... however, the default choice is hard in its own right.

[NicklasWallgren]

@jhoeller How would one go about implementing graceful shutdown while using virtual threads in Spring Boot? It seem as neither VirtualThreadTaskExecutor nor SimpleAsyncTaskExecutor supports SmartLifecycle or similar functionality provided by ExecutorConfigurationSupport#shutdown

Sorry if I'm missing something obvious.

[jhoeller]

General executors do not necessarily need to support lifecycle integration as long the submitters of the tasks are in control of the submitted tasks, setting active or shutdown signals on them and waiting for them to complete on shutdown. This is the case e.g. for our JMS DefaultMessageListenerContainer and its asynchronous invokers, and it is typically also the case for Future-based submissions where the caller interacts with the task through the Future handle. In particular for common framework-submitted tasks, there is no need to track their lifecycle in the executor itself. However, for custom programmatic submissions, it might be unclear which tasks are still running if the original submitter does not hold on to them.

For a non-pooled executor, an executor-controlled shutdown of all tasks would only be possible by holding on to all active tasks within the executor, waiting for their completion that way. To some degree, this goes against the grain of virtual threads, or at least against their idiomatic usage. If such a controlled shutdown is desirable, possibly in conjunction with a concurrency limit as well, I would actually consider a ThreadPoolTaskExecutor setup with a virtual ThreadFactory. The overhead of tracking each individual submitted task would probably outweigh the pooling overhead for virtual threads, so I'd rather go with a ThreadPoolTaskExecutor with pooled virtual threads for such purposes, and likewise with a ThreadPoolTaskScheduler.

In terms of lifecycle management, a scheduler is a bit of a different beast: Since it has a scheduler thread that keeps triggering periodic task executions, it absolutely needs to participate in a controlled shutdown and it should also participate in context-wide pause/resume steps where its triggers need to be suspended. The new SimpleAsyncTaskScheduler implements SmartLifecycle for that reason, not managing the individual task executions that way but rather just its single scheduler thread.

As hinted at above, there is room for several strategies here. It's just that the default choice for a Boot setup is hard: For classic thread pools, it's the question of which pool limit to use depending on how many threads are typically blocked. For a virtual thread executor, it's the question of how important executor-controlled lifecycle management is for a given application. I would not be surprised if common practice will turn out to use thread pools with virtual threads as a compromise between those two.

[NicklasWallgren]

Great, thank you for the detailed explanation!

[jhoeller]

On review, it looks straightforward to provide a waitForTasksToCompleteOnShutdown option in SimpleAsyncTaskExecutor and SimpleAsyncTaskScheduler, tracking the execution Thread for each Runnable and some interruption/notification signals on shutdown so that we can wait for the set of active Threads to be empty. This does introduce some overhead, so it won't be on by default, but it seems worth providing as an opt-in flag for a controlled shutdown in a scenario where unmanaged task submissions are involved.

[vladimirfx]

I've looking for something similar. If we implement such tracking we effectively duplicate the functionality of ScheduledExecutorService.

So how it will be better than this:

Executors.newScheduledThreadPool(0, Thread.ofVirtual().factory())

My tries to implement correct shutdown behavior led me to the conclusion that I reimplement the pooling scheduler from JRE...

So do we need something new? Its nothing wrong in using a pooling scheduler with pool size 0.

[jhoeller]

There is a graceful shutdown option in the form of a taskTerminationTimeout on SimpleAsyncTaskExecutor and SimpleAsyncTaskScheduler now. When set to >0, this will lead to task tracking for every execution thread which we're also using for interrupting the running threads on shutdown. There is an integration test for compatibility with the ThreadPoolTaskScheduler shutdown behavior which looks very promising so far.

As for needing something new, from where I stand right now, I see those two options indeed: either SimpleAsyncTaskScheduler style or ThreadPoolTaskScheduler style. A ConcurrentTaskScheduler wrapping an Executors-created pool is effectively a variant of ThreadPoolTaskScheduler. In such a scenario, the pool size is fixed, whereas SimpleAsyncTaskScheduler allows for dynamic concurrency of scheduled task executions.

[vladimirfx]

What do you think about this variant (sorry for Kotlin):

            bean<ConcurrentTaskScheduler>(name = "taskScheduler", isLazyInit = true) {
                val concurrencyLevel = env.getProperty<Int>("spring.task.scheduling.concurrency") ?: 1
                val executor = ScheduledThreadPoolExecutor(1, Thread.ofVirtual().factory()).apply {
                    this.maximumPoolSize = concurrencyLevel
                }
                ConcurrentTaskScheduler(executor)
            }

1. uses virtual threads only
2. preserves concurrency level
3. preserves shutdown semantics
4. do not pool 'executing' threads

[jhoeller]

Setting the maximum pool size on a ScheduledThreadPoolExecutor does not really have a dynamic scaling effect since the scheduler acts as a fixed-sized pool with an unbounded queue, mixing trigger coordination and actual scheduled task execution on the same threads. In comparison, SimpleAsyncTaskScheduler has quite different scaling behavior where an always-single scheduler thread starts any number of concurrent scheduled task executions on separate worker threads, just potentially bounded by a concurrency limit.

As a side note, ConcurrentTaskScheduler does not provide the same degree of lifecycle integration with the Spring context, in particular as of 6.1. It is missing the pause/resume support and the graceful shutdown signals that ThreadPoolTaskScheduler comes with now. For that reason, I would generally recommend a specifically configured ThreadPoolTaskScheduler for your purposes rather than a wrapper around a custom ScheduledThreadPoolExecutor instance.