Stop wasting time in CI on MIRI runs

[Ted-Jiang]

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
like @alamb mentioned ❤️, it always return true in test.
apache/datafusion#3610 (comment)

Describe the solution you'd like
A clear and concise description of what you want to happen.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.