[SPARK-48887][K8S] Enable `spark.kubernetes.executor.checkAllContaine…

…rs` by default

### What changes were proposed in this pull request?

This PR aims to enable `spark.kubernetes.executor.checkAllContainers` by default from Apache Spark 4.0.0.

### Why are the changes needed?

Since Apache Spark 3.1.0, `spark.kubernetes.executor.checkAllContainers` is supported and useful because [sidecar pattern](https://kubernetes.io/docs/concepts/workloads/pods/sidecar-containers/) is used in many cases. Also, it prevents user mistakes which forget and ignore the sidecars' failures by always reporting sidecar failures via executor status.
- apache#29924

### Does this PR introduce _any_ user-facing change?

- This configuration is no-op when there is no other container.
- This will report user containers' error correctly when there exist other containers which are provided by the users.

### How was this patch tested?

Both `true` and `false` are covered by our CI test coverage since Apache Spark 3.1.0.

### Was this patch authored or co-authored using generative AI tooling?

No.

Closes apache#47337 from dongjoon-hyun/SPARK-48887.

Authored-by: Dongjoon Hyun <[email protected]>
Signed-off-by: Dongjoon Hyun <[email protected]>

docs/core-migration-guide.md

@@ -38,6 +38,8 @@ license: |
 
 - Since Spark 4.0, Spark uses `ReadWriteOncePod` instead of `ReadWriteOnce` access mode in persistence volume claims. To restore the legacy behavior, you can set `spark.kubernetes.legacy.useReadWriteOnceAccessMode` to `true`.
 
+- Since Spark 4.0, Spark reports its executor pod status by checking all containers of that pod. To restore the legacy behavior, you can set `spark.kubernetes.executor.checkAllContainers` to `false`.
+
 - Since Spark 4.0, Spark uses `~/.ivy2.5.2` as Ivy user directory by default to isolate the existing systems from Apache Ivy's incompatibility. To restore the legacy behavior, you can set `spark.jars.ivy` to `~/.ivy2`.
 
 - Since Spark 4.0, Spark uses the external shuffle service for deleting shuffle blocks for deallocated executors when the shuffle is no longer needed. To restore the legacy behavior, you can set `spark.shuffle.service.removeShuffle` to `false`.

docs/running-on-kubernetes.md

@@ -1327,7 +1327,7 @@ See the [configuration page](configuration.html) for information on Spark config
 </tr>
 <tr>
   <td><code>spark.kubernetes.executor.checkAllContainers</code></td>
-  <td><code>false</code></td>
+  <td><code>true</code></td>
   <td>
   Specify whether executor pods should be check all containers (including sidecars) or only the executor container when determining the pod status.
   </td>

resource-managers/kubernetes/core/src/main/scala/org/apache/spark/deploy/k8s/Config.scala

@@ -711,7 +711,7 @@ private[spark] object Config extends Logging {
         "executor status.")
       .version("3.1.0")
       .booleanConf
-      .createWithDefault(false)
+      .createWithDefault(true)
 
   val KUBERNETES_EXECUTOR_MISSING_POD_DETECT_DELTA =
     ConfigBuilder("spark.kubernetes.executor.missingPodDetectDelta")