You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fun fact: this Spark code runs on the executors. But the Spark driver reports exceptions. Make sure to add the information to the exception that the driver shows -- this is what is easiest to see in services such as EMR.
Workarounds
Read logs from the executors, not only from the driver.
The "bad magic bytes" message includes the last 8 bytes. If you can decode these, you may be able to guess what kind of file it is. (This works for formats that use trailers, but sometimes also for ASCII-based formats such as CSV and JSON!)
We recently stumbled upon an error:
This error is missing an important piece of information: the file name. Knowing this would've helped us identify this issue faster.
[@arielshaqed please add more detail here]
The text was updated successfully, but these errors were encountered: