perf: revert fallback optimization

This reverts #151 because it broke big endian targets. CI didn't catch
it because of a misconfiguration that resulted in tests being skipped
for any target that required cross compilation.

This reverts commit 345fab7.

Fixes #152