PERF: Fix regression in datetime ops (#17980)

* PERF: Fix regression in datetime ops

HEAD:

```
In [1]: import pandas as pd; import numpy as np
In [2]: s = pd.Series(pd.to_datetime(np.arange(100000), unit='ms'))
In [3]: %timeit s - s.shift()
2.73 ms ± 30.1 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

0.21.0rc1:

```
527 ms ± 11.6 ms per loop (mean ± std. dev. of 7 runs, 1 loop each)
```

0.20.3

```
2.4 ms ± 57.8 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)
```

* timedelta too

* Clean up the fix

pandas/core/ops.py

@@ -622,7 +622,8 @@ def _is_offset(self, arr_or_obj):
         """ check if obj or all elements of list-like is DateOffset """
         if isinstance(arr_or_obj, ABCDateOffset):
             return True
-        elif is_list_like(arr_or_obj) and len(arr_or_obj):
+        elif (is_list_like(arr_or_obj) and len(arr_or_obj) and
+              is_object_dtype(arr_or_obj)):
             return all(isinstance(x, ABCDateOffset) for x in arr_or_obj)
         return False