Ability to request custom template annoations for StatefulSet of SolrCloud

[sepulworld]

Is your feature request related to a problem? Please describe.
We would like to apply kube2iam roles to pods in solrcloud statefulset. The role will allow access to s3 for backups.

Snippet of what is needed

spec:
  template:
    metadata:
      annotations:
        iam.amazonaws.com/role: role-arn

Describe the solution you'd like

The ability to add any spec/template/metadata annotation for developers to request

[HoustonPutman]

that sounds good. We should already have something like this, but I think just for labels, not annotations. And I could be wrong about that too...

Basically we need to enable a lot more customization of pod preferences

[HoustonPutman]

Spec.customSolrKubeOptions.podOptions and Spec.customSolrKubeOptions.statefulSetOptions have been added.