The blank/missing items unified issue

[horak]

Here are the different types of problems that we've been seeing. Let's refine this list as we try to fix these issues.

1. Unpublished items

Certain items have been unpublished somehow. These items are present in the database and yet don't appear on production. If manually navigated to, they appear blank on production.

• check if these items are present at all?
• publish items while tail -fing solr and python log (/var/log in the sys.log or the apache) file. Also the solr-8080-console.log in ./opt/solrHeatmap/solr-5.5.0/server/logs/solr-8080-console.log

examples: there are some 5k items missing like 1828597
related issues: #1070, #1071, #1073, #1084,

2. Blank item pages

Certain items are displayed without any of their metadata on production, and thus appear blank. I can confirm that this isn't some front-end error as the request payload itself is lacking the proper metadata. I've tested the validity of the lat/lng fields as Steve indicated this maybe have been the cause, unfortunately that doesn't seem to be the problem.
examples: 1828597
related issues: #1070, #1084

3. Tag editing issues

When updated in the admin browser, updates are not propagating to production.
related issues: #1074, #1078, #1079, #1085

4. New blank items

Newly added items do not seem to be a problem. Both uploading and editing the metadata on some test items worked.

5. Missing items from the front of the database (id 2-18901)

This happened while I was importing the Kahoku items and i'm just waiting to restore these since I foresee that work going to waste in lieu of more dramatic restoration. This is an unrelated issue.
related issue: #1062

---

Update (4/23/16) : It seems like issue 1 & 2 might actually be the same problem. I'm merging the two.
Update : After editing an item that classifies as falling under issue 1. above, I was greeted with these errors.
Update (5/2/16): It seems that the solr bug reported above solr-8080-console.log is not reproducible by the same means for some reason. Will look into this further.
Update (5/3/16): Item "1828597" does indeed show the problematic behavior.
Update (6/7/16): Item "707998" - downloaded and reuploaded this blank item. Discovered that downloading the item from the dashboard trims off both attributes and tags. Re-uploading the downloaded item fails to fix the blank page issue.
Update: Uploading a blank item under a different identity doesn't seem to do anything. The new user's page doesn't display the blank item.
Update (6/10/16): Attributes of any kind cause a solr ingest error.
Update (6/11/16): Attributes are now no longer indexed by numeric key values, they are strings instead. Reingestion should fix all the blank items and items not propagating.
Update (6/15/16): Using the pop-up editor to add tags, overwrites existing attributes.
Update (7/25/16): Steve performed a re-ingest, blank items are still not propagating. But I discovered that any edit on an item's metadata will populate all the necessary info. Asking Steve if he thinks we should setup a script to try and do that.