Log of pipeline sessions with Kyle, Julie, Avery & Ivy

[ivyw]

"RGZcatalog.py" stopped at the 668 entry and spat
out the following statement:
...
668
Traceback (most recent call last):
File "RGZcatalog.py", line 223, in
output = '%i entries added.' % RGZcatalog()
File "RGZcatalog.py", line 157, in RGZcatalog
compressed = urllib2.urlopen(str(link_s3)).read() #reads contents
of url to str
File "/usr/lib/python2.7/socket.py", line 351, in read
data = self._sock.recv(rbufsize)
File "/usr/lib/python2.7/httplib.py", line 573, in read
s = self.fp.read(amt)
File "/usr/lib/python2.7/socket.py", line 380, in read
data = self._sock.recv(left)
socket.error: [Errno 104] Connection reset by peer

---

Meanwhile, a quick check on mongo via >db.catalog.count() revealed a
catalog of 667 sources.

Also the 668th source is CI0781 and the .fits file is in the usual place
so then I googled the issue and found that this could be a
"reconnect-on-failure" and this is a sporadic event that is unlikely to replicate.

---

So as a solution, I re-ran RGZcatalog.py again and this is now going for
a bit. It ran for a bit and then encountered the following error message:
...
854
Traceback (most recent call last):
File "RGZcatalog.py", line 223, in
output = '%i entries added.' % RGZcatalog()
File "RGZcatalog.py", line 129, in RGZcatalog
sdss_match = p.getSDSS(entry)
File
"/media/iwong/19c3c716-e6e3-44f1-95f5-79c1822cedc7/RGZ-PIPE/rgz-analysis/RGZcatalogCode/processing.py",
line 112, in getSDSS
df = SDSS_select(query)
File
"/media/iwong/19c3c716-e6e3-44f1-95f5-79c1822cedc7/RGZ-PIPE/rgz-analysis/RGZcatalogCode/processing.py",
line 98, in SDSS_select
return pd.read_csv(file_like, skiprows=1)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py",
line 562, in parser_f
return _read(filepath_or_buffer, kwds)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py",
line 325, in _read
return parser.read()
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py",
line 815, in read
ret = self._engine.read(nrows)
File "/usr/local/lib/python2.7/dist-packages/pandas/io/parsers.py",
line 1314, in read
data = self._reader.read(nrows)
File "pandas/parser.pyx", line 805, in pandas.parser.TextReader.read
(pandas/parser.c:8748)
File "pandas/parser.pyx", line 827, in
pandas.parser.TextReader._read_low_memory (pandas/parser.c:9003)
File "pandas/parser.pyx", line 881, in
pandas.parser.TextReader._read_rows (pandas/parser.c:9731)
File "pandas/parser.pyx", line 868, in
pandas.parser.TextReader._tokenize_rows (pandas/parser.c:9602)
File "pandas/parser.pyx", line 1865, in
pandas.parser.raise_parser_error (pandas/parser.c:23325)
pandas.io.common.CParserError: Error tokenizing data. C error: Expected

1 fields in line 61, saw 4

SOLUTION: Re-run RGZcatalog.py again until it finished. This work for compiling the ATLAS db.