No jobReference when timeout

[sunnychow]

I got the following error when I run a heavy load job which cause a TIMEOUT. Such that the jobComplete is false. And the jobReference is undefined in this case.
https://cloud.google.com/bigquery/querying-data

TypeError: Cannot read property 'jobId' of undefined at responseHandler (.........../node_modules/gcloud/lib/bigquery/index.js:394:43) 

[stephenplusplus]

Is this fixed in #431?

[sunnychow]

It's another issues. it's not fixed in #431
I want to ask the response from google bigquery, according to https://cloud.google.com/bigquery/querying-data

If the query timeout expires before the query has finished, the method will return jobComplete=false, and you must call jobs.getQueryResults as described next. The query job will continue to run even after the timeout period until it finishes, either successfully or because an error has occurred.

I have tested on it before, the response does not return the jobReference.jobId. So it cause undefined error in this line https://github.com/GoogleCloudPlatform/gcloud-node/blob/v0.12.0/lib/bigquery/index.js#L394

[stephenplusplus]

Okay, thanks for explaining, and sorry for the delay. I'll take a look at this asap.

[stephenplusplus]

I can't reproduce so far. I ran a query and specified timeoutMs: 0 to force a jobComplete = false. The response from the API returned, as expected, jobComplete = false, but jobReference was always present.

According to the API docs (Search "jobReference": https://cloud.google.com/bigquery/docs/reference/v2/jobs/query):

jobReference: Reference to the Job that was created to run the query. This field will be present even if the original request timed out...

I'm not sure how the response could have come back without that to cause that error. Can you try to reproduce the error using our master branch (npm install --save googlecloudplatform/gcloud-node), and let me know if you still get the error?

[jdcc]

Update: Sorry, I see now that this is a different issue with the same symptoms. Move if desired.
I can reproduce on master (b094aa2) with this query:

SELECT
          test_id,
          log_time,
          connection_spec.client_ip,
          connection_spec.server_ip,
          paris_traceroute_hop.src_ip,
          paris_traceroute_hop.dest_ip,
          paris_traceroute_hop.src_hostname,
          paris_traceroute_hop.dest_hostname
        FROM [measurement-lab:m_lab.2014_10]
        WHERE 
          project = 3
          AND connection_spec.client_ip = "62.210.74.143" 
          AND connection_spec.server_ip LIKE '217.163.1.%'
        ORDER BY test_id DESC;

and this code:

bigquery.query(query, function(err, rows, nextQuery) {
  if (nextQuery) {
    bigquery.query(nextQuery, function(err, rows, nextQuery) {});
  }
});

The first query will succeed, and the second will hit this:

TypeError: Cannot read property 'jobId' of undefined at responseHandler (.../gcloud/lib/bigquery/index.js:399:43)

[jdcc]

This is what the response object looks like for the second query:

{
kind: 'bigquery#getQueryResultsResponse',
etag: '"uT9StUUc54UCg_Qc3k6xlS99DVE/s-JNbnKyojd-gIIcRpOEggyGqe4"',
jobComplete: false
}

[stephenplusplus]

Sorry for the slow attention to this issue, @jdcc. I'll be taking another look today to try to get to the bottom of this.

[stephenplusplus]

I'm unfortunately still unable to reproduce. According to the docs and in my experience, a jobReference is always returned. @jdcc or @sunnychow would one of you be able to provide a consistently reproducible query I can run from a query on public data? As an example, something like SELECT url FROM [publicdata:samples.github_nested]. Until then, I'm unfortunately unsure how to resolve.

[gquinones]

I'm seeing it consistently on my system. With my data, of course. I'll see if I can generate an example with public data. Meantime, could you put a guard in the code to check when this situation occurs and output to the console? At is is, we can't even tell which of the queries is the one that is timing out.

If as you say, this situation has not happened in your experience, then that message will never trigger. No harm, no foul. If, on the other hand, the message does trigger, then we can provide you with more details to help resolve the issue.

The error message I am seeing is:

TypeError: Cannot read property 'jobId' of undefined
    at responseHandler (/usr/local/brightcove/bigquery-metrics/node_modules/gcloud/lib/bigquery/index.js:384:43)
    at /usr/local/brightcove/bigquery-metrics/node_modules/gcloud/lib/common/util.js:672:11
    at Object.handleResp (/usr/local/brightcove/bigquery-metrics/node_modules/gcloud/lib/common/util.js:193:3)
    at Request._callback (/usr/local/brightcove/bigquery-metrics/node_modules/gcloud/lib/common/util.js:667:12)
    at Request.self.callback (/usr/local/brightcove/bigquery-metrics/node_modules/gcloud/node_modules/request/request.js:354:22)
    at Request.EventEmitter.emit (events.js:98:17)
    at Request.<anonymous> (/usr/local/brightcove/bigquery-metrics/node_modules/gcloud/node_modules/request/request.js:1207:14)
    at Request.EventEmitter.emit (events.js:117:20)
    at IncomingMessage.<anonymous> (/usr/local/brightcove/bigquery-metrics/node_modules/gcloud/node_modules/request/request.js:1153:12)
    at IncomingMessage.EventEmitter.emit (events.js:117:20)

[stephenplusplus]

@gquinones I've put up a repo for debugging: stephenplusplus/gcloud-node-bigquery-debug. You can install this with npm in your project to debug:

$ npm install --save stephenplusplus/gcloud-node-bigquery-debug

A new error should come through when this issue appears. You can log and paste the error.response property here to help us see what the API returns.

@jgeewax do you know someone who can help us with this issue? In summary, when calling BigQuery#query, there is evidently sometimes a response from the API (either Jobs.query or Jobs.getQueryResults) that doesn't include jobReference.

According to the JSON API docs, jobReference is guaranteed to exist, but as seen here, it's not coming back.

[gquinones]

@stephenplusplus Will do.

[gquinones]

@stephenplusplus Bad news, I'm afraid:

TypeError: undefined is not a function
at responseHandler (/Users/gquinones/work/bigquery-metrics/node_modules/gcloud/lib/bigquery/index.js:389:15)
at /Users/gquinones/work/bigquery-metrics/node_modules/gcloud/lib/common/util.js:684:11
at Object.handleResp (/Users/gquinones/work/bigquery-metrics/node_modules/gcloud/lib/common/util.js:205:3)
at Request._callback (/Users/gquinones/work/bigquery-metrics/node_modules/gcloud/lib/common/util.js:679:12)
at Request.self.callback (/Users/gquinones/work/bigquery-metrics/node_modules/gcloud/node_modules/request/request.js:354:22)
at Request.emit (events.js:98:17)
at Request. (/Users/gquinones/work/bigquery-metrics/node_modules/gcloud/node_modules/request/request.js:1207:14)
at Request.emit (events.js:117:20)
at IncomingMessage. (/Users/gquinones/work/bigquery-metrics/node_modules/gcloud/node_modules/request/request.js:1153:12)
at IncomingMessage.emit (events.js:117:20)

[stephenplusplus]

Sorry, if you un- and re-install my fork, it should be fixed.

[gquinones]

NP; Will do.

BTW, I'm wondering if the timeout is coming from http and not bigQuery, and therefore the existence guarantee does not apply.

[gquinones]

@stephenplusplus: Got it!

error: Error encountered for browsers: {"errors":[{}],"message":"API Error: jobReference not returned.","response":{"kind":"bigquery#getQueryResultsResponse","etag":""Gn3Hpo5WaKnpFuT457VBDNMgZBw/BgHAJI-EM7PuCe8xgQX1MgWbQRo"","jobComplete":false}}

[stephenplusplus]

Great, thanks! That appears to be aligned with what @jdcc gets, as well. I think I'll have to defer until @jgeewax can hopefully provide or find us some help.

For now, I've changed the error.response property to be the full request object, including all kinds of details about the http request and response. Feel free to look through there for anything out of the ordinary.

[gquinones]

@stephenplusplus FYI that the query eventually completed and returned results normally, so the issue is dealing with jobs that are in progress. The jobId is available in the response, just not where it was expected. I pulled it from the response and then used the jobs API to status the job.

The response, edited for brevity was:

An error occurred connecting to Big Query or sending data to graphite for browsers: {"errors":[{}],"message":"API Error: jobReference not returned.","response":{"statusCode":200,"body":"{\n "kind": "bigquery#getQueryResultsResponse",\n "etag": "\"Gn3Hpo5WaKnpFuT457VBDNMgZBw/SVn2n7BA1Kb6sfMPuqUQ3Ix7XNI\"",\n "jobComplete": false\n}\n","headers":{"expires":"Wed, 10 Jun 2015 17:42:01 GMT","date":"Wed, 10 Jun 2015 17:42:01 GMT","cache-control":"private, max-age=0, must-revalidate, no-transform","etag":""Gn3Hpo5WaKnpFuT457VBDNMgZBw/SVn2n7BA1Kb6sfMPuqUQ3Ix7XNI"","vary":"Origin","content-type":"application/json; charset=UTF-8","x-content-type-options":"nosniff","x-frame-options":"SAMEORIGIN","x-xss-protection":"1; mode=block","content-length":"143","server":"GSE","alternate-protocol":"443:quic,p=1"},"request":{"uri":{"protocol":"https:","slashes":true,"auth":null,"host":"www.googleapis.com","port":null,"hostname":"www.googleapis.com","hash":null,"search":"?job[bigQuery][projectId]=rising-ocean-426&job[id]=job_XAZJFpqS5nG62vcKXzPADPTsKH4&query=
SELECT
time_stamp, time, platform, event,
...
","query":"job[bigQuery][projectId]=rising-ocean-426&job[id]=job_XAZJFpqS5nG62vcKXzPADPTsKH4&query=
SELECT
time_stamp, time, platform, event,
…","pathname":"/bigquery/v2/projects/rising-ocean-426/queries/job_XAZJFpqS5nG62vcKXzPADPTsKH4","path":"/bigquery/v2/projects/rising-ocean-426/queries/job_XAZJFpqS5nG62vcKXzPADPTsKH4?job[bigQuery][projectId]=rising-ocean-426&job[id]=job_XAZJFpqS5nG62vcKXzPADPTsKH4&query=
SELECT
time_stamp, time, platform, event,
...","href":"https://www.googleapis.com/bigquery/v2/projects/rising-ocean-426/queries/job_XAZJFpqS5nG62vcKXzPADPTsKH4?job[bigQuery][projectId]=rising-ocean-426&job[id]=job_XAZJFpqS5nG62vcKXzPADPTsKH4&query=
SELECT
time_stamp, time, platform, event,
….
"},"method":"GET","headers":{"Authorization":"Bearer ya29.jgHbYohM_g6PPiPhI_HZZwk-ZmS84vrguYzMEiz29fCRbEnfW2hFru-sKkpse_r9GlJX3PzC76I1bg","User-Agent":"gcloud-node/0.15.0"}}}}

[stephenplusplus]

I was able to reproduce this error as soon as I put my code that works locally up to GCE.

@gquinones could you un- and re-install my fork again? I made some changes that should work around the issue.