Integration tests flaky in AppVeyor

[lesv]

#1421 AppVeyor failed on DNS for this, succeeded for Travis, I'm going to approve this.

Tests run: 35, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 3.391 sec <<< FAILURE! - in com.google.cloud.dns.testing.LocalDnsHelperTest
testListRecordSets(com.google.cloud.dns.testing.LocalDnsHelperTest)  Time elapsed: 0.032 sec  <<< FAILURE!
java.lang.AssertionError
	at com.google.cloud.dns.testing.LocalDnsHelperTest.testListRecordSets(LocalDnsHelperTest.java:822)

    assertNull(record.getType());
    assertNotNull(record.getTtl());
    assertNull(listResult.pageToken());

getTtl() appears to be Null.

[garrettjonesgoogle]

This seems to have been happening on every PR.

[garrettjonesgoogle]

We need to either fix the test or stop making this part of the build; it looks really bad to have a Beta product whose builds are failing. /cc @omaray

[pongad]

I'll look into this.

[pongad]

@garrettjonesgoogle I think this test file should simply be deleted. The goal of this repo is to create good client libraries. This file is not testing the DNS library. It's testing the DNS emulator, which was written just to test the client.

I think the emulator is a maintenance burden. It's an HTTP server, meaning it has to deal with synchronization. It also tries to behave like the DNS service. Arguably, it's more complicated than the code it's trying to test.

It's much easier to test the library by making sure that method calls create the right RPC requests and that RPC responses gets translated to the right response objects. We can integration test by calling a couple methods on the real service to make sure we set up HTTP and Auth layers properly. I think this would lead to much better tests.

[garrettjonesgoogle]

I looked through more of the AppVeyor failures. It does look like about half are the particular failure mentioned by @lesv , but I also see other ones too. They occur in ITTranslateTest, ITLoggingTest, LocalPubSubHelperTest, ITBigQueryTest, and BlockingProcessStreamReaderTest. So, just disabling LocalDnsHelperTest won't help. @mziccard , any insights? It seems like there is probably a fair amount of work necessary to work through all of the failures happening.

• https://ci.appveyor.com/project/GoogleCloudPlatform/google-cloud-java/build/276 :

testListSupportedLanguagesWithOptions(com.google.cloud.translate.it.ITTranslateTest)  Time elapsed: 0.594 sec  <<< ERROR!
com.google.cloud.translate.TranslateException: 
400 Bad Request
{
  "error": {
    "code": 400,
    "message": "The API Key and the authentication credential are from different projects.",
    "errors": [
      {
        "message": "The API Key and the authentication credential are from different projects.",
        "domain": "global",
        "reason": "badRequest"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}
	at com.google.cloud.translate.it.ITTranslateTest.testListSupportedLanguagesWithOptions(ITTranslateTest.java:66)

• https://ci.appveyor.com/project/GoogleCloudPlatform/google-cloud-java/build/281

testListRecordSetsBatch(com.google.cloud.dns.testing.LocalDnsHelperTest)  Time elapsed: 0.078 sec  <<< FAILURE!
java.lang.AssertionError
	at com.google.cloud.dns.testing.LocalDnsHelperTest.testListRecordSetsBatch(LocalDnsHelperTest.java:2149)

• https://ci.appveyor.com/project/GoogleCloudPlatform/google-cloud-java/build/297

testWriteAndListLogEntries(com.google.cloud.logging.it.ITLoggingTest)  Time elapsed: 2.484 sec  <<< FAILURE!
java.lang.AssertionError

• https://ci.appveyor.com/project/GoogleCloudPlatform/google-cloud-java/build/274 :

testStartStopReset(com.google.cloud.pubsub.testing.LocalPubSubHelperTest)  Time elapsed: 0.937 sec  <<< FAILURE!
java.lang.AssertionError: Expected test to throw an instance of com.google.cloud.pubsub.PubSubException

• https://ci.appveyor.com/project/GoogleCloudPlatform/google-cloud-java/build/268 :

testCreateAndGetJob(com.google.cloud.bigquery.it.ITBigQueryTest)  Time elapsed: 6.424 sec  <<< FAILURE!
java.lang.AssertionError
	at com.google.cloud.bigquery.it.ITBigQueryTest.testCreateAndGetJob(ITBigQueryTest.java:814)

• https://ci.appveyor.com/project/GoogleCloudPlatform/google-cloud-java/build/262 :

org.junit.runners.model.TestTimedOutException: test timed out after 10 seconds
	at com.google.cloud.testing.BlockingProcessStreamReaderTest.testForwardLogEntry(BlockingProcessStreamReaderTest.java:99)

[garrettjonesgoogle]

It looks like the DNS test in particular is so problematic because we have a DNS emulator that lives within google-cloud-java. It doesn't seem like a good idea for us to be maintaining our own emulator for someone else's service. I'm in favor of nuking it.

[garrettjonesgoogle]

New information: the AppVeyor configuration is set to run integration tests on commits, but not PRs. No AppVeyor run that included integration tests has gotten past the Translate failure, so that needs to be fixed. I am switching AppVeyor to only run unit tests for now because the integration tests are not currently providing value.

#1461

[garrettjonesgoogle]

It looks like users are seeing the Translate failure outside of integration tests: #1405

[garrettjonesgoogle]

At this point, AppVeyor isn't even running tests - the queue is very backlogged.

[garrettjonesgoogle]

We should get AppVeyor integration tests up and running again for GA.

[neozwu]

With this PR, this issue should be resolved.