Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Large error stack traces can cause FAILED BINDER TRANSACTION #269

Closed
brettchabot opened this issue Mar 25, 2019 · 28 comments
Closed

Large error stack traces can cause FAILED BINDER TRANSACTION #269

brettchabot opened this issue Mar 25, 2019 · 28 comments

Comments

@brettchabot
Copy link
Collaborator

Tracking b/118760175

@athkalia
Copy link

Hi @brettchabot , thanks for looking into this. Do you know if this is going to be part of the 1.1.2 release?

@brettchabot
Copy link
Collaborator Author

No sorry, a fix for this isn't currently scheduled for the next release.

@athkalia
Copy link

Hi @brettchabot , thank you for your reply.

How often do you ship your releases? Can you give me a rough time estimate for this? Is there any workaround that I can try? This is at the moment causing 100% of our test runs to fail.

Thanks in advance,
Sakis

@brettchabot
Copy link
Collaborator Author

Sorry I cannot provide a timeline for the fix.

I've seen this error before when a test is failing and dumping a very large stack trace. So I'd suggest to look at the logcat, figure out what test is failing, and fix the root cause.

@athkalia
Copy link

Thank you for the response. If possible please try to prioritize this, the tests are passing locally so we are not sure where the failure comes from.

Thanks again,
Sakis

@danielgomezrico
Copy link

Im having the same issue right now

@athkalia
Copy link

athkalia commented Jan 29, 2020

Hey @brettchabot, still seeing this from time to time. Would it be possible for you to maybe trim the bottom part of the stack trace or something if you know that it cannot fit and it's going to crash? Do you accept contributions? Happy to try to fix this.

@fkirc
Copy link

fkirc commented Jan 29, 2020

I am also facing bug #286 on Firebase test lab.

After a few tests, the orchestrator crashes and no further tests are executed.
At the moment, this leads to a 100% breakage of our Firebase test lab project.
This is hard to reproduce since it only happened on Firebase test lab.

Stacktrace:

java.lang.RuntimeException: android.os.TransactionTooLargeException: data parcel size 4178964 bytes
FATAL EXCEPTION: AndroidTestOrchestrator
Process: androidx.test.orchestrator, PID: 20883
java.lang.RuntimeException: android.os.TransactionTooLargeException: data parcel size 4178964 bytes
at android.app.ActivityThread.finishInstrumentation(ActivityThread.java:6510)
at android.app.Instrumentation.finish(Instrumentation.java:252)
at androidx.test.orchestrator.AndroidTestOrchestrator.finish(AndroidTestOrchestrator.java:451)
at androidx.test.orchestrator.AndroidTestOrchestrator.executeNextTest(AndroidTestOrchestrator.java:334)
at androidx.test.orchestrator.AndroidTestOrchestrator.runFinished(AndroidTestOrchestrator.java:312)
at androidx.test.orchestrator.TestRunnable.run(TestRunnable.java:157)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
at java.lang.Thread.run(Thread.java:919)
Caused by: android.os.TransactionTooLargeException: data parcel size 4178964 bytes
at android.os.BinderProxy.transactNative(Native Method)
at android.os.BinderProxy.transact(BinderProxy.java:510)
at android.app.IActivityManager$Stub$Proxy.finishInstrumentation(IActivityManager.java:5491)
at android.app.ActivityThread.finishInstrumentation(ActivityThread.java:6508)
... 8 more

The logcat is not very useful.

@lwasyl
Copy link

lwasyl commented Feb 8, 2020

So I'd suggest to look at the logcat, figure out what test is failing, and fix the root cause.

This is not as straightforward as it sounds, especially if we don't even get the stacktrace. Is it any reasonably simple way we can work around this in a local fork for example, just to be able to debug issues?

@dmitry-sviridov
Copy link

Having the same issue.
FTL matrix id: matrix-3uxv7ydl011ng

@Subtle-fox
Copy link

Have the same issue too. Reproduced only in TestLab and runs successfully on local machine with androidx.test:orchestrator:1.1.0

@Subtle-fox
Copy link

Subtle-fox commented Mar 17, 2020

We figured out that actual problem was in huge error trace from our failed tests.
I suppose that FTL transfers error log to synthetic overview activity (with device info and other extra info) or service or something similar, which finally leads to TransactionTooLargeException.
P.S. our problem was in wrong orchestrator's parameters on FTL

@tprochazka
Copy link

tprochazka commented Apr 30, 2020

It happens to me too, with this stack trace

04-30 17:46:54.411  9748  9790 I OrchestrationXmlTestRunListener: XML test result file generated at /storage/emulated/0/odo/test_result_2378417933931616849.xml. Total tests 44, failure 36, passed 8, 
04-30 17:46:54.420  9748  9790 D NetworkSecurityConfig: No Network Security Config specified, using platform default
04-30 17:46:54.553  9748  9790 E JavaBinder: !!! FAILED BINDER TRANSACTION !!!  (parcel size = 1697132)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: FATAL EXCEPTION: AndroidTestOrchestrator
04-30 17:46:54.554  9748  9790 E AndroidRuntime: Process: androidx.test.orchestrator, PID: 9748
04-30 17:46:54.554  9748  9790 E AndroidRuntime: java.lang.RuntimeException: android.os.TransactionTooLargeException: data parcel size 1697132 bytes
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at android.app.ActivityThread.finishInstrumentation(ActivityThread.java:6510)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at android.app.Instrumentation.finish(Instrumentation.java:251)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at androidx.test.orchestrator.AndroidTestOrchestrator.finish(AndroidTestOrchestrator.java:451)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at androidx.test.orchestrator.AndroidTestOrchestrator.executeNextTest(AndroidTestOrchestrator.java:334)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at androidx.test.orchestrator.AndroidTestOrchestrator.runFinished(AndroidTestOrchestrator.java:312)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at androidx.test.orchestrator.TestRunnable.run(TestRunnable.java:157)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at java.lang.Thread.run(Thread.java:919)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: Caused by: android.os.TransactionTooLargeException: data parcel size 1697132 bytes
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at android.os.BinderProxy.transactNative(Native Method)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at android.os.BinderProxy.transact(BinderProxy.java:510)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at android.app.IActivityManager$Stub$Proxy.finishInstrumentation(IActivityManager.java:5491)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	at android.app.ActivityThread.finishInstrumentation(ActivityThread.java:6508)
04-30 17:46:54.554  9748  9790 E AndroidRuntime: 	... 8 more

Is it trying to send the whole test_result_2378417933931616849.xml file with Parcelable?

And this is in the loggcat before

04-30 17:46:51.075 31314 31405 E TestRunner: failed: initializationError(com.x.android.test.utils.espresso.BaseEspressoTest)
04-30 17:46:51.075 31314 31405 E TestRunner: ----- begin exception -----
04-30 17:46:51.086 31314 31405 E TestRunner: java.lang.RuntimeException: Delegate runner 'androidx.test.internal.runner.junit4.AndroidJUnit4ClassRunner' for AndroidJUnit4 could not be loaded.
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.ext.junit.runners.AndroidJUnit4.throwInitializationError(AndroidJUnit4.java:92)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.ext.junit.runners.AndroidJUnit4.loadRunner(AndroidJUnit4.java:82)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.ext.junit.runners.AndroidJUnit4.loadRunner(AndroidJUnit4.java:51)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.ext.junit.runners.AndroidJUnit4.<init>(AndroidJUnit4.java:46)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at java.lang.reflect.Constructor.newInstance0(Native Method)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at org.junit.internal.builders.AnnotatedBuilder.buildRunner(AnnotatedBuilder.java:104)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at org.junit.internal.builders.AnnotatedBuilder.runnerForClass(AnnotatedBuilder.java:86)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.internal.runner.junit4.AndroidAnnotatedBuilder.runnerForClass(AndroidAnnotatedBuilder.java:63)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at org.junit.internal.builders.AllDefaultPossibilitiesBuilder.runnerForClass(AllDefaultPossibilitiesBuilder.java:37)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.internal.runner.AndroidRunnerBuilder.runnerForClass(AndroidRunnerBuilder.java:153)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at org.junit.runners.model.RunnerBuilder.safeRunnerForClass(RunnerBuilder.java:70)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.internal.runner.TestLoader.doCreateRunner(TestLoader.java:73)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.internal.runner.TestLoader.getRunnersFor(TestLoader.java:104)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.internal.runner.TestRequestBuilder.build(TestRequestBuilder.java:793)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.runner.AndroidJUnitRunner.buildRequest(AndroidJUnitRunner.java:547)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.runner.AndroidJUnitRunner.onStart(AndroidJUnitRunner.java:390)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at android.app.Instrumentation$InstrumentationThread.run(Instrumentation.java:2189)
04-30 17:46:51.086 31314 31405 E TestRunner: Caused by: java.lang.reflect.InvocationTargetException
04-30 17:46:51.086 31314 31405 E TestRunner: 	at java.lang.reflect.Constructor.newInstance0(Native Method)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at java.lang.reflect.Constructor.newInstance(Constructor.java:343)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.ext.junit.runners.AndroidJUnit4.loadRunner(AndroidJUnit4.java:72)
04-30 17:46:51.086 31314 31405 E TestRunner: 	... 17 more
04-30 17:46:51.086 31314 31405 E TestRunner: Caused by: org.junit.runners.model.InvalidTestClassError: Invalid test class 'com.x.android.test.utils.espresso.BaseEspressoTest':
04-30 17:46:51.086 31314 31405 E TestRunner:   1. No runnable methods
04-30 17:46:51.086 31314 31405 E TestRunner: 	at org.junit.runners.ParentRunner.validate(ParentRunner.java:525)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at org.junit.runners.ParentRunner.<init>(ParentRunner.java:92)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at org.junit.runners.BlockJUnit4ClassRunner.<init>(BlockJUnit4ClassRunner.java:74)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.internal.runner.junit4.AndroidJUnit4ClassRunner.<init>(AndroidJUnit4ClassRunner.java:43)
04-30 17:46:51.086 31314 31405 E TestRunner: 	at androidx.test.internal.runner.junit4.AndroidJUnit4ClassRunner.<init>(AndroidJUnit4ClassRunner.java:48)
04-30 17:46:51.086 31314 31405 E TestRunner: 	... 20 more
04-30 17:46:51.086 31314 31405 E TestRunner: ----- end exception -----

Maybe this is the test that produces too long stacktrace.
But problem Is that com.x.android.test.utils.espresso.BaseEspressoTest is not the name of some specific test, but a common parent class of all our tests. So totally useless information here. This class is not even in test sources, it is part of the test library. So I really don't understand why this name is there.

@tprochazka
Copy link

Is there any way how to configure bigger transaction (parcel data) size on the emulator?

@tprochazka
Copy link

tprochazka commented May 12, 2020

What about just limit result string size in

  private Bundle createResultBundle() {
    OutputStream stream = new ByteArrayOutputStream();
    PrintStream writer = new PrintStream(stream);
    Bundle bundle = new Bundle();

    try {
      resultBuilder.orchestrationRunFinished();
      resultPrinter.orchestrationRunFinished(writer, resultBuilder.build());
    } finally {
      writer.close();
    }

    bundle.putString(
        Instrumentation.REPORT_KEY_STREAMRESULT, String.format("\n%s", stream.toString()));
    return bundle;
  }

to value which fit the maximum bundle size?

At least like a quick fix solution.
Much better would be of course to stop using Bundle.

@tprochazka
Copy link

I removed the test that causing this issue locally (on Windows) and it still failing on another one on the Linux CI server, it looks the limit of binder transaction is different od different platforms or stack trace is different. Both on the same emulators :-( Currently is it is a huge blocker for us. @brettchabot do you thin that makes sense my previous post about limit size in createResultBundle() method? Since this library not using standard builds tools is an investment to build it locally very hard, but if you think that it can help and you are not planning to fix it, I would go in my own fork way.

@sanjeevirajm
Copy link

@tprochazka
What about limiting result string size? - We will miss stack traces that might be important, Right?

@tprochazka
Copy link

@tprochazka
What about limiting result string size? - We will miss stack traces that might be important, Right?

Yes. I proposed it as a quick solution, it is definitely better than crash complete test suit, because then you have nothing. You even don't know which test caused it. You need to investigate logcat from the emulator.

@sanjeevirajm
Copy link

sanjeevirajm commented Jul 15, 2020

@tprochazka i am thinking about writing the entire stack trace in a file. what do you think?
@brettchabot

@tprochazka
Copy link

tprochazka commented Jul 16, 2020

Of course that like a long solution this will be much better.
I already tried to fix it and build it, it is so complicated that I did not have enough time to do that :-(

@vdoshkinn
Copy link

Do we have any working solution for this issue?

@tprochazka
Copy link

No. I'm not running every test as separate TeamCity job, so if one fails, it will not harm all others :-(

copybara-service bot pushed a commit that referenced this issue Sep 2, 2020
… classic/non-orchestrator modes.

This change should clean up test failure reporting by:
  - Remove test runner framework related stack frames
  - Truncate stack traces to a 64KB size when running under orchestrator
    to attempt to avoid binder transaction limits.
    This limit is already enforced when running in classic/non-orchestrator mode

JUnit 4.13 has a really nice getTrimmedStackTrace feature, but androidx.test
is fixed to 4.12 for the time being. So as a temporary workaround, copy
the relevant JUnit utilty class into this project.

Fixes #729, and hopefully #269

PiperOrigin-RevId: 329797783
copybara-service bot pushed a commit that referenced this issue Sep 3, 2020
… classic/non-orchestrator modes.

This change should clean up test failure reporting by:
  - Remove test runner framework related stack frames
  - Truncate stack traces to a 64KB size when running under orchestrator
    to attempt to avoid binder transaction limits.
    This limit is already enforced when running in classic/non-orchestrator mode

JUnit 4.13 has a really nice getTrimmedStackTrace feature, but androidx.test
is fixed to 4.12 for the time being. So as a temporary workaround, copy
the relevant JUnit utilty class into this project.

Fixes #729, and hopefully #269

PiperOrigin-RevId: 329797783
copybara-service bot pushed a commit that referenced this issue Sep 16, 2020
… classic/non-orchestrator modes.

This change should clean up test failure reporting by:
  - Remove test runner framework related stack frames
  - Truncate stack traces to a 64KB size when running under orchestrator
    to attempt to avoid binder transaction limits.
    This limit is already enforced when running in classic/non-orchestrator mode

JUnit 4.13 has a really nice getTrimmedStackTrace feature, but androidx.test
is fixed to 4.12 for the time being. So as a temporary workaround, copy
the relevant JUnit change junit-team/junit4#1028 into this project.

Fixes #729, and hopefully #269

PiperOrigin-RevId: 329797783
copybara-service bot pushed a commit that referenced this issue Sep 16, 2020
… classic/non-orchestrator modes.

This change should clean up test failure reporting by:
  - Remove test runner framework related stack frames
  - Truncate stack traces to a 64KB size when running under orchestrator
    to attempt to avoid binder transaction limits.
    This limit is already enforced when running in classic/non-orchestrator mode

JUnit 4.13 has a really nice getTrimmedStackTrace feature, but androidx.test
is fixed to 4.12 for the time being. So as a temporary workaround, copy
the relevant JUnit change junit-team/junit4#1028 into this project.

Fixes #729, and hopefully #269

PiperOrigin-RevId: 329797783
copybara-service bot pushed a commit that referenced this issue Sep 16, 2020
… classic/non-orchestrator modes.

This change should clean up test failure reporting by:
  - Remove test runner framework related stack frames
  - Truncate stack traces to a 64KB size when running under orchestrator
    to attempt to avoid binder transaction limits.
    This limit is already enforced when running in classic/non-orchestrator mode

JUnit 4.13 has a really nice getTrimmedStackTrace feature, but androidx.test
is fixed to 4.12 for the time being. So as a temporary workaround, copy
the relevant JUnit change junit-team/junit4#1028 into this project.

Fixes #729, and hopefully #269

PiperOrigin-RevId: 329797783
copybara-service bot pushed a commit that referenced this issue Sep 16, 2020
… classic/non-orchestrator modes.

This change should clean up test failure reporting by:
  - Remove test runner framework related stack frames
  - Truncate stack traces to a 64KB size when running under orchestrator
    to attempt to avoid binder transaction limits.
    This limit is already enforced when running in classic/non-orchestrator mode

JUnit 4.13 has a really nice getTrimmedStackTrace feature, but androidx.test
is fixed to 4.12 for the time being. So as a temporary workaround, copy
the relevant JUnit change junit-team/junit4#1028 into this project.

Fixes #729, and hopefully #269

PiperOrigin-RevId: 332051125
@brettchabot
Copy link
Collaborator Author

Please give 1.3.1-alpha01 a try. Stack traces should be slightly smaller by default, and will get truncated to 64KB in size.

https://github.com/android/android-test/releases/tag/androidx-test-1.3.1-alpha01

@tprochazka
Copy link

Thanks a lot! I will test it. We have now 1.2.0 version, so I hope that upgrade will be easy.

@brettchabot
Copy link
Collaborator Author

We try hard to make releases backwards compatible - let us know if you hit any snags upgrading.

@tprochazka
Copy link

I just tested it and it really looks that this fix helps.
It still needs some more testing, but the first test I didn't see any binder transaction issue.
Thanks a lot!

@Dimezis
Copy link

Dimezis commented Nov 19, 2020

1.3.1-alpha01 fixes it indeed, but now the stacktrace is truncated and you can't see where the test failed exactly.
I'd much rather prefer having this info than the huge (truncated) View hierarchy snapshot.

@TWiStErRob
Copy link
Contributor

Agreed, the view dump is not too useful if we don't know which check/perform failed.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests