Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Issue with hooks for cassandra #14

Open
Amoghrd opened this issue Aug 6, 2020 · 0 comments
Open

Issue with hooks for cassandra #14

Amoghrd opened this issue Aug 6, 2020 · 0 comments

Comments

@Amoghrd
Copy link

Amoghrd commented Aug 6, 2020

Tried B/R with noobaa enabled for cassandra and the backup partially failed with the following logs

velero backup logs cassandra --insecure-skip-tls-verify=true | grep error time="2020-08-06T16:05:02Z" level=info msg="stderr: error: ThreadPoolExecutor has shut down\n-- StackTrace --\njava.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down\n\tat org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:58)\n\tat java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)\n\tat java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)\n\tat org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:162)\n\tat org.apache.cassandra.db.ColumnFamilyStore.waitForFlushes(ColumnFamilyStore.java:990)\n\tat org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:956)\n\tat org.apache.cassandra.db.ColumnFamilyStore.forceBlockingFlush(ColumnFamilyStore.java:996)\n\tat org.apache.cassandra.service.StorageService.forceKeyspaceFlush(StorageService.java:3434)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat sun.reflect.misc.Trampoline.invoke(MethodUtil.java:72)\n\tat sun.reflect.GeneratedMethodAccessor5.invoke(Unknown Source)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:276)\n\tat com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112)\n\tat com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46)\n\tat com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237)\n\tat com.sun.jmx.mbeanserver.PerInterface.invoke(PerInterface.java:138)\n\tat com.sun.jmx.mbeanserver.MBeanSupport.invoke(MBeanSupport.java:252)\n\tat com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.invoke(DefaultMBeanServerInterceptor.java:819)\n\tat com.sun.jmx.mbeanserver.JmxMBeanServer.invoke(JmxMBeanServer.java:801)\n\tat javax.management.remote.rmi.RMIConnectionImpl.doOperation(RMIConnectionImpl.java:1468)\n\tat javax.management.remote.rmi.RMIConnectionImpl.access$300(RMIConnectionImpl.java:76)\n\tat javax.management.remote.rmi.RMIConnectionImpl$PrivilegedOperation.run(RMIConnectionImpl.java:1309)\n\tat javax.management.remote.rmi.RMIConnectionImpl.doPrivilegedOperation(RMIConnectionImpl.java:1401)\n\tat javax.management.remote.rmi.RMIConnectionImpl.invoke(RMIConnectionImpl.java:829)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat java.lang.reflect.Method.invoke(Method.java:498)\n\tat sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:357)\n\tat sun.rmi.transport.Transport$1.run(Transport.java:200)\n\tat sun.rmi.transport.Transport$1.run(Transport.java:197)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat sun.rmi.transport.Transport.serviceCall(Transport.java:196)\n\tat sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:573)\n\tat sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:834)\n\tat sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)\n\tat java.security.AccessController.doPrivileged(Native Method)\n\tat sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)\n\tat java.lang.Thread.run(Thread.java:748)\n\n" backup=oadp-operator/cassandra hookCommand="[/bin/bash -c opt/cassandra/bin/nodetool disablegossip && opt/cassandra/bin/nodetool disablethrift && opt/cassandra/bin/nodetool disablebinary && opt/cassandra/bin/nodetool flush]" hookContainer=cassandra hookName="<from-annotation>" hookOnError=Fail hookPhase=pre hookSource=annotation hookTimeout="{30s}" hookType=exec logSource="pkg/podexec/pod_command_executor.go:172" name=cassandra-0 namespace=cassandra-stateful resource=pods time="2020-08-06T16:05:02Z" level=error msg="Error executing hook" backup=oadp-operator/cassandra error="command terminated with exit code 2" hookPhase=pre hookSource=annotation hookType=exec logSource="pkg/backup/item_hook_handler.go:100" name=cassandra-0 namespace=cassandra-stateful resource=pods time="2020-08-06T16:05:02Z" level=error msg="Error backing up item" backup=oadp-operator/cassandra error="command terminated with exit code 2" logSource="pkg/backup/backup.go:454" name=cassandra-0

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant