Memcache timeout #704
Comments
|
15 seconds timeout is very high, timeouts at that indicate serious problems. May be check the state of memcache cluster, and the client. You'd want to make sure you're using a high value of MaxWaitingFutures to make sure you're getting sufficient parallelism on the client. Try different loads and observe how p999 latency varies. I'm afraid there's no silver bullet. These kinds of issues require careful debugging. Taking yourkit profile has been very helpful to me in the past for debugging such issues.
…On Jan 1, 2017, 9:31 PM -0800, leesf ***@***.***>, wrote:
When using memcache(https://github.com/twitter/storehaus) in online layer to store summingbird processed data. timeout of memcache is set to 15 seconds. However, when writing speed to memcache exceeds 5000/s, there will be a individualRequestTimeoutException of Memecache, the details are following
ERROR backtype.storm.daemon.executor -
com.twitter.finagle.IndividualRequestTimeoutException: exceeded 15.seconds to memcache while waiting for a response for an individual request, excluding retries at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
Who knows why? Thanks for replying.
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub (#704), or mute the thread (https://github.com/notifications/unsubscribe-auth/AAojhljzaCnNQdB1ayHPmYwd5FNIGyuUks5rOIu-gaJpZM4LY1-2).
|
|
@pankajroark thanks for replying, i did not use memcache in cluster, but in local machine, and i checked the state of memcache in machine, it seems all goes well, its really odd. |
|
When i use memcache in storehaus alone, there is an error. I changed the memcache port to 12000. And the code is below import com.twitter.storehaus.memcache.MemcacheStore object MemcacheTest { new Thread(new Runnable { new Thread(new Runnable { error info is below com.twitter.finagle.ServiceTimeoutException: exceeded 10.seconds to memcache while creating a service/connection or reserving a service/connection from the service/connection pool memcache It seems the high concurrency read and write operation to memcache causes the problem. Any suggestions to solve it? Thanks for replying. |
|
This error is different, afaict it indicates timeout in establishing a
connection to memcache.
…On Mon, Jan 2, 2017 at 9:09 AM Pankaj Gupta ***@***.***> wrote:
This
On Mon, Jan 2, 2017 at 3:32 AM leesf ***@***.***> wrote:
@pankajroark <https://github.com/pankajroark>
When i use memcache in storehaus alone, there is an error. I changed the
memcache port to 12000. And the code is below
import com.twitter.storehaus.memcache.MemcacheStore
import com.twitter.conversions.time._
import com.twitter.finagle.redis.util.{CBToString, StringToChannelBuffer}
import com.twitter.util.Await
object MemcacheTest {
def main(args: Array[String]): Unit = {
val client = MemcacheStore.
defaultClient("memcache", "localhost:12000", 2, 10.seconds)
val store = MemcacheStore(client)
new Thread(new Runnable {
override def run(): Unit =
try {
for (i <- 1 to 100000)
store.put(i.toString, Some(StringToChannelBuffer(i.toString)))
} catch {
case e: Exception => e.printStackTrace
}
}).start
new Thread(new Runnable {
override def run(): Unit = {
val start = System.currentTimeMillis()
val num = "100"
try {
for (i <- 1 to 10)
Await.result(store.get(num)) match {
case Some(value) => println(CBToString(value))
case None => {
println("None")
}
}
} catch {
case e: Exception => e.printStackTrace
}
val end = System.currentTimeMillis()
println((end - start) + "")
}}).start
}
}
error info is below
com.twitter.finagle.ServiceTimeoutException: exceeded 10.seconds to
memcache while creating a service/connection or reserving a
service/connection from the service/connection pool memcache
at
com.twitter.finagle.client.DefaultClient$$anonfun$14.apply(DefaultClient.scala:155)
at
com.twitter.finagle.client.DefaultClient$$anonfun$14.apply(DefaultClient.scala:153)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:47)
at com.twitter.finagle.factory.Refinery.(Refinery.scala:20)
at
com.twitter.finagle.client.DefaultClient$$anonfun$22.apply(DefaultClient.scala:220)
at
com.twitter.finagle.client.DefaultClient$$anonfun$22.apply(DefaultClient.scala:220)
at
com.twitter.finagle.builder.ClientBuilder.buildFactory(ClientBuilder.scala:849)
at com.twitter.finagle.builder.ClientBuilder.build(ClientBuilder.scala:882)
at com.twitter.finagle.memcached.KetamaClientBuilder.com
$twitter$finagle$memcached$KetamaClientBuilder$$legacyFAClientBuilder$1(Client.scala:890)
at
com.twitter.finagle.memcached.KetamaClientBuilder$$anonfun$build$1.apply(Client.scala:901)
at
com.twitter.finagle.memcached.KetamaClientBuilder$$anonfun$build$1.apply(Client.scala:901)
at
com.twitter.finagle.memcached.KetamaClient$$anonfun$9$$anonfun$10.apply(Client.scala:722)
at
com.twitter.finagle.memcached.KetamaClient$$anonfun$9$$anonfun$10.apply(Client.scala:721)
at scala.Option.map(Option.scala:145)
at
com.twitter.finagle.memcached.KetamaClient$$anonfun$9.apply(Client.scala:721)
at
com.twitter.finagle.memcached.KetamaClient$$anonfun$9.apply(Client.scala:716)
at com.twitter.finagle.Group$$anonfun$map$1.applyOrElse(Group.scala:58)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at
com.twitter.finagle.Group$$anon$2$$anonfun$1$$anonfun$apply$1.applyOrElse(Group.scala:71)
at
scala.runtime.AbstractPartialFunction.apply(AbstractPartialFunction.scala:33)
at
scala.collection.TraversableLike$$anonfun$collect$1.apply(TraversableLike.scala:278)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at
scala.collection.TraversableLike$class.collect(TraversableLike.scala:278)
at scala.collection.AbstractTraversable.collect(Traversable.scala:105)
at com.twitter.finagle.Group$$anon$2$$anonfun$1.apply(Group.scala:70)
at com.twitter.finagle.Group$$anon$2$$anonfun$1.apply(Group.scala:68)
at com.twitter.util.Var$$anonfun$map$1.apply(Var.scala:53)
at com.twitter.util.Var$$anonfun$map$1.apply(Var.scala:53)
at com.twitter.util.Var$$anon$2$$anonfun$1.apply(Var.scala:66)
at com.twitter.util.Var$$anon$2$$anonfun$1.apply(Var.scala:65)
at com.twitter.util.Var$Observer.publish(Var.scala:153)
at com.twitter.util.UpdatableVar.observe(Var.scala:371)
at com.twitter.util.Var$$anon$2.observe(Var.scala:65)
at com.twitter.util.Var$class.observe(Var.scala:31)
at com.twitter.util.Var$$anon$2.observe(Var.scala:62)
at com.twitter.util.Var$class.observeTo(Var.scala:82)
at com.twitter.util.Var$$anon$2.observeTo(Var.scala:62)
at com.twitter.finagle.Group$class.ref(Group.scala:49)
at com.twitter.finagle.Group$$anon$2.ref$lzycompute(Group.scala:65)
at com.twitter.finagle.Group$$anon$2.ref(Group.scala:65)
at com.twitter.finagle.Group$class.members(Group.scala:87)
at com.twitter.finagle.Group$$anon$2.members(Group.scala:65)
at com.twitter.finagle.Group$class.apply(Group.scala:88)
at com.twitter.finagle.Group$$anon$2.apply(Group.scala:65)
at com.twitter.finagle.memcached.KetamaClient.(Client.scala:730)
at
com.twitter.finagle.memcached.KetamaClientBuilder.build(Client.scala:896)
at
com.twitter.storehaus.memcache.MemcacheStore$.defaultClient(MemcacheStore.scala:78)
at
com.hust.grid.leesf.WrappedMemcacheStore.MemcacheTest$.main(MemcacheTest.scala:15)
at
com.hust.grid.leesf.WrappedMemcacheStore.MemcacheTest.main(MemcacheTest.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
It seems the high concurrency read and write operation to memcache causes
the problem. Any suggestions to solve it? Thanks for replying.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#704 (comment)>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAojhuqoOWFNkEDUI7CcAjGRGZ2b23JOks5rOOBggaJpZM4LY1-2>
.
|
|
are you sure you have memcache running? Can you access it any other way? Are you sure you don't have some firewall issues? |
|
@johnynek Thanks for replying. Memcache is running well and i can use telnet command to access to memcache in my machine. |
When using memcache(https://github.com/twitter/storehaus) in online layer to store summingbird processed data. timeout of memcache is set to 15 seconds. However, when writing speed to memcache exceeds 5000/s, there will be a individualRequestTimeoutException of Memecache, the details are following
ERROR backtype.storm.daemon.executor -
com.twitter.finagle.IndividualRequestTimeoutException: exceeded 15.seconds to memcache while waiting for a response for an individual request, excluding retries at com.twitter.finagle.NoStacktrace(Unknown Source) ~[na:na]
Who knows why? Thanks for replying.
The text was updated successfully, but these errors were encountered: