Decouple RateLimiter burst size and refill period (facebook#12379)

Summary:
When the rate limiter does not have any waiting requests, the first request to arrive may consume all of the available bandwidth, despite potentially having lower priority than requests that arrive later in the same refill interval. Then, those higher priority requests must wait for a refill. So even in scenarios in which we have an overall bandwidth surplus, the highest priority requests can be sporadically delayed up to a whole refill period.

Alone, this isn't necessarily problematic as the refill period is configurable via `refill_period_us` and can be tuned down as needed until the max sporadic delay is tolerable. However, tuning down `refill_period_us` had a side effect of reducing burst size. Some users require a certain burst size to issue optimal I/O sizes to the underlying storage system.

To satisfy those users, this PR decouples the refill period from the burst size. That way, the max sporadic delay can be limited without impacting I/O sizes issued to the underlying storage system.

Pull Request resolved: facebook#12379

Test Plan:
The goal is to show we can now limit the max sporadic delay without impacting compaction's I/O size.

The benchmark runs compaction with a large I/O size, while user reads simultaneously run at a low rate that does not consume all of the available bandwidth. The max sporadic delay is measured using the P100 of rocksdb.file.read.get.micros. I just used strace to verify the compaction reads follow `rate_limiter_single_burst_bytes`

Setup: `./db_bench -benchmarks=fillrandom,flush -write_buffer_size=67108864 -disable_auto_compactions=true -value_size=256 -num=1048576`

Benchmark: `./db_bench -benchmarks=readrandom -use_existing_db=true -num=1048576 -duration=10 -benchmark_read_rate_limit=4096 -rate_limiter_bytes_per_sec=67108864 -rate_limiter_refill_period_us=$refill_micros -rate_limiter_single_burst_bytes=16777216 -rate_limit_bg_reads=true -rate_limit_user_ops=true -statistics=true -cache_size=0 -stats_level=5 -compaction_readahead_size=16777216 -use_direct_reads=true`

Results:

refill_micros | rocksdb.file.read.get.micros (P100)
-- | --
10000 | 10802
100000 | 100240
1000000 | 922061

For verifying compaction read sizes: `strace -fye pread64 ./db_bench -benchmarks=compact -use_existing_db=true -rate_limiter_bytes_per_sec=67108864 -rate_limiter_refill_period_us=$refill_micros -rate_limiter_single_burst_bytes=16777216 -rate_limit_bg_reads=true -compaction_readahead_size=16777216 -use_direct_reads=true`

Reviewed By: hx235

Differential Revision: D54165675

Pulled By: ajkr

fbshipit-source-id: c5968486316cbfb7ff8e5b7d75d3589883dd1105

include/rocksdb/rate_limiter.h

@@ -41,9 +41,10 @@ class RateLimiter {
   virtual void SetBytesPerSecond(int64_t bytes_per_second) = 0;
 
   // This API allows user to dynamically change the max bytes can be granted in
-  // a single refill period (i.e, burst)
+  // a single call to `Request()`. Zero is a special value meaning the number of
+  // bytes per refill.
   //
-  // REQUIRED: single_burst_bytes > 0. Otherwise `Status::InvalidArgument` will
+  // REQUIRED: single_burst_bytes >= 0. Otherwise `Status::InvalidArgument` will
   // be returned.
   virtual Status SetSingleBurstBytes(int64_t /* single_burst_bytes */) {
     return Status::NotSupported();
@@ -93,7 +94,7 @@ class RateLimiter {
                               Env::IOPriority io_priority, Statistics* stats,
                               RateLimiter::OpType op_type);
 
-  // Max bytes can be granted in a single burst
+  // Max bytes can be granted in a single call to `Request()`.
   virtual int64_t GetSingleBurstBytes() const = 0;
 
   // Total bytes that go through rate limiter
@@ -143,11 +144,11 @@ class RateLimiter {
 // time. It controls the total write rate of compaction and flush in bytes per
 // second. Currently, RocksDB does not enforce rate limit for anything other
 // than flush and compaction, e.g. write to WAL.
-// @refill_period_us: this controls how often tokens are refilled. For example,
-// when rate_bytes_per_sec is set to 10MB/s and refill_period_us is set to
-// 100ms, then 1MB is refilled every 100ms internally. Larger value can lead to
-// burstier writes while smaller value introduces more CPU overhead.
-// The default should work for most cases.
+// @refill_period_us: This controls how often tokens are refilled. For example,
+//                    when `rate_bytes_per_sec` is set to 10MB/s and
+//                    `refill_period_us` is set to 100ms, then 1MB is refilled
+//                    every 100ms internally. Larger values can lead to sporadic
+//                    delays while smaller values introduce more CPU overhead.
 // @fairness: RateLimiter accepts high-pri requests and low-pri requests.
 // A low-pri request is usually blocked in favor of hi-pri request. Currently,
 // RocksDB assigns low-pri to request from compaction and high-pri to request
@@ -159,10 +160,13 @@ class RateLimiter {
 // @auto_tuned: Enables dynamic adjustment of rate limit within the range
 //              `[rate_bytes_per_sec / 20, rate_bytes_per_sec]`, according to
 //              the recent demand for background I/O.
+// @single_burst_bytes: The maximum number of bytes that can be granted in a
+//                      single call to `Request()`. Zero is a special value
+//                      meaning the number of bytes per refill.
 RateLimiter* NewGenericRateLimiter(
     int64_t rate_bytes_per_sec, int64_t refill_period_us = 100 * 1000,
     int32_t fairness = 10,
     RateLimiter::Mode mode = RateLimiter::Mode::kWritesOnly,
-    bool auto_tuned = false);
+    bool auto_tuned = false, int64_t single_burst_bytes = 0);
 
 }  // namespace ROCKSDB_NAMESPACE

tools/db_bench_tool.cc

@@ -1471,6 +1471,9 @@ DEFINE_bool(rate_limiter_auto_tuned, false,
             "Enable dynamic adjustment of rate limit according to demand for "
             "background I/O");
 
+DEFINE_int64(rate_limiter_single_burst_bytes, 0,
+             "Set single burst bytes on background I/O rate limiter.");
+
 DEFINE_bool(sine_write_rate, false, "Use a sine wave write_rate_limit");
 
 DEFINE_uint64(
@@ -4783,7 +4786,8 @@ class Benchmark {
             // Get()/MultiGet()
             FLAGS_rate_limit_bg_reads ? RateLimiter::Mode::kReadsOnly
                                       : RateLimiter::Mode::kWritesOnly,
-            FLAGS_rate_limiter_auto_tuned));
+            FLAGS_rate_limiter_auto_tuned,
+            FLAGS_rate_limiter_single_burst_bytes));
       }
     }
 

unreleased_history/behavior_changes/generic_rate_limiter_burst.md

@@ -0,0 +1 @@
+* `RateLimiter`s created by `NewGenericRateLimiter()` no longer modify the refill period when `SetSingleBurstBytes()` is called.

unreleased_history/public_api_changes/rate_limiter_burst.md

@@ -0,0 +1 @@
+* `RateLimiter`'s API no longer requires the burst size to be the refill size. Users of `NewGenericRateLimiter()` can now provide burst size in `single_burst_bytes`. Implementors of `RateLimiter::SetSingleBurstBytes()` need to adapt their implementations to match the changed API doc.

util/rate_limiter.cc

@@ -46,13 +46,14 @@ struct GenericRateLimiter::Req {
 GenericRateLimiter::GenericRateLimiter(
     int64_t rate_bytes_per_sec, int64_t refill_period_us, int32_t fairness,
     RateLimiter::Mode mode, const std::shared_ptr<SystemClock>& clock,
-    bool auto_tuned)
+    bool auto_tuned, int64_t single_burst_bytes)
     : RateLimiter(mode),
       refill_period_us_(refill_period_us),
       rate_bytes_per_sec_(auto_tuned ? rate_bytes_per_sec / 2
                                      : rate_bytes_per_sec),
       refill_bytes_per_period_(
           CalculateRefillBytesPerPeriodLocked(rate_bytes_per_sec_)),
+      raw_single_burst_bytes_(single_burst_bytes),
       clock_(clock),
       stop_(false),
       exit_cv_(&request_mutex_),
@@ -108,25 +109,19 @@ void GenericRateLimiter::SetBytesPerSecondLocked(int64_t bytes_per_second) {
 }
 
 Status GenericRateLimiter::SetSingleBurstBytes(int64_t single_burst_bytes) {
-  if (single_burst_bytes <= 0) {
+  if (single_burst_bytes < 0) {
     return Status::InvalidArgument(
-        "`single_burst_bytes` must be greater than 0");
+        "`single_burst_bytes` must be greater than or equal to 0");
   }
 
   MutexLock g(&request_mutex_);
-  SetSingleBurstBytesLocked(single_burst_bytes);
+  raw_single_burst_bytes_.store(single_burst_bytes, std::memory_order_relaxed);
   return Status::OK();
 }
 
-void GenericRateLimiter::SetSingleBurstBytesLocked(int64_t single_burst_bytes) {
-  refill_bytes_per_period_.store(single_burst_bytes, std::memory_order_relaxed);
-  refill_period_us_.store(CalculateRefillPeriodUsLocked(single_burst_bytes),
-                          std::memory_order_relaxed);
-}
-
 void GenericRateLimiter::Request(int64_t bytes, const Env::IOPriority pri,
                                  Statistics* stats) {
-  assert(bytes <= refill_bytes_per_period_.load(std::memory_order_relaxed));
+  assert(bytes <= GetSingleBurstBytes());
   bytes = std::max(static_cast<int64_t>(0), bytes);
   TEST_SYNC_POINT("GenericRateLimiter::Request");
   TEST_SYNC_POINT_CALLBACK("GenericRateLimiter::Request:1",
@@ -137,8 +132,7 @@ void GenericRateLimiter::Request(int64_t bytes, const Env::IOPriority pri,
     static const int kRefillsPerTune = 100;
     std::chrono::microseconds now(NowMicrosMonotonicLocked());
     if (now - tuned_time_ >=
-        kRefillsPerTune * std::chrono::microseconds(refill_period_us_.load(
-                              std::memory_order_relaxed))) {
+        kRefillsPerTune * std::chrono::microseconds(refill_period_us_)) {
       Status s = TuneLocked();
       s.PermitUncheckedError();  //**TODO: What to do on error?
     }
@@ -279,8 +273,7 @@ GenericRateLimiter::GeneratePriorityIterationOrderLocked() {
 void GenericRateLimiter::RefillBytesAndGrantRequestsLocked() {
   TEST_SYNC_POINT_CALLBACK(
       "GenericRateLimiter::RefillBytesAndGrantRequestsLocked", &request_mutex_);
-  next_refill_us_ = NowMicrosMonotonicLocked() +
-                    refill_period_us_.load(std::memory_order_relaxed);
+  next_refill_us_ = NowMicrosMonotonicLocked() + refill_period_us_;
   // Carry over the left over quota from the last period
   auto refill_bytes_per_period =
       refill_bytes_per_period_.load(std::memory_order_relaxed);
@@ -297,10 +290,13 @@ void GenericRateLimiter::RefillBytesAndGrantRequestsLocked() {
     while (!queue->empty()) {
       auto* next_req = queue->front();
       if (available_bytes_ < next_req->request_bytes) {
-        // Grant partial request_bytes to avoid starvation of requests
-        // that become asking for more bytes than available_bytes_
-        // due to dynamically reduced rate limiter's bytes_per_second that
-        // leads to reduced refill_bytes_per_period hence available_bytes_
+        // Grant partial request_bytes even if request is for more than
+        // `available_bytes_`, which can happen in a few situations:
+        //
+        // - The available bytes were partially consumed by other request(s)
+        // - The rate was dynamically reduced while requests were already
+        //   enqueued
+        // - The burst size was explicitly set to be larger than the refill size
         next_req->request_bytes -= available_bytes_;
         available_bytes_ = 0;
         break;
@@ -318,28 +314,13 @@ void GenericRateLimiter::RefillBytesAndGrantRequestsLocked() {
 
 int64_t GenericRateLimiter::CalculateRefillBytesPerPeriodLocked(
     int64_t rate_bytes_per_sec) {
-  int64_t refill_period_us = refill_period_us_.load(std::memory_order_relaxed);
   if (std::numeric_limits<int64_t>::max() / rate_bytes_per_sec <
-      refill_period_us) {
+      refill_period_us_) {
     // Avoid unexpected result in the overflow case. The result now is still
     // inaccurate but is a number that is large enough.
     return std::numeric_limits<int64_t>::max() / kMicrosecondsPerSecond;
   } else {
-    return rate_bytes_per_sec * refill_period_us / kMicrosecondsPerSecond;
-  }
-}
-
-int64_t GenericRateLimiter::CalculateRefillPeriodUsLocked(
-    int64_t single_burst_bytes) {
-  int64_t rate_bytes_per_sec =
-      rate_bytes_per_sec_.load(std::memory_order_relaxed);
-  if (std::numeric_limits<int64_t>::max() / single_burst_bytes <
-      kMicrosecondsPerSecond) {
-    // Avoid unexpected result in the overflow case. The result now is still
-    // inaccurate but is a number that is large enough.
-    return std::numeric_limits<int64_t>::max() / rate_bytes_per_sec;
-  } else {
-    return single_burst_bytes * kMicrosecondsPerSecond / rate_bytes_per_sec;
+    return rate_bytes_per_sec * refill_period_us_ / kMicrosecondsPerSecond;
   }
 }
 
@@ -354,11 +335,10 @@ Status GenericRateLimiter::TuneLocked() {
   std::chrono::microseconds prev_tuned_time = tuned_time_;
   tuned_time_ = std::chrono::microseconds(NowMicrosMonotonicLocked());
 
-  int64_t refill_period_us = refill_period_us_.load(std::memory_order_relaxed);
   int64_t elapsed_intervals = (tuned_time_ - prev_tuned_time +
-                               std::chrono::microseconds(refill_period_us) -
+                               std::chrono::microseconds(refill_period_us_) -
                                std::chrono::microseconds(1)) /
-                              std::chrono::microseconds(refill_period_us);
+                              std::chrono::microseconds(refill_period_us_);
   // We tune every kRefillsPerTune intervals, so the overflow and division-by-
   // zero conditions should never happen.
   assert(num_drains_ <= std::numeric_limits<int64_t>::max() / 100);
@@ -398,13 +378,13 @@ RateLimiter* NewGenericRateLimiter(
     int64_t rate_bytes_per_sec, int64_t refill_period_us /* = 100 * 1000 */,
     int32_t fairness /* = 10 */,
     RateLimiter::Mode mode /* = RateLimiter::Mode::kWritesOnly */,
-    bool auto_tuned /* = false */) {
+    bool auto_tuned /* = false */, int64_t single_burst_bytes /* = 0 */) {
   assert(rate_bytes_per_sec > 0);
   assert(refill_period_us > 0);
   assert(fairness > 0);
-  std::unique_ptr<RateLimiter> limiter(
-      new GenericRateLimiter(rate_bytes_per_sec, refill_period_us, fairness,
-                             mode, SystemClock::Default(), auto_tuned));
+  std::unique_ptr<RateLimiter> limiter(new GenericRateLimiter(
+      rate_bytes_per_sec, refill_period_us, fairness, mode,
+      SystemClock::Default(), auto_tuned, single_burst_bytes));
   return limiter.release();
 }
 

util/rate_limiter_impl.h

@@ -28,8 +28,8 @@ class GenericRateLimiter : public RateLimiter {
  public:
   GenericRateLimiter(int64_t refill_bytes, int64_t refill_period_us,
                      int32_t fairness, RateLimiter::Mode mode,
-                     const std::shared_ptr<SystemClock>& clock,
-                     bool auto_tuned);
+                     const std::shared_ptr<SystemClock>& clock, bool auto_tuned,
+                     int64_t single_burst_bytes);
 
   virtual ~GenericRateLimiter();
 
@@ -47,7 +47,12 @@ class GenericRateLimiter : public RateLimiter {
                Statistics* stats) override;
 
   int64_t GetSingleBurstBytes() const override {
-    return refill_bytes_per_period_.load(std::memory_order_relaxed);
+    int64_t raw_single_burst_bytes =
+        raw_single_burst_bytes_.load(std::memory_order_relaxed);
+    if (raw_single_burst_bytes == 0) {
+      return refill_bytes_per_period_.load(std::memory_order_relaxed);
+    }
+    return raw_single_burst_bytes;
   }
 
   int64_t GetTotalBytesThrough(
@@ -108,10 +113,8 @@ class GenericRateLimiter : public RateLimiter {
   void RefillBytesAndGrantRequestsLocked();
   std::vector<Env::IOPriority> GeneratePriorityIterationOrderLocked();
   int64_t CalculateRefillBytesPerPeriodLocked(int64_t rate_bytes_per_sec);
-  int64_t CalculateRefillPeriodUsLocked(int64_t single_burst_bytes);
   Status TuneLocked();
   void SetBytesPerSecondLocked(int64_t bytes_per_second);
-  void SetSingleBurstBytesLocked(int64_t single_burst_bytes);
 
   uint64_t NowMicrosMonotonicLocked() {
     return clock_->NowNanos() / std::milli::den;
@@ -120,10 +123,12 @@ class GenericRateLimiter : public RateLimiter {
   // This mutex guard all internal states
   mutable port::Mutex request_mutex_;
 
-  std::atomic<int64_t> refill_period_us_;
+  const int64_t refill_period_us_;
 
   std::atomic<int64_t> rate_bytes_per_sec_;
   std::atomic<int64_t> refill_bytes_per_period_;
+  // This value is validated but unsanitized (may be zero).
+  std::atomic<int64_t> raw_single_burst_bytes_;
   std::shared_ptr<SystemClock> clock_;
 
   bool stop_;

util/rate_limiter_test.cc

@@ -35,7 +35,8 @@ class RateLimiterTest : public testing::Test {
 TEST_F(RateLimiterTest, OverflowRate) {
   GenericRateLimiter limiter(std::numeric_limits<int64_t>::max(), 1000, 10,
                              RateLimiter::Mode::kWritesOnly,
-                             SystemClock::Default(), false /* auto_tuned */);
+                             SystemClock::Default(), false /* auto_tuned */,
+                             0 /* single_burst_bytes */);
   ASSERT_GT(limiter.GetSingleBurstBytes(), 1000000000ll);
 }
 
@@ -160,10 +161,10 @@ TEST_F(RateLimiterTest, GetTotalPendingRequests) {
 TEST_F(RateLimiterTest, Modes) {
   for (auto mode : {RateLimiter::Mode::kWritesOnly,
                     RateLimiter::Mode::kReadsOnly, RateLimiter::Mode::kAllIo}) {
-    GenericRateLimiter limiter(2000 /* rate_bytes_per_sec */,
-                               1000 * 1000 /* refill_period_us */,
-                               10 /* fairness */, mode, SystemClock::Default(),
-                               false /* auto_tuned */);
+    GenericRateLimiter limiter(
+        2000 /* rate_bytes_per_sec */, 1000 * 1000 /* refill_period_us */,
+        10 /* fairness */, mode, SystemClock::Default(), false /* auto_tuned */,
+        0 /* single_burst_bytes */);
     limiter.Request(1000 /* bytes */, Env::IO_HIGH, nullptr /* stats */,
                     RateLimiter::OpType::kRead);
     if (mode == RateLimiter::Mode::kWritesOnly) {
@@ -389,7 +390,8 @@ TEST_F(RateLimiterTest, LimitChangeTest) {
       std::shared_ptr<RateLimiter> limiter =
           std::make_shared<GenericRateLimiter>(
               target, refill_period, 10, RateLimiter::Mode::kWritesOnly,
-              SystemClock::Default(), false /* auto_tuned */);
+              SystemClock::Default(), false /* auto_tuned */,
+              0 /* single_burst_bytes */);
       // After "GenericRateLimiter::Request:1" the mutex is held until the bytes
       // are refilled. This test could be improved to change the limit when lock
       // is released in `TimedWait()`.
@@ -430,7 +432,7 @@ TEST_F(RateLimiterTest, AvailableByteSizeExhaustTest) {
       available_bytes_per_period,
       std::chrono::microseconds(kTimePerRefill).count(), 10 /* fairness */,
       RateLimiter::Mode::kWritesOnly, special_env.GetSystemClock(),
-      false /* auto_tuned */);
+      false /* auto_tuned */, 0 /* single_burst_bytes */);
 
   // Step 1. Request 100 and wait for the refill
   // so that the remaining available bytes are 400
@@ -474,7 +476,8 @@ TEST_F(RateLimiterTest, AutoTuneIncreaseWhenFull) {
   std::unique_ptr<RateLimiter> rate_limiter(new GenericRateLimiter(
       1000 /* rate_bytes_per_sec */,
       std::chrono::microseconds(kTimePerRefill).count(), 10 /* fairness */,
-      RateLimiter::Mode::kWritesOnly, mock_clock, true /* auto_tuned */));
+      RateLimiter::Mode::kWritesOnly, mock_clock, true /* auto_tuned */,
+      0 /* single_burst_bytes */));
 
   // verify rate limit increases after a sequence of periods where rate limiter
   // is always drained
@@ -519,7 +522,8 @@ TEST_F(RateLimiterTest, WaitHangingBug) {
       std::make_shared<MockSystemClock>(Env::Default()->GetSystemClock());
   std::unique_ptr<RateLimiter> limiter(new GenericRateLimiter(
       kBytesPerSecond, kMicrosPerRefill, 10 /* fairness */,
-      RateLimiter::Mode::kWritesOnly, mock_clock, false /* auto_tuned */));
+      RateLimiter::Mode::kWritesOnly, mock_clock, false /* auto_tuned */,
+      0 /* single_burst_bytes */));
   std::array<std::thread, 3> request_threads;
 
   ROCKSDB_NAMESPACE::SyncPoint::GetInstance()->LoadDependency(
@@ -559,29 +563,43 @@ TEST_F(RateLimiterTest, RuntimeSingleBurstBytesChange) {
   constexpr int kMicrosecondsPerSecond = 1000000;
 
   const int64_t kRateBytesPerSec = 400;
+  const int64_t kRefillBytes = 100;
 
-  const int64_t kOldSingleBurstBytes = 100;
-  const int64_t kOldRefillPeriodUs =
-      kOldSingleBurstBytes * kMicrosecondsPerSecond / kRateBytesPerSec;
-  const int64_t kNewSingleBurstBytes = kOldSingleBurstBytes * 2;
+  const int64_t kRefillPeriodMicros =
+      kRefillBytes * kMicrosecondsPerSecond / kRateBytesPerSec;
 
-  SpecialEnv special_env(Env::Default(), /*time_elapse_only_sleep*/ true);
+  const int64_t kRefillsPerBurst = 17;
+  const int64_t kBurstBytes = kRefillBytes * kRefillsPerBurst;
+
+  auto mock_clock =
+      std::make_shared<MockSystemClock>(Env::Default()->GetSystemClock());
+
+  // Zero as `single_burst_bytes` is a special value meaning the refill size
   std::unique_ptr<RateLimiter> limiter(new GenericRateLimiter(
-      kRateBytesPerSec, kOldRefillPeriodUs, 10 /* fairness */,
-      RateLimiter::Mode::kWritesOnly, special_env.GetSystemClock(),
-      false /* auto_tuned */));
+      kRateBytesPerSec, kRefillPeriodMicros, 10 /* fairness */,
+      RateLimiter::Mode::kWritesOnly, mock_clock, false /* auto_tuned */,
+      0 /* single_burst_bytes */));
+  ASSERT_EQ(kRefillBytes, limiter->GetSingleBurstBytes());
+
+  // Dynamically setting to zero should change nothing
+  ASSERT_OK(limiter->SetSingleBurstBytes(0));
+  ASSERT_EQ(kRefillBytes, limiter->GetSingleBurstBytes());
 
-  ASSERT_EQ(kOldSingleBurstBytes, limiter->GetSingleBurstBytes());
+  // Negative values are invalid and should change nothing
+  ASSERT_TRUE(limiter->SetSingleBurstBytes(-1).IsInvalidArgument());
+  ASSERT_EQ(kRefillBytes, limiter->GetSingleBurstBytes());
 
-  ASSERT_TRUE(limiter->SetSingleBurstBytes(0).IsInvalidArgument());
-  ASSERT_OK(limiter->SetSingleBurstBytes(kNewSingleBurstBytes));
-  ASSERT_EQ(kNewSingleBurstBytes, limiter->GetSingleBurstBytes());
+  // Positive values take effect as the new burst size
+  ASSERT_OK(limiter->SetSingleBurstBytes(kBurstBytes));
+  ASSERT_EQ(kBurstBytes, limiter->GetSingleBurstBytes());
 
-  // If the updated single burst bytes is not reflected in the bytes
-  // granting process, this request will hang forever.
+  // Initially the supply is full so a request of size `kBurstBytes` needs
+  // `kRefillsPerBurst - 1` refill periods to elapse.
   limiter->Request(limiter->GetSingleBurstBytes() /* bytes */,
                    Env::IOPriority::IO_USER, nullptr /* stats */,
                    RateLimiter::OpType::kWrite);
+  ASSERT_EQ((kRefillsPerBurst - 1) * kRefillPeriodMicros,
+            mock_clock->NowMicros());
 }
 
 }  // namespace ROCKSDB_NAMESPACE