distinction between partition_number and segment_number(k)

[dhfromkorea]

Hi, first of all, thank you for making this available to the world. I am curious about the motivation of having partition number. I think I am confused with the role of this and the segment(-=strata?)

I see the buffer consists of partitions. And each partition consists of segments whose length is batch size.

In addition, I initialized ExperienceReplay with the following setup. And I am getting this error: ValueError: empty range for randrange() (2,2, 0) at this line below.

for n in range(1, self.batch_size + 1):
    index = random.randint(distribution['strata_ends'][n] + 1,
    distribution['strata_ends'][n + 1])
    rank_list.append(index)

and it is because my distribution['strata_ends'] was set up wrong.

distribution['strata_ends']
{1: 0, 2: 1, 3: 1, 4: 1, 5: 1, 6: 2, 7: 3, 8: 4, 9: 5, 10: 7, 11: 8, 12: 10, 13: 12, 14: 15, 15: 17, 16: 20, 17: 24, 18: 28, 19: 32, 20: 36, 21: 41, 22: 46, 23: 52, 24: 58, 25: 65, 26: 72, 27: 80, 28: 88, 29: 97, 30: 107, 31: 117, 32: 128, 33: 140}

And this is because cdf[index] < step evaluates to False in early stratas, not incrementing the index.

for s in range(2, self.batch_size + 1):
    while cdf[index] < step:
        index += 1
    strata_ends[s] = index
    step += 1 / self.batch_size

Is this expected? Would you have any advice on what I am doing wrong?

My ER set up is:

self.config = {'size':10**3,
                         'learn_start': 100,
                         'partition_num': 100,
                         'total_step': config.get('total_step', 10*9),
                         'batch_size': 32,
                         }

[sungjae-cho]

@dhfromkorea 안녕하세요. Hi, I met the same error as yours.
ValueError: empty range for randrange() (2,2, 0)

[dhfromkorea]

Hi, @sungjae-cho and @Damcy I figured out the issue.

Due to the particular way the algorithm was implemented (w.r.t dist_index & partition_number), I discovered there are a few constraints we need to impose on the hyperparameters. Here are assertion statements I set up for my project.

# this must be an integer (otherwise, KeyError or ValueError may be thrown)
dist_index = config['learn_start'] / config['size'] * config['partition_num']
assert math.floor(dist_index) == math.ceil(dist_index), "this must be an integer"
assert config['learn_start'] * config['partition_num'] >= config['size'], "another glitch"

I found it useful to initialize every hyper-parameter as a power of 2. (Technically it can be any integer like 10).

[fstonezst]

@dhfromkorea I put your code in the head of init function, but seems like it doesn't work