Tiktoken wrapper add_eos_token option

[rajammanabrolu]

Modify the tiktoken wrapper to mimic HF Llama tokenizer behavior to add either or both BOS/EOS tokens while encoding (add_eos_token option in build_inputs_with_special_tokens and corresponding change to get_special_tokens_mask)

[dakinggg]

Could you add some simple unit tests to test this functionality? Just that whatever API you want to use (encode_plus I think?) works the way it should.

[rajammanabrolu]

Added tests, mimics how I'll use it in the RL loop. Lmk if there's more things I should be adding