Use save_pretrained to make the model locally available

[tkilias]

Background:

• Currently, we use from_pretrained with the cache_dir option to load the model and upload the cache directory to the BucketFS
• The current approach in general works, but seems to check each time online if the model exists, or we have access and if we have the newest version
• This is not possible in some deployments
• To solve this issue, we could follow the following example using save_pretrained which stores the model into a directory which we then could upload
• We can then load the model with from_pretrained from a local path

Acceptance Criteria

• Change download of model to save_pretrained
• Upload the save model to the BucketFS
• Use a local path to load the model during inference

[MarleneKress79789]

suggested splitting of this ticket:

• familiarize self with huggingface api

• add new functionality (t1: HuggingFaceHubBucketFSModelTransfer with save_pretrained: as duplicate of existing code for download)

• add tests for new functionality

• t2: Extract https://github.com/exasol/transformers-extension/blob/1321af15f1cf0e9ac68ca12c02a3[…]cb1/exasol_transformers_extension/udfs/models/base_model_udf.py load function refac -> make to class and inject it

• t2.5: create new load function feature

• • add test

• change code to use new functionality ,remove check for newest version (should this be toogleable?) (t3)

• • download
• • upload
• • load
• • remove old tests

• remove token for private huggingface model and change tests (only needed for download) (t4)

[MarleneKress79789]

new tickets are

• Add HuggingFaceHubBucketFSModelTransfer with save_pretrained #143
• Extract base_model_udf.load_models into separate class #144
• Add load class which loads model from local path #145
• Use new functions with save_pretrained for model loading #146
• Remove token for private huggingface model from model loading #147