RunwayML seem to have pulled SD 1.5 from HF

[jamesd256]

Hi,

I thought I would draw attention to the fact that the install instructions do not work for me, since the link to the HF SD 1.5 resources is now giving a 404.

As a work-around, I attempted to use a repo which appears to be a fork of the original SD 1.5, although I'm not clear if this is the case:

https://huggingface.co/pt-sk/stable-diffusion-1.5

In any case, running the demo against this alternate repo gives an error

model_config =  {'base_model_type': 'realisticVisionV60B1_v51VAE', 'model_path': 'resources/models', 'text_to_video_mm_path': 'resources/models/fancyvideo_ckpts/vae_3d_61_frames/mp_rank_00_model_states.pt', 'base_model_path': 'resources/models/sd_v1-5_base_models/realisticVisionV60B1_v51VAE.safetensors', 'res_adapter_type': 'res_adapter_v2', 'trained_keys': ['motion_modules.', 'conv_in.weight', 'fps_embedding.', 'motion_embedding.'], 'vae_type': 'vae_3d', 'use_fps_embedding': True, 'use_motion_embedding': True, 'common_positive_prompt': 'Best quality, masterpiece, ultra high res, photorealistic, Ultra realistic illustration, hyperrealistic, 8k', 'common_negative_prompt': '(low quality:1.3), (worst quality:1.3),poorly drawn face, mutation, deformed, blurry, dehydrated, bad anatomy, bad proportions, extra limbs, cloned face,Facial blurring,a large crowd, many people,advertising, information, news, watermark, text, username, signature,out of frame, low res, error, cropped, worst quality, low quality, artifacts, ugly, duplicate, morbid, mutilated, extra fingers, mutated hands, poorly drawn hands, disfigured, gross proportions, malformed limbs, missing arms, missing legs, extra arms, extra legs, fused fingers, too many fingers, long neck, nsfw, breast, naked, eroticism'}
infer_config =  {'infer_mode': 'i2v', 'resolution': [768, 768], 'video_length': 16, 'output_fps': 25, 'cond_fps': 25, 'cond_motion_score': 3.0, 'use_noise_scheduler_snr': True, 'seed': 22, 'prompt_path': 'resources/demos/test_prompts/test_i2v_prompt.txt', 'reference_image_folder': 'resources/demos/reference_images/768x768', 'output_folder': 'resources/demos/samples/i2v/realisticVisionV60B1_v51VAE/768x768'}


Start loading infer pipeline...
device =  cuda
unet trained keys =  ['motion_modules.', 'conv_in.weight', 'fps_embedding.', 'motion_embedding.']
vae_type =  vae_3d


/mnt/storage2/workspace/FancyVideo/env/lib/python3.10/site-packages/diffusers/schedulers/scheduling_ddim.py:142: UserWarning: To copy construct from a tensor, it is recommended to use sourceTensor.clone().detach() or sourceTensor.clone().detach().requires_grad_(True), rather than torch.tensor(sourceTensor).
  self.betas = torch.tensor(trained_betas, dtype=torch.float32)
load scheduler done

Traceback (most recent call last):
  File "/mnt/storage2/workspace/FancyVideo/scripts/demo.py", line 77, in <module>
    main(args)
  File "/mnt/storage2/workspace/FancyVideo/env/lib/python3.10/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/mnt/storage2/workspace/FancyVideo/scripts/demo.py", line 22, in main
    infer_pipeline = InferPipeline(
  File "/mnt/storage2/workspace/FancyVideo/fancyvideo/pipelines/fancyvideo_infer_pipeline.py", line 151, in __init__
    tokenizer    = CLIPTokenizer.from_pretrained(pretrained_model_path, subfolder="tokenizer")
  File "/mnt/storage2/workspace/FancyVideo/env/lib/python3.10/site-packages/transformers/tokenization_utils_base.py", line 1760, in from_pretrained
    resolved_vocab_files[file_id] = cached_file(
  File "/mnt/storage2/workspace/FancyVideo/env/lib/python3.10/site-packages/transformers/utils/hub.py", line 409, in cached_file
    resolved_file = hf_hub_download(
  File "/mnt/storage2/workspace/FancyVideo/env/lib/python3.10/site-packages/huggingface_hub/utils/_deprecation.py", line 101, in inner_f
    return f(*args, **kwargs)
  File "/mnt/storage2/workspace/FancyVideo/env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 106, in _inner_fn
    validate_repo_id(arg_value)
  File "/mnt/storage2/workspace/FancyVideo/env/lib/python3.10/site-packages/huggingface_hub/utils/_validators.py", line 154, in validate_repo_id
    raise HFValidationError(
huggingface_hub.errors.HFValidationError: Repo id must be in the form 'repo_name' or 'namespace/repo_name': 'resources/models/stable-diffusion-v1-5'. Use `repo_type` argument if needed.

Thank you for all the hard work that went into this, I look forward to trying it out

[5ven]

I had the same issue and managed to get past it. I had to download all the files from this repository which is copy of the original runwayml repo:

https://huggingface.co/benjamin-paine/stable-diffusion-v1-5/tree/main

As a precautuion to make sure to have compatible Python environment, I did these extra steps:
python -m venv .venv && source ./.venv/bin/activate

before running the installation.

Then just had to copy the files over to resources/models/stable-diffusion-v1-5 and
CUDA_VISIBLE_DEVICES=0 PYTHONPATH=./ python scripts/demo.py --config configs/inference/i2v.yaml

just worked.

More information

[jamesd256]

Thank you Sven, will try it out