[QUESTION] How to overcome with Access Denied in downloadAddr or playAddr

[jonaskahn]

So, i tried using some test code to download tiktok video. The API seems working well , but I can not access to video url. I do not know whether I did wrong setup or it's a bug
Here is my code

import asyncio
import json

from f2.apps.tiktok.handler import TiktokHandler
from f2.apps.tiktok.utils import DeviceIdManager


async def main():
    device_ids = await DeviceIdManager.gen_device_ids(1, full_cookie=True)
    print(json.dumps(device_ids, indent=4))
    kwargs = {
        "headers": {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36 Edg/126.0.0.0",
            "Referer": "https://www.tiktok.com/",
        },
        "proxies": {"http://": None, "https://": None},
        "cookie": device_ids["cookie"][0],
    }
    video = await TiktokHandler(kwargs).fetch_one_video(itemId="7095819783324601605")
    print("=================_to_raw================")
    print(video._to_json())
    # print("=================_to_dict================")
    # print(video._to_dict())
    # print("=================_to_list================")
    # print(video._to_list())


if __name__ == "__main__":
    asyncio.run(main())

Here is the error

You don't have permission to access "http://v16-webapp-prime.tiktok.com/video/tos/useast2a/tos-useast2a-ve-0068c003/f0351788fbbb48829b73ca7d951aec41/?" on this server.
Reference #18.d522ca17.1721691509.2fde4e5e

https://errors.edgesuite.net/18.d522ca17.1721691509.2fde4e5e```

[CherryBoros]

please refer to the following links

[QUESTION] 使用 /docs/snippets/tiktok/user-post.py 得到的 “video_playAddr” 无法访问 #97

tiktok 403 Forbidden

[Johnserf-Seed]

The tiktok video address is required to carry headers to the request, to include the cookie parameter bound to the devic_id to return the video content correctly

[jonaskahn]

@Johnserf-Seed
Well, I am back with F2 today, and problem may still there.
Here is the demonstration code

import asyncio

from f2.apps.tiktok.dl import TiktokDownloader
from f2.apps.tiktok.handler import TiktokHandler
from f2.apps.tiktok.utils import DeviceIdManager


async def main():
    device_id = await DeviceIdManager.gen_device_id(full_cookie=True)
    kwargs = {
        "headers": {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0",
            "Referer": "https://www.tiktok.com/",
        },
        "timeout": 1000,
        "cookie": f"{device_id['cookie']}",
    }

    video = await TiktokHandler(kwargs=kwargs).fetch_one_video("7421889378101808404")
    res = await TiktokDownloader(kwargs=kwargs).create_download_tasks(kwargs={"interval": "all"},aweme_datas=video._to_dict(), user_path="/home/jonas/Downloads")

if __name__ == "__main__":
    asyncio.run(main())

and I still got error

/home/jonas/miniconda3/envs/t2scrapper/bin/python /mnt/workspaces/Project/Research/t2scrapper/tests/test.py 
ERROR    Traceback (most recent call last):                                     
           File                                                                 
         "/home/jonas/miniconda3/envs/t2scrapper/lib/python3.10/site-packages/f2
         /utils/_dl.py", line 42, in get_content_length                         
             response.raise_for_status()                                        
           File                                                                 
         "/home/jonas/miniconda3/envs/t2scrapper/lib/python3.10/site-packages/ht
         tpx/_models.py", line 761, in raise_for_status                         
             raise HTTPStatusError(message, request=request, response=self)     
         httpx.HTTPStatusError: Client error '403 Forbidden' for url            
         'https://v16-webapp-prime.tiktok.com/video/tos/alisg/tos-alisg-pve-0037
         c001/osoCf0QAP9aDv7higRIjSZfiIxDFf2ENAQMI0m/?a=1988&bti=ODszNWYuMDE6&ch
         =0&cr=3&dr=0&lr=all&cd=0%7C0%7C0%7C&cv=1&br=3934&bt=1967&cs=0&ds=6&ft=-
         Csk_m7nPD12NulwOE-Ux472LY6e3wv25QcAp&mime_type=video_mp4&qs=0&rc=OmVnPG
         lpZ2Y4NTc1PDY0ZUBpM3JvdGo5cjl3dTMzODczNEAyY2BjYS82Xi8xNi8vYTQ2YSNnaC5yM
         mRjZXNgLS1kMTFzcw%3D%3D&src=1&btag=e00088000&expire=1735379642&l=202412
         26095326811D52919EA6FCDE341B&ply_type=2&policy=2&signature=06464052293a
         75607c56200d6bba0a2a&tk=tt_chain_token'                                
         For more information check:                                            
         https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403           
                                                                                

Next try, i use "cookie": f"{device_id}". But same error occur. Do you have any suggestion?

[Johnserf-Seed]

@Johnserf-Seed Well, I am back with F2 today, and problem may still there. Here is the demonstration code

import asyncio

from f2.apps.tiktok.dl import TiktokDownloader
from f2.apps.tiktok.handler import TiktokHandler
from f2.apps.tiktok.utils import DeviceIdManager


async def main():
    device_id = await DeviceIdManager.gen_device_id(full_cookie=True)
    kwargs = {
        "headers": {
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36 Edg/122.0.0.0",
            "Referer": "https://www.tiktok.com/",
        },
        "timeout": 1000,
        "cookie": f"{device_id['cookie']}",
    }

    video = await TiktokHandler(kwargs=kwargs).fetch_one_video("7421889378101808404")
    res = await TiktokDownloader(kwargs=kwargs).create_download_tasks(kwargs={"interval": "all"},aweme_datas=video._to_dict(), user_path="/home/jonas/Downloads")

if __name__ == "__main__":
    asyncio.run(main())

and I still got error

/home/jonas/miniconda3/envs/t2scrapper/bin/python /mnt/workspaces/Project/Research/t2scrapper/tests/test.py 
ERROR    Traceback (most recent call last):                                     
           File                                                                 
         "/home/jonas/miniconda3/envs/t2scrapper/lib/python3.10/site-packages/f2
         /utils/_dl.py", line 42, in get_content_length                         
             response.raise_for_status()                                        
           File                                                                 
         "/home/jonas/miniconda3/envs/t2scrapper/lib/python3.10/site-packages/ht
         tpx/_models.py", line 761, in raise_for_status                         
             raise HTTPStatusError(message, request=request, response=self)     
         httpx.HTTPStatusError: Client error '403 Forbidden' for url            
         'https://v16-webapp-prime.tiktok.com/video/tos/alisg/tos-alisg-pve-0037
         c001/osoCf0QAP9aDv7higRIjSZfiIxDFf2ENAQMI0m/?a=1988&bti=ODszNWYuMDE6&ch
         =0&cr=3&dr=0&lr=all&cd=0%7C0%7C0%7C&cv=1&br=3934&bt=1967&cs=0&ds=6&ft=-
         Csk_m7nPD12NulwOE-Ux472LY6e3wv25QcAp&mime_type=video_mp4&qs=0&rc=OmVnPG
         lpZ2Y4NTc1PDY0ZUBpM3JvdGo5cjl3dTMzODczNEAyY2BjYS82Xi8xNi8vYTQ2YSNnaC5yM
         mRjZXNgLS1kMTFzcw%3D%3D&src=1&btag=e00088000&expire=1735379642&l=202412
         26095326811D52919EA6FCDE341B&ply_type=2&policy=2&signature=06464052293a
         75607c56200d6bba0a2a&tk=tt_chain_token'                                
         For more information check:                                            
         https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/403           
                                                                                

Next try, i use "cookie": f"{device_id}". But same error occur. Do you have any suggestion?

@jonaskahn
Let me explain it to you in detail and you'll understand.
The first point is that the generation of device_id is actually bound to ua, as you can see in the source code of gen_device_id, using the ua filled in conf.yaml.
The second point is that the cookie that accompanies the generation of device_id is also bound. That is to say, you only use the accompanying cookie and do not fill in the bound device_id in conf.yaml, resulting in the video playaddr 403.
The third point, you can not generate the device_id every time, too often will cause your ip to be blocked. the core function of the method of gen_device_id is in fact to run after the download fails!

Important

The core function of the gen_device_id method is to run the method once after the download fails to get the new device_id and then go to continue the download.

Once you understand this, you can modify your code:

import asyncio

from f2.apps.tiktok.dl import TiktokDownloader
from f2.apps.tiktok.handler import TiktokHandler
from f2.apps.tiktok.utils import DeviceIdManager

# from f2.log.logger import logger

# set logger level to DEBUG
# logger.setLevel("DEBUG")


async def print_device_id():
    # if not device_id, generate one, not every request need to generate device_id
    device_id = await DeviceIdManager.gen_device_id(full_cookie=True)
    # print device_id and fill in the cookie field in kwargs, fill device_id field in conf.yaml
    print(device_id)


async def main():

    kwargs = {
        "headers": {
            # UA is very important, you should keep it up to date from conf.yaml
            "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36",
            "Referer": "https://www.tiktok.com/",
        },
        "timeout": 1000,
        "interval": "all",
        # Here you should fill in the cookie you got from the web page, or the cookie returned by binding with device_id
        "cookie": "",
    }

    video = await TiktokHandler(kwargs).fetch_one_video("7421889378101808404")
    await TiktokDownloader(kwargs).create_download_tasks(
        # You should full kwargs in here
        kwargs,
        aweme_datas=video._to_dict(),
        user_path="/home/jonas/Downloads",
    )


if __name__ == "__main__":
    # first step, you should run print_device_id to set up
    # asyncio.run(print_device_id())
    # second step, after you set up device_id, you can run main
    asyncio.run(main())

Not surprisingly, you will successfully download this video!

[jonaskahn]

Oh, I see. What a great explanation. I'll try by myself

[jonaskahn]

Well, I made it. After binding deviceID into conf.yaml. It just worked :D . Thank you.

One more thing is f2.F2_CONFIG_FILE_PATH = "conf.yaml" seems does not work (even if I try to absolute path). I must directly edit DeviceID in the library conf/conf.yaml ? Do you have any suggestions?

[Johnserf-Seed]

Well, I made it. After binding deviceID into conf.yaml. It just worked :D . Thank you.

One more thing is f2.F2_CONFIG_FILE_PATH = "conf.yaml" seems does not work (even if I try to absolute path). I must directly edit DeviceID in the library conf/conf.yaml ? Do you have any suggestions?

Important

The last important thing is that in order to maintain standards, f2 has agreed that yaml-related files must be placed in the conf/ directory in the developer interface, whether they are customized or default. So you need to create a conf directory dedicated to the configuration file, I think this is a reason why you failed before, I will specifically point this out in the documentation.

The directory should look like this

│ test-conf-yaml.py
│─conf
└──test_conf.yaml

These example methods can all read the custom path conf.yaml, you can try it out! @jonaskahn

import f2
from f2.utils.conf_manager import ConfigManager

# Example test yaml file path by absolute path & relative path
path_abs = r"H:\GitHub\Clone\f2\f2\conf\test_conf.yaml"
path_rel = r"conf\test_conf.yaml"

# Test default path useing f2.F2_CONFIG_FILE_PATH
print(f2.F2_CONFIG_FILE_PATH)
print(ConfigManager(f2.F2_CONFIG_FILE_PATH).get_config("f2").get("tiktok"))

# Test absolute path useing f2.F2_CONFIG_FILE_PATH
f2.F2_CONFIG_FILE_PATH = path_abs
print(f2.F2_CONFIG_FILE_PATH)
print(ConfigManager(f2.F2_CONFIG_FILE_PATH).get_config("f2").get("tiktok"))

# Test relative path useing f2.F2_CONFIG_FILE_PATH
f2.F2_CONFIG_FILE_PATH = path_rel
print(f2.F2_CONFIG_FILE_PATH)
print(ConfigManager(f2.F2_CONFIG_FILE_PATH).get_config("f2").get("tiktok"))

# Test absolute path useing path_abs
print(ConfigManager(path_abs).get_config("f2").get("tiktok"))

# Test relative path useing path_rel
print(ConfigManager(path_rel).get_config("f2").get("tiktok"))

[jonaskahn]

Oh I see. Thank for your supporting. I highly appreciated that (y)