web_base loader is broken

[DmitryKatson]

System Info

Name: langchain
Version: 0.0.299

Who can help?

@eyurtsev @hwchase17

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

Just follow https://python.langchain.com/docs/integrations/document_loaders/web_base

from langchain.document_loaders import WebBaseLoader
loader = WebBaseLoader("https://www.espn.com/")
data = loader.load()

Expected behavior

The standard WebLoader is broken when we pass webpath as described in the docs as
loader = WebBaseLoader("https://www.espn.com/")

However it workes if we pass as
loader = WebBaseLoader(["https://www.espn.com/"])

The reason for that is this commit

Also because of this breaking change the other Custom Web Loaders are broken as well IMSDbLoader, AZLyricsLoader, and CollegeConfidentialLoader

[dosubot]

🤖

Hello,

Thank you for bringing this issue to our attention. It seems like the problem is due to the way the web_paths attribute is set in the __init__ method of the WebBaseLoader class. If web_path is a string, it is not considered a Sequence and hence, it is not converted to a list. Instead, it is directly assigned to self.web_paths. This means that self.web_paths becomes a string instead of a list of strings. This could cause issues in methods like lazy_load, load, and aload where self.web_paths is iterated over, expecting a list of strings.

This issue would affect the functionality of other custom web loaders like IMSDbLoader, AZLyricsLoader, and CollegeConfidentialLoader if they inherit from WebBaseLoader and use the web_paths attribute, as they would also face the same issue when a single webpath is passed as a string.

To fix this issue, you could modify the __init__ method to always convert web_path to a list, even if it's a single string:

if web_paths:
    self.web_paths = list(web_paths)
else:
    self.web_paths = [web_path] if isinstance(web_path, str) else list(web_path)

This way, self.web_paths will always be a list of strings, whether web_path is a single string or a sequence of strings.

As for the commit you mentioned, I'm unable to find the specific changes that could have caused the WebBaseLoader to break. Could you please provide more details about the changes in that commit?

Thank you for your patience and cooperation.

Sources

• UnstructuredURLLoader can't load data from url
• libs/langchain/langchain/document_loaders/web_base.py

---

This response is meant to be useful and save you time. It is not meant to be a precise solution, but rather a starting point for your own research.

Help me be more useful! Please leave a 👍 if this is helpful and 👎 if it is irrelevant.

If you want to continue the conversation, start your reply with @dosu-bot.

[DmitryKatson]

This is not very useful as web_base loader doesn't work as it's written in the docs. The team should fix the web_base loader or change the docs and all dependent loaders

[JahangirJadi]

from langchain.document_loaders import WebBaseLoader
from langchain.indexes import VectorstoreIndexCreator

url = 'https://lilianweng.github.io/posts/2023-06-23-agent/'
loader = WebBaseLoader(URL)

This made me crazy.

loader.load not working. Please fix it.

error:
436 raise InvalidURL(*e.args)
438 if not scheme:
--> 439 raise MissingSchema(
440 f"Invalid URL {url!r}: No scheme supplied. "
441 f"Perhaps you meant https://github.com/langchain-ai/langchain/issues/3542{url}?"
442 )
444 if not host:
445 raise InvalidURL(f"Invalid URL {url!r}: No host supplied")

MissingSchema: Invalid URL 'h': No scheme supplied. Perhaps you meant https://h?

[DmitryKatson]

from langchain.document_loaders import WebBaseLoader from langchain.indexes import VectorstoreIndexCreator

url = 'https://lilianweng.github.io/posts/2023-06-23-agent/' loader = WebBaseLoader(URL)

This made me crazy.

loader.load not working. Please fix it.

error: 436 raise InvalidURL(*e.args) 438 if not scheme: --> 439 raise MissingSchema( 440 f"Invalid URL {url!r}: No scheme supplied. " 441 f"Perhaps you meant https://github.com/langchain-ai/langchain/issues/3542{url}?" 442 ) 444 if not host: 445 raise InvalidURL(f"Invalid URL {url!r}: No host supplied")

MissingSchema: Invalid URL 'h': No scheme supplied. Perhaps you meant https://h?

@JahangirJadi
So as a quick workaround solution this works.

url = 'https://lilianweng.github.io/posts/2023-06-23-agent/'
loader = WebBaseLoader([URL])

[selfmeister]

worked for me:

if isinstance(web_path, str):
     self.web_paths = [web_path]
else:
     self.web_paths = list(web_paths)```

[dosubot]

Hi, @DmitryKatson

I'm helping the LangChain team manage their backlog and am marking this issue as stale. From what I understand, the issue was raised due to the WebBaseLoader being broken when using the webpath as described in the documentation. There were discussions around potential fixes and workarounds provided by you and other users, but the resolution status remains unresolved.

Could you please confirm if this issue is still relevant to the latest version of the LangChain repository? If it is, please let the LangChain team know by commenting on the issue. Otherwise, feel free to close the issue yourself, or it will be automatically closed in 7 days. Thank you for your understanding and contributions.

I (Dosu)