SOSSE (Selenium Open Source Search Engine) is a web archiving software, crawler, and search engine. It’s hosted on both GitLab and GitHub. Feel free to use either platform to submit feature requests, bug reports, merge requests, or start a discussion.
-
🌍 Web Page Search: Search the content of web pages, including dynamically rendered ones, with advanced queries. (doc)
-
🕑 Recurring Crawling: Crawl pages at fixed intervals or adapt the rate based on content changes. (doc)
-
🔖 Web Page Archiving: Archive HTML content, adjust links for local use, download required assets, and support dynamic content. (doc)
-
📂 File Downloads: Batch download binary files from web pages. (doc)
-
🔔 Atom Feeds: Generate content feeds for websites that don’t have them, or receive updates when a new page containing a keyword is published. (doc)
-
🔒 Authentication: The crawler can authenticate to access private pages and retrieve content. (doc)
-
👥 Permissions: Admins can configure crawlers and view statistics, while authenticated users can search or do so anonymously. (doc)
-
👤 Search Features: Includes private search history (doc), and external search engine shortcuts (doc), etc.
Explore the 📚 documentation and check out some 📷 screenshots.
SOSSE is written in Python and is distributed under the GNU AGPLv3 license. It uses browser-based crawling with Mozilla Firefox or Google Chromium alongside Selenium to index pages that rely on JavaScript. For faster crawling, Requests can also be used. SOSSE uses PostgreSQL for data storage.
To quickly try the latest version with Docker:
docker run -p 8005:80 biolds/sosse:latest
Then, open http://127.0.0.1:8005/ and log in with the username admin and password admin.
For persistence of Docker data or alternative installation methods, please refer to the installation guide.
Join the Discord server to get help, share ideas, or discuss SOSSE!