This is an open-source free alternative to Go Index. What I noticed was google and bing were being quite slow at indexing and by using their APIs the whole process can speed up. Furthermore, sometimes google and bing do not handle the whole sitemap updates correctly resulting at very slow overall (and sometimes inexistant) indexing for fully dynamic-based websites that generate content frequently.
This code opts in to do the indexing manually by crawling your target's sitemaps, getting all content, checking if the content of the url has already been pushed to google/bing before and if not it pushes it. Runs every 5 minutes (edit the index.ts for the frequency of your choice) and saves data on an sqlite database as MD5 hashes.
Setup your .env file like the .env.example:
GOOGLE_CLIENT_EMAIL="blahblah.gserviceaccount.com"
GOOGLE_PRIVATE_KEY="-----BEGIN PRIVATE KEY-----\nMI0....ILE=\n-----END PRIVATE KEY-----\n"
BING_API_KEY="25e...338d"
TARGET="https://unitedgpt.app"
To get the Google credentials follow this guide using your google account with which the Google Search Console TARGET is verified. Export them as JSON and extract the fileds client_email and private_key.
You can install this by:
bun install
bun run db
bun run start