Skip to content

Car Information telegram bot that scrapes Russian car services

Notifications You must be signed in to change notification settings

ReDestroyDeR/CarMeta

Repository files navigation

CarMeta

Car Information telegram bot that scrapes Russian car services

This is my final project for "Scala Developer" course at Otus.

Architecture

architecture

Done:

[ x ] Scraping Service (AutoRussia only, no Advertisement suport)
[ x ] Aggregation Service
[ - ] Warehouse
[ - ] Telegram Frontend

Project is built using SBT

Service decomposition is done poorly, development is done like it's Mutli-Module Monolith, but .jars should be deployed separetely

Technologies

  • Scala 2.13
  • Cats Effect 3
  • fs2
  • Ciris
  • Circe
  • ScalaPB
  • Kafka
  • Redis
  • Elasticsearch

Run

Docker compose includes Kafka, Confluent Control Panel, Redis, Kibana and Elasticsearch. You can run these dependencies by running docker compose up -d

Start scraper and aggregation services, you should see that Scraper is scraping data from AutoRussia and Aggregation service is getting records from definitions topic. Data Flow could be seen in Control Panel, after that, Definitions of cars could be seen in Kibana.

About

Car Information telegram bot that scrapes Russian car services

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages