Crawls web pages and saves the URL and a description in an elasticsearch database. Additionally provides a web frontend to search for homepages, URLs and keywords.
Built with
Usage:
- run local elasticsearch instance on
localhost:9000 - run frontend server:
python3 server.py - run crawler:
python3 main.py [root-url]