Starbelly is a user-friendly web crawler that is easy to deploy and configure.
For information about security considerations and best practices for deploying Starbelly, please see docs/SECURITY.md.
If you discover a security vulnerability, please email acaceres@hyperiongray.com rather than opening a public issue.
- Policy-Based Crawling: Define custom crawl policies to control crawler behavior
- Graphical User Interface: Easy-to-use web interface for managing crawls
- WebSocket API: Real-time communication and streaming results
- Docker Deployment: Simple deployment using Docker containers
- RethinkDB Backend: Scalable database for storing crawl data
- Asynchronous I/O: Built on Trio for high-performance concurrent crawling
Starbelly is deployed using Docker and Docker Compose. See the Installation Guide for detailed instructions.
Quick start:
git clone https://github.com/hyperiongray/starbelly-docker.git
cd starbelly-docker/starbelly
docker-compose up -dAfter installation, navigate to your server's address in a web browser. The default credentials are:
- Username: admin
- Password: admin
For detailed usage instructions, see the documentation.
Complete documentation is available at starbelly.readthedocs.io.
Example notebooks and scripts are available in the notebooks and examples directories.
Starbelly provides a WebSocket API for programmatic access. See the WebSocket API documentation for details.
Python client library: starbelly-python-client
We welcome contributions! Please see CONTRIBUTING.md for guidelines.
Starbelly is under the MIT License. See LICENSE for details.
For commercial support or inquiries, please contact Hyperion Gray at acaceres@hyperiongray.com
