zeminerbot is ZeMiner.com's web crawling robot. It fetches millions of pages per day looking for fresh data so you don't have to run expensive and time consuming data retrieval jobs.
Ethical data retrieval is a top priority at ZeMiner.com. Therefore, when you purchase our datasets, you can rest assured the websites from which the data was gathered were not crawled in a
Our crawler honors the robots.txt standard. To exclude any path on your web server from being crawled by zeminerbot, just use the zeminerbot User Agent string. For exemple:
User-agent: zeminerbot Disallow: /path/
Research any web site, 100% risk-free. Your privacy is protected at ZeMiner.com.