The Crawler

Ethical data retrieval is our priority.


zeminerbot is's web crawling robot. It fetches millions of pages per day looking for fresh data so you don't have to run expensive and time consuming data retrieval jobs.

Ethical data retrieval is a top priority at Therefore, when you purchase our datasets, you can rest assured the websites from which the data was gathered were not crawled in a

zeminerbot Obeys Robots.txt

Our crawler honors the robots.txt standard. To exclude any path on your web server from being crawled by zeminerbot, just use the zeminerbot User Agent string. For exemple:

User-agent: zeminerbot
Disallow: /path/

Research any web site, 100% risk-free. Your privacy is protected at

Million Domain Names
Indicators Analyzed Per URL
Analytical Algorithms Employed