In this post today I am explaining all about another GitHub repository for enumerating robots.txt file URLs present through a python script. The name of the script called robots.txt tester.
Robots.txt tester:
With this script, you can enumerate all URLs present in robots.txt files, and test whether you can access them or not.
Setup
Clone the repository and install the dependencies :
git clone https://github.com/p0dalirius/robotstester cd robotstester python3 setup.py install
Usage
robotstester -u http://www.example.com/
You can find here a complete list of options :
[~] Robots.txt tester, v1.2.0
usage: robotstester.py [-h] (-u URL | -f URLSFILE) [-v] [-q] [-k] [-L] [-t THREADS] [-p] [-j JSONFILE] [-x PROXY] [-b COOKIES]
This Python script can enumerate all URLs present in robots.txt files, and test whether they can be accessed or not.
optional arguments:
-h, --help show this help message and exit
-u URL, --url URL URL to the robots.txt to test e.g. https://example.com:port/path
-f URLSFILE, --urlsfile URLSFILE
List of robots.txt urls to test
-v, --verbose verbosity level (-v for verbose, -vv for debug)
-q, --quiet Show no information at all
-k, --insecure Allow insecure server connections when using SSL (default: False)
-L, --location Follow redirects (default: False)
-t THREADS, --threads THREADS
Number of threads (default: 5)
-p, --parsable Parsable output
-j JSONFILE, --jsonfile JSONFILE
Save results to specified JSON file.
-x PROXY, --proxy PROXY
Specify a proxy to use for requests (e.g., http://localhost:8080)
-b COOKIES, --cookies COOKIES
Specify cookies to use in requests. (e.g., --cookies "cookie1=blah;cookie2=blah")
