Штатные серверы Python и немного о python-proxy-rotator

Здесь я начал было изучать файлы Python Proxy Rotator, потом нашел ссылки на документацию Python. При беглом знакомстве с объектами BaseHTTPServer, CGIHTTPServer, SimpleHTTPServer, SocketServer... завяз, но нашел видео "Create a simple custom HTTPServer with Python" и решил, что все это надо записать для будущего изучения.

20.18. BaseHTTPServer — Basic HTTP serve
Module CGIHTTPServer
Module SimpleHTTPServer
Video ...Create a simple custom HTTPServer with Python
20.17. SocketServer — A framework for network servers
weakref.proxy

In [5]:

%load "/media/SL-63-X86_6/w8/GitHub/python-proxy-rotator/README.md"

In []:

#Python Proxy Rotator
####An automatic python proxy rotator. (Supports SSL proxies)

##Currently known bugs
* Has currently a static set of proxies and lacks an efficient way to check if the remote proxy server is alive.
* You may experience various "server errors" due to unstable proxy servers

##Introduction
This is a multithreaded, man-in-the-middle proxy rotator acting as a relay for HTTP-requests. The goal is to make it harder to track you as a web user.

#####Not only is it unnecessarily overkill, but it is also really cool. 

A single webpage includes mulitple files: css, images, scripts and so on. This server creates a new request for each and everyone of those objects through a new proxy, making it appear as your request acctually originates for mulitple clients ("omni browsing"). By using non-transparent SSL proxies, this will be a tough nut to crack if someone decided to listen to you outgoing data traffic.  

#####Planned features: 
* SOCKS4 and SOCKS5 support
* SMTP support

####Current features: 
* FTP support
* HTTP & HTTPS support
* Proxy chaining (technically limited to only HTTPS proxies)
* Accepts globally trusted certificates
* Correctly handles SSL
* Ability to act as a SSL Certificate Authority (for the locally recieved requests)
* Lets the browser/user decide whether to use HTTPS or not. This is a necessary feature due to the fact that not all servers support SSL encryption. And there is no reason to encrypt the data yourself, if your HTTPS proxy decrypts the data before it reaches the goal.

##It's not magic
This example describes the life of a HTTP GET/POST request.

![How it works graphics](https://github.com/jorgenkg/python-proxy-rotator/blob/master/magic.png?raw=true)

##Requirements
* `Python with SSL compiled socket.`
* `pyOpenSSL` High-level wrapper around a subset of the OpenSSL library

`$ pip install pyOpenSSL`
 
## Usage
Change directory to the download destination and run:  
```bash
$ python badger.py [-n repetitions]

OPTIONS
-p, --local_proxies    # default operation
-f, --force_proxy_refresh  # force the program to reverify the proxies 
-n CHAINLENGTH, --chainlength CHAINLENGTH # number of servers to bounce through
```

Set `127.0.0.1:8080` as the proxy in your favourite browser.

##Kudos
This project is inspired by [pymiproxy](https://github.com/allfro/pymiproxy).  
This project contains the `CertificateAuthority()` defined in `pymiproxy`

In []:

In [2]:

%load "/media/SL-63-X86_6/w8/GitHub/python-proxy-rotator/badger.py"

In []:

import argparse, sys, threading, cmd
from pipe_server.server import ThreadedPipeServer

parser = argparse.ArgumentParser()
parser.add_argument('-p', '--local_proxies', action='store_true')
parser.add_argument('-f', '--force_proxy_refresh', action='store_true')
parser.add_argument('-n', '--chainlength')
parser.add_argument('-D', '--debug', action='store_true')
args = parser.parse_args()

proxy = ThreadedPipeServer(
            try_local_proxylist = False if args.force_proxy_refresh and not args.local_proxies else True,
            chainlength = args.chainlength,
            DEBUG = args.debug
        )
try:
    print "Server is running."
    proxy.serve_forever()
    print 'visible?'
except KeyboardInterrupt:
    sys.stdout.write("\nServer is shuting down. Please wait...")
    proxy.terminate()
    sys.stdout.write(" Done!\n")
    sys.stdout.flush()

In [4]:

%load "/media/SL-63-X86_6/w8/GitHub/python-proxy-rotator/pipe_server/server.py"

In []:

#!/usr/bin/env python
from BaseHTTPServer import HTTPServer, BaseHTTPRequestHandler
from SocketServer import ThreadingMixIn
from handler import ProxiedRequestHandler
from proxy.proxier import ProxyManager
from ca_generator import CertificateAuthority

class PipeServer(HTTPServer):
    def __init__(self, server_address=('', 8080), try_local_proxylist=True, chainlength=0, DEBUG=False):
        HTTPServer.__init__(self, 
                        server_address, 
                        ProxiedRequestHandler,
                    )
        self.ca = CertificateAuthority()
        self.proxy_fetcher = ProxyManager(try_local_proxylist)
        self.CHAIN = chainlength
        self.DEBUG = DEBUG


class ThreadedPipeServer(ThreadingMixIn, PipeServer):
    def stop_proxy(self):
        self.proxy_fetcher.terminate()
    
    def terminate(self):
        self.stop_proxy() # stop loading proxies
        self.socket.close()
    
    def setchainlength(self,length):
        self.CHAIN = length

In [7]:

%load "/media/SL-63-X86_6/w8/GitHub/python-proxy-rotator/pipe_server/handler.py"

In []:

from BaseHTTPServer import BaseHTTPRequestHandler
from urlparse import urlparse, urlunparse, ParseResult
from httplib import HTTPResponse
import os, random
from ssl import wrap_socket, PROTOCOL_TLSv1
from socket import socket, timeout as TimeoutException
from proxy.badgersocket import ProxySocket

class ProxiedRequestHandler(BaseHTTPRequestHandler):

    def __init__(self, request, client_address, server):
        self.connect_through_ssl = False
        self._proxy_fetcher = server.proxy_fetcher
        self.CHAIN = server.CHAIN
        self.DEBUG = server.DEBUG
        try: BaseHTTPRequestHandler.__init__(self, request, client_address, server)
        except Exception as e: 
            if server.DEBUG: print e
    
    def _connect_to_host(self):
        if self.connect_through_ssl:
            self.hostname, self.port = self.path.split(':')
        else:
            u = urlparse(self.path)
            if u.scheme != 'http': print 'ERROR Unknown request scheme: %s' % repr(u.scheme)
            self.hostname = u.hostname
            self.port = u.port or 80
            self.path = urlunparse( 
                                ParseResult(
                                    scheme='http',
                                    netloc='%s' % u.hostname,
                                    params=u.params,
                                    path=u.path or '/',
                                    query=u.query,
                                    fragment=u.fragment
                                )
                            )
        # Create a pipe to the remote server
        self._pipe_socket = ProxySocket( self._proxy_fetcher, chainlength=self.CHAIN, use_ssl=self.connect_through_ssl, DEBUG=self.DEBUG )
        self._pipe_socket.connect( (self.hostname, int(self.port)) )

        # Wrap socket if SSL is required
        if self.connect_through_ssl:
           self._pipe_socket = wrap_socket(self._pipe_socket)

    def do_CONNECT(self):
        self.connect_through_ssl = True
        try:
            self._connect_to_host()
            # http://curl.haxx.se/ca/cacert.pem
            self.request = wrap_socket(
                                        self.request, 
                                        server_side = True,
                                        certfile = self.server.ca[self.path.split(':')[0]],
                                        ssl_version=PROTOCOL_TLSv1,
                                        ca_certs=os.path.join(os.getcwd(),'cacert.pem')
                                    )
        except Exception, e:
            self.send_error(500, str(e))
            return
        
        # Reload!
        self.setup()
        self.ssl_host = 'https://%s' % self.path
        self.handle_one_request()
    
    def do_RELAY(self, retried=False):
        if not self.connect_through_ssl:
            try:
                self._connect_to_host()
            except TimeoutException, e:
                if retried: self.send_error(504, "Gateway Timeout")
                else: self.do_RELAY(True)
                return
            except Exception, e:
                self.send_error(500, str(e))
                return


        request = '%s %s %s\r\n' % (self.command, self.path, self.request_version)
        request += '%s\r\n' % self.headers
        if 'Content-Length' in self.headers: request += self.rfile.read(int(self.headers['Content-Length']))
        self._pipe_socket.sendall(request)

        http_response = HTTPResponse(self._pipe_socket)
        http_response.begin()
        response = '%s %s %s\r\n' % (self.request_version, http_response.status, http_response.reason)
        response += '%s\r\n' % http_response.msg
        response += http_response.read()
        
        
        http_response.close()
        self._pipe_socket.close()

        try:
            self.request.sendall(response)
        except Exception, e:
            print 'ERROR request relay: ',e

    def do_GET(self):
        self.do_RELAY()
    
    def do_POST(self):
        self.do_RELAY()
    
    def finish(self):
        try: self._pipe_socket.close()
        except Exception: pass
        BaseHTTPRequestHandler.finish(self)

In []:

Посты чуть ниже также могут вас заинтересовать