How to make a request in Python 3 through an HTTP proxy

2

In Python, we can retrieve the content of a URL quite simply:

from urllib import request
response = request.urlopen('http://www.google.com.br')

However, in corporate environments we can not access the internet directly, we need to go through a proxy. How to do this?

    
asked by anonymous 05.02.2014 / 21:03

2 answers

2

Solution in Python 3 using urllib

You can create a request using a opener which, in turn, receives the information of a Proxyhandler :

from urllib import request

#configura um "opener" com um proxy
proxy  = request.ProxyHandler({"http":"proxy:8080"})
auth = request.HTTPBasicAuthHandler()
opener = request.build_opener(proxy, auth, request.HTTPHandler)

#faz uma requisição usando o "opener"
response = opener.open('http://www.google.com.br')
print(response.read().decode('ISO-8859-1'))

You can still set a global proxy setting as follows:

from urllib import request

#configura um "opener" com um proxy
proxy  = request.ProxyHandler({"http":"proxy:8080"})
auth = request.HTTPBasicAuthHandler()
opener = request.build_opener(proxy, auth, request.HTTPHandler)

#instala o "opener" globalmente
request.install_opener(opener)

#faz uma requisição genérica (sem o opener)
response = request.urlopen('http://www.google.com.br')
print(response.read().decode('ISO-8859-1'))

Solution in Python 2 using urllib2

It changes a bit in relation to Python 3, remembering that urllib2 2 is equivalent to urllib of version 3, although functions and classes have changed location and name:

import urllib2

#proxy
proxy = urllib2.ProxyHandler({"http":"proxy:8080"})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

response = urllib2.urlopen("http://www.google.com")
print response.read()

Solution using the library requests

To add the proxy to a simple request:

import requests

response = requests.get("http://www.google.com",
                        proxies = { "http": "proxy:8080" })
print(response.text)

To add the proxy to multiple requests:

import requests

sessao = requests.Session()
sessao.proxies = {"http": "proxy:8080"}

response = sessao.get("http://www.google.com")
print(response.text)

Note: Remembering that this is a separate installed library.

General note: The examples are for an HTTP proxy that does not require a username and password.

    
05.02.2014 / 21:03
0

Simply define the proxy that you want to use through an environment variable. In Windows proxy settings are obtained directly from the internet explorer configuration.

But it also depends on the proxy type. If it's a MS Proxy server-like thing to use NTLM for authorization, then there's going to have to use anything like this:

link

    
08.02.2014 / 03:31