Import csv from bucket s3

Question

Import csv from bucket s3

Navigation

#1 by (0 votes)

0

I'm trying to import this one .csv file from the Amazon S3 bucket using the code below:

import csv
with open('https://s3.amazonaws.com/carto-1000x/data/yellow_tripdata_2016-01.csv', 'rb') as teste:
    reader = csv.reader(teste)
    for linha in reader:
        print linha

I'm using the PyCharm IDE with Python 2.7.13 which already comes installed on the Mac. And I get a message that the link does not exist and when I call the link in the browser it opens quietly.

Does anyone know if I need any library to be able to read this csv that is in S3?

Remembering that I ran a test with a local csv and it worked.

python

asked by anonymous 02.06.2017 / 17:16

1 answer

How to automatically scroll in the page using javascript? Send combobox and checkbox data to another table

score 0 · Answer 1

When you do open('qualquer string') python tries to 'solve' 'any string' as a local file, as you put a url, the file is not found. Here's a solution to reading the url file:

from contextlib import closing    
import csv, io, requests, urllib.request, codecs
from contextlib import closing

url = 'https://s3.amazonaws.com/carto-1000x/data/yellow_tripdata_2016-01.csv'

with closing(requests.get(url, stream=True)) as data:
    reader = csv.reader(codecs.iterdecode(data.iter_lines(), 'utf-8'))
    for row in reader:
        print (row)