Alternatives for storing and analyzing data

3

One part of a project I'm developing is acquiring voltage and current data from a solar panel. By day I have around 40 measures like these that are here.

14/dez  16:54   18.21   0.12    18.35   0.0641
14/dez  16:54   17.82   0.27    17.98   0.2041
14/dez  16:54   17.42   0.44    17.61   0.3041
14/dez  16:54   14.23   0.39    15.30   0.3841
14/dez  16:54   5.92    0.63    7.45    0.6741
14/dez  16:54   4.97    0.54    6.50    0.6441
14/dez  16:54   4.56    0.48    5.91    0.5641
14/dez  16:54   3.51    0.68    4.59    0.7541
14/dez  16:55   2.32    0.60    2.97    0.6841
14/dez  16:55   2.19    0.51    2.76    0.6041
14/dez  16:55   2.13    0.44    2.67    0.5041
14/dez  16:55   0.71    0.67    0.90    0.7641
14/dez  16:55   0.61    0.64    0.77    0.7441
14/dez  16:55   0.00    0.69    0.00    0.821

15/dez  06:04   17.67   0.13    17.61   0.051
15/dez  06:04   15.82   0.25    15.82   0.181
15/dez  06:04   12.41   0.30    13.00   0.231
15/dez  06:04   8.09    0.22    8.72    0.161
15/dez  06:04   2.84    0.31    3.11    0.281
15/dez  06:04   2.38    0.25    2.60    0.191
15/dez  06:04   2.19    0.17    2.40    0.111
15/dez  06:04   1.69    0.36    1.79    0.301
15/dez  06:04   1.07    0.27    1.16    0.241
15/dez  06:04   1.02    0.21    1.10    0.171
15/dez  06:05   1.01    0.10    1.09    0.061
15/dez  06:05   0.33    0.32    0.35    0.311
15/dez  06:05   0.28    0.29    0.30    0.251
15/dez  06:05   0.00    0.33    0.00    0.312

I have to store this data so that in the future it is possible to draw charts by choosing the day and time for example.

What I want is to store and then retrieve the data more or less like this:

Ex grosseiro para recuperar: plot(data['14/12'].hora['07:00'].medidas)

Does anyone know if I can do this using a database? Or suggest another solution to store?

If it's in Python it would be even better.

Thank you

    
asked by anonymous 28.12.2016 / 20:51

2 answers

1

For this example I placed (stored) exactly the data that you put in a file and I went through it and remove the data, this file being tests.txt is in the same folder as the file with the code, store) in ficheiro.csv :

dados = {}
with open('tests.txt', 'r') as f:
    for line in f:
        params = [i.strip() for i in line.split('  ') if i.strip() != '']
        if not params:
            continue
        data, hora, v1, l1, v2, l2 = params
        dados[data] = dados.get(data, {})
        dados[data][hora] = dados[data].get(hora, [])
        dados[data][hora].append((v1, l1, v2, l2))

In this line [i.strip() for i in line.split(' ') if i.strip() != ''] . Note that if you can, it would be much simpler to separate the data per line by only ex: ";" which would be: [i.strip() for i in line.split(';')] instead of fickle spaces like puseste, and without empty lines could also be checked, if not params: continue .

However this code is fully functional for the data / format you put

You can access every day and / or time like this:

Day:

print(dados['15/dez'])

Output:

  

{'06:05': ('1.01', '0.33', '0.28', '0.00'), ('0.10', '0.32', '0.29', '0.33' '' 0.35 ',' 0.30 ',' 0.00 '), (' 0.061 ',' 0.311 ',' 0.251 ',' 0.312 ')], '06: 04': [ '' .17 '', '' 0.051 '), ('15 .82', '0.25', '15 .82 ',' 0.181 '), ('12 .41', '0.30', '13 .00 ',' '0.22', '8.72', '0.161'), ('2.84', '0.31', '3.11', '0.281'), ('2.38', '0.25', '2.60', ' (1.69, 0.36, 1.79, 0.301), ('1.07', '0.27', '1.16', '0.241' '), (' 1.02 ',' 0.21 ',' 1.10 ',' 0.171 ')]}

Time (here you go to the various measures of each hour):

print(dados['15/dez']['06:05'])

Output:

  ('0.10', '0.32', '0.29', '0.33'), ('1.09', '0.35', ' 0.30 ',' 0.00 '), (' 0.061 ',' 0.311 ',' 0.251 ',' 0.312 ')]

Here I will leave an alternative using defaultdict :

from collections import [defaultdict][1]

dados = defaultdict(dict)
with open('tests.txt', 'r') as f:
    for line in f:
        params = [i.strip() for i in line.split('  ') if i.strip() != '']
        if not params:
            continue
        data, hora, v1, l1, v2, l2 = params
        if hora not in dados[data]:
            dados[data][hora] = []
        dados[data][hora].append((v1, l1, v2, l2))
print(dados)
    
28.12.2016 / 21:17
0

It does, but I find it unnecessary to use a database, since you do not seem to have much experience with it.

Search for CSV, you can save your data in a saved .txt within the same project, when you need it again just reopen csv. There are several libraries in python to assist you in managing CSV files, it is the easiest way to store small data.

If you want a database you can use Firebase, it works by the cloud and it's very simple to use. For local banks I recommend SQLite.

    
28.12.2016 / 21:19