There is no magic, the tshark was smart and read the file in chunks using pointers.
The tshark was written in C and certainly has a better performance in loops than python. The fact is that the tshark had to allocate chunks or buffers in memory to read the file piece by piece and go separating the data within the range of interest.
This line pcap = dpkt.pcap.Reader(f) tells Python to read the whole file and put everything in the pcap variable, that is, to turn to allocate 5GB of data: - (
The smart way to do this is to move the pointer to some other part of the file so you can read from the location you pointed to.
In python you can do this:
from scapy.all import *
import dpkt
f = open("capture21dez2016.pcap")
pcap = f.read(4096)
while pcap:
#processe cada pedaço aqui
pcap = f.read(4096)
f.close()
fix on line pcap = f.read(4096) we are opening the file by chunks, to be exact every 4096 bytes, f.read() uses pointer to know where exactly the last position was read always to start reading the last position file, you can set how many bytes at a time you want to read, I used 4096 for example. You can continue to use this code answer to find your interest range, convert your date and time of interest in timestamp to get easier and remember if you have already found the data within the desired range you can exit the loop and no longer need to read the rest of the file: -)
