Python: improving time required to load 1.5G CSV file

In previous post we discussed how we can search for specific item in 1.5G CSV file with 132M records. And we improved search from 73625.309 ms (more than a minute) to just ~0.005 ms - almost 15 million times faster. Which is pretty impressive improvement as per my understanding.

But there is still one bottle neck that can be improved - time required for first initial scan of the file. Let's try to improve this in this article.

All source files can be found in JFF-Bohdan/item_lookup

Fast autocomplete using Python and Redis

Fast autocomplete using Python and Redis

General information

Problem

Imagine situation when you need provide autocomplete for some data list. In case when you using SQL probably you will do something like this:

select * from table_name where table_field like 'search%'

Probably you know, that when you database contains significant amount of data you will get essential decreasing of data processing.

pynvr - Python NVR (Network Video Recorder) based on OpenCV.

pynvr - Python NVR (Network Video Recorder) based on OpenCV.

Source can be found here: https://github.com/JFF-Bohdan/pynvr

General information

pynvr can interract with cameras using OpenCV (supports many types of connection including connection to USB cameras and network cameras using RTSP, for example)

Captured video stream must be analysed for motion and then saved to files according to specific rules.

When motion detected (or rule triggered) not only video after start must be saved but some frames before (pre-alarm/pre-event frames) must be saved to output file.

Qt 5.1 zlib compression compatible with Python zlib.decompress. Uploading files to the server.

Qt 5.1 zlib compression compatible with Python zlib.decompress. Uploading files to the server.

Task

I want transfer file from Qt 5.1 application to the server. File must be packed before transmission (in Qt application), then must be unpacked in server's (in Python script). We will use zlib compression in both sides.