Row major and Column major Explained, using Python against Matlab

In both cases, a 2d array of 2 rows, 4 columns is created.
Then, it is reshaped to 1 row, 8 columns.
This clearly demonstrates how Matlab stores the array data column-wise, and Python stores the array data row-wise after it is flattened out to 1d.

Matlab (Column Major)

Screen Shot 2017-06-04 at 10.51.20 AM

>> a = = [1 2 3 4 ; 5 6 7 8]

a = 
 1 2 3 4
 5 6 7 8

a = a(:)

a =

1 
5 
2 
6 
3 
7 
4 
8

Python (Row-major)

Screen Shot 2017-06-04 at 10.51.13 AM

>> import numpy as np
>> a = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
>> a
Out[]: 
array([[1, 2, 3, 4], 
        5, 6, 7, 8]])

>> a = a.tostring() # shows the memory arrangement, in a string

Out[] : b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00'

If you look at the string printed, it shows that the elements are arranged in the fashion 1, 2, 3, 4, 5, 6, 7, 8.

Why is this important to know, you ask?

For optimization, it is important. In Python, C/C++, the elements will be laid out contiguously row-wise in memory. So you should strive to process the elements row-wise. The entire row could be copied to cache and worked on from there by the OS. If you processed the elements column wise, contrary to how it is laid out in memory, you would incur expensive context switching as the OS copies in the elements you require into cache for processing for every column-wise element you process!

 

Conquer CSV files in Python

copywriting-guide-cartoon-comma

CSV files are common for data manipulation in Python, in cases where you extract the data from an excel sheet. Here is a short tutorial on how to extract some data out of a csv file, along with other nifty tricks along the way.

1) Easy Binary Conversion

You can use this to convert to binary.

your_binary_string = "0100000010001"
int_value = int(your_binary_string, 2)

Of course, you can extend this to octal, etc.

2) File reading and list comprehension

Suppose you have a whole csv of binary numbers. You need to read it out to python as a list.
Read the csv as string, and convert it to int easily.

Your csv data looks like this :

0,0,1,1111100110101010,0000111111101111,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
1,1,0,1111000110000000,0001000100000001,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
2,2,0,1111001010001000,0000111001100011,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
3,3,0,1111000010011000,0000111011100101,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
4,4,0,1111000000010011,0000110010011111,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
5,5,0,1110111100101100,0000110010000001,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
6,6,0,1110111001000010,0000101010111110,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
7,7,0,1110110111010000,0000100111110110,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
8,8,0,1110110011101111,0000100010011001,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
9,9,0,1110110010101011,0000011101010010,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
....

and you just want to get the values in bold. Meaning putting each of the the fourth and third column values in a tuple, converted to integer value.

import matplotlib.pyplot as plt
import numpy as np
import csv

filename = 'yourfile.csv'

with open(filename, 'rt') as csvfile:
 reader = csv.reader(csvfile)
 # the following line uses list comprehension to iterate through every row in the csv
 # and creates row number of tuples in a list
 # You can call next(reader) to skip a row, for example if the 1st row is just labels
 iq = [(int(row[4], 2), int(row[3], 2)) for row in reader]

You get something like this:

[(4079, 63914), (4353, 61824), (3683, 62088), (3813, 61592) .... ]

At this point, if for some reason you want to get all the 1st values in each tuple in one list, and the 2nd values in each tuple in another list, you can do:

i, q = zip(*iq)
i = list(i)
q = list(q)

You will get for i :

[4079, 4353, 3683, 3813, 3231, 3201, 2750, 2550, 2201, 1874, 1559, .... ]

and q:

[63914, 61824, 62088, 61592, 61459, 61228, 60994, 60880, 60655, 60587... ]

3) Writing CSV files

With the previous i and q lists you extracted, you can now write it out as a csv.

with open('complex_out.csv', 'w') as csvfile:
 fieldnames = ['i', 'q']
 writer = csv.DictWriter(csvfile, fieldnames = fieldnames)
 writer.writeheader()
 for row_num in range(len(i)):
 writer.writerow({'i':i[row_num], 'q':q[row_num]})

Your csv file looks like this :

i,q
4079,63914
4353,61824
3683,62088
3813,61592
3231,61459
3201,61228
2750,60994
2550,60880
2201,60655
1874,60587
1559,60434
1213,60367
898,60284
529,60238
195,60224
65371,60196
65033,60237
64701,60281
64372,60335
64024,60430
63705,60535
63385,60660
63065,60826
62763,61000
62474,61192
62198,61410
61935,61644
61680,61883
61468,62145
61232,62433
61040,62710
60863,63013
60702,63328
60566,63648
60459,63979
60361,64304
60292,64646
60247,64994
.....

3) Converting data types

From the above data, I know that my number is really a 16bit binary representation.
Now, I am told that this is a 2’s complement representation.
So each tuple should be really (int16, int16), with +ve and -ve values possible. Fortunately Python allows us to do this easily.

iq = np.array(iq, np.int16) # creates an array with iq existing values, converted to int16
array([[ 4079, -1622],
 [ 4353, -3712],
 [ 3683, -3448],
 ..., 
 [ -567, 5301],
 [ -216, 5329],
 [ 146, 5332]], dtype=int16)

Now we want to convert it to complex64, for further processing down the line.

sig = iq.astype(np.float32).view(np.complex64) # convert the values to float32, before viewing it as a complex64 (2 float32s)
array([[ 4079.-1622.j],
 [ 4353.-3712.j],
 [ 3683.-3448.j],
 ..., 
 [ -567.+5301.j],
 [ -216.+5329.j],
 [ 146.+5332.j]], dtype=complex64)
sig = sig.ravel() # flatten it to 1D
array([ 4079.-1622.j, 4353.-3712.j, 3683.-3448.j, ..., -567.+5301.j,
 -216.+5329.j, 146.+5332.j], dtype=complex64)

Hooray, now we’re ready to do further processing on this data!

Python: Using ioloop To Send to 2 Protocols Without Threading

ioloop is very useful for pre-scheduling functions to be executed in the future. In this manner, you can simulate interspersing sending to 2 protocols without threading. (in this case, “serial” and “tcp” at 100Hz and 1000Hz respectively.)

In this simple code, io_loop.add_timeout keeps pre-scheduling events to happen in the future. Do note that this all runs in one thread, so make sure your sending functions printTCP() and printSerial() are never blocking. I use this for simple use cases where I am just reading from a file and sending to two different sources. Also, in my code I use a lot of zmq, and only required the simple use cases of zmq.eventloop. For more advanced use cases, you may want to take a look at tornado.ioloop. http://www.tornadoweb.org/en/stable/ioloop.html (on which pyzmq’s event loop is based on)

import functools
import datetime
from zmq.eventloop import ioloop


def printTCP():
 print('Sent TCP')


def printSerial():
 print('Sent Serial')


io_loop = ioloop.IOLoop.instance()

# Send serial messages at 100Hz
serial_deadline = datetime.timedelta(seconds=0.1)
for x in range(1000):
 io_loop.add_timeout(serial_deadline, functools.partial(printSerial))
 serial_deadline += datetime.timedelta(seconds=0.1)

# Send tcp messages at 1000Hz
tcp_deadline = datetime.timedelta(seconds=0.01) # separate deadline for sending ins pulses
for x in range(10000):
 io_loop.add_timeout(tcp_deadline, functools.partial(printTCP))
 tcp_deadline += datetime.timedelta(seconds=0.01)

io_loop.start()

You will get something like this:

 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 ....

Quickly Plot in Python with Text File Values

Python has this wonderful command in numpy, np.loadtxt.
In this case, I use np.loadtxt to load a file consisting of several ADC real values to do a quick plot for visualization. The rule is that each row in the text file has to have the same number of elements. In this case it is simply.

2304
4233
3252
and so on...

 

import matplotlib.pyplot as plt
import numpy as np

data = np.loadtxt('/path/to/your/file')

# get x-axis bins
x_axis = np.arange(data.size) * 960 / data.size

#plot time domain
plt.plot(data)

# plot frequency domain
plt.figure()
spc = np.fft.fft(data)
plt.plot(x_axis, 20*np.log10(abs(spc)))

That’s all. Pretty quick and nifty!

Ten Reasons to Learn Python

python_640x400

    1. It’s Matlab!

      Users of Matlab will find the Python console familiar. For Matlab-esque applications, numpy and scipy are easily available for Python. matplotlib covers the graphing stuff that you need.

      Lazy to install all of this stuff? Just download WinPython or Anaconda – these are all-in-one installers designed to work out of the box for your scientific needs. To me, Python represents the glue between the academic and the developer world. Python is likely to chip away at Matlab slowly but surely for one reason – newly graduated computer science students are more likely to already know Python. It is one of the favourite languages used to teach.

    2. It’s a Scripting Language!

      In the course of my work, I frequently have to quickly script up simulators or quick interface testers to test production code. This is where Python comes into play. The sheer convenience of having everything and anything you could ever possibly need for a tester makes Python an easy choice. You can collect the data via the tester and quickly display a graph of the data via matplotlib. If the work is I/O bound rather than compute bound, Python does the job well. This is a pretty good article on what I think is going on.

      However, if it is compute bound and you really need the speediest, look to C/C++. Or you may try one of the alternative Python interpreters, Pypy. The main aim of Pypy is to improve upon the default Python interpreter, CPython.

    3. It’s a Application Programming Language!

      You can make games with PyGame. With PyQt, you can basically write a cross-platform GUI application. Another GUI app development framework which gets a lot of love is wxPython. BitTorrent was developed using wxPython. TkInter is another windowing framework. Another app that was created using Python is Calibre, a capable e-book reader.

    4. It’s a Web Development Language!

      Of course you’ve heard of Django. It’s one of the most popular and powerful web application frameworks. Here are some web Python frameworks, compared.

      Dropbox, YouTube, Google, Quora, Reddit, Pinterest, Spotify – they all use Python. Need I say more?

    5. It’s a Calculator!

      Python has basically replaced the calculator on my desktop for any advanced calculations. All I need to do is to start the terminal and :

      >python
      >>> import numpy as np
      >>> np.sqrt(6)
      2.4494897427831779

      Ok, not so advanced, but you get the idea. Cosine, sine, arctangent, matrices, they’re all there. And it’s right there in the terminal.

    6.  Batteries included!

      For ZeroMQ messaging protocol – You have PyZMQ.
      For GPU programming (CUDA-based), you have Continuum Analytics’s Accelerate GPU solution and PyCUDA here.
      For QT-based GUIs, you can use PyQT and generate GUIs with a few dozen lines of code.

      Anything you can imagine, it probably has a python binding at the very least. Google’s newly-open-sourced TensorFlow for machine learning has Python front and center. There’s a python framework for pretty much anything.

      For programmers, the most important thing is to get shit done, so anytime you can leverage on 3rd party libraries, you’re probably going to want to do it.

      Oh, and it comes with any respectable desktop linux distribution, so you get it out-of-the-box.

    7. Syntactic Sugar !

      Indentation in Python essentially is the curly bracket equivalent in C/C++. This results in code that is generally easier to read and free of the individual programmer’s syntactic quirks. This also results in generally neater code. Other language features like dynamic typing helps a lot in reducing unnecessary syntax. The less code you write, the less probability of error. In fact, Python reminds me of my own pseudo code.
      Being easier to learn, Python reduces the time to market.

      Type :

      >>> import this

      for the Zen of Python – it kind of illustrates what Python is all about.

    8. Python is free, and many of its IDEs are free!

      PyCharm Community is free. This is from one of my favourite IDE developers, JetBrains. Another one is Spyder, which tries to mimic the Matlab programming environment. Spyder comes installed with the WinPython and Anaconda packages. Many of these are cross-platform, and come with modern useful code-completion features.

      Python is kept alive by the following companies. The Python source code is all open-source.  So it probably isn’t going away anytime soon. Python actually is around 26 years old, so yeah, that’s an eternity in computer software.

    9. Show me the money!

      Based on Quartz, here are programming languages listed next to their average annual salary from lowest to highest. Number 3 ain’t bad, considering Objective C is proprietary to Apple. Anyway a lot of Swift seems to be inspired by Python.

      main-qimg-ad55198f335d14ca73d191d08d071651

    10. It’s Everywhere!

      It truly is, and it will get even more pervasive as more students graduate with Python in their pocket.

 

 

 

PyCUDA Windows Installation (Offline)

PyCUDA
PyCUDA is a Python extension for CUDA which is useful for prototyping GPU solutions with Python. Here are some things it is being used for. http://wiki.tiker.net/PyCuda/ShowCase It isn’t officially supported by NVIDIA though.
Because my environment is offline, many of the things I mention here may be extraneous.

First, install CUDA Toolkit for windows – I assume that you have done so or you probably wouldn’t be here.

There is the official installation page for Windows here : http://wiki.tiker.net/PyCuda/Installation/Windows , which you really should follow if you have internet access on your installation computer. If you follow that and you still have trouble, come back here to see if the rest of this helps.

You need Microsoft Visual Studio 2010 for this installation. Or at least, the correct cl.exe compiler and any other dependencies it may need. I have not tested with later versions. If you have a 64-bit system and Python installation, make sure that you are using the 64-bit compiler and not the 32-bit one. The 64-bit cl.exe is located here :

c:\Program Files (x86)\Microsoft Visual Studio 10.0\VC\bin\amd64\

You can run cl.exe to check whether you are using the 32 or 64 bit version.

Offline Installation

In the case that you are as unfortunate as me to have to install this stuff offline, here are the various files that I used to install pycuda. https://onedrive.live.com/redir?resid=692F268A60881F2D!15820&authkey=!AJpd-O9yUi_O_1w&ithint=file%2czip

In the case you need any other additional wheel files, download them here: http://www.lfd.uci.edu/~gohlke/pythonlibs/

Do note that you should download the correct wheel file for your python distribution.

For example,

boost_python-1.59-cp27-none-win_amd64.whl

cp27 means Python 2.7. For my installation I downloaded those with cp34 as I had a Python 3.4 installation.

You can use pip, located under ‘scripts’ in the Python directory to install whl (known as wheel) files.

pip install pathandfilenamewhatever.whl

Boost Libraries Installation

After the various installations, I was not able to find the boost thread library when I tried to compile PyCUDA. Hence, I had to compile the boost libraries manually.

Refer to section 5.1 here http://www.boost.org/doc/libs/1_55_0/more/getting_started/windows.html

for the easiest method to compile the boost libraries.

After the boost libraries were compiled, I went to the ‘boost/lib’ directory and manually copied out the files I wanted to the python installation\libs directory :

  • libboost-thread-vc100-mt-1_59
  • libboost_thread-vc100-mt-gd-1_59

PyCUDA Compilation

Now, we can start compiling PyCUDA.

Unzip the PyCUDA source files, and at the command prompt inside the PyCUDA directory, type this:
configure

This will create a siteconf.py at your directory which houses the information required by PyCUDA for installation.

Here’s how my siteconf.py looks like:

BOOST_INC_DIR = ['']
BOOST_LIB_DIR = ['']
BOOST_COMPILER = 'msvc'
USE_SHIPPED_BOOST = True
BOOST_PYTHON_LIBNAME = ['libboost_python3-vc100-mt-1_59']
BOOST_THREAD_LIBNAME = ['libboost_thread-vc100-mt-1_59']
CUDA_TRACE = False
CUDA_ROOT = 'C:\\Program Files\\NVIDIA GPU Computing Toolkit\\CUDA\\v7.5'
CUDA_ENABLE_GL = False
CUDA_ENABLE_CURAND = True
CUDADRV_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib/x64']
CUDADRV_LIBNAME = ['cuda']
CUDART_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib/x64']
CUDART_LIBNAME = ['cudart']
CURAND_LIB_DIR = ['${CUDA_ROOT}/lib', '${CUDA_ROOT}/lib/x64']
CURAND_LIBNAME = ['curand']
CXXFLAGS = ['/EHsc']
LDFLAGS = ['/FORCE']

Set USE_SHIPPED_BOOST to true so that it will look for the boost libraries inside the python directory. I had to set the CXXFLAGS and LDFLAGS as such so that the code would compile. Sounds not so good, I know, but what the heck, it compiles.
Then type,
python setup.py install

It should compile at this stage. Hooray!!!

After the installation:

Edit nvcc.profile (in CUDA\v7.5\bin) (or whereever you installed the Cuda GPU Computing toolkit) and set INCLUDES flag to this:
INCLUDES += "-I$(TOP)/include" "-I$(TOP)/include/cudart" "-IC:/Program Files (x86)/Microsoft Visual Studio 10.0/VC/include" $(_SPACE_)

 

And that is my personal, painful story of PyCUDA Windows Installation.

This page helped me a lot! Thanks Marty! I didn’t need to install the Windows SDK for this installation though.