Row major and Column major Explained, using Python against Matlab

In both cases, a 2d array of 2 rows, 4 columns is created.
Then, it is reshaped to 1 row, 8 columns.
This clearly demonstrates how Matlab stores the array data column-wise, and Python stores the array data row-wise after it is flattened out to 1d.

Matlab (Column Major)

Screen Shot 2017-06-04 at 10.51.20 AM

>> a = = [1 2 3 4 ; 5 6 7 8]

a = 
 1 2 3 4
 5 6 7 8

a = a(:)

a =

1 
5 
2 
6 
3 
7 
4 
8

Python (Row-major)

Screen Shot 2017-06-04 at 10.51.13 AM

>> import numpy as np
>> a = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
>> a
Out[]: 
array([[1, 2, 3, 4], 
        5, 6, 7, 8]])

>> a = a.tostring() # shows the memory arrangement, in a string

Out[] : b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00'

If you look at the string printed, it shows that the elements are arranged in the fashion 1, 2, 3, 4, 5, 6, 7, 8.

Why is this important to know, you ask?

For optimization, it is important. In Python, C/C++, the elements will be laid out contiguously row-wise in memory. So you should strive to process the elements row-wise. The entire row could be copied to cache and worked on from there by the OS. If you processed the elements column wise, contrary to how it is laid out in memory, you would incur expensive context switching as the OS copies in the elements you require into cache for processing for every column-wise element you process!

 

Conquer CSV files in Python

copywriting-guide-cartoon-comma

CSV files are common for data manipulation in Python, in cases where you extract the data from an excel sheet. Here is a short tutorial on how to extract some data out of a csv file, along with other nifty tricks along the way.

1) Easy Binary Conversion

You can use this to convert to binary.

your_binary_string = "0100000010001"
int_value = int(your_binary_string, 2)

Of course, you can extend this to octal, etc.

2) File reading and list comprehension

Suppose you have a whole csv of binary numbers. You need to read it out to python as a list.
Read the csv as string, and convert it to int easily.

Your csv data looks like this :

0,0,1,1111100110101010,0000111111101111,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
1,1,0,1111000110000000,0001000100000001,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
2,2,0,1111001010001000,0000111001100011,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
3,3,0,1111000010011000,0000111011100101,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
4,4,0,1111000000010011,0000110010011111,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
5,5,0,1110111100101100,0000110010000001,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
6,6,0,1110111001000010,0000101010111110,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
7,7,0,1110110111010000,0000100111110110,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
8,8,0,1110110011101111,0000100010011001,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
9,9,0,1110110010101011,0000011101010010,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
....

and you just want to get the values in bold. Meaning putting each of the the fourth and third column values in a tuple, converted to integer value.

import matplotlib.pyplot as plt
import numpy as np
import csv

filename = 'yourfile.csv'

with open(filename, 'rt') as csvfile:
 reader = csv.reader(csvfile)
 # the following line uses list comprehension to iterate through every row in the csv
 # and creates row number of tuples in a list
 # You can call next(reader) to skip a row, for example if the 1st row is just labels
 iq = [(int(row[4], 2), int(row[3], 2)) for row in reader]

You get something like this:

[(4079, 63914), (4353, 61824), (3683, 62088), (3813, 61592) .... ]

At this point, if for some reason you want to get all the 1st values in each tuple in one list, and the 2nd values in each tuple in another list, you can do:

i, q = zip(*iq)
i = list(i)
q = list(q)

You will get for i :

[4079, 4353, 3683, 3813, 3231, 3201, 2750, 2550, 2201, 1874, 1559, .... ]

and q:

[63914, 61824, 62088, 61592, 61459, 61228, 60994, 60880, 60655, 60587... ]

3) Writing CSV files

With the previous i and q lists you extracted, you can now write it out as a csv.

with open('complex_out.csv', 'w') as csvfile:
 fieldnames = ['i', 'q']
 writer = csv.DictWriter(csvfile, fieldnames = fieldnames)
 writer.writeheader()
 for row_num in range(len(i)):
 writer.writerow({'i':i[row_num], 'q':q[row_num]})

Your csv file looks like this :

i,q
4079,63914
4353,61824
3683,62088
3813,61592
3231,61459
3201,61228
2750,60994
2550,60880
2201,60655
1874,60587
1559,60434
1213,60367
898,60284
529,60238
195,60224
65371,60196
65033,60237
64701,60281
64372,60335
64024,60430
63705,60535
63385,60660
63065,60826
62763,61000
62474,61192
62198,61410
61935,61644
61680,61883
61468,62145
61232,62433
61040,62710
60863,63013
60702,63328
60566,63648
60459,63979
60361,64304
60292,64646
60247,64994
.....

3) Converting data types

From the above data, I know that my number is really a 16bit binary representation.
Now, I am told that this is a 2’s complement representation.
So each tuple should be really (int16, int16), with +ve and -ve values possible. Fortunately Python allows us to do this easily.

iq = np.array(iq, np.int16) # creates an array with iq existing values, converted to int16
array([[ 4079, -1622],
 [ 4353, -3712],
 [ 3683, -3448],
 ..., 
 [ -567, 5301],
 [ -216, 5329],
 [ 146, 5332]], dtype=int16)

Now we want to convert it to complex64, for further processing down the line.

sig = iq.astype(np.float32).view(np.complex64) # convert the values to float32, before viewing it as a complex64 (2 float32s)
array([[ 4079.-1622.j],
 [ 4353.-3712.j],
 [ 3683.-3448.j],
 ..., 
 [ -567.+5301.j],
 [ -216.+5329.j],
 [ 146.+5332.j]], dtype=complex64)
sig = sig.ravel() # flatten it to 1D
array([ 4079.-1622.j, 4353.-3712.j, 3683.-3448.j, ..., -567.+5301.j,
 -216.+5329.j, 146.+5332.j], dtype=complex64)

Hooray, now we’re ready to do further processing on this data!

Sign Bit Extension

Sometimes you can get a value like this :

For example, something that is 12-bit in a 16-bit short int.

shortIntValue = 0b0000 1100 1110 1100

The latter 12 bits are the actual value. It is a negative value by the 12th bit being 1. And you really want :

0b1111 1100 1110 1100

You can do

shortIntValue <<= 4;

followed by

shortIntValue >>= 4;

Volia! Sign bit extension! This is useful for 2’s complement values which are put into a primitive integer size. (As above)

iOS Launch Images Madness

I was struggling with the iOS launch images. The easiest way I found was to not use image assets, or the launch image file. Just put these images under Resources in your project in XCode and iOS will find them.

  • Default.png (iPhone)
  • Default@2x.png (iPhone Retina 3.5 inch)
  • Default@3x.png (iPhone 6 Plus landscape)
  • Default-568h@2x.png (iPhone Retina 4 inch) 
  • Default-667h@2x.png (iPhone 6 portrait)
  • Default-736h@3x.png (iPhone 6 Plus portrait)
  • Default-Portrait.png (iPad in portrait orientation)
  • Default-Portrait@2x.png (iPad Retina in portrait orientation)
  • Default-Portrait@3x.png (iPhone 6 Plus portrait)
  • Default-Landscape.png (iPad in landscape orientation)
  • Default-Landscape@2x.png (iPad Retina in landscape orientation)

Oh my god, such a trivial thing.. why did Apple make it so difficult??

Thoughts on Open Source

opensource-svg

I used to not understand open source.

Why would people give away their source code for free? Going through the software revolution of the 90s, Microsoft was the king of software. I followed the Microsoft’s decimation of WordPerfect, Borland, Lotus 1-2-3 and Netscape as a kid. And Microsoft didn’t give away software for free, no, let alone source code! Microsoft charged folks for Office, Windows, Visual Studio, etc. To their credit, they still do to a certain extent. So to me, it all seemed counter-intuitive – something that was free, in my mind, would not see any quick progress due to the lack of economic motivation behind it. My overall train of thought could be summarised by this grumpy nerd’s post.

Then Linux Torvalds came along with this :

Hello everybody out there using minix –

I’m doing a (free) operating system (just a hobby, won’t be big and professional like gnu) for 386(486) AT clones. This has been brewing since april, and is starting to get ready. I’d like any feedback on things people like/dislike in minix, as my OS resembles it somewhat (same physical layout of the file-system (due to practical reasons) among other things).

— Linus Torvalds

And it’s amazing how others have contributed to the project ever since. Linux is probably the most successful example of open-source software. From the Linux kernel, countless distributions have emerged. Heck, even Android has the Linux kernel in it. Red Hat, CentOS, Ubuntu, Debian, Linux Mint they all share the linux kernel, which is actively being maintained by the Linux Foundation.

There’s a lot of open source software out there, and it’s all a beautiful mess. Companies leveraging on open-source software have been successfully built. Red Hat is a prime example of this, which generates revenue through support and training contracts. http://www.zdnet.com/article/red-hat-nearing-1-billion-in-revenue-not-bad-for-free-software/ And did you know that WordPress, the blog software that you are reading this on, is open-source?

The problem with proprietary software is this : When you don’t know what’s going on, you don’t know what’s going on. Proprietary APIs usually hide the implementation details of a function, so you are at the mercy of whoever holds the proprietary software implementation. You would be in danger even. I’m sure Snowden would agree. With open source, all the implementation details are out there, whether you want to understand it or not. Furthermore, with proprietary software, once the company decides that it is no longer in their financial interests to continue with the work, it is thrown away.

I’ve concluded that to make large scale software without a whole Fortune 500 company like Microsoft or Google behind you, open source is the way to go. I use ZeroMQ in my work, and this is a great example of this. The author has successfully built a community from which many other projects have sprung.  Once folks find that the open source code is useful to them, they will have it in their hearts to contribute back. Even if it is just fixing a bug or two.

My other open source loves are CSound and  AudioKit , all great audio programming open source projects from which one can learn. AudioKit is fairly new, but has enjoyed quick growth due to many developers wanting a Swift audio development framework (pun intended).  It was the CSound code that inspired me to delve into DSP code.

Revenue for the little guy? I’d like to think of somehow who buys stuff I make as paying me to keep it going. It is not easy maintaining a piece of software amidst the ever-changing tech landscape.I would say it takes more energy to maintain software than to shit one out.

Open source is simply a developmental methodology that was ahead of its time. It’s a methodology that places community over software. It’s a reflection of the human need to contribute and make a difference. I that software, like nature, given enough time undergoes evolution. Great software stays and gets better with time.