Row major and Column major Explained, using Python against Matlab

In both cases, a 2d array of 2 rows, 4 columns is created.
Then, it is reshaped to 1 row, 8 columns.
This clearly demonstrates how Matlab stores the array data column-wise, and Python stores the array data row-wise after it is flattened out to 1d.

Matlab (Column Major)

Screen Shot 2017-06-04 at 10.51.20 AM

>> a = = [1 2 3 4 ; 5 6 7 8]

a = 
 1 2 3 4
 5 6 7 8

a = a(:)

a =

1 
5 
2 
6 
3 
7 
4 
8

Python (Row-major)

Screen Shot 2017-06-04 at 10.51.13 AM

>> import numpy as np
>> a = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
>> a
Out[]: 
array([[1, 2, 3, 4], 
        5, 6, 7, 8]])

>> a = a.tostring() # shows the memory arrangement, in a string

Out[] : b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00'

If you look at the string printed, it shows that the elements are arranged in the fashion 1, 2, 3, 4, 5, 6, 7, 8.

Why is this important to know, you ask?

For optimization, it is important. In Python, C/C++, the elements will be laid out contiguously row-wise in memory. So you should strive to process the elements row-wise. The entire row could be copied to cache and worked on from there by the OS. If you processed the elements column wise, contrary to how it is laid out in memory, you would incur expensive context switching as the OS copies in the elements you require into cache for processing for every column-wise element you process!

 

Conquer CSV files in Python

copywriting-guide-cartoon-comma

CSV files are common for data manipulation in Python, in cases where you extract the data from an excel sheet. Here is a short tutorial on how to extract some data out of a csv file, along with other nifty tricks along the way.

1) Easy Binary Conversion

You can use this to convert to binary.

your_binary_string = "0100000010001"
int_value = int(your_binary_string, 2)

Of course, you can extend this to octal, etc.

2) File reading and list comprehension

Suppose you have a whole csv of binary numbers. You need to read it out to python as a list.
Read the csv as string, and convert it to int easily.

Your csv data looks like this :

0,0,1,1111100110101010,0000111111101111,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
1,1,0,1111000110000000,0001000100000001,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
2,2,0,1111001010001000,0000111001100011,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
3,3,0,1111000010011000,0000111011100101,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
4,4,0,1111000000010011,0000110010011111,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
5,5,0,1110111100101100,0000110010000001,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
6,6,0,1110111001000010,0000101010111110,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
7,7,0,1110110111010000,0000100111110110,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
8,8,0,1110110011101111,0000100010011001,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
9,9,0,1110110010101011,0000011101010010,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
....

and you just want to get the values in bold. Meaning putting each of the the fourth and third column values in a tuple, converted to integer value.

import matplotlib.pyplot as plt
import numpy as np
import csv

filename = 'yourfile.csv'

with open(filename, 'rt') as csvfile:
 reader = csv.reader(csvfile)
 # the following line uses list comprehension to iterate through every row in the csv
 # and creates row number of tuples in a list
 # You can call next(reader) to skip a row, for example if the 1st row is just labels
 iq = [(int(row[4], 2), int(row[3], 2)) for row in reader]

You get something like this:

[(4079, 63914), (4353, 61824), (3683, 62088), (3813, 61592) .... ]

At this point, if for some reason you want to get all the 1st values in each tuple in one list, and the 2nd values in each tuple in another list, you can do:

i, q = zip(*iq)
i = list(i)
q = list(q)

You will get for i :

[4079, 4353, 3683, 3813, 3231, 3201, 2750, 2550, 2201, 1874, 1559, .... ]

and q:

[63914, 61824, 62088, 61592, 61459, 61228, 60994, 60880, 60655, 60587... ]

3) Writing CSV files

With the previous i and q lists you extracted, you can now write it out as a csv.

with open('complex_out.csv', 'w') as csvfile:
 fieldnames = ['i', 'q']
 writer = csv.DictWriter(csvfile, fieldnames = fieldnames)
 writer.writeheader()
 for row_num in range(len(i)):
 writer.writerow({'i':i[row_num], 'q':q[row_num]})

Your csv file looks like this :

i,q
4079,63914
4353,61824
3683,62088
3813,61592
3231,61459
3201,61228
2750,60994
2550,60880
2201,60655
1874,60587
1559,60434
1213,60367
898,60284
529,60238
195,60224
65371,60196
65033,60237
64701,60281
64372,60335
64024,60430
63705,60535
63385,60660
63065,60826
62763,61000
62474,61192
62198,61410
61935,61644
61680,61883
61468,62145
61232,62433
61040,62710
60863,63013
60702,63328
60566,63648
60459,63979
60361,64304
60292,64646
60247,64994
.....

3) Converting data types

From the above data, I know that my number is really a 16bit binary representation.
Now, I am told that this is a 2’s complement representation.
So each tuple should be really (int16, int16), with +ve and -ve values possible. Fortunately Python allows us to do this easily.

iq = np.array(iq, np.int16) # creates an array with iq existing values, converted to int16
array([[ 4079, -1622],
 [ 4353, -3712],
 [ 3683, -3448],
 ..., 
 [ -567, 5301],
 [ -216, 5329],
 [ 146, 5332]], dtype=int16)

Now we want to convert it to complex64, for further processing down the line.

sig = iq.astype(np.float32).view(np.complex64) # convert the values to float32, before viewing it as a complex64 (2 float32s)
array([[ 4079.-1622.j],
 [ 4353.-3712.j],
 [ 3683.-3448.j],
 ..., 
 [ -567.+5301.j],
 [ -216.+5329.j],
 [ 146.+5332.j]], dtype=complex64)
sig = sig.ravel() # flatten it to 1D
array([ 4079.-1622.j, 4353.-3712.j, 3683.-3448.j, ..., -567.+5301.j,
 -216.+5329.j, 146.+5332.j], dtype=complex64)

Hooray, now we’re ready to do further processing on this data!

Python: Using ioloop To Send to 2 Protocols Without Threading

ioloop is very useful for pre-scheduling functions to be executed in the future. In this manner, you can simulate interspersing sending to 2 protocols without threading. (in this case, “serial” and “tcp” at 100Hz and 1000Hz respectively.)

In this simple code, io_loop.add_timeout keeps pre-scheduling events to happen in the future. Do note that this all runs in one thread, so make sure your sending functions printTCP() and printSerial() are never blocking. I use this for simple use cases where I am just reading from a file and sending to two different sources. Also, in my code I use a lot of zmq, and only required the simple use cases of zmq.eventloop. For more advanced use cases, you may want to take a look at tornado.ioloop. http://www.tornadoweb.org/en/stable/ioloop.html (on which pyzmq’s event loop is based on)

import functools
import datetime
from zmq.eventloop import ioloop


def printTCP():
 print('Sent TCP')


def printSerial():
 print('Sent Serial')


io_loop = ioloop.IOLoop.instance()

# Send serial messages at 100Hz
serial_deadline = datetime.timedelta(seconds=0.1)
for x in range(1000):
 io_loop.add_timeout(serial_deadline, functools.partial(printSerial))
 serial_deadline += datetime.timedelta(seconds=0.1)

# Send tcp messages at 1000Hz
tcp_deadline = datetime.timedelta(seconds=0.01) # separate deadline for sending ins pulses
for x in range(10000):
 io_loop.add_timeout(tcp_deadline, functools.partial(printTCP))
 tcp_deadline += datetime.timedelta(seconds=0.01)

io_loop.start()

You will get something like this:

 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent Serial
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 Sent TCP
 ....

Super Easy Sequence Diagrams – PlantUML

http://plantuml.com/ is a very easy to use sequence diagram maker that is free. It has a extremely intuitive syntax that will allow you to create complex diagrams within minutes.

There’s a online version here : https://www.planttext.com

Take for example :

@startuml

participant FDC_FPGA

box "FPGA" #LightYellow
 participant FDC_FPGA
end box

box "MPC" #LightBlue
 participant Kernel
 participant User_Space
end box

User_Space -> Kernel : Wait Data Ready ioctl
activate User_Space
activate Kernel
FDC_FPGA -> Kernel : Data Ready Interrupt
Kernel -> User_Space : Data Ready Awake
deactivate Kernel
deactivate User_Space
User_Space -> Kernel : Setup Read Descriptors ioctl
Kernel -> FDC_FPGA : Set Registers
User_Space -> Kernel : Enable DMA Start ioctl
Kernel -> FDC_FPGA : Set Registers
User_Space->Kernel : Wait DMA Done ioctl
activate Kernel
activate User_Space
FDC_FPGA -> Kernel : Dma Done Interrupt
Kernel -> User_Space : Dma Done Awake
deactivate User_Space
deactivate Kernel

@enduml

This gives a very nice diagram that is ready for presentation :

Kernel_Driver_Flow

I use it all over the place for Software Design Documents. The best thing is – it has “source code” , so anytime there’s a change done, all I have to do is to use the “source” to regenerate the diagram. Happiness!

NCD: 2016 Macbook Pro 15″ Review! (refurb)

macbook-pro-2016-5

Got me a 2016 MacBook Pro 15″. Got it refurbished but it is really looks good as new. I really couldn’t justify paying the full price for what you get. No complains here, Apple’s refurbished stuff is top-notch.

Here are some of my thoughts on it after the one month honeymoon period.

Pretty

Apple knows how to make desirable, beautiful hardware. And the new MacBook Pro is no exception. Apple stuck to the brushed aluminium feel and the whole package looks elegant. Not much to pick at regarding looks.

Keyboard

The new low-travel keys take some getting used to – I’m actually still not used to them. I like chiclet keys, but these are even lower travel than those. It almost feels like they are trying to steer you in the direction of on-screen keys on a digital screen by introducing these. Perhaps such keys are less failure-prone?

They do work functionally, but give off a clickety-clack sound when you are typing. Those with mechanical keyboards may like this “feature” that it has. Overall, I still prefer the feel of the keys of previous models.

Touchpad

The 15″‘s touchpad, or trackpad as Apple would like you to call it, is slightly too big in my opinion. It has built-in palm rejection to prevent accidental triggering of the cursor when you are typing, although I still do mange to trigger it, once in a while. No major complains here, and Apple’s gesture integration with OSx is top-notch, as always.

Touchbar

And here comes the latest gimmick from Apple’s stable. The touch bar. I have to admit, when I just got this machine, I wanted to use the touch bar so much that I was on the look out for stuff appearing on it. But as time went by, I realised that I never actually look at the keyboard when I am typing, let alone look at the touch-bar. In this respect, the touch bar doesn’t serve its intended purpose. It isn’t a selling point for me. If I could get the same model for 200USD less without the touch-bar, I would.

Dongles, Dongles, Dongles!

USB3.1 is excellent technology. You can interface lots of things to it – HDMI / power / just about anything you can think of.  It has a transfer speed of up to 5GBits/s, which translates to 625MB/s. This is comparable to a SSD’s write speed. What this means is that you can expect to write to an external USB3.1 SSD drive with the same speed as you would to an onboard SSD, with both drives being the same.

Apple loves USB3.1, and they loved it so much they put nothing but USB3.1 connectors on the laptop. Even the power adaptor is USB-3.1, which is a shame – I loved the MagSafe laptop power connector on older models.  On the flip side, it also means that you don’t have to lug a power adaptor around, any wall power with usb3.1 socket and be used to charge your Macbook.

The problem is : These ports are ahead of their time. There really isn’t a lot of USB3.1 devices out there at the moment. So for now, we will have to use dongles to convert these connectors to the intended connector type. In time to come, I don’t see why devices not all converge to USB3.1 as a standard connector. It just makes everything easier – but currently it is a case of the real world needing to catch up the the tech.

Last Words

The real reason I got Apple’s laptops is that I make iOS apps – there isn’t a real alternative  that doesn’t involve tons of hair pulling when all I want to do is to be productive.  Regarding this, I got what I wanted. No more waiting for Xcode to load interface builder, or web-page freezing up when I too much is going on in a web-page. Rendering stuff is much faster than on my late 2011 MacBook Pro.

I got six good years out of my 2011 MacBook Pro, and it is the only computer that I can profess to use the term “love” on. In my opinion, OSx is a better OS than Windows. It feels as stable as a Linux-based system, with the software that I need to do my stuff. It is an added bonus that you can plug in a USB keyboard and have it play music software with little latency immediately. The Microsoft lovers will say, but windows runs on all PCs, so it inevitably has to do more heavy lifting compared to OSx. I don’t care. I just want to get things done, and the machine that allows me to do that is my weapon of choice.

TF8 Synth is out!

I’m happy to say that the iPhone synthesizer I was working on for 2 years is out.

screen696x696

Lots of hard work was put into this. A lot of hard work is still on-going to polish it to a shine. Check it out here, it’s a free download.

 

Loading tmux on Boot in Linux

ss-tmux2

tmux is a wonderful tool for displaying virtual consoles on the linux command prompt screen. It’s the next best thing to getting actual GUI windows controllable with a mouse.

Mainly, I use it for ssh purposes. Where I can ssh to a pc that I know has tmux already launched in the background and type.

tmux a

which attaches the session to the on-going tmux background session, allowing you to see everything that is going on in that process. This is especially useful for embedded systems where there are multiple processes launched in the background and you want to monitor them all.

So I have a tmux script launcher.sh here:

#!/bin/bash

SESSION="MPC1"

#allow re-launch
/usr/bin/tmux has-session -t $SESSION 2> /dev/null && /usr/bin/tmux kill-session -t $SESSION
/usr/bin/tmux -2 new-session -d -s $SESSION

echo "Launching tmux"

/usr/bin/tmux split-window -h
/usr/bin/tmux split-window -v
/usr/bin/tmux select-pane -t 0

/usr/bin/tmux send-keys -t $SESSION.0 "cd /path/to/binary1folder" C-m
/usr/bin/tmux send-keys -t $SESSION.0 "./binary1" C-m

/usr/bin/tmux send-keys -t $SESSION.1 "cd /path/to/binary2folder" C-m
/usr/bin/tmux send-keys -t $SESSION.1 "./binary2" C-m

/usr/bin/tmux send-keys -t $SESSION.2 "cd /path/to/binary3folder" C-m
/usr/bin/tmux send-keys -t $SESSION.2 "./binary3" C-m

This basically opens up , three panes and splits the window horizontally first, then splitting again one of the split windows vertically. It then launches a binary in each of the window panes. I won’ t go too much into the scripting here as there are plenty of resources for doing so, like this.

Configuring tmux to boot on startup on CentOS 7

Normally this should be pretty straightforward, but I ran into some hiccups.

First,

sudo nano /etc/rc.local

And edit the rc.local file to include

su -c /path/toyourscript/launcher.sh -l your_user_id

-l your_user_id means that you do the launch of the script launcher.sh as the user your_user_id.

And by right it should launch when CentOS boots, launching launcher.sh in the background which in turn launches tmux. However, I found that one of the abrt startup scripts, abrt-console-notification.sh was interfering with the launching of the tmux process/binaries. It would hang at the console terminal of the tmux screens. Doing the following resolved the problem for me.

cd /etc/profile.d/
chmod -r abrt-console-notification.sh

Basically, make abrt-console-notification.sh non-readable, allowing the profile.d startup process to skip over this particular script. It’s kind of a hack, but it worked. I reckon at most I don’t get the automatic bug reporting tool notifications at the console. Note that this script is run depending on what type of installation you chose when installing CentOS. I think that the minimal installl doesn’t run into this issue.

Hope this it useful to you, let me know!

ps. Here’s a tmux cheat sheet. https://gist.githubusercontent.com/afair/3489752/raw/e7106ac93c8f9602d3843696692a87cfb43c2d21/tmux.cheat

Sign Bit Extension

Sometimes you can get a value like this :

For example, something that is 12-bit in a 16-bit short int.

shortIntValue = 0b0000 1100 1110 1100

The latter 12 bits are the actual value. It is a negative value by the 12th bit being 1. And you really want :

0b1111 1100 1110 1100

You can do

shortIntValue <<= 4;

followed by

shortIntValue >>= 4;

Volia! Sign bit extension! This is useful for 2’s complement values which are put into a primitive integer size. (As above)

Create a free website or blog at WordPress.com.

Up ↑