Fix for Low Resolution in Console After Installing NVIDIA drivers (CentOS 7)

Installing NVIDIA requires the blacklisting of the default nouveau drivers.

It is the fault of the nvidia driver that the console resolution became 640x 480. Here is a hack to get it back to your desired resolution.

Set the following grub kernel parameter:

vga = 791

> sudo vim /boot/grub2/grub.cfg

For your desired kernel launching menuentry and between rhgb and quiet insert vga=791

ie.

menuentry 'CentOS Linux......{
.....
    linux16 ...  rhgb vga=791 quiet 
}

 

See here for the full list of settings and corresponding vga=? numbers

https://en.wikipedia.org/wiki/VESA_BIOS_Extensions#Linux_video_mode_numbers

Reboot and you should see that you will get a higher default resolution when you boot to the command line console.

Note that this is intended to be a quick fix, the next time you run grub2-mkconfig the new parameter will likely be removed as the grub.cfg will be re-generated.

Advertisements

Compiling Static vs Dynamic Libraries on CMake

tv

Why compile statically?

This allows for ease of deployment, at the expense of a larger binary executable.
You don’t have to copy the libraries that you use manually to the target system

CMake link_libraries() Magic

I use CLion, which (currently) enforces the use of CMake in compiling C/C++ projects. In your CMakeLists.txt file, first make sure you link the directory to find your files:

link_directories("/usr/local/lib")

CMake has a magic link_libraries() function which takes in the library specified and determines how you want it to be compiled (statically or dynamically linked).

If you type

link_libraries(ev)

It is interpreted as a dynamic linked library.

link_libraries(libev.a)

Tells CMake to look for this static library file in the linked directories, and build it statically into your binary.

Order of Static Linking Matters

Doing

link_libraries(libPocoFoundation.a)
link_libraries(libPocoXML.a)

I met this error.

/usr/local/lib/libPocoXML.a(XMLWriter.o): In function `Poco::XML::XMLWriter::XMLWriter(std::ostream&, int)':XMLWriter.cpp:(.text+0x28b3): undefined reference to `Poco::UTF8Encoding::UTF8Encoding()'XMLWriter.cpp:(.text+0x28cc): undefined reference to `Poco::UTF8Encoding::UTF8Encoding()'
......

It seems that the libPocoXML.a static library is trying to call functions in libPocoFoundation.a but can’t find them.
Reversing the order of linking the libraries helps.

link_libraries(libPocoXML.a)
link_libraries(libPocoFoundation.a)

This is because when CMake links the libPocoXML.a library, it makes a note of the external functions that are called and looks for them to be linked in the subsequent libraries. One example here is the Poco::UTF8Encoding::UTF8Encoding() function.

What is happening here is that CMake links the libPocoXML.a, looks for the function in subsequent libraries that are linked and finds nothing. Reversing the order allows libPocoXML.a to find the desired function later on in libPocoFoundation.a.

This only happens for static library compilation due to how CMake interprets it.
Check your binary using ldd :

ldd DSPBox

linux-vdso.so.1 =>  (0x00007fff2eb46000) libippi.so.9.0 => /opt/intel/compilers_and_libraries_2016.0.109/linux/ipp/lib/intel64/libippi.so.9.0 (0x00007f4a209cd000) libipps.so.9.0 => /opt/intel/compilers_and_libraries_2016.0.109/linux/ipp/lib/intel64/libipps.so.9.0 (0x00007f4a2078c000) libippcore.so.9.0 => /opt/intel/compilers_and_libraries_2016.0.109/linux/ipp/lib/intel64/libippcore.so.9.0 (0x00007f4a20580000) libippvm.so.9.0 => /opt/intel/compilers_and_libraries_2016.0.109/linux/ipp/lib/intel64/libippvm.so.9.0 (0x00007f4a2036a000) libcufft.so.7.5 => /usr/local/cuda/lib64/libcufft.so.7.5 (0x00007f4a1972f000) libtiff.so.3 => /usr/local/lib/libtiff.so.3 (0x00007f4a194d3000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007f4a19299000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f4a19094000) librt.so.1 => /lib64/librt.so.1 (0x00007f4a18e8c000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007f4a18b84000) libm.so.6 => /lib64/libm.so.6 (0x00007f4a18881000) libgomp.so.1 => /lib64/libgomp.so.1 (0x00007f4a1866a000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007f4a18454000) libc.so.6 => /lib64/libc.so.6 (0x00007f4a18091000) libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007f4a17e3c000) libz.so.1 => /lib64/libz.so.1 (0x00007f4a17c25000) /lib64/ld-linux-x86-64.so.2 (0x00007f4a20c53000)

 

Statically linking a file which has dynamic file dependencies.

This worked

link_libraries(tiff)

but not the static version.

 link_libraries(libtiff.a)

/usr/local/lib/libtiff.a(tif_jpeg.o): In function `TIFFjpeg_destroy’:/home/pier/Software/Development/tiff-3.8.2/libtiff/tif_jpeg.c:377: undefined reference to `jpeg_destroy’/usr/local/lib/libtiff.a(tif_jpeg.o): In function `TIFFjpeg_write_raw_data’:/home/pier/Software/Development/tiff-3.8.2/libtiff/tif_jpeg.c:320: undefined reference to `jpeg_write_raw_data’/usr/local/lib/libtiff.a(tif_jpeg.o): In function `TIFFjpeg_finish_compress’:…..
If you dig in deeper, you are able to find the dependancies using readelf

 cd /usr/local/
libreadelf -d libtiff.so | grep 'NEEDED'

0x0000000000000001 (NEEDED)             Shared library:
[libjpeg.so.62] 0x0000000000000001 (NEEDED)             Shared library:
[libz.so.1] 0x0000000000000001 (NEEDED)             Shared library:
[libm.so.6] 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]

So apparently we still need the above dynamic libraries even for libtiff.a. Fortunately these files come preinstalled in CentOS and most distributions.
So all you need to do now is :

link_libraries(libtiff.a jpeg z)

libc and libm are linked by default by gcc. CMake interprets this as : compile libtiff.a statically into the binary, but its dependancies libjpeg and libz are still dynamically linked. Get them dynamically from the linked system folders.
To be sure, check using ldd again:

ldd DSPBox

linux-vdso.so.1 =>  (0x00007ffdfc5ee000) libippi.so.9.0 => /opt/intel/compilers_and_libraries_2016.0.109/linux/ipp/lib/intel64/libippi.so.9.0 (0x00007fa41cf23000) libipps.so.9.0 => /opt/intel/compilers_and_libraries_2016.0.109/linux/ipp/lib/intel64/libipps.so.9.0 (0x00007fa41cce2000) libippcore.so.9.0 => /opt/intel/compilers_and_libraries_2016.0.109/linux/ipp/lib/intel64/libippcore.so.9.0 (0x00007fa41cad6000) libippvm.so.9.0 => /opt/intel/compilers_and_libraries_2016.0.109/linux/ipp/lib/intel64/libippvm.so.9.0 (0x00007fa41c8c0000) libcufft.so.7.5 => /usr/local/cuda/lib64/libcufft.so.7.5 (0x00007fa415c85000) libjpeg.so.62 => /lib64/libjpeg.so.62 (0x00007fa415a12000) libz.so.1 => /lib64/libz.so.1 (0x00007fa4157fc000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fa4155df000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fa4153db000) librt.so.1 => /lib64/librt.so.1 (0x00007fa4151d3000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fa414eca000) libm.so.6 => /lib64/libm.so.6 (0x00007fa414bc8000) libc.so.6 => /lib64/libc.so.6 (0x00007fa414806000) libgomp.so.1 => /lib64/libgomp.so.1 (0x00007fa4145ee000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fa4143d8000) /lib64/ld-linux-x86-64.so.2 (0x00007fa41d1a9000)

No more dynamic library requirement for libtiff!

Row major and Column major Explained, using Python against Matlab

In both cases, a 2d array of 2 rows, 4 columns is created.
Then, it is reshaped to 1 row, 8 columns.
This clearly demonstrates how Matlab stores the array data column-wise, and Python stores the array data row-wise after it is flattened out to 1d.

Matlab (Column Major)

Screen Shot 2017-06-04 at 10.51.20 AM

>> a = = [1 2 3 4 ; 5 6 7 8]

a = 
 1 2 3 4
 5 6 7 8

a = a(:)

a =

1 
5 
2 
6 
3 
7 
4 
8

Python (Row-major)

Screen Shot 2017-06-04 at 10.51.13 AM

>> import numpy as np
>> a = np.array([[1, 2, 3, 4], [5, 6, 7, 8]])
>> a
Out[]: 
array([[1, 2, 3, 4], 
        5, 6, 7, 8]])

>> a = a.tostring() # shows the memory arrangement, in a string

Out[] : b'\x01\x00\x00\x00\x02\x00\x00\x00\x03\x00\x00\x00\x04\x00\x00\x00\x05\x00\x00\x00\x06\x00\x00\x00\x07\x00\x00\x00\x08\x00\x00\x00'

If you look at the string printed, it shows that the elements are arranged in the fashion 1, 2, 3, 4, 5, 6, 7, 8.

Why is this important to know, you ask?

For optimization, it is important. In Python, C/C++, the elements will be laid out contiguously row-wise in memory. So you should strive to process the elements row-wise. The entire row could be copied to cache and worked on from there by the OS. If you processed the elements column wise, contrary to how it is laid out in memory, you would incur expensive context switching as the OS copies in the elements you require into cache for processing for every column-wise element you process!

 

Conquer CSV files in Python

copywriting-guide-cartoon-comma

CSV files are common for data manipulation in Python, in cases where you extract the data from an excel sheet. Here is a short tutorial on how to extract some data out of a csv file, along with other nifty tricks along the way.

1) Easy Binary Conversion

You can use this to convert to binary.

your_binary_string = "0100000010001"
int_value = int(your_binary_string, 2)

Of course, you can extend this to octal, etc.

2) File reading and list comprehension

Suppose you have a whole csv of binary numbers. You need to read it out to python as a list.
Read the csv as string, and convert it to int easily.

Your csv data looks like this :

0,0,1,1111100110101010,0000111111101111,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
1,1,0,1111000110000000,0001000100000001,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
2,2,0,1111001010001000,0000111001100011,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
3,3,0,1111000010011000,0000111011100101,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
4,4,0,1111000000010011,0000110010011111,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
5,5,0,1110111100101100,0000110010000001,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
6,6,0,1110111001000010,0000101010111110,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
7,7,0,1110110111010000,0000100111110110,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
8,8,0,1110110011101111,0000100010011001,0000000000000000,0000000000000000,001,1,0,1,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
9,9,0,1110110010101011,0000011101010010,0000000000000000,0000000000000000,001,1,0,0,1,0,0,1,0,0,0,0,0,0,0000000000000000,0000000000000000,0,0000000000000000,0000000000000000
....

and you just want to get the values in bold. Meaning putting each of the the fourth and third column values in a tuple, converted to integer value.

import matplotlib.pyplot as plt
import numpy as np
import csv

filename = 'yourfile.csv'

with open(filename, 'rt') as csvfile:
 reader = csv.reader(csvfile)
 # the following line uses list comprehension to iterate through every row in the csv
 # and creates row number of tuples in a list
 # You can call next(reader) to skip a row, for example if the 1st row is just labels
 iq = [(int(row[4], 2), int(row[3], 2)) for row in reader]

You get something like this:

[(4079, 63914), (4353, 61824), (3683, 62088), (3813, 61592) .... ]

At this point, if for some reason you want to get all the 1st values in each tuple in one list, and the 2nd values in each tuple in another list, you can do:

i, q = zip(*iq)
i = list(i)
q = list(q)

You will get for i :

[4079, 4353, 3683, 3813, 3231, 3201, 2750, 2550, 2201, 1874, 1559, .... ]

and q:

[63914, 61824, 62088, 61592, 61459, 61228, 60994, 60880, 60655, 60587... ]

3) Writing CSV files

With the previous i and q lists you extracted, you can now write it out as a csv.

with open('complex_out.csv', 'w') as csvfile:
 fieldnames = ['i', 'q']
 writer = csv.DictWriter(csvfile, fieldnames = fieldnames)
 writer.writeheader()
 for row_num in range(len(i)):
 writer.writerow({'i':i[row_num], 'q':q[row_num]})

Your csv file looks like this :

i,q
4079,63914
4353,61824
3683,62088
3813,61592
3231,61459
3201,61228
2750,60994
2550,60880
2201,60655
1874,60587
1559,60434
1213,60367
898,60284
529,60238
195,60224
65371,60196
65033,60237
64701,60281
64372,60335
64024,60430
63705,60535
63385,60660
63065,60826
62763,61000
62474,61192
62198,61410
61935,61644
61680,61883
61468,62145
61232,62433
61040,62710
60863,63013
60702,63328
60566,63648
60459,63979
60361,64304
60292,64646
60247,64994
.....

3) Converting data types

From the above data, I know that my number is really a 16bit binary representation.
Now, I am told that this is a 2’s complement representation.
So each tuple should be really (int16, int16), with +ve and -ve values possible. Fortunately Python allows us to do this easily.

iq = np.array(iq, np.int16) # creates an array with iq existing values, converted to int16
array([[ 4079, -1622],
 [ 4353, -3712],
 [ 3683, -3448],
 ..., 
 [ -567, 5301],
 [ -216, 5329],
 [ 146, 5332]], dtype=int16)

Now we want to convert it to complex64, for further processing down the line.

sig = iq.astype(np.float32).view(np.complex64) # convert the values to float32, before viewing it as a complex64 (2 float32s)
array([[ 4079.-1622.j],
 [ 4353.-3712.j],
 [ 3683.-3448.j],
 ..., 
 [ -567.+5301.j],
 [ -216.+5329.j],
 [ 146.+5332.j]], dtype=complex64)
sig = sig.ravel() # flatten it to 1D
array([ 4079.-1622.j, 4353.-3712.j, 3683.-3448.j, ..., -567.+5301.j,
 -216.+5329.j, 146.+5332.j], dtype=complex64)

Hooray, now we’re ready to do further processing on this data!

Sign Bit Extension

Sometimes you can get a value like this :

For example, something that is 12-bit in a 16-bit short int.

shortIntValue = 0b0000 1100 1110 1100

The latter 12 bits are the actual value. It is a negative value by the 12th bit being 1. And you really want :

0b1111 1100 1110 1100

You can do

shortIntValue <<= 4;

followed by

shortIntValue >>= 4;

Volia! Sign bit extension! This is useful for 2’s complement values which are put into a primitive integer size. (As above)