Category: /dev/random

Entropy lacking write stream

Building Python 3.5 for CentOS

Update: As per Alexander Todorov’s suggestion using https://www.softwarecollections.org/ is much easier and saves a lot of trouble compiling.

So after git it’s time to upgrade python. CentOS 7 has python 3.4, but there are some small but very annoying incompatibilities with 3.5. And it seems that 3.5 will be the major python 3 release. So here we go:

RPM build chroot

First lets create the chroot directory and  initialize the  rpm database   :

mkdir /root/buildroot
rpm --root=/root/buildroot --rebuilddb

Next  install the latest centos-release package,  yum and add the EPEL inside the chroot:

rpm --root=/root/buildroot -i http://mirror.centos.org/centos/7.2.1511/os/x86_64/Packages/centos-release-7-2.1511.el7.centos.2.10.x86_64.rpm
yum --installroot=/root/buildroot install yum
rpm --root=/root/buildroot -i http://dl.fedoraproject.org/pub/epel/7/x86_64/e/epel-release-7-8.noarch.rpm

Next we continue inside the chroot :

systemd-nspawn -D /root/buildroot/

Building RPMs

Next we need to install gcc, and few other packages that are needed to compile RPMs and pyton.

yum install autoconf bluez-libs-devel  bzip2-devel libdb4-devel  gcc-c++ gmp-devel libffi-devel libGL-devel libX11-devel ncurses-devel net-tools readline-devel sqlite-devel tcl-devel tix-devel tk-devel valgrind-devel xz-devel  python-rpm-macros openssl-devel make gcc perl-ExtUtils-MakeMaker  rpmdevtools

Pick a python 3.5 SRPM from https://apps.fedoraproject.org/packages/python3 .

And install it :

rpm -i  https://kojipkgs.fedoraproject.org//packages/python3/3.5.2/4.fc25/src/python3-3.5.2-4.fc25.src.rpm

You can just ignore the warning about missing users. Go to SPECS directory and edit python3.spec so that :

%global with_rewheel 1

becomes

%global with_rewheel 0

This has something to do with a circular dependency of the rewheel package. Now we can build python with:

cd rpmbuild/SPECS/
rpmbuild -ba python3.spec

After the build is done we can check if the packages are installable:

cd /root/rpmbuild/RPMS/x86_64
yum install python3-rpm-macros-3 *.rpm

Now let’s try to rebuild with the rewheel package. We will need python-rpm-macros, python3-pip and python3-setuptools  and a bunch of other packages for building them 🙂

# python-rpm-macros
rpm -i https://kojipkgs.fedoraproject.org//packages/python-rpm-macros/3/11.el6/src/python-rpm-macros-3-11.el6.src.rpm
rpmbuild -D 'rpmmacrodir /etc/rpm/' -ba python-rpm-macros.spec

yum install ../RPMS/noarch/python3-rpm-macros-3-11.el7.centos.noarch.rpm ../RPMS/noarch/python-rpm-macros-3-11.el7.centos.noarch.rpm ../RPMS/noarch/python-srpm-macros-3-11.el7.centos.noarch.rpm


# install some build dependencies for setuptools and pip
yum -y install bash-completion
yum -y install https://kojipkgs.fedoraproject.org//packages/python-pip/8.1.2/4.fc26/noarch/python3-pip-8.1.2-4.fc26.noarch.rpm
yum -y install https://kojipkgs.fedoraproject.org//packages/python-setuptools/28.6.0/1.fc26/noarch/python3-setuptools-28.6.0-1.fc26.noarch.rpm
yum -y install https://kojipkgs.fedoraproject.org//packages/python-wheel/0.30.0a0/1.fc26/noarch/python3-wheel-0.30.0a0-1.fc26.noarch.rpm
yum -y install https://kojipkgs.fedoraproject.org//packages/python-mock/2.0.0/2.fc25/noarch/python3-mock-2.0.0-2.fc25.noarch.rpm https://kojipkgs.fedoraproject.org//packages/python-funcsigs/1.0.2/2.fc25/noarch/python3-funcsigs-1.0.2-2.fc25.noarch.rpm h
ttps://kojipkgs.fedoraproject.org//packages/python-six/1.10.0/4.fc26/noarch/python3-six-1.10.0-4.fc26.noarch.rpm
yum -y install https://kojipkgs.fedoraproject.org//packages/pytest/3.0.3/1.fc26/noarch/python3-pytest-3.0.3-1.fc26.noarch.rpm https://kojipkgs.fedoraproject.org//packages/python-py/1.4.31/3.fc24/noarch/python3-py-1.4.31-3.fc24.noarch.rpm

# python3-setuptools
rpm -i https://kojipkgs.fedoraproject.org//packages/python-setuptools/28.6.0/1.fc26/src/python-setuptools-28.6.0-1.fc26.src.rpm
rpmbuild -D 'with_python3 1'  -D 'python3_other_pkgversion 3' -D '__python3 /usr/bin/python3.5' -D 'fedora 1'  -bb python-setuptools.spec

# python3-pip
rpm -i https://kojipkgs.fedoraproject.org//packages/python-pip/8.1.2/4.fc26/noarch/python3-pip-8.1.2-4.fc26.noarch.rpm
rpmbuild -D 'rhel 8' -D 'python3_other_pkgversion 3' -D '__python3 /usr/bin/python3.5' -ba python-pip.spec

Change back python3.spec to:

%global with_rewheel 1

And now rebuild python3 again:

rpmbuild -ba python3.spec

After the build finishes we have the RPMs in ~/rpmbuild/RPMS:

# find rpmbuild/RPMS/
rpmbuild/RPMS/
rpmbuild/RPMS/noarch
rpmbuild/RPMS/noarch/python3-pip-8.1.2-4.el7.centos.noarch.rpm
rpmbuild/RPMS/noarch/python2-pip-8.1.2-4.el7.centos.noarch.rpm
rpmbuild/RPMS/noarch/python-srpm-macros-3-11.el7.centos.noarch.rpm
rpmbuild/RPMS/noarch/python-rpm-macros-3-11.el7.centos.noarch.rpm
rpmbuild/RPMS/noarch/python2-rpm-macros-3-11.el7.centos.noarch.rpm
rpmbuild/RPMS/noarch/python3-rpm-macros-3-11.el7.centos.noarch.rpm
rpmbuild/RPMS/noarch/python2-setuptools-28.6.0-1.el7.centos.noarch.rpm
rpmbuild/RPMS/noarch/python3-setuptools-28.6.0-1.el7.centos.noarch.rpm
rpmbuild/RPMS/x86_64
rpmbuild/RPMS/x86_64/python3-test-3.5.2-6.el7.centos.x86_64.rpm
rpmbuild/RPMS/x86_64/python3-tools-3.5.2-6.el7.centos.x86_64.rpm
rpmbuild/RPMS/x86_64/python3-debuginfo-3.5.2-6.el7.centos.x86_64.rpm
rpmbuild/RPMS/x86_64/python3-libs-3.5.2-6.el7.centos.x86_64.rpm
rpmbuild/RPMS/x86_64/system-python-libs-3.5.2-6.el7.centos.x86_64.rpm
rpmbuild/RPMS/x86_64/python3-devel-3.5.2-6.el7.centos.x86_64.rpm
rpmbuild/RPMS/x86_64/python3-debug-3.5.2-6.el7.centos.x86_64.rpm
rpmbuild/RPMS/x86_64/python3-tkinter-3.5.2-6.el7.centos.x86_64.rpm
rpmbuild/RPMS/x86_64/python3-3.5.2-6.el7.centos.x86_64.rpm
rpmbuild/RPMS/x86_64/system-python-3.5.2-6.el7.centos.x86_64.rpm

Exit the chroot and install the newly build python 3.5 packages :

yum install rpmbuild/RPMS/x86_64/python3-3.5.2-6.el7.centos.x86_64.rpm  rpmbuild/RPMS/x86_64/python3-libs-3.5.2-6.el7.centos.x86_64.rpm rpmbuild/RPMS/x86_64/system-python-3.5.2-6.el7.centos.x86_64.rpm rpmbuild/RPMS/x86_64/system-python-libs-3.5.2-6.el7.centos.x86_64.rpm rpmbuild/RPMS/x86_64/python3-devel-3.5.2-6.el7.centos.x86_64.rpm  rpmbuild/RPMS/x86_64/python3-tools-3.5.2-6.el7.centos.x86_64.rpm  rpmbuild/RPMS/x86_64/python3-tkinter-3.5.2-6.el7.centos.x86_64.rpm rpmbuild/RPMS/noarch/python3-setuptools-28.6.0-1.el7.centos.noarch.rpm  rpmbuild/RPMS/noarch/python3-pip-8.1.2-4.el7.centos.noarch.rpm

Building latest git for CentOS

CentOS is a great operating system, but recent versions of software packages is not one of its virtues. For example the current CentOS 7.2 ships with git version 1.8.3 released 2013, almost 3 years ago. And by that time git gained quite some useful features. So lets see how to build RPMs with latest git.

RPM build chroot

First thing is to make a chroot where to build the RPM, because there are a lot of dependencies required for building, which needs to be cleaned up afterwards. Much easier is to just delete the whole chroot dir 🙂

First lets create the chroot directory and  initialize the  rpm database   :

mkdir /root/buildroot
rpm --root=/root/buildroot --rebuilddb

Next  install the latest centos-release package and yum inside :

rpm --root=/root/buildroot -i http://mirror.centos.org/centos/7.2.1511/os/x86_64/Packages/centos-release-7-2.1511.el7.centos.2.10.x86_64.rpm
yum --installroot=/root/buildroot install yum

Next we continue inside the chroot. Lucky us, systemd-nspawn makes it trivial to work inside the chroot :

systemd-nspawn -D /root/buildroot/

 

Building RPMs

Next we need to install gcc, and few other packages that are needed to compile RPMs .

yum install make gcc perl-ExtUtils-MakeMaker  rpmdevtools curl-devel expat-devel gettext-devel openssl-devel zlib-devel

And now we need a git SRPM to build. We can just grab the Fedora one which has git 2.7.1. Here https://apps.fedoraproject.org/packages/git/overview/ are all Fedora builds, and I picked the Rawhide version. Clicking on it takes us to a page with build information and there is also a link to download the SRPM for this build. Lets install it :

rpm -i  https://kojipkgs.fedoraproject.org//packages/git/2.7.1/1.fc24/src/git-2.7.1-1.fc24.src.rpm

You can just ignore the warning about missing users. Go to SPECS directory and try to build the RPMs:

cd rpmbuild/SPECS/
rpmbuild -ba git.spec

error: Failed build dependencies:
        asciidoc >= 8.4.1 is needed by git-2.7.1-1.el7.centos.x86_64
        xmlto is needed by git-2.7.1-1.el7.centos.x86_64
        desktop-file-utils is needed by git-2.7.1-1.el7.centos.x86_64
        emacs is needed by git-2.7.1-1.el7.centos.x86_64
        libgnome-keyring-devel is needed by git-2.7.1-1.el7.centos.x86_64
        pkgconfig(bash-completion) is needed by git-2.7.1-1.el7.centos.x86_64

Ok, we need to install some dependencies to be able to build. Lets just install them:

yum install asciidoc  xmlto desktop-file-utils emacs libgnome-keyring-devel pkgconfig bash-completion

and run rpmbuild again:

rpmbuild -ba git.spec

After the build finishes we have the RPMs in ~/rpmbuild/RPMS:

cd ~/rpmbuild/RPMS/
find
.
./noarch
./noarch/gitweb-2.7.1-1.el7.centos.noarch.rpm
./noarch/gitk-2.7.1-1.el7.centos.noarch.rpm
./noarch/git-email-2.7.1-1.el7.centos.noarch.rpm
./noarch/git-gui-2.7.1-1.el7.centos.noarch.rpm
./noarch/git-cvs-2.7.1-1.el7.centos.noarch.rpm
./noarch/git-p4-2.7.1-1.el7.centos.noarch.rpm
./noarch/perl-Git-SVN-2.7.1-1.el7.centos.noarch.rpm
./noarch/git-all-2.7.1-1.el7.centos.noarch.rpm
./noarch/perl-Git-2.7.1-1.el7.centos.noarch.rpm
./x86_64
./x86_64/git-core-doc-2.7.1-1.el7.centos.x86_64.rpm
./x86_64/git-svn-2.7.1-1.el7.centos.x86_64.rpm
./x86_64/git-core-2.7.1-1.el7.centos.x86_64.rpm
./x86_64/git-daemon-2.7.1-1.el7.centos.x86_64.rpm
./x86_64/git-2.7.1-1.el7.centos.x86_64.rpm
./x86_64/git-debuginfo-2.7.1-1.el7.centos.x86_64.rpm

Exit the chroot and install the following packages for a minimal git installation:

yum install buildroot/root/rpmbuild/RPMS/x86_64/git-2.7.1-1.el7.centos.x86_64.rpm buildroot/root/rpmbuild/RPMS/x86_64/git-core-* buildroot/root/rpmbuild/RPMS/noarch/perl-Git-2.7.1-1.el7.centos.noarch.rpm

Not, so bad 🙂 Next time I will check how to create a repository .

Quck and (not so) dirty compressing reverse proxy.

While working on a couchapp recently I found out quite interesting fact. CouchDB doesn’t support   the gzip/deflate HTTP responses. And with a view that’s several MB, on a slow connection it was a lot pain using the app.

My first thought was, no problem, my CouchDB is behind nginx anyway, lets just turn on gzip compression in nginx. And while that was super simple to do, it yielded an undesired effect. The responses were now compressed, but nginx strips off Etags headers, and there is no way around it. Without the Etag queries always return full responses, even if the data hasn’t been modified. With them, a short 304 reponse is sent when there is no change.

Unhappy with nginx approach to way too strict HTTP, I decided to write my own compressing proxy. Fortunately thats super simple with nodejs/iojs. Its just a matter of gluing few modules together 🙂

First install the modules:

npm install http-proxy connect compression morgan

Then save this as proxy.js in the same directory:

var	http = require('http'),
		connect = require('connect'),
		compression = require('compression'),
		morgan = require('morgan'),
		httpProxy = require('http-proxy');

var port = 8012;

var proxy = httpProxy.createProxyServer({
	target: 'http://localhost:5984/'
});

var app = connect();

// Log the requests, useful for debugging
app.use(morgan('combined'));

app.use(compression());
app.use(
	function(req, res) {
		proxy.web(req, res);
	}
).listen(port);

console.log('proxy  started  on port ' +  port);

And voila , now run:

node proxy.js

And you have gzipped responses from CouchDB on port 8012.

 

Unfrozen, or how to get that file back from Eclipse

I was happily working on a small java project, and after just creating a new Java class, and typed around 100 lines in the Eclipse editor, the whole IDE  froze.  It seems there is a bug in Luna that will freeze the IDE on out of memory condition, and I have just hit it.

100 lines is not that much of a loss generally, I can write them again. But why? Aren’t computers supposed to save us from the manual work. I wanted this file back 🙂 Quick check on the file system showed that after creating the class Eclipse saved and empty class inside. I had to find another way I guess.

What if I somehow manage to connect a Java debugger and search for the file in memory. Unfortunately I am not that proficient in Java and  doing all the research on how to do it may take quite some time.  So lets see do I really need a debugger ? I just need the file, and it is somewhere in the Eclipse process memory. Getting the whole memory of a process is quite easy in Linux :

gcore `pgrep java`

And voila, we have the whole memory of the process saved to core.<PID> . Now lets find that file. strings comes to mind instantly :

strings core.5607 | gview -

And I started searching for occurrence of ‘class MetadataInfo’ ( that was my class name) . Unfortunately while  MetadataInfo had plenty of instances none of them was preceded by a class keyword and none was actually in the content of my file.  Hmm, maybe the file isn’t stored as plain string, maybe it is some kind of tree for easier syntax t highlighting, as there were pieces of the file all over the place. But then it occurred to  me that the file is probably stored in UTF16 in memory as Java is very much into UTF16. Quick check shows that strings supports several encodings :

$ strings --help
......
  -e --encoding={s,S,b,l,B,L} Select character size and endianness:
                            s = 7-bit, S = 8-bit, {b,l} = 16-bit, {B,L} = 32-bit
.......

Nice! Lets try 16-bit LE  as I am on an Intel machine:

strings -e l core.5607  | gview -

Now searching  for ‘class MetadataInfo’ finds several occurrences of the full file, I guess because of the Local History that Eclipse keeps for each file.  Just copied one that seems good enough, saved it to file and killed the frozen Eclipse w/o significant data loss.

 

Btrfs defragment

I have BTRFS for root and home volumes on my latop, and today I found out that  BTRFS supports defragmentation. It’s interesting to check whether defragmentation has measurable performance impact, so I did a quick test.

The easiest thing was to measure the boot time as reported by systemd .  So I did a couple of restarts, recorded the boot times systemd-analyze reports :

Before defragment Kernel initrd user Total
Boot 1 2.743 2.015 4.81 9.568
Boot 2 2.749 1.913 4.757 9.419
Boot 3 2.79 1.882 3.209 7.881
Avegare 2.761 1.937 4.259 8.956

Then I did a :

sudo btrfs filesystem defragment -v -r /

And recorded the new boot times :

After defragment Kernel initrd user Total
Boot 1 2.751 1.734 3.37 7.855
Boot 2 2.743 1.76 4.287 8.79
Boot 3 2.746 1.747 2.847 7.34
Average 2.747 1.747 3.501 7.995

That’s a whole second less, or about 11% improvement.  Not so bad at all, given that the disk is Samsung 840 Pro SSD, and is quite fast anyway.

tmux on Solaris

In the last month my work is somewhat related to Solaris and since I really got used to tmux I kind of miss it. Unfortunately it turns out Solaris is not the happy Linux land I am used to. One does not simply  dowload, ./configure and make install stuff in Solaris. Not without   the proper spells 🙂 So here is an attempt to document my experience to build static tmux binary, that doesn’t require libevent or ncurses shared libraries. I used a VM I had with gcc installed from openCSW.

First thing libevent, download and unpack it. Then:

$ cd libevent-2.0.21-stable
$ ./configure --prefix=/tmp/tmux-install
$ make && make install

Next goes ncurses,  download  and unpack. Then:

$ export AR=gar
$ ./configure --prefix=/tmp/tmux-install --without-cxx-binding
$ make && make install

And finally we can compile tmux itself. This is abit trickier , as it seems tmux doesn’t directly support Solaris. Download and unpack.  You need to apply the following patch:

diff -ur tmux-1.8/client.c tmux-1.8-patched/client.c
--- tmux-1.8/client.c	Sun Mar 17 16:03:37 2013
+++ tmux-1.8-patched/client.c	Mon Nov 18 22:29:59 2013
@@ -33,6 +33,39 @@
 
 #include "tmux.h"
 
+/* BEGIN SOLARIS 11 FIX */
+#ifndef LOCK_SH
+#define LOCK_SH 1 /* shared lock */
+#define LOCK_EX 2 /* exclusive lock */
+#define LOCK_NB 4 /* don’t block when locking */
+#define LOCK_UN 8 /* unlock */
+#endif
+int flock(int fd, int cmd);
+void cfmakeraw(struct termios *termios_p);
+int
+flock(int fd, int cmd)
+{
+    struct flock f;
+    memset(&f, 0, sizeof (f));
+    if (cmd & LOCK_UN)
+        f.l_type = F_UNLCK;
+    if (cmd & LOCK_SH)
+        f.l_type = F_RDLCK;
+    if (cmd & LOCK_EX)
+        f.l_type = F_WRLCK;
+    return fcntl(fd, (cmd & LOCK_NB) ? F_SETLK : F_SETLKW, &f);
+}
+void
+cfmakeraw(struct termios *termios_p)
+{
+    termios_p->c_iflag &= ~(IGNBRK|BRKINT|PARMRK|ISTRIP|INLCR|IGNCR|ICRNL|IXON);
+    termios_p->c_oflag &= ~OPOST;
+    termios_p->c_lflag &= ~(ECHO|ECHONL|ICANON|ISIG|IEXTEN);
+    termios_p->c_cflag &= ~(CSIZE|PARENB);
+    termios_p->c_cflag |= CS8;
+}
+/* END SOLARIS 11 FIX */
+
 struct imsgbuf	client_ibuf;
 struct event	client_event;
 struct event	client_stdin;
diff -ur tmux-1.8/server-client.c tmux-1.8-patched/server-client.c
--- tmux-1.8/server-client.c	Tue Mar 26 21:22:31 2013
+++ tmux-1.8-patched/server-client.c	Mon Nov 18 22:34:13 2013
@@ -25,7 +25,20 @@
 #include 
 #include 
 #include 
+#include 
 
+#ifndef timersub
+# define timersub(a, b, result)						\
+	do {								\
+		(result)->tv_sec = (a)->tv_sec - (b)->tv_sec;		\
+		(result)->tv_usec = (a)->tv_usec - (b)->tv_usec;        \
+		if ((result)->tv_usec < 0) { \ + --(result)->tv_sec;				\
+			(result)->tv_usec += 1000000;			\
+		}                                                       \
+	} while (0)
+#endif
+
 #include "tmux.h"
 
 void	server_client_check_focus(struct window_pane *);

Use gpatch, obviuosly patch in Solaris carries some braindead Unix legacy which I don’t even want to know about:

$ gpatch -p1 < ../tmux-1.8-solaris.patch

Then we setup the CPP/LDFLAGS, ./configure and make:

$ export LIBEVENT_CFLAGS=-I/tmp/tmux-install/include
$ export LIBEVENT_LIBS="-lsendfile /tmp/tmux-install/lib/libevent.a -L/tmp/tmux-install/include"
$ CPPFLAGS="-D_XPG6" LDFLAGS="-D_XPG6" ./configure --prefix=/tmp/tmux-install/
$ make

Thats it 🙂 In the tmux source directory we have a tmux binary that depends only on system libraries so you can copy it to other Solaris boxes and use it there.

$ ldd tmux
        libxnet.so.1 =>  /lib/libxnet.so.1
        libsocket.so.1 =>        /lib/libsocket.so.1
        libnsl.so.1 =>   /lib/libnsl.so.1
        libcurses.so.1 =>        /lib/libcurses.so.1
        libsendfile.so.1 =>      /lib/libsendfile.so.1
        librt.so.1 =>    /lib/librt.so.1
        libresolv.so.2 =>        /lib/libresolv.so.2
        libc.so.1 =>     /lib/libc.so.1
        libmp.so.2 =>    /lib/libmp.so.2
        libmd.so.1 =>    /lib/libmd.so.1
        libscf.so.1 =>   /lib/libscf.so.1
        libaio.so.1 =>   /lib/libaio.so.1
        libdoor.so.1 =>  /lib/libdoor.so.1
        libuutil.so.1 =>         /lib/libuutil.so.1
        libgen.so.1 =>   /lib/libgen.so.1
        libm.so.2 =>     /lib/libm.so.2