Convert SPSS .SAV file to CSV (for Excel)

SPSS alternative, GNU PSPP include a very easy to use tool that can convert the SPSS SAV file to CSV. This can be further read in to excel. Just open the csv file in Excel. If excel does not automatically split data, choose the first column, go to Data tab and select “text to columns” and use comma as separator.

Usage is very simple:

pspp-convert <input.sav> <output.csv>

More instructions in documentation. You can download the Windows version of SPSS at least from pspp4windows Source Forge project page.

Posted in Misc |

Filezilla GnuTLS error -15: An unexpected TLS packet was received and VsFTP

Too-long-don’t-want-to-read–give-me-commands-to-paste-to-my-terminal:

sudo /usr/sbin/setsebool -P ftp_home_dir 1

Long(er) story

I ran into “interesting” problem on CentOS while trying to setup VsFTP with FTPS. I was using FileZilla to connect to the server. The VsFTPd version that comes with my CentOS version seems to be rather old, 2.2.2. The setup however was smooth. But then the problems started. The main issue I hit was FileZilla complaining about “GnuTLS error -15: An unexpected TLS packet was received” after the connection was established (NOTE: GnuTLS error -12 indicates a different problem, and a common solution is to add ssl_ciphers=HIGH to vsftpd.conf)

To an untrained eye this might indicate there is something wrong with the encryption. But no – apparently the “GnuTLS error -15″ on FileZilla can be also caused by other issues which have nothing to do with the encryption. The way to find the solution was to take a step back, enable regular FTP and try to connect with that one. The regular connection attempt showed an interesting error message: “500 OOPS: cannot change directory:/home/myaccount”. Since I had already checked the permissions on the directory, the problem started to sound like some issue with SELinux. And indeed, that was the case. FTP daemon could not access the home directory of the user due to missing permissions.

Some googling revealed that “sudo /usr/sbin/setsebool -P ftp_home_dir 1″ was the correct thing to do. And indeed, after that command both FTP and FTP-S started working.

Lessons learned: very the unencrypted stuff works before sprinkling security to your solution.

Some resources:

Posted in Misc |

Enable debug log in Azure Storage Emulator

Azure Storage Emulator has logging capability but this is not enabled by default and it does not seem to be available via user interface.

In order to enable the log,

  • Head to the Storage Emulator data directory, which is located at %LOCALAPPDATA%\DevelopmentStorage.
  • Edit the the developmentstorage configuration file located in the directory and change the LoggingEnabled to true
  • Restart Strorage Emulator

The logs will appear in the logs directory.

<?xml version="1.0"?>
<DevelopmentStorage xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" v
ersion="2009-03-18">
  <SQLInstance>(localdb)\v11.0</SQLInstance>
  <PageBlobRoot>C:\Users\juhap\AppData\Local\DevelopmentStorage\PageBlobRoot</PageBlobRoot>
  <BlockBlobRoot>C:\Users\juhap\AppData\Local\DevelopmentStorage\BlockBlobRoot</BlockBlobRoot>
  <LogPath>C:\Users\juhap\AppData\Local\DevelopmentStorage\Logs</LogPath>
  <LoggingEnabled>true</LoggingEnabled>
</DevelopmentStorage>
Posted in Azure |

Ubuntu, disable special effects

I installed Ubuntu under Hyper-V and for reason the UI with all the effects is S-L-O-W.

Quick steps for disabling the UI effects:

sudo apt-get install compizconfig-settings-manager
ccsm

Then head to the effects tab and disable those that you want. This does not make the UI blazing fast, but helps a bit. NOTE: You may not want to take away the window decorations, as that means losing the title bar.

(Tip via IT World)

Posted in Ubuntu |

Keymapper mappings for using Apple keyboard with Windows

I fell in love with the full size Apple keyboard. It is low profile and I really like how it works. Plus it’s pretty affordable (compared to premium keyboards).

Unfortunately by default few of the keys are out of place compared to standard PC keyboard. In order to fix this I created a mapping to be used with Keymapper tool that fixes these issues.

NOTE: I’m using the finnish/swedish keyboard layout.

Save the following to for example mappings.reg file and read it to registry. After that you need to at least log out, maybe also restart windows to make the mappings effective.

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,04,00,00,00,38,00,5B,E0,5B,E0,38,00,38,E0,5C,E0,00,00,00,00

[HKEY_CURRENT_USER\Keyboard Layout]
"Scancode Map"=hex:00,00,00,00,00,00,00,00,06,00,00,00,38,E0,5C,E0,38,00,5B,E0,5B,E0,38,00,37,E0,64,00,46,E0,66,00,00,00,00,00
Posted in Web development |

Change SQL Server Express authentication mode

When you install SQL Server Express you can select the authentication mode. If you want to enable SQL Server authentication mode (username and password) later on, you can use T-SQL to do it:

ALTER LOGIN sa ENABLE;
GO
ALTER LOGIN sa WITH PASSWORD = '<strong password>';
GO
Posted in Web development |

Digging Liferay article categories from database

In order to investigate certain issues I wanted to find out the journal categories directly from database.

Below is the select I came up with. This is for MySQL. The string operations on the second line try to extract the actual content that is inside the extra markup. It is not perfect, but you get an idea about the title (which was the goal).

SELECT 
		replace(substr(ja.title, LENGTH(ja.title) - LOCATE('"',REVERSE(ja.title)) + 3),"</Title></root>","") as Title
	   ,ac.name as Category,
	   ac.categoryId,
	   ac.vocabularyId
  FROM journalarticle ja,
  	   journalarticleresource jar,
  	   assetentry ae,  	   
  	   classname_ cn,  	   
  	   assetentries_assetcategories anac,
  	   assetcategory ac
 WHERE 
 	   jar.resourcePrimKey = ja.resourcePrimKey  
 	   AND jar.groupId = ja.groupId
 	   AND jar.uuid_ = ae.classUuid
 	   AND cn.classNameId = ae.classNameId
 	   AND anac.entryId = ae.entryId
 	   AND ac.categoryId = anac.categoryId
 	   AND ac.groupId = jar.groupId
 	   AND ja.title like '%Your title here%'
ORDER by ja.id_ 	 
; 	 
Posted in Liferay |

Write log from Liferay velocity templates

A simple example for writing log entries from Liferay velocity templates

#set ($logFactory = $portal.getClass().forName('com.liferay.portal.kernel.log.LogFactoryUtil'))
#set ($log = $logFactory.getLog('myTemplate'))
$log.info('Here we go again')
Posted in Liferay |

Visio 2013 disable rerouting

Drop something to the drawing surface, all other shapes freak out, running away from your new shape which you just wanted to throw in the background. Sounds familiar?

If this something you don’t like, then disable connector splitting. Head to File-menu, select options. Then go to advanced and uncheck the “Enable connector splitting”.

visio_connectorsplit

(The title might not be accurate but that’s the question I was trying to ask from Google)

Posted in Misc |

Disable Tomcat thread pool

I recently ran into issues with a third party application that had certain issues with threadlocals. Namely certain parts of the application were using threadlocal to store information, but did not clean up the stored information. This information then caused issues on other part of the application. Now this certainly was a bug in the application, but fixing the actual bug was not possible in the given time frame. I also did not want to introduce some custom filters/valves that could have been used to solve the issue.

The first idea was to disable thread pooling all together and to create a new thread for each request. This would have some kind of overhead, but it would solve the issues with thread local. There is good discussion on Stack Overflow about the performance issues related to creating threads. Result from some testing was that the overhead for most cases would be negligible. This testing obviously only covered pure threads, Tomcat might be doing some things with threads which have impact on performance if thread pooling is disabled.

Disabling thread pooling is not easy, as there does not seem to be direct configuration options for this. There are certain thread related configuration settings for the connectors, but these does not seem to allow disabling pooling. Also there are some reports that the threads never die.

A solution seems to be to switch to using Executors. Executors have been added to Tomcat to give better control over thread pools. Each Executor has its own thread pool and connectors can be associated with executors. This means you can either make multiple connectors share a thread pool or you can create separate thread pools for each connector.

With executors you have a little bit more control over the tread pools. The settings are still not comprehensive enough to really disable pooling of the threads, but it seems to be you can get pretty close. My solution was to make the threads as short lived as possible. So minimise spare threads and make sure the threads are killed as soon as they are released.

<Executor name="myThreadPool" 
          namePrefix="my-" 
          maxThreads="100" 
          minSpareThreads="0"
          maxIdleTime="1"
/>

What this means is that by-default we want all threads to die. We don’t want to have threads waiting to serve incoming requests. Once the thread has been idle for 1ms, it should be thrown away.

Associating a connector with executor is fairly simple. Just add executor parameter to the connector. If the executor specified exists, then the thread settings in connector are ignored and executor is used. If the executor specified does not exist this parameter is just ignored and connector settings are used.

There is one caveat related to AJP connectors, which is pretty well documented by Haxx. Namely certain AJP connectors only support limited set of configuration options. If you are using just “AJP/1.3″ in the protocol field of the connector, you may end up with JkCoyoteHandler which does does not support the executor parameter. You can go around this issue by explicitly specifying org.apache.coyote.ajp.AjpProtocol in the protocol field.

One more thing to check is the keepAliveTimeout in the AJP connector. This specifies how the connector will wait for another AJP connection before closing down the connector. Default value is the same connectionTimeout attribute. The default value for connectionTimeout is to wait forever. So it seems to be that if you omit both settings, then the connector will keep the thread running forever while it is waiting for new connection.

<Service name="Catalina">
  ...
  <Executor name="myThreadPool" 
            namePrefix="my-" 
            maxThreads="100" 
            minSpareThreads="0"
            maxIdleTime="1"
  />
  ...
  <Connector port="8010" 
             executor="myThreadPool"
             protocol="org.apache.coyote.ajp.AjpProtocol" 
             keepAliveTimeout="1" /> 
  ...
</Service>

In my tests this did not completely eliminate the thread pool. If requests are coming in quickly enough, the same thread may be used to serve two requests.

In order to debug things you might want to enable access log to see which threads are actually handling each request. You can do this with access log valve.

<Valve className="org.apache.catalina.valves.AccessLogValve" 
       directory="logs" 
       prefix="localhost_access_log." 
       suffix=".txt" 
       pattern="%h %l %u %t %I %r %s %b" 
       resolveHosts="false"/>

Important part is %I which includes the thread name. You can verify that Tomcat is using your executor by checking that the thread names match then name prefix specified for executor.

Posted in Misc |