Carl's notes

I’ve been told I will have a very large file to import into MySQL, 160 million rows. I don’t have the file yet but I wanted to see how my queries would work with such a database so I generated some content. It’s a database of SHA1 hashes which I will store as strings of length 32 in base 32 (26 letter alphabet and digits 2-7). In base 16 (0-9 and A-F), they would be 40 characters long.

First task is to generate the data. I first wrote some SQL queries to generate random strings (stored procedure, RAND(), array of valid characters). Based on some smaller scale tests, I estimate this would take 80 hours on an InnoDB table. Much too long, so I tried taking the SHA1() of a random number. This was estimated to take 47 hours on an InnoDB table but only 75 minutes on a MyISAM table. Apparently, this is due to InnoDB keeping a log of everything so it can roll back the transaction if something goes wrong. Still too long so another option I tried was to generate the entries with a short Python script using hashlib.sha1(str(i)).hexdigest() where i is an int between 0 and 159,999,999. This only took about 9 minutes and the total file size was 6.5 GB.

Second task is to get the data into the database. InnoDB is too slow. MyISAM took about 10 minutes to import the text file (one value per line) using the load data infile local command. This is conveniently handled via the GUI in HeidiSQL. The total database size was 22 GB, but I think I made the char field a little too big (40 characters long instead of 32).

Searching for a single value in this table took close to 4 minutes, so I added an index on the only column. The index took several hours to generate and added 7 GB to the table size. The basic select query now takes less than 0.1 seconds.

The next day, I wanted to repeat this and see if increasing the value of the myisam_sort_buffer_size variable would speed up the index creation. Unfortunately, I created an InnoDB table (by default) and didn’t change the engine to MyISAM before starting the bulk import. Several hours later, I killed HeidiSQL, then loaded the MySQL command line client, ran show full processlist and kill 2 where 2 was the process id of my import query. This didn’t stop the constant hard drive access because the InnoDB engine then started to roll back the import. Apparently, this can take even longer than the import itself and is impossible to stop even if you stop and start mysqld or the whole server. The rollback operation always continues automatically when you restart the server. This isn’t quite true and I came across a page in the documentation which explains that you can set innodb_force_recovery to the value 3 in the configuration file to prevent rollback. Then you restart the server, go to the database and drop the problematic table, delete the innodb_force_recovery setting or put it back to zero, then restart the server again. There were a few issues with the server not stopping correctly and I had to reboot a few times but it did work and I was able to access the database again.

Sorry for the lack of references.

The MySQL data files and logs (slow log, global log, error log…) are under C:\ProgramData\MySQL\MySQL Server 5.5\data. The ProgramData directory is hidden so you need to change the folder settings. There is a data folder in C:\Program Files too but it doesn’t seem to be active.

The university I work for provides virtual servers using VMware. I was given one to use as a web application server running Glassfish. I encountered a few difficulties along the way.

First, I couldn’t get the server to boot from my Debian netinst ISO file. I was able to mount it while the server was booting, using the vSphere client, but I never got there fast enough for it to boot from it. I tried rebooting (from inside vSphere) but that disconnects the ISO. With a bit of help from a colleague, I learnt that you can reboot with ctrl-alt-insert (translated to ctrl-alt-del) and that this doesn’t unmount the disc image.

Second, I assumed that I needed the IA64 release of Debian since the virtual machine was 64-bit and Intel-based. However, I then learnt that IA64 is only for Itanium processors, and that most other 64-bit CPUs need the amd64 release, even if they are Intel. I was then able to install Debian.

Third, the keyboard layout of the virtual machine is independent of my laptop’s layout (Dvorak), so I assume vSphere client accesses low level key presses. Configuring keyboard layout in Debian is achieved via dpkg-reconfigure keyboard-configuration.

Fourth, there wasn’t a package for Glassfish on Debian. Well, there are a few packages with that name but they are out of date and only provide some jar files. Fortunately, Glassfish can be installed easily by hand within a single folder. Create a new glassfish user, set it up with a home directory, su to that user and run the installer in silent mode. The default settings worked for me and everything was installed in one subdirectory of the glassfish home directory. There is a way to run the installer with the -n option on another machine (with a GUI) and create a configuration file that can then be used on a headless machine like the one I was using. I couldn’t get that option to work on Windows so I just let it install with the default options. I expect you can configure it later anyway. If you want to access and manage it via Netbeans, you need to set an admin password. Apparently, it can’t contain any spaces. Have a look at the asadmin enable-secure-admin command and asadmin change-admin-password --user admin command. asadmin is in the glassfish installation folder so you will need to call it using its full path. I found a very useful page by Nabi Zamani about installing Glassfish on Ubuntu. Not exactly Debian but close. The start up script and iptables scripts are very helpful, although by searching a bit more you can find init scripts that include the dependencies and other meta-info at the top (avoids the OS complaining about missing LSB data). If glassfish doesn’t start or iptables isn’t making it accessible on port 80 (instead of the default port 8080), check that executable permissions are set correctly on the init and network files mentioned in the link above. There’s a password file in glassfish/domains/domain1/ which needs to be readable only by glassfish. I made the mistake of running some command as root and messed up the permissions. Also make sure that iptables has loaded the rules that should have been saved into /etc/iptables-save.rules (or similar).

Finally, I wanted to install the VMware tools. No idea what they do but apparently they increase performance of the client OS. Couldn’t find where to download them or how to install them until I came across Wolfsrudel’s comment on debiantutorials.com. He (male judging by the very small photo) explains that the tools are available after you add non-free and contrib to your apt/sources.list file. Then run the following as root.

1. apt-get update
2. apt-get install module-assistant open-vm-source open-vm-tools
3. module-assistant prepare
4. module-assistant build open-vm-source
5. module-assistant install open-vm-source
6. reboot

More trouble getting decent figures out of Matlab. Solved a few of my problems.

Dashed and dotted lines look weird when exported to EPS. There are several scripts available on the Mathworks FileExchange. One that I’ve just tried by Oliver Woodford, and an earlier one mentioned by Loren Shure in her article on making pretty figures. You could edit the EPS files manually to change the parameters of the lines but these scripts will do it for you.

The PNG export appears to suffer from the same problem regarding dashed or dotted lines but without the option to modify the final file (because it’s not a vector format). Since the EPS files look good, I might as well convert them to PNG. Using ImageMagick for batch conversion under Linux wouldn’t be a problem but I’m currently using Windows. But it’s just as easy and with the mogrify command you won’t even need to write a loop. Open a command terminal, change to the directory where your files are, and run mogrify -format png -resize 2048 -density 600 *.eps. The resize option determines the dimensions of the PNG, and the density option ensures that it’s not blurry. The PNG files will be named the same as the EPS files but with their own extension.

One of my figures needed to be rotated 90 degrees to look better on the screen and make exporting it easier (although it’s perhaps not absolutely necessary for it to be entirely visible on the screen). I didn’t want the trouble of changing all my coordinates. Running camroll(90) has pretty much the desired effect but messes up the axes’ labels. view(90,90) worked better for me. Thanks andyras and Rachel on StackOverflow. You may need to set limits and labels after performing the rotation.

Just started using MS PowerPoint for a presentation. I must admit it is more pleasant to use than OpenOffice Impress was last time I tried and the feature I describe here doesn’t seem to exist in OpenOffice 3.

In this particular presentation, I wanted to insert a dozen screenshots from a video game. I wanted these pictures to appear on successive slides in exactly the same position and size. Doing this by hand is slow because you need to insert the pictures one by one and adjust each of their positions and sizes using the text fields. In PowerPoint 2010, you can insert a photo album. This is exactly what I was looking for. It allows you to select any number of images and create a new presentation with one image per slide. I don’t know what happens if the images are different sizes but it worked well for my screenshots. You can configure a few options, such as whether to have a border around the image and whether to include captions. I needed to copy the slides from the newly created presentation into my original one but that’s hardly a problem.

Windows includes a Dvorak keyboard layout but it doesn’t allow you to type accented characters (directly or using dead keys). A guy called Arjen has created a layout which allows you to access all the additional symbols that I can imagine wanting to type. Download the zip archive from his site and run the setup.exe contained in one of the folders. You will need to be admin on your machine. After confirmation, it will automatically add the custom layout to your selection.

The layout is illustrated and described on his website. It’s not quite the same as the Dvorak layouts included with most Linux distributions but it looks quite intuitive.

When you use Subversion (or other version control systems), be sure to use them consistently. Checked out directories contain a hidden directory containing information about the repository, the original checked out files, which files have been deleted or added… Copying the complete directory to a different location or even a different computer might work seamlessly, copying individual files is also acceptable. But as soon as you copy subdirectories (with their special hidden directory) that have been updated or committed on another computer, the upper directory and the subdirectory become inconsistent as far as version control is concerned. You will get messages asking you to clean up, and claiming that some subdirectories are not under version control.

• Best practice is to use the version control tools to exchange files: commit on one computer, checkout on another, make changes, commit, update on the first computer.
• If individual files need to be processed on another computer (eg. batch converting images from one format to another on a *nix machine) then proceed as above, or select all and copy the individual files, not the whole directory.
• If you need to provide the whole version controlled directory to someone who doesn’t need version control (eg. sending LaTeX sources to an editor), use the export feature to create a clean directory tree without all the hidden version control subdirectories.

A collaborator sent me an ASP webapp to demo some of his ideas. As a non-Windows user I struggled to get it running. But it’s not that difficult.

1. Get a Windows computer (I’m using Windows XP Professional). You’ll need the installation CD that goes with it if the IIS webserver isn’t already installed.
2. Install the IIS webserver. Go to the Control Panel, Add or Remove Programs, Add/Remove Windows Components. Check the IIS box, then click Next and follow the instructions. You’ll need to insert the installation CD.
3. Extract the zip file I received to C:\Inetpub\wwwroot. This should create a new folder which we’ll call MyWebApp. This folder has several pairs of .aspx and .aspx.cs files with matching names. The .aspx files seem to contain the web page HTML with a bit of ASP (similar to the way PHP pages work). The .aspx.cs files contain C# functions that are used by the .aspx pages. The App_Code subfolder contains some .cs with a few C# function definitions. The App_Data subfolder is empty. The Bin subfolder contains a .dll, specifically LinqToWikipedia.dll. I found I needed to replace this DLL with one downloaded from the official LinqToWikipedia website.
4. Configure IIS. Control Panel, Administrative Tools, Internet Information Services, left column, Web Sites, Default Web Site, MyWebApp. Right
   click, Properties. Check the Directory Browsing box. Click the Create button next to Application Name in the lower half of the page. The Application Name box should now display ‘MyWebApp’ and the button should change to read Remove. Validate changes with OK.
5. Point a browser to http://localhost/MyWebApp. You should be able to view all the .aspx pages.
6. Some pages need editing. Visual Studio not required but gives nice syntax highlighting. No compilation required. Just save the modified file and refresh the corresponding page in the browser.

Having spent a long night calibrating Ubisense, here is a checklist to make it faster next time. This is with version 2.0.4 of the software. Many thanks to our resident surveying expert Yukang for staying very late to help me out with this.

Doublecheck offsets to apply to surveyed points. Get signs right. Remember which points were surveyed with the reflector pointing up or down, take into account the height of the totalstation.

Take into account the height of the antenna inside the tag. This may add 1-2 centimetres to the height of surveyed points.

Check that the antenna boards inside the sensors haven’t come loose. If they have you need to open the case and clip them back into place.

Check that the sensors are horizontal, roll should be zero.

Optimal pitch of the sensors is 45 degrees downwards. I seem to not point them down enough so should try to emphasise this in future.
Read more…

Some computers come with Norton Antivirus and Internet Security preinstalled but with no uninstall link in the start menu and no entry in the add/remove software list. Symantec provide a removal tool that will cleanly uninstall their software.