On sorting files... part 1

For months now, I've been looking for a way to automaticaly sort my files. There is a lot of software for creating and maintaining some sort of media library for pictures, movies or music, but there is almost nothing else for "the rest". On my home computer, "the rest" is composed of 2600 files that occupies a grand total of 45G (although 23G of that is composed of iso files of Linux and BSD). Almost all of these files were downloaded from a website somewhere, either application from sourceforge.net, game add-ons from The Elder Scroll Nexus , Fallout 3 Nexus or Simtropolis or drivers from various hardware manifacturers.

Right now, the way I manage these files is mostly by hand. Everything goes into a "Download" folder (that I have since my BBS days), and it's then sorted by hand into several different folders... and that folder is in a perpetual "TO DO:not completely sorted" state... (since those BBS days...)

I wish I could just simply drop these files in a folder where they would automatically get sorted...

...And I don't think it's impossible to implement. Each of the files I have specifically mentioned comes from websites where the content is already classified. Sourceforge classifies projects among various categories and so does all the gaming communities, with Mods, maps, etc. The only thing required is a method of exchanging metadata.

Seems easy enough, but it's not. When you start to think about that problem you realize quickly that it's not really a technical problem. It's more a problem of getting everyone to work together with the same standard.

I have prepared a series of blog posts for each of part of the problem, a sort of brain dump of what I came up with in the past months. A single post would create the "Wall of Text" or TL;DR (too Long; Didn't Read). (And if I try to write something too big, as the Thinkgeek t-shirt says "I never finish anyth")

Choosing Software

Wikipedia has really changed the way I do things... And I'm not talking about articles, I'm talking about choosing software.

Let me explain with an example. I recently installed Windows 7 on my computer (see previous post) and I wanted to try a new Media player. I've always been running Winamp and wanted something different...

The Wikipedia article on Winamp points to both a List of Media players and a Comparison of Media players. Using these lists, I found out that the best choice for me was Songbird. I also took notes of aTunes (and of course Winamp). I decided to use Songbird but I might change that decision because Songbird some interesting bugs on Windows 7 x64 (unsupported platform).

The same exercise can be done with CD burning Software points to a List of optical disc authoring software, which, in turn points to both CDBurnerXP and ImgBurn, both applications I have used in the past. It also shows that there is a CD Burning Open Source application on Windows called InfraRecorder (which I had never heard about).

I find this little "technique" of using "list of" article in Wikipedia very convenient.

Re-installing Windows

I have a book behind me called "Backup & Recovery" from O'Reilly Publishing. I have learn a very basic yet very important lesson from that book... And If my memory is right it goes something like:
"Your backups are as good as your last restoration attempt."

The whole concept being that even though you have backups, unless your actually tried to restore those backups, they don't mean anything.

Since:

• Microsoft came out with an RC of Windows 7
• I ran Windows XP which could only see 3.5G out of 8G of physical RAM
• I'm been making backups of everything on my computer

I decided it was time to test that backup strategy with a fresh install of Windows. Of course, because Windows 7 RC will shutdown in 2010, I will have to re-do this process all over again.

Thus, I am writing here everything that I have to do in case I need to do it again. (This posting will be updated if anything changes during the year)

This Machine is use for: Gaming, Web, E-mail and the occasional programming in C++ / QT.

List of software I need to get online (in alphabetical order):

• 7zip
• Adobe Air
• Allsnap
• ClipX
• Boinc
• Dropbox
• Firefox
• Flash player
• Foxit Reader
• gnucash
• Launchy (no 64bit version, but 32bit seems to work)
• Nokia Qt SDK
• OpenOffice.org
• Pidgin
• Picasa
• Process Explorer
• Putty
• QDevelop
• SMplayer
• Steam
• Thunderbird
• VirtualBox
• Vlc
• WinDirStat
• WinSCP
• Xming

List of Backup procedures:

• Backed-up "Documents and Settings" Folder from Windows XP
• Backed-up "My Documents" Folder from Windows XP
• Backup list of mods in Oblivion Mod Manager and Fallout 3 mod Manager
  
• Imaged the C: Partition with CloneZilla (just in case)

List of Restore procedures:

• Moved the "My Documents", "My Pictures", etc as sub-folders of "D:\My Documents"
• Show File Extentions. Always. (Control Panel -> Folder options)
• Add "path=C:\Qt\2009.02\mingw\bin;C:\Qt\2009.02\qt\bin" variable in Control Panel -> System -> Advance -> Environment Variables
  
• Re-installed Firefox extentions (including: Xmarks, Noscript, Delicious bookmarks, IE tab ,etc)
• Restore Thunderbird directories:
  

1. Documents and Settings\Ve2dmn\Application Data\Thunderbird to C:\Users\Ve2dmn\AppData\Roaming\Thunderbird
2. Documents and Settings\Ve2dmn\Local Settings\Application Data\Thunderbird to C:\Users\Ve2dmn\AppData\Local\Thunderbird

The rest is mostly tweaking.

I guess I have proven to myself I'm somewhat protected against data loss now... Eventually I will have something like Time Machine, TimeVault or Flyback for Windows, but for now, I guess this will have to do :)

Bad English to French Translation...

Living in Quebec we sometime get very bad English to French mistranslations... Things like:
"Lamp Oil" translated to "Lamping Petrol"
"Made in Turkey" translated with "Made with turkey meat"
But the best one yet is on the washing instruction for one of my shirt.

Original text:
"See reverse for Care.
-Machine Wash cold with like color only
-Non-Chlorine bleach when needed
-Hang or Tumble dry low, Remove Promply
-Cool Iron when needed
-May be dry cleaned
-Do not Iron Decoration"

Was Translated to(For the French impraired):
"See Contrary for love
-The machine washes the cold with the loved color only
-The decoration not-chlorine when necessary
-Hang or fall dry, the low level removed on time
-Cool down some iron metal when necessary
-Cleaned. The month of May is dry.
-The decoration is not made of Iron"

 
I though I had seen it all, but this is just a bit too much in my opinion.

Small live stream recorder with Qt

I decided to try to write a small App with Qt (no, quicktime, but the Nokia toolkit, Qt)

The result is the screenshot you see. It's an app called, the "Twit Live Recorder V0.1 (Proof of Concept)". It's a live stream preview and recorder, with the ability to "split" the current file (it create a new file and add -1 or -2, -3 and so on...). It is very rough code, but it works. (you could probably crash it if you try hard enough)

Note: I took Twit Live as an example because I know that Leo gaves his O.K. to have his stream recorded (ODTV is great! You guys rules!). This could probably apply to any live stream that can be recorded with 'wget' or 'curl'. I won't give the urls here because I have no idea if it violates the terms of service for Stickam, Ustream or Bitgravity (or even Justin.tv). As always, uses your jugement and don't do anything illegal.

Also note that simply copying and pasting the code will probably not work. I removed some characters that were interpreted as htnl code and replace them with "quotes". If someone wants the original files, I could post them somewhere.

Final Note: This was done for fun, as an exercise. I'm probably not going to maintaint this application, as I don't see the use for it. (The OdTV people are already working on automation...) I'm not a full-time C++ developper and probably lack the experience to answer. If you have C++ or Qt Question you are probably better off asking it on a forum dedicated to that like StackOverflow.

The Code Behind the App
I used QDevelop to create the App, but it could probably be built with QtCreator, Kdevelop, MS Visual Studio or the command line.

The entire app is basically 5 files: a .pro project ( Twit Live Recorder.pro), a main.cpp , the .ui file (twit_live_recorder.ui), and the .cpp and .h of the main object (twitliverecorderimpl.cpp and .h )

The .pro "project" file:

TEMPLATE = app
QT += core webkit network
FORMS += ui/twit_live_recorder.ui
SOURCES += src/twitliverecorderimpl.cpp src/main.cpp
HEADERS += src/twitliverecorderimpl.h
CONFIG += build_all debug_and_release
TARGET = Twit_Live_Recorder

The file is pretty straightforward. Build a Qt app, with modules "core", "webkit" and "network", with name "Twit_Live_Recorder" (You can also see that I like to build both debug and release.) The qmake from Nokia can turn this file in the coresponding Makefile for the correct architecture and OS.

The main.cpp is also pretty straightforward:

#include "QtGui/QApplication"
#include "twitliverecorderimpl.h"

int main(int argc, char *argv[])
{
QApplication a(argc, argv);
TwitLiveRecorderImpl w;
w.show();
return a.exec();
}

Include the "Twit Live Recorder Implementation" called twitliverecorderimpl, create a QApp (to handle most of the OS stuff) , create a TwitLiveRecorderImpl dialog and show it.

I won't copy the content of the .ui because the XML code is not meant to be read. I can tell you it's a QMainWindows with a QWebview as a central widget (to get a preview of the stream). Also includes a QLabel and QLineEdit at the bottom to list the current file. Finally, the main windows includes a MenuBar and Toolbar with the following Qaction: "Record", "Split", "Stop","Quit", "About", and "About Qt"

Now on the the meat of the app, the Dialog Implementation. First the .h:

#ifndef TWITLIVERECORDERIMPL_H
#define TWITLIVERECORDERIMPL_H

#include "QMainWindow"
#include "QFile"
#include "QHttp"
#include "ui_twit_live_recorder.h"

class TwitLiveRecorderImpl : public QMainWindow, public Ui::TwitLiveRecorder
{
Q_OBJECT

public:
TwitLiveRecorderImpl( QWidget * parent = 0, Qt::WFlags f = 0 );

public slots:
void Http_error(bool error);
void Http_stateChanged (int state);

private slots:
void on_action_Quit_activated();
void on_actionAbout_activated();
void on_actionRecord_activated();
void on_actionSplit_Current_Recording_activated();
void on_actionStop_Current_Recording_activated();

private:
QString Host_url;
QString Stream_url;
QString BaseFileName;
QString ActiveFileName;
int SplitNumber;
QFile *ActiveFile;
QHttp *CurrentConnection;

};
#endif

As you can see, this object includes both a QFile and a QHttp private objects. What we do to save the stream is simply open a Http connection to the server and pass the QFile as the destination IO device. The on_action_ slots are function the Qt precompiler will automagically connect with the button on the UI. The Http_error and Http_stateChanged are there mostly to help debugging the Http stream.

Finally, the .ccp file. Let's go trough it one function at a time. First the header and Object Constructor:

#include "twitliverecorderimpl.h"
#include "QMessageBox"
#include "QWebSettings"
#include "QFileDialog"
#include "QFileInfo"
#include "QDate"
#include "QDebug"

TwitLiveRecorderImpl::TwitLiveRecorderImpl( QWidget * parent, Qt::WFlags f)
: QMainWindow(parent, f)
{
setupUi(this);

//Set Urls
webView->setUrl(QUrl("http://live.twit.tv"));
Host_url = "Server.whatever.com";
Stream_url = "http://Server.whatever.com/stream-url";

//Create a New Connection (unused until the record button is pressed)
CurrentConnection = new QHttp();
CurrentConnection->setHost(Host_url);

//Set Filename. Default file name looks like: "Twit-2009-03-25.flv"
BaseFileName = "Twit-" +
QDate::currentDate().toString("yyyy-MM-dd");
ActiveFileName ="";

//Split Number to have Twit-2009-03-25-1.flv, etc.
SplitNumber = 0 ;

//Enable Plugins so we can have Flash working
QWebSettings::globalSettings()->setAttribute(QWebSettings::PluginsEnabled,true);

//Connect to "About QT" in the menu
connect(actionAbout_Qt,SIGNAL(activated())
,qApp,SLOT(aboutQt ()));

//Connect Signal from http for debugging
connect(CurrentConnection,SIGNAL(done(bool))
,this,SLOT(Http_error(bool)));
connect(CurrentConnection,SIGNAL(stateChanged(int))
,this,SLOT(Http_stateChanged(int)));
}

Set the inital Values of each Variables as well as connect every signal not automagically connected.

Functions for the "Quit" and "About" Menu Options:

void TwitLiveRecorderImpl::on_action_Quit_activated()
{
close();
}

void TwitLiveRecorderImpl::on_actionAbout_activated()
{
QMessageBox::about( this, tr("Twit Live Recorder"),
tr("Twit Live Recorder v0.1(Proof of Concept)"
"Using QT"));
}

Function to help with Debugging the http connection:

void TwitLiveRecorderImpl::Http_error(bool error)
{
if(error)
{
//Debug Code goes here
}
else
//Debug Code goes here
}
void TwitLiveRecorderImpl::Http_stateChanged ( int state ) { //Debug Code goes here
}

3 Function are left for "Record", "Stop" and "Split". First the "Record" function:

void TwitLiveRecorderImpl::on_actionRecord_activated()
{
ActiveFileName = QFileDialog::getSaveFileName(
this, tr("Please Select File Name"),
"/"+ BaseFileName + ".flv");
if(ActiveFileName == "")
return;

//Change the look of the Buttons:
actionRecord->setDisabled(true);
actionSplit_Current_Recording->setDisabled(false);
actionStop_Current_Recording->setDisabled(false);

ActiveFile = new QFile(ActiveFileName);

//Open Connection to server
CurrentConnection->get(Stream_url,ActiveFile);

//Set Visible "Currently Recording"
Line_CurrentFileName->setText(ActiveFile->fileName());

SplitNumber = 0 ;
}

The important part is the File dialog where the user choose the name and location of the file, that file is then passed-on as a destination for a new connection. The button are also activated or deactivated, depending on which one. The "Stop" function does things in reverse:

void TwitLiveRecorderImpl::on_actionStop_Current_Recording_activated()
{
//Abort Connection to Server
CurrentConnection->abort();

//Reset Displayed File Name
Line_CurrentFileName->setText("");

//Change the look of the Buttons:
actionRecord->setDisabled(false);
actionSplit_Current_Recording->setDisabled(true);
actionStop_Current_Recording->setDisabled(true);
}

Finally the "split" Function:

void TwitLiveRecorderImpl::on_actionSplit_Current_Recording_activated()
{
QFile *PreviousFile = ActiveFile;
SplitNumber++;

//Create new File name based on the Active File
QFileInfo OldFile = QFileInfo(ActiveFileName);
QString NewFileName = OldFile.absolutePath() + OldFile.baseName() + "-" + QString::number( SplitNumber, 16 ) + "." + OldFile.completeSuffix();

//Next File is the new Active File
ActiveFile = new QFile(NewFileName);
ActiveFile->open(QIODevice::WriteOnly);

//Close the Old Connection and open a new one
CurrentConnection->abort();
CurrentConnection->setHost(Host_url);
CurrentConnection->get(Stream_url,ActiveFile);

//Set Visible "Currently Recording"
Line_CurrentFileName->setText(ActiveFile->fileName());
PreviousFile->close();
}

This function opens a new file based on the previous name. Right now it is probably pretty buggy: It does not check if the file already exist nor give any sort of feedback, but the basic functionality is there.

It was a fun app to write and I was surprised at how little I had in the end to get a functioning application. Of course, this doesn't show the all the debugging time I spent on it (a ratio of around 1:5 to 1:6). And I should also give credit where credit is due, becasue this would not have been possible with work from Qt and Webkit. I am standing on the shoulder of giant with this sort of work.

International Radio Operator Alphabet

If you work on the phone and must spell out names all the time, use the NATO, FAA and ICAO approved phonetic alphabet:

• Alpha
• Bravo
• Charlie
• Delta
• Echo
• Foxtrot
• Golf
• Hotel
• India
• Juliet
• Kilo
• Lima
• Mike
• November
• Oscar
• Papa
• Quebec
• Romeo
• Sierra
• Tango
• Uniform
• Victor
• Whiskey
• Xray
• Yankee
• Zulu

As a bonus, it works in several languages and it saves the embarrassment of mistranslation when spelling out words: "A as in Ananas"
(Pineapple is a commonly mistranslated word. Aside from the Japanese パイナップル (Literally PA-I-NA-T-PU-RU), almost every other language uses a variation of Ananas, Nanas or sometimes Pina.)

We live in an amazing world, yet nobody is happy

This "viral video" is going around on the internet (It's been taken out of Youtube because of copyright violation from NBC, but can be found elsewhere):

http://vodpod.com/watch/1335606-louis-ck-%E2%80%9Ceverything%E2%80%99s-amazing-nobody%E2%80%99s-happy%E2%80%9D
http://barefootmeg.multiply.com/video/item/56

The stand up comedian has a point. Technology has come a LONG way in 100 years. Yet we have so high expectations that we are never happy with what we got...
Just 100 years ago, the average plane would not go very far... Barely enough for military reconnaissance.
Just 50 years ago, The average computer was AT LEAST several cabinet and had less than 10K of memory, operating well under 1 MIPS.
Just 25 years ago, The average personnal computer didn't have a hard drive in it!
Just 20 years ago, cell phone were all analog.
Just 15 years ago, the website for CBC was created out of HTML 1.0
Just 10 years ago, the first blackberry was released
Just 5 years ago, No intel mac, no Iphones, first color Ipod
Just 4 years ago, Xbox 360
Just 3 years ago, Intel Macs

And yet we complain when, on the passenger seat of a car, an iPhone has trouble finding the nearest gas station in a densely populated area. Yet the poor thing has to compensate for relativity to calculate GPS coordinates, compensate for Doppler shift while registering with 40 differents cell towers along the way and transferring huge among of data, and using algorithms developed over the years to estimate the shortest route, just so we can find cheaper gas.

A spoiled generation :)

Blogged with the Flock Browser

A political question unanwered

Here's is one project I plan not to continue. I simply lost the motivation to do it. The reason I blog about it is that I wish someone will read this and will finally make it work.

But first a disclaimer: This is a really good and bad idea at the same time. It create a political minefield and possibly something controversial.

It's about the electoral map.

It all started back in November. There was talks in the media that the Quebec electoral map wasn't 100% correct. Some district were too populated and the map had to be re-drawn... The process was started but was too slow for it to be done for the upcoming election. It got me intrigued. I'm pretty sure the way the electoral map is drawn influence a lot the outcome of the election. But how? I wanted to find out.

I wasn't the first time this sort of questioning crossed my mind. Back in 2005, while I was still a student in my final year of Computer Engineering degree, I had a course on algorithms and graph theory. The kind of course were you learn that some problems are unsolvable, unless you have infinite time to solve them. I remember asking the teacher this very question: "Can't we use graphs to map were the votes are and tell if the electoral map is unjust?". He answered that it was possible, but no one would be willing to do this sort of stuff because it's too political.

And so, when the medias started talking about the electoral map, I remembered that talk I had with the teacher. I had an idea at the time of how one would analyze the various information sources to determine how well the electoral map is divided. Here's how I wanted to do it:

You make point of interests, containing all the votes in a giving sector. It could be a municipality of any other sort of subdivision. The more there is, the better. For example, in Quebec city, one could create a Point-of-interest at for voting spot, containing all the various boxes that were in that spot. Each spot has a weight based on the number of registered voters and is linked to the rest of the electoral district to form a single district. Once you have this map, you create "potential links", links from each P-O-I to various neighbors in other district (for example, the 3 closest neighbors not in the district). The create the various potential maps, you turn on and off each link until you have a legal, but different map.

OF course, there is a couple of problems with this approach.First, to do all this you need:

• Coordinates for each voting poll
• Number of voters for each voting poll
• Results for each poll
• District for each voting poll

Almost all this information is public, except for one: the exact location of each voting poll.The only information giving for each box is the municipality where the voting poll was. So all the maping information for the voting poll in Montreal, only says : "Location: Montreal". Hard to place the voting voting polls with so little information.

The second problem is the complexity of the entire process. My initial list had 1848 potentials P-O-I on the map. If you limits the links to the 5 closest neighbors, you still get somewhere between 3000 and 9000 links... Which means calculating between 2^3000 and 2^9000 possibilities... Which could take a very, very long time. (2^3000 is over 10^600, or 1 with 600 zeros)

After a couple of days of trying to find ways to optimize the algorithm, I simply gave up. There is not enough information to create a meaningful map, since the location information in large urban area is not very detailed and leads only to a partial answer...

It was a great idea, but I can't go further.

I guess my time is better spent elsewhere...

Hellsing (anime) is a perfect match for E Nomine

The music by E Nomine seem to be a perfect match for any sort of Hellsing AMV.

I found several, but I think this one is far better than the rest (I HAD to share this):