Fer-de-Lance. Intelligence for the free desktop

Wolfgang Müller

1. Our vision: the content-aware desktop

It's simple: while current file managers help you browse the (hierarchical) file system, and some content-based desktop search engines help you find text files (much faster than findgrep, like e.g. ht://dig), a content-aware desktop will help you find files of any kind.

The base technology is already there. I will speak about this below.

1. Find some file by visual browsing through thumbnails
2. Find a piece of program containing a given construct
3. Find all the gnumeric tables in which a certain function was used
4. Find an image that is visually similar to a given image.
5. Find all html (sgml/LaTeX ...) files containing images similar to a given image, and containing the word "desktop".
6. Find all html (sgml/LaTeX ...) files which are visually similar to a given file if rendered in a browser.
7. Instead of a purely history-based "recently used document" menu, provide a "related documents" menu on the desktop

2. The Toolbox

In this section I outline which tools need to be combined to build a content-based desktop.

2.1 The two big free desktop projects GNOME and KDE

These two projects, probably are known to any one reading this document. The competition between the GNOME and KDE projects has helped create two desktops which combine great looks with great day-to-day usability, configurability and the choice between interactive tools and command line tools. Both projects provide combined web/desktop browsers that will give a great basis for using FdL services. However, a flexible content-based search engine for files other than text is missing.

2.2 Content-based search engines for text

Much work has been invested in successful search engines for text. There are several free ones on the market, like

• Swish++
• ht://dig
• medusa
• mifluz

2.3 The GNU Image-Finding Tool, MRML, and theSnakeCharmer client

The heart of FdL, and the expertise I can offer, is the GNU Image-Finding Tool (developed at the University of Geneva, see http://www.gnu.org/software/gift). Most of the brains needed to make FdL a success you can already see at work in the GIFT demo (see the GIFT 0.1.6pre (server) and SnakeCharmer 0.2b (client) at work under the URL http://viper.unige.ch/demo.html. The surrounding pages give an overview about other research in content-based multimedia retrieval).

What's great about the GIFT is that the search method employed is the complete analog of what's done in text retrieval (this David Squire's idea, who founded the Viper project). While the GIFT is currently specialized in still images, its search method makes it simple to combine textual and image/sound/video search. Furthermore, the GIFT provides an open framework that allows integration of new query engines.

However, the current version of the GIFT is conceived as a retrieval engine. Indexation has to be done offline, there is no user/group/others rights administration etc., to speak of just a few things that have need to be done.

3. Combining the bits for making Fer-de-Lance a success

According to my analysis we would need to do the following:

• Integrate a client for the GIFT into Nautilus/Konqueror. The protocol used by the GIFT, (MRML, see http://www.mrml.net) is XML-based and designed for extensibility. In my opinion it would be an advantage for the Free Software community to use this protocol for communication with the search engine, for two reasons:
  • MRML is easy to port, as it doesn't need an ORB or desktop to work
  • Currently the group at the University of Geneva in which I am working is working hard for attracting other research groups towards the use of MRML. Given this effort will be a success, FdL will benefit from this directly, as fresh retrieval engines from all over the world become plugins for FdL.
• Work with the GNOME/KDE people to find an API which e.g. will help FdL find out which functions where employed by the user for generating a given file.
• Work with GNOME/KDE people for adding "what's related" functionality to the "file" menu of your desktop
• Extend MRML to accommodate FdL functionality (adding items)
• Build a flexible feature extraction framework. Presently, the number of features and the kind of features that are extracted are determined offline, we need a framework which does this online to give query engines the chance to require new features under certain circumstances. While being a difficult task, this task is rather of technical/software-engineering nature. Free software people tend to find solutions to these problems :-)
• Adapt the current GIFT query engines to use the new feature extraction framework, and extend them to incorporate multiple levels of documents (e.g. and HTML document contains Images etc.), based on work that is already published, but where the first prototype was insatisfactory

4. Help wanted!

All the bits described above need people, if possible good people. The more there are the faster we will get something "cool", a unique combination between cutting edge science, first-class software engineering and the free software spirit.

5. Companies and Universities

We hope to attract universities and R&D departments of enterprises to join our effort. Major contributors will be cited on our credits page (minor contributors will be in the AUTHORS file, as always)

6. Combining bits? What's new, then?

In the last section, I have been speaking a lot about combining and extending existing software. What's new, then, in this project?

It's looking at the big picture in a different way

I've been preparing this announcement for quite some time, yesterday there was somebody (user quarnap not related to me) on Slashdot commenting on Eazel, expressing what I would call Fer-de-Lance goals in quite harsh language:

I've had just about enough desktop icons and file managers. Is this it? Is this all there is? Is this the future of desktop computing--more of the same, but prettier? But it does themes!!! OOooohhhh, I'm impressed now! I don't give a shit about file managers. I'm sick of having to track every file I create. Why can't the computer do that for me? Why do I have to give a name and location to every single document I create? Why can't the computer do that? Isn't it smart enough to see what I'm typing, tell what sort of document I'm creating, and act accordingly? Here is my idea, and keep in mind, I'm not a programmer.

• IMHO, URLs and other forms of names have been proven are useful for giving pointers to any kind of objects: consider saying "great free operating system" instead of simply saying "GNU/Linux", and how would you draw the distinction with FreeBSD?
• for many cases hierarchical organization is very useful, but not every hierarchy fits all purposes (this is why we have softlinks)
• sometimes hierarchies are not adapted at all (this is why I have my paperwork organized as heaps on my desktop in my office ;-) ).

FdL will enable the user to rely filenames where appropriate, and to use content based retrieval where this is faster.

7. Peace with you

I have been writing about GNOME and KDE. I do not intend to cause flame wars. This project is about creating something unique and extremely innovative within the free software world, one of the first times that a new consumer product is invented and not reinvented in a free software process. For making this a fast success, bundling our activities seems to be the call of the hour. It will benefit everyone if FdL services are easy to port from GNOME to KDE and vice versa, as it will be of advantage to use the same protocol for communication between the search engine and the desktop. Like that, we are more likely to attract researchers from other universities and research labs to contribute their newest research to our effort.

Please help us to show that the time has come for the intelligent desktop.