Nutch, in its default configuration, will sort pages by relevance using Lucene scores. You can get Nutch to sort by another field in its index using the sort parameter, but first, you need a field to sort by. (For an example of how sorting works, add &sort=url to the end of your Nutch query. Obviously sorting by url isn’t very useful, but you can see how the query works. Note that you can also add &reverse=true to sort the results in reverse order.)

Sorting by date in Nutch essentially involves two parts: first, using a plugin to get Nutch to index dates into a new field in your index, and second, getting your query page to add the additional parameter to search queries.

Indexing dates with a Nutch plugin

NOTE: I figured out most of the information here from the tutorial on writing a Nutch plugin available on the Nutch Wiki. If you need more detailed information on any of the steps here I recommend reading that tutorial.

All Nutch plugins implement an interface. These interfaces are called extension points. The Nutch Wiki has a list of available extension points. The goal of our plugin is to add an additional date field to the Lucene index, so we want to implement the IndexingFilter interface.

Nutch plugins consist of xml description files and your Java source code files. File structure is as follows:

/nutch-dir
  /src
    /plugin
      /yourpluginname
        build.xml
        plugin.xml
        /src
          /org
            /apache
              /nutch
                /yourpluginname
                  ImplementationClassName.java

Here’s an example build.xml file:

<?xml version="1.0"?>
<project name="yourpluginname" default="jar">
  <import file="../build-plugin.xml"/>
</project>

And here’s an example plugin.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<plugin
   id="yourpluginname"
   name="longer/human-readable name of your plugin"
   version="0.0.1"
   provider-name="your_website">

   <runtime>
      <library name="yourpluginname.jar">
         <export name="*"/>
      </library>
   </runtime>

   <extension id="org.apache.nutch.urldateindexer.yourpluginname"
              name="longer/human-readable name of your plugin"
              point="org.apache.nutch.indexer.IndexingFilter">
      <implementation id="ImplementationClassName"
                      class="org.apache.nutch.yourpluginname.ImplementationClassName"/>
   </extension>
</plugin>

NOTE: I had some trouble using my own package name, but it might have been due to another problem. When in doubt I recommend sticking with org.apache.nutch.yourpluginname.

And here’s the code for the Java source file, ImplementationClassName.java. (You’ll want to change it to your own name to match what you put in the plugin.xml file.)

package org.apache.nutch.yourpluginname;

//imports
import java.util.Date;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.Text;
import org.apache.lucene.document.DateTools;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.DateTools.Resolution;
import org.apache.nutch.crawl.CrawlDatum;
import org.apache.nutch.crawl.Inlinks;
import org.apache.nutch.indexer.IndexingException;
import org.apache.nutch.indexer.IndexingFilter;
import org.apache.nutch.parse.Parse;

public class UrlDateIndexer implements IndexingFilter {

  private Configuration conf;

  public UrlDateIndexer() {
  }

  //this is where your code to add a new field to the index goes
  public Document filter(Document doc, Parse parse, Text url,
    CrawlDatum datum, Inlinks inlinks)
    throws IndexingException {

    // Get url from the method inputs
    String urlString = url.toString();

    // Use regex to identify type of date and parse date from url format (code for the TdcDateParser class not shown)
    TdcDateParser tdcDateParser = new TdcDateParser();
    Date publishedDate = tdcDateParser.parseUrl(urlString);

    // Store date in field
    if (null != publishedDate){
    	//create a new field
      Field publishedDateField = new Field("published_date",DateTools.dateToString(publishedDate, Resolution.DAY),
           Field.Store.YES, Field.Index.UN_TOKENIZED);

      //add the field to the index
    	doc.add(publishedDateField);
    }

    return doc;
  }

  public void setConf(Configuration conf) {
    this.conf = conf;
  }

  public Configuration getConf() {
    return this.conf;
  }
}

In the case of the plugin I build, the goal was to parse the date from the url of a page. You’ll need to modify this method to get the date from wherever is appropriate for your search engine. The basic flow is to create a field, add that field to a document and then return the document.

There are a few things to note about the line that creates the new field:

Field publishedDateField = new Field("published_date",DateTools.dateToString(publishedDate, Resolution.DAY),
    Field.Store.YES, Field.Index.UN_TOKENIZED);

• dateToString: More recent versions of Lucene don’t store dates in the index directly. Therefore, you need to use the dateToString method to store dates.
• Field.Store.YES: This stores the text of the field in the index. You wouldn’t do this for a long field that you don’t need to view in its entirety in the search engine (e.g. page content). I believe setting Field.Store to YES is mandatory to be able to sort by a field.
• Field.Index.UN_TOKENIZED: Tells Lucene not to break up the field into individual words, as it would with a page’s content. I also believe this is mandatory to be able to sort by a field.

Now that you have finished creating your files, upload them to the nutch directory on your server if you haven’t already done so. Then, add a line like the following to the build.xml file in the /nutch-dir/src/plugin directory:

  <ant dir="yourpluginname" target="deploy"/>

Then launch ant from the command line in that same directory (/nutch-dir/src/plugin). This should compile the classes and jar file for your plugin and place them in the /nutch-dir/build directory. It should also place a copy of the jar file and plugin.xml file in the /nutch-dir/plugins directory.

NOTE: I had to modify the deploy.dir property in the /nutch-dir/src/plugin/build-plugin.xml file to get ant to put the jar and plugin.xml file in the right plugins directory. Originally it was set to put them in build/plugins. I’m not sure which directory is formally correct, but the /nutch-dir/plugins directory is where Nutch was set to read from on my system. If this is the case for you, just remove /build from the deploy.dir line. My resulting deploy.dir line is as follows:

<property name="deploy.dir" location="${nutch.root}/plugins/${name}"/>

Next, you need to modify the /nutch-dir/conf/nutch-site.xml file to recognize your new plugin. Just add your plugin name to the plugin.includes property. For example:

<property>
  <name>plugin.includes</name>
  <value>yourpluginname|protocol-http|urlfilter-regex|parse-(text|html|js)|index
-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|reg
ex|basic)</value>
</property>

Launch a Nutch crawl of just a few pages to test your new plugin. To see if the plugin was activated by Nutch, open /nutch-dir/logs/hadoop.log and look for a line like the following:

2009-04-17 10:54:27,477 INFO  plugin.PluginRepository - Indexer of dates from URLs (urldateindexer)

If the plugin didn’t show up, make sure your plugin jar and xml files are in the directory indicated in these log file lines:

2009-04-17 10:54:23,617 INFO  plugin.PluginRepository - Plugins: looking in: /nutch-0.9/plugins
2009-04-17 10:54:24,062 INFO  plugin.PluginRepository - Plugin Auto-activation mode: [true]
2009-04-17 10:54:24,062 INFO  plugin.PluginRepository - Registered Plugins:

Finally, test to see whether your plugin actually added fields to the index. You can do this using Luke. Download the yourcrawl/index directory from your crawl to your local machine and open it with Luke. Check that the name of your new field is listed in the fields list and that the field holds the expected values for your sample documents. If not, modify your source code, run ant in the /nutch-dir/src/plugin directory to rebuild the jar file and run your crawl again.

Sorting by date from your query page

You’re almost done! Now that your new field is in the index, you just need to modify your query page to sort by the new field. The default nutch query pages are /apache-tomcat-6.0.14/webapps/ROOT/search.jsp and (language)/search.html in that directory.

If you always want to sort by date, just find the line of code with input name="query" in it and add a parameter to sort by date after it. For example:

<input name="query" size="44" /> <input type="submit" value="Search" />
<input type="hidden" name="sort" value="published_date" />

Note that you may also want to add the following parameter to make Nutch put the most recent dates first.

<input type="hidden" name="reverse" value="true" />

If you want to do something more sophisticated, like only sort by date in certain situations, you can modify the code in search.jsp. Look for the following line, and modify it as needed:

String sort = request.getParameter("sort");

For instance, I added a checkbox to my site that let users chose whether to sort by date or not. Since this checkbox returned a value only if it was checked, I wrote code that used the presence of that parameter to determine whether to assign a value to sort.

  if (request.getParameter("sort_by_date") != null){
    sort = "published_date";
    reverse = true;
  }

Good luck with your own Nutch sorting projects! Feel free to leave a comment or contact me if you have any questions.

The basic idea of Do It Tomorrow is that you have a defined task list to accomplish each day. It’s a “closed list” because you aren’t supposed to add new items to it unless they absolutely must be done that day (and even then, you’re supposed to put them on a separate list so you don’t lose sight of your original goals). Whenever possible, you’re supposed to put off any new work until “tomorrow.” The theory is that most time management problems come from the fact that we spend too much of our day doing unplanned things. Pushing new work off until a future time gives you a “buffer” that allows you to plan when to do it.

I think this approach solves one of the key problems I’ve been having with GTD. GTD is an excellent knowledge capture and task creation system, but I don’t think it fully addresses what (or how much) to do on a particular day. With the closed list, you have a defined list of goals for each day. It stops you from overworking, but it also stops you from slacking off.

Check it out!

I created a Ruby script that reads in my tasks from the RTM private feed and my events from the Google Calendar private ical feed. It then assembles a text file of everything happening that day and prints it out. For fun, I rigged it up with the Microsoft text-to-speech API, so you can have your agenda read to you too.

If you’re interested in using it, I’ve released a copy of the source code under the Apache 2.0 license. Get it here.

Note that you’ll need a copy of the Ruby iCalendar library for this to work.

Here’s a sample of the printer output and the speech output.

I’m releasing it under the Apache license. Some of it is originally inspired from Google tutorials anyway. The portion of the code you may find useful is the self.GetAuthentication method. Basically, what you need to be able to do to permanently authenticate a user is:

1. Get the user to sign in to a Google account.
2. Get the user to approve your Web site for access to a url scope (in my case, the Google Contacts feed)
3. Upgrade the single use token that Google returns to a session token.
4. Store that token in a database (in Google App Engine’s case, just use the built-in datastore).

Then, next time a user visits the Web site, just look up the session key from your database and you’re good to go.

When Google upgraded to a new version of its contact manager, it bizarrely took away the ability to search fields like “notes” and “company.” For anyone who files people by company or puts contact “tags” in the notes field like I do, this was a major inconvenience. One work around to this problem is to open up the old version of Gmail (just type ?ui=1 after the gmail url) and use that contact manager, but who knows how long that’s going to be around.

So, until Google comes out with the be-all, end-all of contact managers, this Web site should serve as a nice holdover.

UPDATE: I’ve posted the source code with some notes. Based on my experience, I think some of Google App Engine’s tutorials need updating.

I’m probably going to use this for group projects this semester.

My main issue with MCL was that it was annoying to create keywords. Executor fixes that problem: you can either add shortcuts to a folder it scans, drag and drop shortcuts into its dialog box, or right-click a shortcut and hit “Send To…Executor.” Very handy. Plus, it has some great built in keywords. For instance, typing in “ip” will make your IP address appear in the bar, ready to be copy and pasted.

In terms of memory, it takes up less than 10 MB. Not as small as MCL, but acceptable.


wget http://wordpress.org/latest.tar.gz
rm -rf wp-includes
rm -rf wp-admin
tar -xzvf latest.tar.gz
cp -rpf wordpress/* .
rm latest.tar.gz


Of course, you’ll want to replace “.” in the second-to-last line with the address of your wordpress directory. In my case, I use the root directory.

UPDATE: Another tip I thought of to make this process even more streamlined: paste all of the above commands into a text file and call it “wp-upgrade” or something like that. Then type in:
chmod 755 wp-upgrade
That will make the file executable. From then on, all you have to do to upgrade WordPress is type ./wp-upgrade.

If you are sending a text message to an e-mail address and put words in parentheses at the start of your message, they will show up as the subject of the e-mail.

For example:
(this is the subject) this is the body

I haven’t tested this on other phones, but it works on my old Samsung phone on the Verizon Wireless network. I spent quite a bit of time Googling this before I just tried sending an email to my cell phone with a subject and then copied the syntax of how it appeared.

This feature is especially handy when using Remember The Milk’s “e-mail a task” feature. RTM gives you a unique e-mail address. Any e-mails sent to the address will be added to your task list. According to the documentation, you’re supposed to put the task name as the subject and any other information (e.g. due date) in the body.

My problem was I wanted the tasks to show up as due today, rather than due anytime. So, using the text message subject syntax, I wrote:

(remember the milk) due: today

Another alternative is to use the twitter interface for RTM, but it stopped working for me recently.

A productivity tip: Store the RTM e-mail as “@Task” in your phone contact list. That way, you won’t have to scroll through your contacts list to get to it.