Nutch, in its default configuration, will sort pages by relevance using Lucene scores. You can get Nutch to sort by another field in its index using the sort parameter, but first, you need a field to sort by. (For an example of how sorting works, add &sort=url to the end of your Nutch query. Obviously sorting by url isn’t very useful, but you can see how the query works. Note that you can also add &reverse=true to sort the results in reverse order.)

Sorting by date in Nutch essentially involves two parts: first, using a plugin to get Nutch to index dates into a new field in your index, and second, getting your query page to add the additional parameter to search queries.

Indexing dates with a Nutch plugin

NOTE: I figured out most of the information here from the tutorial on writing a Nutch plugin available on the Nutch Wiki. If you need more detailed information on any of the steps here I recommend reading that tutorial.

All Nutch plugins implement an interface. These interfaces are called extension points. The Nutch Wiki has a list of available extension points. The goal of our plugin is to add an additional date field to the Lucene index, so we want to implement the IndexingFilter interface.

Nutch plugins consist of xml description files and your Java source code files. File structure is as follows:

/nutch-dir
  /src
    /plugin
      /yourpluginname
        build.xml
        plugin.xml
        /src
          /org
            /apache
              /nutch
                /yourpluginname
                  ImplementationClassName.java

Here’s an example build.xml file:

<?xml version="1.0"?>
<project name="yourpluginname" default="jar">
  <import file="../build-plugin.xml"/>
</project>

And here’s an example plugin.xml file:

<?xml version="1.0" encoding="UTF-8"?>
<plugin
   id="yourpluginname"
   name="longer/human-readable name of your plugin"
   version="0.0.1"
   provider-name="your_website">

   <runtime>
      <library name="yourpluginname.jar">
         <export name="*"/>
      </library>
   </runtime>

   <extension id="org.apache.nutch.urldateindexer.yourpluginname"
              name="longer/human-readable name of your plugin"
              point="org.apache.nutch.indexer.IndexingFilter">
      <implementation id="ImplementationClassName"
                      class="org.apache.nutch.yourpluginname.ImplementationClassName"/>
   </extension>
</plugin>

NOTE: I had some trouble using my own package name, but it might have been due to another problem. When in doubt I recommend sticking with org.apache.nutch.yourpluginname.

And here’s the code for the Java source file, ImplementationClassName.java. (You’ll want to change it to your own name to match what you put in the plugin.xml file.)

package org.apache.nutch.yourpluginname;

//imports
import java.util.Date;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.io.Text;
import org.apache.lucene.document.DateTools;
import org.apache.lucene.document.Document;
import org.apache.lucene.document.Field;
import org.apache.lucene.document.DateTools.Resolution;
import org.apache.nutch.crawl.CrawlDatum;
import org.apache.nutch.crawl.Inlinks;
import org.apache.nutch.indexer.IndexingException;
import org.apache.nutch.indexer.IndexingFilter;
import org.apache.nutch.parse.Parse;

public class UrlDateIndexer implements IndexingFilter {

  private Configuration conf;

  public UrlDateIndexer() {
  }

  //this is where your code to add a new field to the index goes
  public Document filter(Document doc, Parse parse, Text url,
    CrawlDatum datum, Inlinks inlinks)
    throws IndexingException {

    // Get url from the method inputs
    String urlString = url.toString();

    // Use regex to identify type of date and parse date from url format (code for the TdcDateParser class not shown)
    TdcDateParser tdcDateParser = new TdcDateParser();
    Date publishedDate = tdcDateParser.parseUrl(urlString);

    // Store date in field
    if (null != publishedDate){
    	//create a new field
      Field publishedDateField = new Field("published_date",DateTools.dateToString(publishedDate, Resolution.DAY),
           Field.Store.YES, Field.Index.UN_TOKENIZED);

      //add the field to the index
    	doc.add(publishedDateField);
    }

    return doc;
  }

  public void setConf(Configuration conf) {
    this.conf = conf;
  }

  public Configuration getConf() {
    return this.conf;
  }
}

In the case of the plugin I build, the goal was to parse the date from the url of a page. You’ll need to modify this method to get the date from wherever is appropriate for your search engine. The basic flow is to create a field, add that field to a document and then return the document.

There are a few things to note about the line that creates the new field:

Field publishedDateField = new Field("published_date",DateTools.dateToString(publishedDate, Resolution.DAY),
    Field.Store.YES, Field.Index.UN_TOKENIZED);

• dateToString: More recent versions of Lucene don’t store dates in the index directly. Therefore, you need to use the dateToString method to store dates.
• Field.Store.YES: This stores the text of the field in the index. You wouldn’t do this for a long field that you don’t need to view in its entirety in the search engine (e.g. page content). I believe setting Field.Store to YES is mandatory to be able to sort by a field.
• Field.Index.UN_TOKENIZED: Tells Lucene not to break up the field into individual words, as it would with a page’s content. I also believe this is mandatory to be able to sort by a field.

Now that you have finished creating your files, upload them to the nutch directory on your server if you haven’t already done so. Then, add a line like the following to the build.xml file in the /nutch-dir/src/plugin directory:

  <ant dir="yourpluginname" target="deploy"/>

Then launch ant from the command line in that same directory (/nutch-dir/src/plugin). This should compile the classes and jar file for your plugin and place them in the /nutch-dir/build directory. It should also place a copy of the jar file and plugin.xml file in the /nutch-dir/plugins directory.

NOTE: I had to modify the deploy.dir property in the /nutch-dir/src/plugin/build-plugin.xml file to get ant to put the jar and plugin.xml file in the right plugins directory. Originally it was set to put them in build/plugins. I’m not sure which directory is formally correct, but the /nutch-dir/plugins directory is where Nutch was set to read from on my system. If this is the case for you, just remove /build from the deploy.dir line. My resulting deploy.dir line is as follows:

<property name="deploy.dir" location="${nutch.root}/plugins/${name}"/>

Next, you need to modify the /nutch-dir/conf/nutch-site.xml file to recognize your new plugin. Just add your plugin name to the plugin.includes property. For example:

<property>
  <name>plugin.includes</name>
  <value>yourpluginname|protocol-http|urlfilter-regex|parse-(text|html|js)|index
-basic|query-(basic|site|url)|summary-basic|scoring-opic|urlnormalizer-(pass|reg
ex|basic)</value>
</property>

Launch a Nutch crawl of just a few pages to test your new plugin. To see if the plugin was activated by Nutch, open /nutch-dir/logs/hadoop.log and look for a line like the following:

2009-04-17 10:54:27,477 INFO  plugin.PluginRepository - Indexer of dates from URLs (urldateindexer)

If the plugin didn’t show up, make sure your plugin jar and xml files are in the directory indicated in these log file lines:

2009-04-17 10:54:23,617 INFO  plugin.PluginRepository - Plugins: looking in: /nutch-0.9/plugins
2009-04-17 10:54:24,062 INFO  plugin.PluginRepository - Plugin Auto-activation mode: [true]
2009-04-17 10:54:24,062 INFO  plugin.PluginRepository - Registered Plugins:

Finally, test to see whether your plugin actually added fields to the index. You can do this using Luke. Download the yourcrawl/index directory from your crawl to your local machine and open it with Luke. Check that the name of your new field is listed in the fields list and that the field holds the expected values for your sample documents. If not, modify your source code, run ant in the /nutch-dir/src/plugin directory to rebuild the jar file and run your crawl again.

Sorting by date from your query page

You’re almost done! Now that your new field is in the index, you just need to modify your query page to sort by the new field. The default nutch query pages are /apache-tomcat-6.0.14/webapps/ROOT/search.jsp and (language)/search.html in that directory.

If you always want to sort by date, just find the line of code with input name="query" in it and add a parameter to sort by date after it. For example:

<input name="query" size="44" /> <input type="submit" value="Search" />
<input type="hidden" name="sort" value="published_date" />

Note that you may also want to add the following parameter to make Nutch put the most recent dates first.

<input type="hidden" name="reverse" value="true" />

If you want to do something more sophisticated, like only sort by date in certain situations, you can modify the code in search.jsp. Look for the following line, and modify it as needed:

String sort = request.getParameter("sort");

For instance, I added a checkbox to my site that let users chose whether to sort by date or not. Since this checkbox returned a value only if it was checked, I wrote code that used the presence of that parameter to determine whether to assign a value to sort.

  if (request.getParameter("sort_by_date") != null){
    sort = "published_date";
    reverse = true;
  }

Good luck with your own Nutch sorting projects! Feel free to leave a comment or contact me if you have any questions.

The basic idea of Do It Tomorrow is that you have a defined task list to accomplish each day. It’s a “closed list” because you aren’t supposed to add new items to it unless they absolutely must be done that day (and even then, you’re supposed to put them on a separate list so you don’t lose sight of your original goals). Whenever possible, you’re supposed to put off any new work until “tomorrow.” The theory is that most time management problems come from the fact that we spend too much of our day doing unplanned things. Pushing new work off until a future time gives you a “buffer” that allows you to plan when to do it.

I think this approach solves one of the key problems I’ve been having with GTD. GTD is an excellent knowledge capture and task creation system, but I don’t think it fully addresses what (or how much) to do on a particular day. With the closed list, you have a defined list of goals for each day. It stops you from overworking, but it also stops you from slacking off.

I created a Ruby script that reads in my tasks from the RTM private feed and my events from the Google Calendar private ical feed. It then assembles a text file of everything happening that day and prints it out. For fun, I rigged it up with the Microsoft text-to-speech API, so you can have your agenda read to you too.

If you’re interested in using it, I’ve released a copy of the source code under the Apache 2.0 license. Get it here.

Note that you’ll need a copy of the Ruby iCalendar library for this to work.

Here’s a sample of the printer output and the speech output.

I’m releasing it under the Apache license. Some of it is originally inspired from Google tutorials anyway. The portion of the code you may find useful is the self.GetAuthentication method. Basically, what you need to be able to do to permanently authenticate a user is:

1. Get the user to sign in to a Google account.
2. Get the user to approve your Web site for access to a url scope (in my case, the Google Contacts feed)
3. Upgrade the single use token that Google returns to a session token.
4. Store that token in a database (in Google App Engine’s case, just use the built-in datastore).

Then, next time a user visits the Web site, just look up the session key from your database and you’re good to go.

I’m probably going to use this for group projects this semester.

My main issue with MCL was that it was annoying to create keywords. Executor fixes that problem: you can either add shortcuts to a folder it scans, drag and drop shortcuts into its dialog box, or right-click a shortcut and hit “Send To…Executor.” Very handy. Plus, it has some great built in keywords. For instance, typing in “ip” will make your IP address appear in the bar, ready to be copy and pasted.

In terms of memory, it takes up less than 10 MB. Not as small as MCL, but acceptable.


wget http://wordpress.org/latest.tar.gz
rm -rf wp-includes
rm -rf wp-admin
tar -xzvf latest.tar.gz
cp -rpf wordpress/* .
rm latest.tar.gz


Of course, you’ll want to replace “.” in the second-to-last line with the address of your wordpress directory. In my case, I use the root directory.

UPDATE: Another tip I thought of to make this process even more streamlined: paste all of the above commands into a text file and call it “wp-upgrade” or something like that. Then type in:
chmod 755 wp-upgrade
That will make the file executable. From then on, all you have to do to upgrade WordPress is type ./wp-upgrade.

Obvious quote of the day:

“Why do so many feel a connection — be it kinship or competition — with utter strangers just because they share a name? Social science, it turns out, has an answer. It is because human beings are unconsciously drawn to people and things that remind us of ourselves.”

Topics covered:

• Hosting your site (where to save your files)
• Creating and updating files
• To dot.com or not to dot.com (or .org, or .net)

Hosting your site

For people to be able to view your site on the Internet, you need to put your files on a Web server. If you’re at Penn State, the university provides some free space for you to use. If you’re not (or you’ll be graduating soon — your PSU space ends 6 months after graduation), I recommend some free and cheap alternatives at the bottom of this section.

At Penn State

1. First you have to apply for Web space at https://www.work.psu.edu/webspace/ The quiz questions are pretty obvious. It takes a few days to set up.

2. Now you need to open the folder Penn State created for you to store your files. On Windows, Open My Network Places from your desktop. Double-click “Add Network Place.” Type in: \\win.pass.psu.edu\xyz123 (replace xyz123 with your user id). Hit next and type in your username and password.

* NOTE: If you’re off-campus, you probably won’t be able to access your files directly like this. Instead, you’ll have to use SSH to upload your files (see below).

3. Your Penn State file space (Penn State calls it the “PASS space”) will show up. Open the www folder. This is where you will store all your Web site files.

Now you’re ready to go. Jump down to the section on creating and updating files. They will appear at http://www.personal.psu.edu/xyz123 (again, replacing xyz123 with your user name).

Not at Penn State

If you’re not at Penn State, you’ll need to find a place to host your files. Free Web hosts abound, but the only one I’ve used is Yahoo! Geocities. It puts an annoying ad on your page, but the service has been around for a while so it’s probably pretty stable. You’ll also have to manually upload your files using their Web interface.

As an alternative to free Web hosts, I recommend “pay as you go” Web hosting from NearlyFreeSpeech.NET. You make an initial deposit and then they just draw against it. Fees are about $1 a gigabyte of transfer, so even five dollars could last you for years. I use this service for this Web site and I’ve had a great experience. The one downside is they have no Web-based tools for uploading files, so you’ll have to learn to use SSH.

Using SSH

If you can’t access your Web space using the “My Network Places” method above, you’ll have to use SSH to upload your files to your server. I recommend WinSCP for Windows users. Download the standalone application.

To get files on your Web space, open the program and type in the server (ftp.personal.psu.edu for Penn State) and your user name and password. Click connect, accept the encryption key and your files should show up. Then just drag files from your hard drive to the Web server directory. If you need to make changes to files, just re-upload them and overwrite the old files.

Note that unlike with the “My Network Places” method, you won’t be able to edit your files directly on the server. Instead, you’ll have to edit them on your hard drive and then upload them.

Creating and updating files

Download my basic template to get started. You can use your own files if you want, but I’m going to reference the template specifically.

• index.html is the main file.
• works.html is where the list of stories is.
• site.css is the stylesheet that controls how the pages look.
• the png file is the mug shot that appears on the right side of the page.

Just edit these files and drag them to your Web space. Also upload any other files you referenced. I recommend using a text editor to edit your files, but if you use a graphical editor, make sure your links are relative (e.g. don’t start with file://). Otherwise they won’t work on the Web site.

For an introduction to HTML and CSS editing, check out HTML Dog. I’ll write a more specific tutorial later if I get around to it.

To dot.com or not to dot.com

Domains are cool. The address looks impressive on your resumé, you get every address @yourname.com and your url is a lot easier to remember.

You get a domain by registering it with a company called a registrar. My current favorite is 1&1 (which is also a hosting company). I like it because it’s reliable and its prices are on the cheaper end. Dot.com domains are $7/year; less common suffixes like .mobi are sometimes less. The company will keep your credit/debit card and charge you each year. Another perk of 1&1 is that they offer private registration, so you don’t have to reveal your contact information on domain name registry listings.

Once you’ve registered your domain, you have two options. The first is to forward your domain to the url where your files are hosted. If you take this route, you have two options: masked forward or redirect. Under the first option, your domain stays in the address bar but the file inside come from your Web host. Under the second option, the real URL of your Web site appears in the address bar.

If you care about your domain staying in the address bar, pick the first option. Just keep in mind that urls of pages won’t show up in the address bar. So if someone likes your “works” page and they copy and paste the url, it will be the one for your main page, not the works page specifically. Of course if you have a Web site with just a few pages it’s probably not a big issue.

The second route for forwarding your domain is to use DNS. This works with paid Web hosts like NearlyFreeSpeech.NET. This method will keep your domain url in the address bar and let you type in specific urls. It takes a little doing to set up (maybe I’ll write up a tutorial later) but basically you need to have 1&1 forward your name servers or DNS to your Web host.

UPDATE: Looks like pfist.er is out as well.