Lion tweaks

This is a quick summary of all the tweaks handed down from generation to generation (aka found on the Internet), that I applied to a fresh Lion install to get it into a usable state.

Turn firewall on

Go to System Preferences -> Security & Privacy, then click the lock. Now you can start the firewall via the Start button. Also, click on Advanced... and then check Enable stealth mode.

Turn off Autocorrect

Go to System Preferences > Language & Text > Text, then uncheck Correct spelling automatically.

Trackpad tweaks

Go to System Preferences -> Trackpad, then uncheck Scroll direction: natural.

Terminal tweaks

Start a new terminal, then open the preferences.

Theme

• Select Pro, then click Default at the bottom.
• On the left, click Change next to font and select 14pt font size.
• Check Antialias text.
• Uncheck Use bold fonts.

Dimensions

Go to the Window tab, then enter 120 in the Columns input field.

Keybindings

Go to the Keyboard tab. Select the first entry control cursor left, then click Edit at the bottom
Delete the last three characters via the Delete one character button, then add a single b (the result should be 33b). Do the same with control cursor right and f so that you get 33f. These two changes give you Control+Left cursor/Right cursor for going to the previous/next word (quite useful for SSH, for instance).

Check Use option as meta key.

Other stuff

On the Advanced tab, uncheck Audible bell and possibly Visual bell.

Use function keys instead of feature keys

Go to System Preferences -> Keyboard, then check Use all F1, F2, etc. keys as standard function keys.

Re-enable key repeat

In a terminal, enter this:

defaults write -g ApplePressAndHoldEnabled -bool false 

Then go to System Preferences -> Keyboard, then move the sliders for Key Repeat and Delay Until Repeat both to the rightmost setting.

Finally, restart the computer.

Disable the new window animation

In a terminal, enter this:

defaults write NSGlobalDomain NSAutomaticWindowAnimationsEnabled -bool NO 

Mission Control tweaks

Go to System Preferences -> Mission Control, then uncheck Show Dashboard as a space and Automatically rearrange spaces based on most recent use.

Go to System Preferences -> Keyboard -> Keyboard Shortcuts, then select Spotlight. Double click the key binding for Show Spotlight window and press Shift+Command+Space.

Now, select Mission Control. Double click the key binding for Mission Control and press Option+Command+Space.

Double click the key binding for Move left a space and then press Option+Command+Left cursor. Similarly, for Move right a space, use Option+Command+Right cursor.
This will enable Option–Left cursor/Right cursor for jumping between words (in addition to the Control–Left cursor/Right cursor that we set up above).

Internal DNS in Amazon EC2 via tags

The recent EC2 API update from August 31st introduced the concept of tags which allow us to attach somewhat arbitrary metadata to instances and other resources. One useful application that came to my mind is to use them for maintaining internal DNS. The two blog posts here and here describe how to do this using the name of the ssh key that was used to create the instance. However that means that a new ssh key has to be used for each instance which is cumbersome. Tags make this a lot easier, especially since they are automatically returned as part of the instance metadata.

The only problem was that Glenn Rempe’s AWS ruby library doesn’t support the new API version yet. So forked the library and updated it to support the new API version and added the necessary functions.

So assuming that your instances have a tag named “hostname” that contains the desired short hostname (e.g. db1), with this new gem, the script from the above two blog posts becomes:

#!/usr/bin/env ruby
%w(optparse rubygems AWS resolv pp).each { |l| require l }
options = {}
parser = OptionParser.new do |p|
  p.banner = "Usage: hosts [options]"
  p.on("-a", "--access-key USER", "The user's AWS access key ID.") do |aki|
    options[:access_key_id] = aki
  end
  p.on("-s",
       "--secret-key PASSWORD",
       "The user's AWS secret access key.") do |sak|
    options[:secret_access_key] = sak
  end
  p.on_tail("-h", "--help", "Show this message") {
    puts(p)
    exit
  }
  p.parse!(ARGV)
end
if options.key?(:access_key_id) and options.key?(:secret_access_key)
  puts "127.0.0.1 localhost"

  AWS::EC2::Base.new(options).describe_instances.reservationSet.item.each do |r|
    r.instancesSet.item.each do |i|
      if i.instanceState.name =~ /running/
        tagSet = i.tagSet
        if (!tagSet.nil? && !tagSet.item.nil?)
          tagSet.item.each do |hash|
            if hash.key == 'hostname'
              puts(Resolv::DNS.new.getaddress(
                i.privateDnsName).to_s +" #{hash.value}.ec2 #{hash.value}")
            end
          end
        end
      end
    end
  end
else
  puts(parser)
  exit(1)
end

P.S.: The documentation for the API can be found here.

How to install Scribe with HDFS support on Ubuntu Karmic

Prerequisites

Install some pre-requisites (more might be needed, my system had a bunch of things already):

sudo apt-get install bison flex sun-java6-jdk ruby1.8-dev ant

Create a build folder

We won’t install scribe or thrift on the machine itself, instead keep it confined to a folder. For this we should

mkdir scribe-build
cd scribe-build
mkdir dist

The dist folder will contain the binary distribution of scribe once we’re done, including all libraries.

Install Boost

On Ubuntu, you can simply install boost via the package manager:

sudo apt-get install libboost1.40-dev libboost-filesystem1.40-dev

These are the only two parts of boost that are needed. Also, please make sure to get at least version 1.40.

If you want to install from source instead, download boost version 1.40 or newer from http://www.boost.org/ (current version is 1.41.0) and then unpack it into the scribe-build folder. After that, cd to the created folder and build it:

cd boost_1_41_0
./bootstrap.sh --prefix=`pwd`/../dist
./bjam install
cd ..

Install Libevent

Again, libevent can simply be installed via the package manager:

sudo apt-get install libevent-dev

On Karmic, this will install libevent (if not installed already) and libevent development files for version 1.4.11 or newer. If you want to install it from source, download the 1.4.x source distribution from http://www.monkey.org/~provos/libevent/ (1.4.13 is the current version) and unpack it into the scribe-build folder. Then cd into the generated folder and build it:

cd libevent-1.4.13-stable
./configure --prefix=`pwd`/../dist
make
make install
cd ..

Thrift and FB303

Download version 0.2.0-incubating from http://incubator.apache.org/thrift/download and unpack it into scribe-build. This should generate a folder scribe-build/thrift-0.2.0. To build it, run:

cd thrift-0.2.0
export PY_PREFIX=`pwd`/../dist
export JAVA_PREFIX=`pwd`/../dist
./configure --prefix=`pwd`/../dist \
    --with-boost=`pwd`/../dist \
    --with-libevent=`pwd`/../dist
make
make install
cd ..

This will most likely throw an error when trying to setup the ruby binding since it won’t be allowed to write into the system directory. This is due to a bug in the thrift build scripts – there is no way that I could find to tell it to install the ruby bindings locally. However, the things that we want will have been installed successfully, so let’s move on.

Next build the FB303 project:

cd contrib/fb303
export PY_PREFIX=`pwd`/../../../dist
./bootstrap.sh \
    --with-thriftpath=`pwd`/../../../dist \
    --with-boost=`pwd`/../../../dist \
    --prefix=`pwd`/../../../dist
make
make install
cd ../../..

Libhdfs

Scribe currently requires libhdfs 0.20.1 with patches applied – the stock version from the Hadoop 0.20.1 distribution won’t work. You can either use the Cloudera 0.20.1 distribution which has these patches applied, or use a newer version – presumably 0.21 works, but I haven’t tried it.

On Ubuntu, you can either install the Cloudera Hadoop distribution via debian packages, or you can compile it from source. The Debian/Ubuntu setup steps are described here:
http://archive.cloudera.com/docs/_apt.html.

We however are going to compile libhdfs from source to get an independent library. Download from
http://archive.cloudera.com/cdh/testing/hadoop-0.20.1+152.tar.gz
and unpack it into the scribe-build folder. This will create a hadoop-0.20.1+152 folder, so let’s go there:

cd hadoop-0.20.1+152

Unfortunately, we also need to tweak two files by adding this line

#include <stdint.h>

right under the existing

#include <stdint.h>

in these two files

src/c++/utils/api/hadoop/SerialUtils.hh
src/c++/pipes/api/hadoop/Pipes.hh

Once you’ve done that, run:

cd src/c++/libhdfs
./configure --enable-shared \
    JVM_ARCH=tune=k8 \
    --prefix=`pwd`/../../../../dist
make
make install
cd ../../../..

Note that this seems to have been fixed in the 0.20.1+168.89 cloudera release.

Build scribe

Download scribe 2.1 from http://github.com/facebook/scribe/downloads or clone the git repository (git://github.com/facebook/scribe.git). If you download the distribution, unpack it into the scribe-build directory, yielding a scribe-build/scribe- folder. cd to the scribe folder and the run:

cd scribe-2.1
export LD_LIBRARY_PATH="`pwd`/../dist/lib:"\
"/usr/lib/jvm/java-6-sun/jre/lib/amd64:"\
"/usr/lib/jvm/java-6-sun/jre/lib/amd64/server"
export CFLAGS="-I/usr/lib/jvm/java-6-sun/include/ "\
"-I/usr/lib/jvm/java-6-sun/include/linux/"
export LDFLAGS="-L`pwd`/../dist/lib "\
"-L/usr/lib/jvm/java-6-sun/jre/lib/amd64 "\
"-L/usr/lib/jvm/java-6-sun/jre/lib/amd64/server"
export LIBS="-lhdfs -ljvm"
./bootstrap.sh --enable-hdfs \
    --with-hadooppath=`pwd`/../dist \
    --with-boost=`pwd`/../dist \
    --with-thriftpath=`pwd`/../dist \
    --with-fb303path=`pwd`/../dist \
    --prefix=`pwd`/../dist
make
make install
cd ..

Adjust the jre/lib paths in the LDFLAGS to match your environment (e.g. 32bit vs. 64bit). The HDFS/Hadoop path in there is optional (i.e. enabled via the –enable-hdfs option) and only required if you want hdfs support.

Test that it works

Simply start scribe with the library path set correctly:

cd dist
export LD_LIBRARY_PATH="`pwd`/lib"
./bin/scribed ../scribe-2.1/examples/example1.conf

This should generate output like this:

[Tue Jan 19 00:31:07 2010] "STATUS: STARTING"
[Tue Jan 19 00:31:07 2010] "STATUS: configuring"
[Tue Jan 19 00:31:07 2010] "got configuration data from file "
[Tue Jan 19 00:31:07 2010] "CATEGORY : default"
[Tue Jan 19 00:31:07 2010] "Creating default store"
[Tue Jan 19 00:31:07 2010] "configured  stores"
[Tue Jan 19 00:31:07 2010] "STATUS: "
[Tue Jan 19 00:31:07 2010] "STATUS: ALIVE"
[Tue Jan 19 00:31:07 2010] "Starting scribe server on port 1463"