Hadoop on Mountain Lion

Environment

  • Mac OSX 10.8.3
  • Homebrew
  • Java 6
  • Hadoop 1.1.2

Setting up SSH for public key login

generate public key in order to login to localhost without password.

#ssh-keygen

#cd ~/.ssh
#cat "id_rsa.pub" >> authorized_keys2

#ssh localhost

Setting up Hadoop configuration

all conf files are located in /usr/local/Cellar/hadoop/1.1.2/libexec/conf．
core-site.xml

<configuration>
  <property>
    <name>fs.default.name</name>
    <value>hdfs://localhost:9000</value>
  </property>
</configuration>

hdfs-site.xml

<configuration>
  <property>
    <name>dfs.name.dir</name>
    <value>/Users/${user.name}/hdfs</value>
  </property>
  <property>
    <name>dfs.replication</name>
    <value>1</value>
  </property>
</configuration>

mapred-site.xml

<configuration>
  <property>
    <name>mapred.job.tracker</name>
    <value>localhost:9001</value>
  </property>
  <property>
    <name>mapred.tasktracker.map.tasks.maximum</name>
    <value>4</value>
  </property>
</configuration>

hadoop-env.sh

export JAVA_HOME="$(/usr/libexec/java_home -v 1.6)"
export HADOOP_OPTS="-Djava.security.krb5.realm=OX.AC.UK -Djava.security.krb5.kdc=kdc0.ox.ac.uk:kdc1.ox.ac.uk"

You can download Apple Java 6 here.

Initialization

hadoop namenode -format

RUN

start-all.sh

Test

Calculation of Pi

hadoop jar /usr/local/Cellar/hadoop/1.1.2/libexec/hadoop-examples-1.1.2.jar pi 4 400000

ref
http://lizan.asia/blog/2012/11/13/mountain-lion-setup-hadoop/
http://borrelli.org/2012/05/02/hadoop-osx-sshkey_setup/