onizo#

Build a Hadoop single node cluster with Ubuntu 12.04LTS running in VirtualBox on Mac OSX 10.8.5

 

This is a rough draft. I will clean it up soon!

 

What I'm working with:

Macbook Air 1.7Ghz Intel Core i7 8GB 1600 MHz DDR3 OSX 10.8.5

VirtualBox 4.3.6

 

Prerequisites:

Build a VirtualBox machine to run Ubuntu 12.04.LTS 64-bit

Build Ubuntu Server 12.04LTS

 

From the newly installed server:

Create hadoop user and group:

sudo addgroup hadoop

sudo adduser --ingroup hadoop hadoop

sudo adduer hadoop sudo

 

Create ssh keys for the hadoop user (must be done as hadoop user):

ssh-keygen -t rsa -P “”

cp .ssh/id_rsa.php .ssh/authorized_keys

 

Update and install prerequisites:

ssh and rsync should already be installed.

sudo apt-get install python-software-properties

 

sudo apt-get install gcc

sudo apt-get install g++

 

wget htt://protobuf.googlecode.com/files/protobuf-2.5.0.tar.gz

tar -xvzf protobuf-2.5.0.tar.gz

cd protobuf-2.5.0

sudo ./configure

sudo make

sudo make check

sudo make install

sudo ldconfig

 

sudo add-apt-repository ppa:webupd8team/java

sudo apt-get update

sudo apt-get install oracle-java7-installer

 

Download hadoop from apache mirror:

wget http://www.eng.lsu.edu/mirrors/apache/hadoop/common/hadoop-2.2.0/hadoop-2.2.0.tar.gz

 

Unpack the hadoop tarball into /usr/local/:

sudo tar -xvzf hadoop-2.2.0.tar.gz -C /usr/local

 

Change the name of the hadoop directory in /usr/local/:

sudo mv /usr/local/hadoop-2.2.0 /usr/local/hadoop

 

Give the hadoop user and group ownership of the directory /usr/local/hadoop/:

sudo chown -R hadoop:hadoop /usr/local/hadoop

 

Edit hadoop users .bashrc and add the following to the end of the file:

# Hadoop Environment

export JAVA_HOME=/usr/lib/jvm/java-7-oracle/

export HADOOP_INSTALL=/usr/local/hadoop

export PATH=$PATH:$HADOOP_INSTALL/bin

export PATH=$PATH:$HADOOP_INSTALL/sbin

export HADOOP_MAPRED_HOME=$HADOOP_INSTALL

export HADOOP_COMMON_HOME=$HADOOP_INSTALL

export HADOOP_HDFS_HOME=$HADOOP_INSTALL

export YARN_HOME=$HADOOP_INSTALL

 

Edit /usr/local/hadoop/etc/hadoop/hadoop-env.sh to add java path:

export JAVA_HOME=/usr/lib/jvm/java-7-oracle/

 

Login as the hadoop user and test if hadoop is installed:

hadoop version

 

Hadoop is now installed and needs to be configured.

 

Edit /usr/local/hadoop/etc/hadoop/core-site.xml and add the following inside the <configuration> tags:

<property>

  <name>fs.default.name</name>

  <value>hdfs://localhost:9000</value>

</property>

 

Edit /usr/local/hadoop/etc/hadoop/yarn-site.xml and add the following inside the <configuration> tags:

<property>

  <name>yarn.nodemanager.aux-services</name>

  <value>mapreduce_shuffle</value>

</property>

<property>

  <name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>

  <value>org.apache.hadoop.mapred.ShuffleHandler</value>

</property>

 

Copy the mapreduce template to site xml

cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template /usr/local/hadoop/etc/hadoop/mapred-site.xml 

 

Edit the mapreduce-site.xml and add the following inside the <configuration> tags:

<property>

  <name>mapreduce.framework.name</name>

  <value>yarn</value>

</property>

 

Create a temp datadirs in the hadoop users home directory for testing:

mkdir -p datadir/hdfs/namenode

mkdir -p datadir/hdfs/datanode

 

Edit /usr/local/hadoop/etc/hadoop/hdfs-site.xml and add the following inside the <configuration> tags:

<property>

  <name>dfs.replication</name>

  <value>1</value>

</property>

<property>

  <name>dfs.namenode.name.dir</name>

  <value>file:/home/hadoop/datadir/hdfs/namenode</value>

</property>

<property>

  <name>dfs.datanode.data.dir</name>

  <value>file:/home/hadoop/datadir/hdfs/datanode</value>

</property>

 

Format the namenode as the hadoop user:

hdfs namenode -format

 

Start hadoop service:

start-dfs.sh

start-yarn.sh

 

Show hadoop services running:

jps

 

Run example for testing from /usr/local/hadoop as the hadoop user:

hadoop jar ./share/hadoop/mapreduce/hadoop-mapreduce-examples-2.2.0.jar pi 2 5

 

How to build a php soapclient object to connect to a .NET web service

I was having a lot of problems building a form that interacts with objects from a third party API via SOAP. I finally resolved the connection problems by making a change to the soap header.

Here is the XML that is required for the request:

<soap:Envelope xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
 <soap:Header>
  <AuthHeader xmlns="namespace.url">
   <Username>string</Username>
   <Password>string</Password>
  </AuthHeader>
 </soap:Header>
 <soap:Body>
  <SomeFunction xmlns="namespace.url" />
 </soap:Body>
</soap:Envelope>

Here is the PHP I used along with some debug:

<?php
$wdsl = "http://servicedomain.service?wsdl";
$client = new SoapClient($wdsl, array("soap_version"=> SOAP_1_1,
        'trace' => 1,
        'exceptions' => 0
        ));

$ns = 'namespace.url';
$headerbody = array( 'AuthHeader' => array('Username' => 'user', 'Password' => 'pass'));
$header = new SoapHeader($ns, 'AuthHeader', $headerbody);
$client->__setSoapHeaders($header);

$response = $client->SomeFunction();
var_dump($response);

echo "REQUEST HEADER:\n" . $client->__getLastRequestHeaders() . "\n";
echo "REQUEST:\n" . $client->__getLastRequest() . "\n";
echo "RESPONSE HEADER:\n" . $client->__getLastResponseHeaders() . "\n";
echo "RESPONSE:\n" . $client->__getLastResponse() . "\n";
?>

F5 Big-IP 6400 LTM 9.3.1- Asymetric routing across multiple VLANs

Yesterday, a coworker discovered an issue where packets routed outbound on one VLAN were not being processed when returning on a different VLAN.

here's the fix:

# disable Connection.VlanKeyed
[root@bigip:Active] config # bigpipe db Connection.VlanKeyed disable

# disable the Packet Velocity ASIC (PVA)
[root@bigip:Active] config # bigpipe db Pva.Acceleration none

F5 Big-IP 6400 LTM 9.3.1 - Save configuration using the bigpipe command

save your configuration after making any changes using the 'bigpipe' command:

# bigpipe save all
# this command will save 3 different files.
[root@bigip:Active] config # b save all
/config/bigip_base.conf was renamed to /config/bigip_base.conf.bak (62 lines).
/config/bigip.conf was renamed to /config/bigip.conf.bak (1643 lines).
/config/bigip_local.conf was renamed to /config/bigip_local.conf.bak (0 lines).

# bigpipe save
# this command will save 2 different files.
[root@bigip:Active] config # b save
/config/bigip.conf was renamed to /config/bigip.conf.bak (1643 lines).
/config/bigip_local.conf was renamed to /config/bigip_local.conf.bak (0 lines).

I tend to use 'b save all' out of habit. though, which, logically, you only need to use when using bigpipe command changing the bigip_base.com file.  'b save' works just as well for most cases. I'm really not sure why there is a difference. to be on the save side; it is usually a good idea to save *all* of your work.

some changes are not runtime, and need to be loaded in order for the switch to take the new configuration:

# bigpipe load
# this command will load all of your configuration files
[root@bigip:Active] config # b load
Reading configuration from /config/bigip_base.conf.
Reading configuration from /usr/bin/monitors/builtins/base_monitors.conf.
Reading configuration from /config/profile_base.conf.
Reading configuration from /config/daemon.conf.
Reading configuration from /config/bigip.conf.
Reading configuration from /config/bigip_local.conf.
Loading the configuration ...

as you can see. there 3 additional files loaded than just the one backed up from a 'b save all'. there are some changes that require a restart of the LTM base:

# bigstart restart
[root@bigip:Active] config # bigstart restart
[root@bigip:INOPERATIVE] config #
[root@bigip:Active] config #

the terminal will return after being unreachable for a short time. wait a few more moments and hit <enter> a few times to confirm the load balancer has recovered into 'active' mode.

if you see any errors running this command from the console interface, you should probably contact support and get the issue resolved as quickly as possible.

only a handful of cases require a complete reboot of the system. usually limited to system level changes such as code upgrades and hotfixes.

# full_box_reboot

F5 Big-IP 6400 LTM 9.3.1 - Remote reboot and reload via command line

when making configuration changes via command line, you will need to save and load the configuration into memory.

after ssh'ing into the load balancer:
# save your work
[root@bigip:Active] config # b save all

# load the config into memory
[root@bigip:Active] config # b load

# to completely reboot the box:
[root@bigip:Active] config # full_box_reboot