Thursday, November 14, 2013

install jdk7 in ubuntu

To install jdk7 in ubuntu follow next step
sudo apt-get install python-software-properties
if dont install this error is show "Ubuntu: add-apt-repository: command not found" when execute next step
 sudo add-apt-repository ppa:webupd8team/java
 sudo apt-get update
 sudo apt-get install oracle-java7-installer

Monday, September 2, 2013

make a mirror dbpedia in 5 steps

to make a mirror, you follow 5 simples steps. you must install virtuoso database. and change virtuoso.ini to read files
DirsAllowed              = ., /path/files/script 
1º download files from dbpedia with command
wget -r -np -nd -nc -A'*.nt.bz2' http://downloads.dbpedia.org/3.8/en/
2º decompress files
for i in `ls ./download/*.bz2` ; do
  bzcat $i | gzip --fast > ${i%.bz2}.gz && mv ./download/*.gz ./decompress/ && rm $i ;
done &
3º deletes triplets more long 1024 chars.
cd decompress;
for i in *.gz  ; do
  echo -n "cleaning $i..."
  zcat $i | grep -v -E '^<.+> <.+> <.{1025,}> \.$'  > $i.clean
  mv *clean ../fixed/
  echo "done."
done
4º split files in small files.
cd fixed
for i in *clean ; do

  split_size=$(stat -c%s "$i")

  echo "Size = $split_size"

  echo -n "slipting  $i..."

  if [ $split_size -gt 5000000 ]; then
        echo "File is large."

        echo "Performing split on file $i"

        split -a 10 -l 50000 $i "split/$i" ;
  else
        mv $i split/ ;
   fi

done;
5º execute insert command in virtuoso database
echo "creating load statement"

cd  slipt

for file in `ls *`
do
        load_target="/home/user/split/$file"
        load_query="EXEC=TTLP_MT(file_to_string_output('$load_target'), '', 'http://dbpedia.org', 255);commit WORK;"
        isql-vt 1111  dba dba  "$load_query"
     #  echo "$virt_isql" "$virt_port" "$virt_userName" "$virt_passWord" "$load_query"
 if [ $?  -ne 1 ]; then
  #echo "execution ok. move data"
  ls  split/ | wc -l
  mv "$load_target"  process-files/
  echo "procesando $file"
 else
   echo "error to process file $file"
   exit;
 fi
done;
echo "done"

Tuesday, August 27, 2013

Eclipse + maven: error compilation suddenly

Sometimes, you can see this error in eclipse.
[INFO] -------------------------------------------------------------
[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] Unable to locate the Javac Compiler in:
  C:\Program Files (x86)\Java\jre7\..\lib\tools.jar
Please ensure you are using JDK 1.4 or above and
not a JRE (the com.sun.tools.javac.Main class is required).
In most cases you can change the location of your Java
installation by setting the JAVA_HOME environment variable.
[INFO] 1 error
This error happen when you compile eclipse and maven with jre. You must change to JDK compile in Eclise.

Wednesday, August 21, 2013

read a resource inisde a jar

In both examples the file has locate in root of jar file
this.getClass().getClassLoader().getResourceAsStream("file.xml"))
this.getClass().getResourceAsStream("/file.xml"))



this.getClass().getClassLoader().getResource("file.xml").getFile())

creacion de un cluster con Lily en cloudera

Encontre hace poco un proyecto muy interesante http://www.lilyproject.org/lily/index.html
Para instalarlo en clustes hay que seguir la guía pero te encontrarás con algún problema.
http://lilyproject.org/books/lily_book--2_0--20130124-120744/publications/html-chunked/output/dsy432-lily.html#dsy432-lily Recientemente he creado un cluster con 3 equipos en rack de Lily. Primero instalo cloudera en cada uno de los 3 equipos pero me pedia actualizar a la version de 12 de ubuntu.
Una vez actualizado me han ido dando distintos errores:

  • sncronizacion de los relojes.
    La sincronización de los relojes de los equipos, tiene que dar una diferencia menor de 3 segundos. Con un servicio de NTP en ubuntu, esto se ha corregido. El resto de los equipos se conecta a el. Si no se sincroniza correctamente verifica la fecha hora de tu equipo. Verifica que si tienes un dominio, estén todos en dominio, ya que si uno de ellos no lo está, fallará constantemente y no usará el servicio de NTP.
  • hdfs al arrancar da un error.
    se soluciona borrando la carpeta a mano.
  • mapreduce-traker da un error de versiones distinta, se solucion actualizan todo el sistema.
  • HBase no arranca bien,
    HRegionServer Failed deleting my ephemeral node org.apache.zookeeper.KeeperException$SessionExpiredException: KeeperErrorCode = Session expired esto se debe a un error de sincronización.
  • Para poder lanzar Lily en Cloudera se debe tener autenticacion con Kerberos si te da un error parecido a este "Authentication failed. Cannot get kdc for realm" es que no tienes kerberos instalados. http://www.cloudera.com/content/cloudera-content/cloudera-docs/CDH4/latest/CDH4-Security-Guide/cdh4sg_topic_11_1.html tutorial de kerberos https://we.riseup.net/debian/kerberos-for-birds-not-dummies primero se debe instalar kerberos install kerbero atp-get install krb5-admin-server

    change version java runtime in tomcat7

    Tomcat dont execute default version java in linux machine. to view what version java use proccess tomcat type this
    #ps -ef | grep tomcat
    
    tomcat7  13692     1  1 10:57 ?        00:00:07 /usr/lib/jvm/default-java/bin/java -Djava.util.logging.config.file=/var/lib/tomcat7/conf/logging.properties -Djava.awt.headless=true -Xmx128m -XX:+UseConcMarkSweepGC -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.endorsed.dirs=/usr/share/tomcat7/endorsed -classpath /usr/share/tomcat7/bin/bootstrap.jar:/usr/share/tomcat7/bin/tomcat-juli.jar -Dcatalina.base=/var/lib/tomcat7 -Dcatalina.home=/usr/share/tomcat7 -Djava.io.tmpdir=/tmp/tomcat7-tomcat7-tmp org.apache.catalina.startup.Bootstrap start
    
    To view default java version
    #java -version 
    
    java version "1.7.0_21"
    Java(TM) SE Runtime Environment (build 1.7.0_21-b11)
    Java HotSpot(TM) 64-Bit Server VM (build 23.21-b01, mixed mode)
    
    and execute default java
    #/usr/lib/jvm/default-java/bin/java -version
    
    
    java version "1.6.0_27"
    OpenJDK Runtime Environment (IcedTea6 1.12.5) (6b27-1.12.5-0ubuntu0.12.04.1)
    OpenJDK 64-Bit Server VM (build 20.0-b12, mixed mode)
    
    #update-java-alternatives  -l
    
    j2sdk1.6-oracle 315 /usr/lib/jvm/j2sdk1.6-oracle
    java-1.6.0-openjdk-amd64 1061 /usr/lib/jvm/java-1.6.0-openjdk-amd64
    java-1.7.0-openjdk-amd64 1051 /usr/lib/jvm/java-1.7.0-openjdk-amd64
    java-7-oracle 1062 /usr/lib/jvm/java-7-oracle
    
    show soft link to java 1.6
    #/usr/lib/jvm/ ls -la 
    default-java -> java-1.6.0-openjdk-amd64
    
    remove soft link to 1.6 and create a new link 1.7
    # unlink default-java
    # ln -f  -s java-7-oracle /usr/lib/jvm/default-java
    
    show new link to 1.7
    #/usr/lib/jvm/ ls -la 
    default-java -> java-7-oracle
    

    Monday, July 15, 2013

    unix script file exists

    I have a lists of files. Inside each file I have a list of files. I have to test that this files exist in directory /tmp I create this scprit to test pattern list of files dbpediaa? #pattern=`ls -la | awk '{print $9}' | grep -vE "\."`
    
    for t in dbpediaa?
    do
     echo "existe fichero $t ? "
    
     ficheros=`grep -i .nt.gz  $t | awk -F "'"  '{ print $4 }'`
    
     
     for fichero in $ficheros
     do
       if [ -f /tmp/$fichero  ]; then
         echo "."
        else
         echo "no existe $fichero"
       fi
     done;
    done;