How To Install/configure Apache Oozie Workflow Scheduler
{{postValue.id}}
To Install and Configure Apache Oozie Workflow Scheduler for CDH 4.X on RHEL/CentOS
Oozie is a Hadoop' s open source schedular, which simplifies the workflow and define the dependency between the jobs for an input data. Steps to Install and Configure Apache Oozie Workflow Scheduler for CDH 4.X on RHEL/CentOS is discussed in this article.
To Install Apache Oozie on RHEL/CentOS
Download CDH repository from the official site or alternatively run the following command.
[root@linuxhelp ~]# wget http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm
--2016-06-10 17:43:07-- http://archive.cloudera.com/cdh4/one-click-install/redhat/6/x86_64/cloudera-cdh-4-0.x86_64.rpm
Resolving archive.cloudera.com... 151.101.8.167
Connecting to archive.cloudera.com|151.101.8.167|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 9108 (8.9K) [application/x-redhat-package-manager]
Saving to: “ cloudera-cdh-4-0.x86_64.rpm”
100%[================================================================================> ] 9,108 --.-K/s in 0.02s
2016-06-10 17:43:07 (401 KB/s) - “ cloudera-cdh-4-0.x86_64.rpm” saved [9108/9108]
Once the rpm package is downloaded, install using the following command.
[root@linuxhelp ~]# yum --nogpgcheck localinstall cloudera-cdh-4-0.x86_64.rpm
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Local Package Process
Examining cloudera-cdh-4-0.x86_64.rpm: cloudera-cdh-4-0.x86_64
Marking cloudera-cdh-4-0.x86_64.rpm to be installed
Loading mirror speeds from cached hostfile
* base: centos.excellmedia.net
* extras: centos.excellmedia.net
* updates: centos.excellmedia.net
Resolving Dependencies
--> Running transaction check
---> Package cloudera-cdh.x86_64 0:4-0 will be installed
.
.
.
Installed:
cloudera-cdh.x86_64 0:4-0
Complete!
Next, install the Oozie
[root@linuxhelp ~]# yum install oozie -y
Loaded plugins: fastestmirror, refresh-packagekit, security
Setting up Install Process
Loading mirror speeds from cached hostfile
* base: centos.excellmedia.net
* extras: centos.excellmedia.net
* updates: centos.excellmedia.net
Resolving Dependencies
--> Running transaction check
---> Package oozie.noarch 0:3.3.2+108-1.cdh4.7.1.p0.7.el6 will be installed
--> Processing Dependency: oozie-client = 3.3.2+108 for package: oozie-3.3.2+108-1.cdh4.7.1.p0.7.el6.noarch
.
.
.
Installed:
oozie.noarch 0:3.3.2+108-1.cdh4.7.1.p0.7.el6
Dependency Installed:
bigtop-jsvc.x86_64 0:1.0.10-1.cdh4.7.1.p0.13.el6
bigtop-tomcat.noarch 0:6.0.37-1.cdh4.7.1.p0.14.el6
bigtop-utils.noarch 0:0.6.0+248-1.cdh4.7.1.p0.13.el6
hadoop.x86_64 0:2.0.0+1612-1.cdh4.7.1.p0.12.el6
hadoop-0.20-mapreduce.x86_64 0:2.0.0+1612-1.cdh4.7.1.p0.12.el6
hadoop-client.x86_64 0:2.0.0+1612-1.cdh4.7.1.p0.12.el6
hadoop-hdfs.x86_64 0:2.0.0+1612-1.cdh4.7.1.p0.12.el6
hadoop-mapreduce.x86_64 0:2.0.0+1612-1.cdh4.7.1.p0.12.el6
hadoop-yarn.x86_64 0:2.0.0+1612-1.cdh4.7.1.p0.12.el6
nc.x86_64 0:1.84-24.el6
oozie-client.noarch 0:3.3.2+108-1.cdh4.7.1.p0.7.el6
parquet.noarch 0:1.2.5-1.cdh4.7.1.p0.9.el6
parquet-format.noarch 0:1.0.0-1.cdh4.7.1.p0.10.el6
zookeeper.x86_64 0:3.4.5+28-1.cdh4.7.1.p0.13.el6
Complete!
To Configure Oozie on RHEL/CentOS
Install MySql database, then create oozie DB and grant privileges as shown below.
[root@linuxhelp ~]# mysql -u root -p Enter password: Welcome to the MySQL monitor. Commands end with or g. Your MySQL connection id is 3 Server version: 5.1.73 Source distribution Copyright (c) 2000, 2013, Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners. Type ' help ' or ' h' for help. Type ' c' to clear the current input statement. mysql> create database oozie Query OK, 1 row affected (0.00 sec) mysql> grant all privileges on oozie.* to ' oozie' @' localhost' identified by ' linuxc' Query OK, 0 rows affected (0.00 sec) mysql> grant all privileges on oozie.* to ' oozie' @' %' identified by ' linuxc' Query OK, 0 rows affected (0.00 sec) mysql> exit Bye
Run the below commands to download and add the MySQL JDBC connectivity driver JAR to Oozie lib directory.
[root@linuxhelp ~]# cd /tmp [root@linuxhelp tmp]# wget http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.31.tar.gz --2016-06-10 18:06:41-- http://dev.mysql.com/get/Downloads/Connector-J/mysql-connector-java-5.1.31.tar.gz Resolving dev.mysql.com... 137.254.60.11 Connecting to dev.mysql.com|137.254.60.11|:80... connected. HTTP request sent, awaiting response... 302 Found Location: http://cdn.mysql.com//archives/mysql-connector-java-5.1/mysql-connector-java-5.1.31.tar.gz [following] --2016-06-10 18:06:42-- http://cdn.mysql.com//archives/mysql-connector-java-5.1/mysql-connector-java-5.1.31.tar.gz Resolving cdn.mysql.com... 23.207.128.216 Connecting to cdn.mysql.com|23.207.128.216|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 3764875 (3.6M) [application/x-tar-gz] Saving to: “ mysql-connector-java-5.1.31.tar.gz” 100%[================================================================================> ] 3,764,875 719K/s in 5.1s 2016-06-10 18:06:47 (723 KB/s) - “ mysql-connector-java-5.1.31.tar.gz” saved [3764875/3764875] [root@linuxhelp tmp]# tar -zxf mysql-connector-java-5.1.31.tar.gz [root@linuxhelp tmp]# cd mysql-connector-java-5.1.31 [root@linuxhelp mysql-connector-java-5.1.31]# cp mysql-connector-java-5.1.31-bin.jar /var/lib/oozie/
Execute the below commands as oozie user to create the oozie database schema.
[root@linuxhelp ~]# sudo -u oozie /usr/lib/oozie/bin/ooziedb.sh create -run
setting OOZIE_CONFIG=/etc/oozie/conf
setting OOZIE_DATA=/var/lib/oozie
setting OOZIE_LOG=/var/log/oozie
.
.
.
setting CATALINA_OPTS=" $CATALINA_OPTS -Doozie.https.keystore.pass=${OOZIE_HTTPS_KEYSTORE_PASS}"
Validate DB Connection
DONE
Check DB schema does not exist
DONE
Check OOZIE_SYS table does not exist
DONE
Create SQL schema
DONE
Create OOZIE_SYS table
DONE
Now Oozie DB has been successfully created.
Download the ExtJS to enable oozie web console from official website or by using wget command.
[root@linuxhelp ~]# cd /tmp
[root@linuxhelp tmp]# wget http://archive.cloudera.com/gplextras/misc/ext-2.2.zip
--2016-06-10 18:10:35-- http://archive.cloudera.com/gplextras/misc/ext-2.2.zip
Resolving archive.cloudera.com... 151.101.8.167
Connecting to archive.cloudera.com|151.101.8.167|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 6800612 (6.5M) [application/zip]
Saving to: “ ext-2.2.zip”
100%[================================================================================> ] 6,800,612 664K/s in 9.5s
2016-06-10 18:10:44 (697 KB/s) - “ ext-2.2.zip” saved [6800612/6800612]
[root@linuxhelp tmp]# unzip ext-2.2.zip
Archive: ext-2.2.zip
creating: ext-2.2/
creating: ext-2.2/adapter/
creating: ext-2.2/adapter/ext/
inflating: ext-2.2/adapter/ext/ext-base.js
creating: ext-2.2/adapter/jquery/
.
.
.
inflating: ext-2.2/air/src/SystemMenu.js
inflating: ext-2.2/air/src/SystemTray.js
inflating: ext-2.2/adapter/yui/yui-utilities.js
Move the extracted directory to /var/lib/oozie/.
[root@linuxhelp tmp]# mv ext-2.2 /var/lib/oozie/
Atlast issue the following command to start the oozie server,
[root@linuxhelp tmp]# service oozie status not running. [root@linuxhelp tmp]# service oozie start Setting OOZIE_HOME: /usr/lib/oozie Sourcing: /usr/lib/oozie/bin/oozie-env.sh setting OOZIE_CONFIG=/etc/oozie/conf setting OOZIE_DATA=/var/lib/oozie setting OOZIE_LOG=/var/log/oozie setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat setting CATALINA_TMPDIR=/var/lib/oozie setting CATALINA_PID=/var/run/oozie/oozie.pid setting CATALINA_BASE=/usr/lib/oozie/oozie-server-0.20 setting CATALINA_OPTS=-Xmx1024m setting OOZIE_HTTPS_PORT=11443 setting OOZIE_HTTPS_KEYSTORE_PASS=password setting CATALINA_OPTS=" $CATALINA_OPTS -Doozie.https.port=${OOZIE_HTTPS_PORT}" setting CATALINA_OPTS=" $CATALINA_OPTS -Doozie.https.keystore.pass=${OOZIE_HTTPS_KEYSTORE_PASS}" Using OOZIE_CONFIG: /etc/oozie/conf Sourcing: /etc/oozie/conf/oozie-env.sh setting OOZIE_CONFIG=/etc/oozie/conf setting OOZIE_DATA=/var/lib/oozie setting OOZIE_LOG=/var/log/oozie setting OOZIE_CATALINA_HOME=/usr/lib/bigtop-tomcat setting CATALINA_TMPDIR=/var/lib/oozie setting CATALINA_PID=/var/run/oozie/oozie.pid setting CATALINA_BASE=/usr/lib/oozie/oozie-server-0.20 setting CATALINA_OPTS=-Xmx1024m setting OOZIE_HTTPS_PORT=11443 setting OOZIE_HTTPS_KEYSTORE_PASS=password setting CATALINA_OPTS=" $CATALINA_OPTS -Doozie.https.port=${OOZIE_HTTPS_PORT}" setting CATALINA_OPTS=" $CATALINA_OPTS -Doozie.https.keystore.pass=${OOZIE_HTTPS_KEYSTORE_PASS}" Setting OOZIE_CONFIG_FILE: oozie-site.xml Using OOZIE_DATA: /var/lib/oozie Using OOZIE_LOG: /var/log/oozie Setting OOZIE_LOG4J_FILE: oozie-log4j.properties Setting OOZIE_LOG4J_RELOAD: 10 Setting OOZIE_HTTP_HOSTNAME: www.linuxhelp1.com Setting OOZIE_HTTP_PORT: 11000 Setting OOZIE_ADMIN_PORT: 11001 Using OOZIE_HTTPS_PORT: 11443 Setting OOZIE_BASE_URL: http://www.linuxhelp1.com:11000/oozie Using CATALINA_BASE: /usr/lib/oozie/oozie-server-0.20 Setting OOZIE_HTTPS_KEYSTORE_FILE: /var/lib/oozie/.keystore Using OOZIE_HTTPS_KEYSTORE_PASS: password Setting CATALINA_OUT: /var/log/oozie/catalina.out Using CATALINA_PID: /var/run/oozie/oozie.pid Using CATALINA_OPTS: -Xmx1024m -Doozie.https.port=11443 -Doozie.https.keystore.pass=password -Dderby.stream.error.file=/var/log/oozie/derby.log Adding to CATALINA_OPTS: -Doozie.home.dir=/usr/lib/oozie -Doozie.config.dir=/etc/oozie/conf -Doozie.log.dir=/var/log/oozie -Doozie.data.dir=/var/lib/oozie -Doozie.config.file=oozie-site.xml -Doozie.log4j.file=oozie-log4j.properties -Doozie.log4j.reload=10 -Doozie.http.hostname=www.linuxhelp1.com -Doozie.admin.port=11001 -Doozie.http.port=11000 -Doozie.https.port=11443 -Doozie.base.url=http://www.linuxhelp1.com:11000/oozie -Doozie.https.keystore.file=/var/lib/oozie/.keystore -Doozie.https.keystore.pass=password -Djava.library.path= Using CATALINA_BASE: /usr/lib/oozie/oozie-server-0.20 Using CATALINA_HOME: /usr/lib/bigtop-tomcat Using CATALINA_TMPDIR: /var/lib/oozie Using JRE_HOME: /usr/lib/jvm/jre-openjdk Using CLASSPATH: /usr/lib/bigtop-tomcat/bin/bootstrap.jar Using CATALINA_PID: /var/run/oozie/oozie.pid [root@linuxhelp tmp]# service oozie status running
To check the status of Oozie admin, run the following command.
[root@linuxhelp tmp]# oozie admin -oozie http://localhost:11000/oozie -status
System mode: NORMAL
Open a browser and navigate to http://< ip_address> :11000/oozie
Comments ( 0 )
No comments available