Skip to content

Installing custom sqoop database drivers for Import Data Scheduler

Jump to bottom
Daniel Smith edited this page Aug 26, 2016 · 9 revisions

Initially, the Data Import Job Scheduler is configured with PostgreSQL drivers only.

Installing additional supported drivers, including MySQL, Oracle, and Teradata, requires some manual steps, as described below:

  1. Check for enabled databases (optional step), using the following commands (as appropriate).

$ cf target -o trustedanalytics -s platform

$ cf env workflow-scheduler|grep SQOOP_DATABASE

SQOOP_DATABASE_MYSQL: false

SQOOP_DATABASE_ORACLE: false

SQOOP_DATABASE_POSTGRESQL: true

SQOOP_DATABASE_TERADATA: false

  1. Download drivers refer to following list of databases with required files:
  1. (Kerberos environment only) Get the HDFS superuser keytab (can be obtained from env of auth-gateway or hdfs-broker - it's base64 encoded), as follows:
  • $ cf target -o trustedanalytics -s platform
  • $ cf env auth-gateway|grep HDFS_KEYTAB| awk '{print $2}' |base64 --decode > authgateway.keytab

  1. Upload your driver and keytab files to cdh-master-2, as follows:

    $scp *.jar cdh-jumpbox:

    $ssh cdh-jumpbox

    $sudo scp *.jar ec2-user@cdh-master-2:

  2. Ssh into cdh-master-2, as follows:

    $sudo ssh ec2-user@cdh-master-2

  3. (No-kerberos environment only) Move the driver files to /tmp/sqoop_drivers directory, as follows:

    $mkdir /tmp/sqoop_drivers

    $mv *.jar /tmp/sqoop_drivers

  4. Obtain the HDFS superuser identity, executing the following commands:

  • Environments with Kerberos:

    $ kinit -kt authgateway.keytab authgateway/sys

  • Environments without Kerberos:

    $sudo bash

    $su hdfs

    $cd /tmp/sqoop_drivers

  1. Put the drivers in the HDFS directory /user/oozie/share/lib/, as follows:

$ hadoop fs -put *.jar /user/oozie/share/lib

  1. Restart Oozie in Cloudera Manager.

  2. Enable the database in the TAP workflow-scheduler app, using the following commands:

$ cf set-env workflow-scheduler SQOOP_DATABASE_MYSQL true

Setting env variable 'SQOOP_DATABASE_MYSQL' to 'true' for the workflow-scheduler in org trustedanalytics / space platform as system admin...

**Tip:** Use 'cf restage' to ensure your env variable changes take effect, as follows:
`$ cf restage workflow-scheduler`

Once the workflow-scheduler has been restaged, you can import data using the TAP Import Job Scheduler using the newly installed drivers.

Home | FAQs

Clone this wiki locally