Job Scheduler

The Job Scheduler allows you to import data from a SQL database into HDFS connected to TAP. Data can be imported in batch mode or by scheduling periodic, automatic updates.

Import data from a SQL database

From the TAP console main menu, navigate to Job Scheduler and then Import data.

TAP displays a form for you to fill out, starting with a job name, as shown below. Pick a unique name for your job.

• JDBC URI - You can enter the URI directly, or you can fill in the fields above the URI field to create the required schema for the jdbc uri:

jdbc:driver://host:port/database_name

• Username and Password - These are the credentials to connect to the data source

• Table - This is the name of the database table to be imported into HDFS.

• Destination dir - This is the directory in the target HDFS where you will store the imported data. Note: Make sure you have write access rights to this directory.

• Choose import mode - There are 3 import modes available: Append, Overwrite, and Incremental

  • Append - Each import will fetch the whole table into a separate file. Results of previous imports will not be overwritten. Files on HDFS will have names in the pattern: part-m-00000, part-m-00001, and so on.
  • Overwrite - Each import will fetch the entire source table and overwrite results of the previous import, using part-m-00000 for the filename each time.
  • Incremental - The import will fetch records having, in a column identified by Column name parameter, values not lower than the value provided by the Value parameter. Each subsequent import will fetch records with values in the aforementioned column higher than the previously fetched values. For this purpose (identifying ) we recommend using a numeric column, which is auto-incremented.
    • Column name - The column from the database (unique numeric format), against which Value will be checked; used for unique identification of data to be imported.
    • Value - A reference value used to filter out records from the source database - only records with values (in a column identified by ‘Column name’) not smaller than this reference ‘Value’ will be imported.

• Start time - The start time of your job.
  • Note: When you enter a Start time prior to the current time,, Oozie will try to “catch up” by executing jobs from the past.
• End time - The end time of your job.
  • End time should always be later than Start time.
• Frequency - The frequency with which your job will be submitted.
• Timezone - The id of the time zone for the entered start and end time.

Job browser

Selecting >Job browser from the main menu will allow you to view scheduled jobs. There are two tabs available from the >Job browser page: Workflow jobs and Coordinator jobs.

• Coordinator jobs - Coordinator jobs contain configuration and manage workflow jobs. You can click on See details to get additional information.
  • Details - Additional information about coordinator job

• Started workflow jobs - List of workflow jobs spawned by coordinator job. Each workflow job on the list have See details field, whcich will redirect you to selected workflow job details.

• Workflow jobs - In this page you can see list of workflow jobs. Workflow job represent import from database to hdfs. You can click on See details to get additional information:
  • Details - Additional information about workflow job
  • See logs - Here you can see logs related to workflow job

See also

Job Scheduler FAQ

Installing custom database drivers