RE: Use case that will read data from number of files and load into RDBMS table

We have a data ingestion use case that will read data from number of files and load into RDBMS table. The files are of same structure but the file names to read data from are updated daily in the RDBMS table. What options are available in Diyotta to handle this use case

RaviP Beginner Asked on July 15, 2019 in Diyotta Studio.
Add Comment
1 Answers

While there are multiple ways available in Diyotta Studio to accomplish this use case, one commonly used solution is detailed below

    • Create a DBCommand for the RDBMS that contains the list of files to be processed

 

    • Write the select query that will return the list of files

 

    • Choose the output directory where the results of the select query is to be save

      RE: Use case that will read data from number of files and load into RDBMS table

 

    • In data flow job that does the file to RDBMS ingestion, navigate to “Connections à Properties à E (for Extract properties)

 

    • Choose “Indirect” for the property “File Data Reference”

      RE: Use case that will read data from number of files and load into RDBMS table

 

We can also achieve this by  looping through the list of files and loading them sequentially/parallelly by using the following approach 

  • Once the DBCommand is writing the list of filenames to the output file.
  • Place the FILE_to_RDBMS  job between the looper and list the contents of the file as  shown below

RE: Use case that will read data from number of files and load into RDBMS table

  • In the File_to_RDMS job navigate to connections –> properties –E[Extract properties] and include the looper parameter in the File_Name textbox.

RE: Use case that will read data from number of files and load into RDBMS table

Expert Answered on July 15, 2019.
Add Comment

Your Answer

By posting your answer, you agree to the privacy policy and terms of service.