RE: Use case that will read data from number of files and load into RDBMS table
We have a data ingestion use case that will read data from number of files and load into RDBMS table. The files are of same structure but the file names to read data from are updated daily in the RDBMS table. What options are available in Diyotta to handle this use case
While there are multiple ways available in Diyotta Studio to accomplish this use case, one commonly used solution is detailed below
-
- Create a DBCommand for the RDBMS that contains the list of files to be processed
-
- Write the select query that will return the list of files
-
- Choose the output directory where the results of the select query is to be save
- Choose the output directory where the results of the select query is to be save
-
- In data flow job that does the file to RDBMS ingestion, navigate to “Connections à Properties à E (for Extract properties)
-
- Choose “Indirect” for the property “File Data Reference”
- Choose “Indirect” for the property “File Data Reference”
We can also achieve this by looping through the list of files and loading them sequentially/parallelly by using the following approach
- Once the DBCommand is writing the list of filenames to the output file.
- Place the FILE_to_RDBMS job between the looper and list the contents of the file as shown below
- In the File_to_RDMS job navigate to connections –> properties –E[Extract properties] and include the looper parameter in the File_Name textbox.
