how do we create a trailer file for AVRO target with DFS as source.
I need a trailer file for the AVRO target to be generated in which the DFS is a source in the dataflow.
Diyotta has added an option in data flow with DFS source and Avro target to generate trailer file
- DFS source will correspond to the Avro file in hdfs.
- DFS data object will be created using “copy as other” option from Hive data object corresponding to the Hive table build on these avro hdfs files.
- Avro target data object will be created using “create as target” or “copy as other” option from the DFS source.
- The Avro files in the hdfs will be moved as is to the local agent location. No processing will be done on these files in Diyotta.
- A single trailer file will be generated for the Avro files in the same format as generated for any other Avro generation.
To generate Avro files from dfs files set below properties for the data flow with source as DFS and target as Avro.
- In the dfs source instance’s extract property provide the hdfs location to pick the files from
- In the avro target instance’s load properties provide the avro file and trailer file details to be used to generate the files
