how do we create a trailer file for AVRO target with DFS as source.

I need a trailer file for the AVRO target to be generated in which the DFS is a source in the dataflow.

 

 

Beginner Asked on June 3, 2020 in Diyotta Studio.
Add Comment
1 Answer(s)

Diyotta has added an option in data flow with DFS source and Avro target to generate trailer file

  1. DFS source will correspond to the Avro file in hdfs.
  2. DFS data object will be created using “copy as other” option from Hive data object corresponding to the Hive table build on these avro hdfs files.
  3. Avro target data object will be created using “create as target” or “copy as other” option from the DFS source.
  4. The Avro files in the hdfs will be moved as is to the local agent location. No processing will be done on these files in Diyotta.
  5. A single trailer file will be generated for the Avro files in the same format as generated for any other Avro generation.

To generate Avro files from dfs files set below properties for the data flow with source as DFS and target as Avro.

  • In the dfs source instance’s extract property provide the hdfs location to pick the files from

RE: how do we create a trailer file for AVRO target with DFS as source.

  • In the avro target instance’s load properties provide the avro file and trailer file details to be used to generate the files

RE: how do we create a trailer file for AVRO target with DFS as source.

Expert Answered on June 3, 2020.
Add Comment

Your Answer

By posting your answer, you agree to the privacy policy and terms of service.