When we create a Data Object from the file, we notice that the delimiters are not correct.
Currently having this issue with DB Command from job flow. The query is executing successfully and loading the data into a .dat file. However, when we want to create a Data Object from the file, we notice that the delimiters are not correct. Note: They are correct during the Show Preview section but not after that part.
Can you please provide some insight into how we can for example specify a delimiter like “||” or any special character that can make it easier to differentiate the columns in the Data Object phase.
Our use case is to load the data from DB2 source to BigQuery target after the DBCMD
The suggested approach is to use the dataflow instead of DBCMD for pushing the file into the Big Query as we are inserting this into BigQuery and not into a Filesystem. Also in dataflow, you can include the column delimiters and text qualifiers as part of the extract properties.
For the use case: SELECT SQL in DBCMD and using that file to push it into a big query. You can use the below approach.
create a dataflow with source [DB2] and target [big query] and include the select statement in override query as shown below