Can you please tell me if you might be getting the multiple record file from mainframe ? If yes, then I think it would be easy to use the PWX NRDB reader using a datamap that has all the 12 record type layouts plus header and trailer and write it to HDFS/Hive.
A successive job in Hadoop pushdown mode (or Native) can be used to process that data in HDFS/Hive, if needed. Please note that we cannot use NRDB sources in Hadoop pushdown mode.
If not, I am afraid we might not have an easy way to parse the different types of records Natively or using just the PDO and the logic to separate/identify the 12 record types will have to be built into the mapping itself.
I am getting the file from an external source not an internal mainframe source.
I can consistently identify the record type in the same positions for every non header/footer row.
Is there a way to dynamically route each record type through a different pipeline and somehow apply a control file to each pipeline?
I know I can manually code a router transformation with 12 groups. Then from here, can I somehow apply the record format using some sort of control file? I would prefer not having to substring all of the fields for the 12 different formats.
I'm also wondering if I could use a parser or data parser transformation? These transformations are new to me and I still don't understand them so I'm not sure if they could work for my use case.