Are you trying to recon the source data with the target? And you are not using BDM as ETL?
How currently records are moved from source to target?
Well DEI has all the monitoring stats in the execution monitor for you to validate jobs. You also have a REST operations API to retrieve mapping execution stats.
So you have a variety of options built into the tool just like our other products.
Currently the business area is NOT using DEI/BDM for ETL. We are just now coming to them to start using EDC/Axon/BDQ.
I don't know how they are currently moving data, it could be a script or some other ETL tool.
I know that in order for them to reconcile Source to Target verification/validation (basically checking to make sure that row counts match) they would need to use BDM. I don't know what format the data is landing in, I imagine that if it is a nested file that this would be a bit more difficult vs a flat file or RDBMS.
I think they want to be notified when the data gets dropped not have to go look in the monitor service to find the job and then hunt around for the info, to be honest I don't blame them.
I will go look for the REST operations that you mention for mappings. I may have to come back if I can't find what I am looking for.
An audit step can be built to automatically pull this data. You can send it directly to them in an email if needed per session, you can route the information to a scorecard so they can have a dashboard for all mappings/loads which can notify them if something goes wrong.
All depends on what they are looking to achieve but we can do it.
I dug through several of the documents looking for the REST API operations to get the mapping execution stats and I can't find them. The only REST API stuff I could find was related to the business glossary and for Web Service nothing to get execution stats. I am looking in the 10.2.1 for BDQ/BDM docs because not all the docs are available for 10.4.
Can you tell me the name of the Document? I have the full doc set downloaded for 10.2.1.
I assume when you say Audit step you are talking about creating a mapping that checks the number of reads equals the writes. Correct?
You mentioned sending an email per session, do you mean per run?
In order to get the information to a scorecard you would have to create a profile correct?
I am looking into this now. Thank you this gives me a starting point.
Here you have the link to the documentation: Monitoring REST API
"With the REST API, you can get the mapping execution statistics for mappings that run on the native, Blaze, or Spark engines. You can get mapping statistics, mapping advanced statistics, mapping execution steps, and mapping execution plans."
Hope this helps.