I find myself needing to process key/value pairs to generate an unknown number of ports.
At run time, I will have a parameter to pass in key/value pairs to indicate the port name and the columns that will be concatenated to make up the port value.
I have not started coding this yet, but I'm assuming that what I wish to do can be accomplished with either a Python or Java transformation.
Assuming I can accomplish my goal with either, is one better than the other? Will java perform better? Will it make a difference if the data is small vs. large vs. really large?
I'm not sure about the performance but keep in mind that you can use Python transformation only if the execution environment is the Spark or Databricks Spark engine. So if you plan to run this on native execution environment you can only use Java for this