I'm not really sure I understood your question correctly, so please bear with me if some of the following points are not valid for your case. I'm trying to clarify for myself what problems you have.
As of my understanding Key masking with numeric data should deliver identical values when the same input values are used with the same seed. So I fully understand that you're puzzled about the results. The only reason I can imagine for a difference is that a Decimal value may not be treated as a Decimal value but that the binary representation of that Decimal value is taken as input for key masking; that would explain differences. But that's just a very rough guess.
You should not need to set up a mapplet to perform what you need. An advanced masking rule should be sufficient. Did you try that?
Regarding the need for some dictionary: I don't see any reason for this request. Can you leave this empty?
Again, a mapplet for this masking rule is (in my opinion) unnecessary, and for an advanced rule you should not be asked for a dictionary.
As your input is a mainframe file, just forget about the connection type. For flat files you can choose here whether the file shall be read / written via (S)FTP instead of being a local file; for relational sources / targets a connection variable can be set up via a parameter file, meaning that the same masking process can run against different source / target databases just by naming a different connection in the parameter file, no need to re-generate the workflows or anything similar. But - as far as I know - this feature probably doesn't apply to mainframe source files.