About the javalib folder I can't tell, but I can answer your second question.
This Add Classpath setting is needed if your Java code depends on some JAR file which is no standard JAR file. For example, let's suppose you have to program a FTPS connection in a JTX. There are JAR files available for FTPS on the web, and of course when compiling resp. running your Java code you need this JAR file being appended to the CLASSPATH of your session.
For the following example let's assume that the JAR file you need is named FtpS.jar; on your Windows PC running the PowerCenter cients it's located in H:\Public\ExternalJars , on your server machine it's located in /app/java/externals/jars .
So in order to compile your Java code on the Windows PC, you have to add the path H:\Public\ExternalJars\FtpS.jar via this button Add Classpath.
In order to run this Java code on the PowerCenter server, you need to add the JAR file to the CLASSPATH of the session. This can be done in two ways. If your admin prefers, you can add the JAR path /app/java/externals/jars/FtpS.jar to the CLASSPATH of the integration Service. Otherwise my recommendation is to add it in the session settings (there's an extra field named Additional Class Libraries or so, I can't look it up at this moment).
Clear by now?
My apologies, I just recognised an error in my previous response. Here's the corrected part:
In order to compile the JTX in the Designer, you have to add the complete file path and name H:\Public\ExternalJars\FtpS.jar via the Add Classpath button (and not only the "directory path").
And the correct location in the session settings for the class path is (under the tab Properties) the text field Java Classpath ; here you have to enter the full path name of each JAR file you need in any JTX of the mapping.