2 Replies Latest reply on Aug 8, 2019 6:19 PM by Bob Zhang

    ICRT - process-engine is stuck when too many DAS service process request in parallel

    Bob Zhang New Member

      Hi GCS,

      We are using ICRT to integrate our system into Salesforce, we use ICRT process to sync data bi-direction in real time.

      We also install DAS (Data Access Service) bpr to execute custom sql to our DB.

       

      Now I have a simple process as below as demo.

      The "execSQL" is a DAS service and execute a simple sql, such as "select * from table where id = 1"

      I'm doing the performance test. When I send  100 above process API requests in parallel. The process are stuck in "Running". And after minutes, the processes timeout. After that, any more process request will be stuck, that means my process-engine was stuck at this time, I have to restart secure agent.

       

      After investigate, I found that there were 50 Worker Threads stuck at "java.net.SocketInputStream.socketRead0(Native Method)". And after analysis, I think the reason is because

      1. DAS deploy as a API in tomcat. When Informatica process-engine resolve the DAS step, it send post request to DAS API and wait for response.

      2. DAS and other Informatica process step are use same worker thread pool which's max is 50.

      3. When process-engine receive process request, the worker thread begin to resolve, and if the step is DAS, it send a http request to itself tomcat, DAS API receive the request and waiting for worker thread to do work. but at this time, all these worker threads are the threads who sending the DAS request.

      Process flow like below, just like a dead lock problem.

       

      So I think the solutions to fix this problem can be:

      1. Separate DAS as a independent server. This require a new server and increase cost.
      2. Use independent worker manager for DAS.
      3. Increase more worker threads to reduce probability of this issue.

       

      The questions are:

      1. How to increase default worker manager thread, I cannot update the max thread for it.

      2. If question 1 is not possible, then is it possible to config custom worker manager and how?

      3. Even more, can I use custom worker manager only for DAS service an how?

       

      Thanks.