3 Replies Latest reply on Nov 11, 2019 3:29 AM by Paolo Moretti

    Lookup Data cache size setting(Manual)

    Sourav Saha New Member

      Hi All,

       

      I am having a doubt regarding the Lookup Data cache size option that we get in the workflow settings:

       

      By default, the Integration Service allocates 1,000,000 bytes to the index cache
      and 2,000,000 bytes to the data cache when the Data cache size is set to 'Auto'.

      If I want more memory to be allocated for the same, for e.g. I allocate around 130 MB of cache for any lookup in the memory settings, overriding the default settings, to increase the performance of the Session, please let me know the consequences or changes to be faced while I run with such a manual setting, for e.g. will there be any space crunch in other load processes running in parallel under the same Integration Service, or any other consequences that you might know of, if I don't use the default memory settings.

       

      Please suggest on the above doubt.

       

      Regards,

      Sourav.

        • 1. Re: Lookup Data cache size setting(Manual)
          Paolo Moretti Seasoned Veteran

          Those values are the minimum amounts allocated by the Integration Service for each transformation in a session whose cache is set to "Auto", no matter the values defined for "Maximum Memory Allowed for Auto Memory Attributes" and "Maximum Percentage of Total Memory Allowed for Auto Memory Attributes".

           

          You can set the above "Maximums" to a higher value in order to get more physical memory allocated for each transformation. By reviewing the session logs, you can fine tune those Maximums.

           

          Every additional memory need that goes above that will be overflowed to a cache file on disk.

           

          If you know how much you want to allocate beforehand (e.g. by means of "Cache Calculator" or by reviewing past logs), you can define a specific values for Data/Index cache size at the transformation level. This will ensure such amount is allocated in memory (i.e. "Auto" configuration at the session level won't apply for that specific transformation).

           

          See  => Advanced Workflow Guide - Session Cache Limits

          • 2. Re: Lookup Data cache size setting(Manual)
            Sourav Saha New Member

            Hi Paolo,

             

            1. Can you tell me as to what might be the impact if I set the cache size manually? As the disk will have a given amount of space, also on the same disk many other loads will be running, so will the manual setting have any impact on other loads running under the same Integration Service?

            2. Or any other possible impacts I might see on manually increasing the cache size, that you know of?

            3. If we increase these two settings, "Maximum Memory Allowed for Auto Memory Attributes" and "Maximum Percentage of Total Memory Allowed for Auto Memory Attributes" , to a higher value, what is its physical impact on disk and what is its relation with the lookup data cache size?

             

            Please suggest on the above.

            • 3. Re: Lookup Data cache size setting(Manual)
              Paolo Moretti Seasoned Veteran

              The effects on disk are difficult to foresee. Unless you know beforehand the cache is definitely going to disk (e.g. persistent lookup), in most of the cases you get on disk only what cannot be allocated on memory.

               

              From my point of view the best approach is to leave everything set to Auto and configure the two maximums depending on how much memory is available on your system and how many jobs you run at the same time.

               

              Then you start reviewing the session logs to check whether a transformation's cache needs to be tuned. The session logs give you information about the cache's size, how much you need to keep all the cache in memory, and how busy a transformation was.

               

              If you reckon that a specific specific transformation needs to be tuned independently of the other transformations in the same session, then you start  thinking about configuring a specific value for its cache (i.e. you can use the cache calculator).

               

              With regards to "sizes", you are the only one who knows how much cache a transformation needs. No matter whether it is set to "Auto" or not, the main point is that what cannot be allocated in memory is overflowed to disk.

               

              If a cache is not persistent, then the cache files are usually deleted from disk once the session terminates.