2 Replies Latest reply on Aug 22, 2018 4:55 AM by Steve Ford

    UMS 6.5 - native memory leak

    Seshu Kumar Velagapudi New Member

      hi All,

                 there is a significant native memory leak in our Java processes. Below are the details:

      OS: Solaris

      Ultramessaging: UMP6.5

       

      Symptoms:

                - Heap memory is well under the limits. Heap dump confirms this.

                - top command shows the significant increase in the size without ever coming down.

      ex:

      PID USERNAME LWP PRI NICE  SIZE   RES

      3914 svcdrun  113  53    2 3920M 3322M

                - the services are crashing with unable to allocate memory for different reasons.

      ex:

      Native memory allocation (mmap) failed to map 131072 bytes for committing reserved memory

      Native memory allocation (mmap) failed to map 603914240 bytes for committing reserved memory

      .

      .

      root cause:

           - I tried using the dtrace and run the malloc and free for some time (~1minute) on a process with heavy LBM publication on multiple topics

           - after the run, collected and eliminated the malloc with matching free entries. Majority of the remaining entries pointing to, lbm_src_send or lbtrm_src_send. below are the few allocation entries which were never freed.

       

      Ptr=0x2d8d3140 Size=266

                    libc.so.1`malloc+0x45

                    liblbm.so.7.0.1`lbmc_create_buff+0xca

                    liblbm.so.7.0.1`lbm_any_src_sendv_ex_post_block+0x4fd

                    liblbm.so.7.0.1`lbm_any_src_sendv_ex+0x305

                    liblbm.so.7.0.1`lbm_src_sendv_ex+0x70

                    liblbm.so.7.0.1`lbm_src_send+0x3b

       

      Ptr=0x2d350280 Size=16

                    libc.so.1`malloc+0x45

                    liblbm.so.7.0.1`lbtrm_rate_ctlr_handle_pre_send+0x121

                    liblbm.so.7.0.1`lbm_rate_ctlr_sec_send+0x425

                    liblbm.so.7.0.1`lbtrm_src_send+0x225

                    liblbm.so.7.0.1`lbm_imbq_send+0x11b

                    liblbm.so.7.0.1`lbm_imbq_handle_tsni_timer+0x412

                    liblbm.so.7.0.1`lbm_timer_expire+0x40d

       

      Ptr=0x3f622a0 Size=16

                    libc.so.1`malloc+0x45

                    liblbm.so.7.0.1`lbtrm_rate_ctlr_handle_pre_send+0x121

                    liblbm.so.7.0.1`lbm_rate_ctlr_sec_send+0x425

                    liblbm.so.7.0.1`lbtrm_src_send+0x225

                    liblbm.so.7.0.1`lbm_imbq_send+0x11b

                    liblbm.so.7.0.1`lbm_src_implicit_batch_send+0x32a

                    liblbm.so.7.0.1`lbm_src_send_buff_internal+0x1ee

                    liblbm.so.7.0.1`lbm_any_src_sendv_ex_post_block+0x54d

                    liblbm.so.7.0.1`lbm_any_src_sendv_ex+0x305

                    liblbm.so.7.0.1`lbm_src_sendv_ex+0x70

                    liblbm.so.7.0.1`lbm_src_send+0x3b

       

      configuration:

      we are using late join for some of the topics and particularly when we send the data, we are using BLOCK, FLUSH flags.

       

      can somebody pls check this one, if they have any idea and point us in right direction?

       

       

      Best Regards,

      Seshu

        • 1. Re: UMS 6.5 - native memory leak
          Seshu Kumar Velagapudi New Member

          Hi ,

                 Thanks for your update. as this issue is causing significant problem in production, how to contact a dedicated customer support person (from our company perspective) who can help us in resolving this?

           

          Best Regards,

          Seshu

          • 2. Re: UMS 6.5 - native memory leak
            Steve Ford New Member

            Hello Seshu.  Have you gotten an adequate reply through the support ticket?

             

            In summary, there are two places where an Ultra Messaging publisher consumes potentially large amounts of memory: topic retention buffer (late join), and transport transmission window.  The sizes of both are configurable, but be aware that usually those sizes are specified in terms of bytes of application data, not total memory footprint.  Each message you send also consumes some overhead structures.  So sending very small messages can actually consume significantly more memory footprint than just the message bytes alone.

             

            Assuming the LBT-RM protocol, the configuration parameters of interest are: retransmit_retention_age_threshold (pre-6.10), retransmit_retention_size_threshold, retransmit_retention_size_limit, transport_lbtrm_transmission_window_size, transport_lbtrm_tranmission_window_limit.  The transport sizes are multiplied by the number of transport sessions, which is either explicitly configured by the option  transport_lbtrm_multicast_address, or is implicitly configured by  transport_lbtrm_multicast_address_low and  transport_lbtrm_multicast_address_high.