1 Reply Latest reply on Jun 16, 2015 1:41 PM by Sherwin Clutter

    Unrecoverable lost bursts when recovering from store

    New Member

      We are getting unrecoverable lost bursts when recovering messages from the store.

      Basically, there are around 250000 total messages from around 8 topics which we are recovering from the store. All topics recover from sequence id zero. After around 50000/60000 messages, we start to get unrecoverable lost burst from LBM. It looks like received messages are dropped due to a buffer size limitations.

       

      We have increased OS level receive buffers for TCP/UDP sockets to 64MB, but issue still exist. I have changed various configurations related to buffer sizes, but was not successful. In our case, both the store and receiver are running in the same machine.

        • 1. Re: Unrecoverable lost bursts when recovering from store
          Sherwin Clutter Active Member

          Hello Manusha,

           

          Recovery of messages for the stores occurs over TCP so you should not be losing messages due to buffer size.

           

          Recovery of large numbers of messages requires careful tuning of a number of configuration options.  One of the issues that you will run into when trying to recovery lots of data quickly is the "delivery_control_total_map_entries" threshold.  Which is not to say that you should just increase that configuration option.

           

          The option controls how many UM datagrams will be buffered by a context.  When the number of datagrams exceeds that threshold, the context will immediately deliver any loss records for the transport stream which received the last incoming datagram.

           

          This threshold is readily exceeded when trying to recover large numbers of messages simultaneously from multiple stores.

           

          The solution will involve some analysis of your particular recovery needs with tuning of the late-join configuration options to appropriately throttle the recovery rate of messages.

           

          You should raise a support request for more detailed assistance.

           

          Kind Regards,

          Sherwin Clutter