6 Replies Latest reply on Feb 2, 2017 11:05 AM by mpappi

    Datomic (ETL)

    Santosh Nuli New Member

      Hi All,

       

      We have client requirement to extract data out of Datomic. I'm fairly new to this and appreciate if experts shed some light on this.

       

      Very much appreciate your time and support

       

      Santosh

        • 1. Re: Datomic (ETL)
          Sai Harsha New Member

          Hi Santosh,

          I have no idea on Datomic, but i found that Datomic is the upcoming new database and informatica powercenter will not support that database.

           

          Thanks,

          Harsha

          • 2. Re: Datomic (ETL)
            mpappi New Member

            Although Informatica may not support it using a native connection, you can look for an ODBC connection.

            Here you have the link for Support Statements from Informatica. There you will find one specific for ODBC drivers. Basically, if you face issues with the driver or connection, you will need support from your ODBC vendor as well.

             

            https://network.informatica.com/docs/DOC-16182

             

             

            hope it helps.

            cheers

            • 3. Re: Datomic (ETL)
              Santosh Nuli New Member

              Thank you Mpappi and Krishna. I will look into it.

              • 4. Re: Datomic (ETL)
                Nowell Henry New Member

                I'm on Santosh's team and came here looking for the same answer on how to integrate with Datomic through PowerCenter.  Datomic is an immutable data store that keeps track of all changes (http://www.datomic.com).  It is not an RDBMS like SQL Server or Oracle so interfacing with Datomic is very different.  Datomic does not provide an ODBC driver that we are aware of.


                We understand if Datomic isn't officially supported but maybe we can accomplish integration through a supported method such as publish/subscribe, custom java, etc.

                 

                We met with the application team today and discussed our options.  We have 3 routes we are exploring currently:

                 

                1.  Datomic developers will create a publish and subscribe through asyncrhonous message queue protocol? (AMQP I think it's called?).  I think this might require a PowerExchange adapter that can handle messaging queues as a source for publish/subscribe.

                 

                2.  Datomic has a rest api we can use (http transformation?).  This might not be good for performance reasons.  How would we determine change data?

                 

                3.  Datomic will build an interface in custom java.  This might be the best option.  Could we use custom transformation or java transformation in PowerCenter along with the custom code or would we need a different PowerExchange adapter to support this?

                 

                Any suggestions on the appropriate direction would be greatly appreciated.  None of us have experience with immutable databases like Datomic.

                • 5. Re: Datomic (ETL)
                  Guru

                  Neither do I (have experience with immutable data stores)

                   

                  Thanks for the detailed explanation, that makes sense.

                   

                  In general terms I think a message queue might be a good way. Informatica has several products which might fit in here, but that's not my area of expertise. I only seem to recall (off the top of my head, never worked with it myself) that AMQP should work fine.

                   

                  Calling a REST API from the Informatica Developer shouldn't be hard to do (again, not my area of expertise, I'm a PowerCenter guy), but indeed this may become troublesome in terms of performance if larger chunks of data need to be processed.

                   

                  Integrating Java code using the so-called Java Transformation (JTX) is not very difficult. Everyone who has a few months of experience with PowerCenter and basically understands how to write Java code (not related to GUIs but to back-end processing) should be able to understand how that works and can be done.

                  There's a (not 100% up-to-date but still correct) Best Practices article about the JTX available in Velocity, the Best Practices collection maintained by Informatica Professional Services. You can find a link under the top-level link Resources to be found near the upper border of the Informatica Network home page. As soon as you have browsed through the PowerCenter Transformation Guide (chapter about JTX), I suggest you read this Velocity article to "round up" your first knowledge about the JTX in general.

                  Whatever problem you face in this regard, don't hesitate to ask here, that's what we are here for. I just can't write the code for free in my spare time, I don't have enough spare time for that. *g*

                   

                  But as mentioned earlier, first I would check whether a message queue might be the fastest, easiest, and best-supportable solution. Even though it would first cost money for the respective PowerExchange products. In the end you might be able to save months of development and go-to-market time this way.

                   

                  Regards,

                  Nico

                  • 6. Re: Datomic (ETL)
                    mpappi New Member

                    Hi,

                     

                    I've gone thru some Datomic documents and it seems that there's no ODBC indeed. It does work differently than regular databases and even if Informatica ends up supporting it nativelly, it won't be like a regular database.

                     

                    Using a MOM like an IBM MQ Series or MS MQ as the middle guy between Datomic and PowerCenter may be the best choice in terms of performance. Also, it will be the most smooth way to connect to it. The rest API work good, but may not be very fast. Also, you'll need to use Informatica Data Services, and not purely PowerCenter.

                    The Java Tx is great also, but a little tricky with you having to manage the java libs and connection on your Java code. Another option is to export the Datomic to an XML and use PWC to process it.