1 2 Previous Next 20 Replies Latest reply on Mar 31, 2021 4:24 PM by Vasudevan TS

    Shell Script to move files from Infa_shared directory to another server

    Saurabh Shrivastava Guru

      Hello All,

       

      ######## I want to achieve this via Shell Scripting ########

       

      I have a requirement to move a file(which will come on a daily basis) in unix box location (data/infa_shared/SrcFiles/ABC/). The format of the file is DailyRenewalsYYYYMMDD.txt. This file does not contains any header.

       

      Now I want to move this file a server 10.14.5.135 inside a folder ABC/incoming/

      I have the username and password to connect to this server.

      Below is the ask

      1. Move the file from path data/infa_shared/SrcFiles/ABC/ (as and when it comes) to remote server location 10.14.5.135 inside a folder ABC/incoming/

      2. If a file doesn't gets moved to remote server location then an failure email alert should be sent out automatically to a group (eg: abc@xyz.com)

      3. If a file gets moved to remote server location then a success email alert should be sent out automatically to a group (eg: abc@xyz.com).

       

      Someone please help me on this....

       

      Inside a workflow, I have created a file watcher job that would look for the file DailyRenewals*

      Once it finds the file, it gets succeeded and triggers a command task. Inside this command task, I would need to call the script which would move the file.

        • 1. Re: Move files from Infa_shared directory to another server
          H H Seasoned Veteran

          1) You can use FTP/SFTP to transfer the file : FTP

          2) WF email configuration : Using Email Tasks in a Workflow or Worklet

          • 2. Re: Move files from Infa_shared directory to another server
            Syed Aziz Guru

            Hello Saurabh,

             

            When there is a requirement to send file attachment through On Success E-mail, On Failure E-mail, or E-mail task and if the file contains some lengthy data in some lines inside, there would be a line break added after 990th character in the attachment file that is received at the recipient end.

             

            To send an alert message or email when the Informatica services stop unexpectedly configure the SMTP settings for the outgoing mail server to enable alerts.

             

            To configure SMTP settings for the domain, do the following:

            1. On the Domain tab, click Properties.
              The Properties tab appears.
            2. In the SMTP Configuration area, click Edit.
            3. Enter the following settings:
              • Server Host Name
                SMTP outbound mail server hostname.
                For example, enter the Microsoft Exchange Server for Microsoft Outlook.
              • Port
                Port used by the outgoing mail server. Enter any valid port number up to 65535. The default is 25.
              • User NameUsername for authentication upon sending if required by the outbound mail server.
              • Password
                User password for authentication upon sending if required by the outbound mail server.
              • Sender EmailThe address Email address the Service Manager uses in the From field when sending notification emails.
                If you leave this field blank, the Service Manager uses the default “Administrator@” as the sender.
            4. Click OK.

            This sends the email or alert message when the services stop unexpectedly.

            Subscribing to Alerts

            To subscribe to the alert do the following:

            1. In the Administrator Console, click on Manage. (Top left corner)
            2. Select Preferences > Edit.
            3. Select the Subscribe for Alerts option.
            4. Click OK.
            5. Enter the Email Address under Personal profile.
            6. Click OK.
            7. Start the services.

            Viewing Alerts

            When you subscribe to alerts, you can receive domain and service notification emails for certain events. When a domain or service event occurs that triggers a notification, you can track the alert status in the following ways:

            • The Service Manager sends an alert notification email to all subscribers with the appropriate privilege and permission on the domain or service.
            • The Log Manager logs alert notification delivery success or failure in the domain or service log.

            For example, the Service Manager sends the following notification email to all alert subscribers with the appropriate privilege and permission on the service that failed:

            From: Administrator@<database host>

            To: Jon Smith

            Subject: Alert message of type [Service] for object [HR_811].

            The service process on node [node01] for service [HR_811] terminated unexpectedly.

            In addition, the Log Manager writes the following message to the service log:

            ALERT_10009 Alert message [service process failover] of type [service] for object [HR_811] was successfully sent.

             

            You can review the domain or service logs for undeliverable alert notification emails. In the domain log, filter by Alerts as the category. In the service logs, search on the message code ALERT. When the Service Manager cannot send an alert notification email, the following message appears in the related domain or service log:

            ALERT_10004: Unable to send alert of type [alert type] for object [object name], alert message [alert message], with error [error].

             

            When are alerts generated?

             

            When a user subscribes to alerts in PowerCenter, the following emails are received in the following scenarios:

             

            MASTER GATEWAY NODEOTHER GATEWAY NODE S
            Email TypeGraceful ShutdownAbnormal TerminationGraceful ShutdownAbnormal Termination
            NODEThe master gateway node [node02] was terminated.The master gateway node [node02] was terminated.------
            DOMAINNode [node01] is elected as the master gateway node for domain [Domain_1].Node [node01] is elected as the master gateway node for domain [Domain_1].The domain node [node01] has become inactiveThe domain node [node01] has become inactive

            In case of Application Service failure (for example, Repository Service going down), the following alerts are received:

            EMAIL TYPEABNORMAL TERMINATION OF THE APPLICATION SERVICEGRACEFUL DISABLING OF THE APPLICATION SERVICE
            SERVICEThe service process on node [node01] for service [repo_1] terminated unexpectedly.
            Informatica_SMTP_Configuration.pdfInformatica_SMTP_Configuration.pdf

             

            Beat regards,

            Syed

            • 3. Re: Shell Script to move files from Infa_shared directory to another server
              Saurabh Shrivastava Guru

              Thanks for replying....


              I have updated my post to make the requirement more clear

              • 4. Re: Shell Script to move files from Infa_shared directory to another server
                H H Seasoned Veteran

                Command tasks are 2 types, post and pre session and others which can be called anywhere in WF. Use the command task to call the shell script which transfers the file using ftp. Once the ftp transfer is done from inside shell script, you can use post email task based on whether file was sent or not. You can set a WF variable after the command task run, which indicates whether the command task was successful or not.

                 

                Excerpt from developer guide :

                 

                "For example, a Command task produces an exit code output value that indicates whether the command ran successfully. The workflow cannot directly access this Command task output data. To use the data in the remainder of the workflow, you assign the exit code output to a workflow variable named CommandExitCode. Then use the CommandExitCode workflow variable in an expression for a conditional sequence flow. The Data Integration Service runs the next object in the workflow if the command ran successfully."

                 

                So basically if CommandExitCode is 1 then send success mail, else send failure mail.

                 

                Developer guide : Command Task Output

                • 5. Re: Shell Script to move files from Infa_shared directory to another server
                  Nico Heinze Guru

                  First of all the Developer guide ...(whatever) won't help here, this is the PowerCenter forum and not the Platform forum. So I dare to suppose that the original poster would need support for a PowerCenter workflow and not a platform workflow.

                   

                  Second the file watcher event cannot wait for a file mask, as far as I know, only for one particular file name.

                  Assuming my understanding is correct (I can't check myself at this moment, please check the documentation to make sure whether or not this works), you would need some other approach.

                  For example, a shell script which not only tries to FTP the file(s) but also sends the respective email.

                  Finally you can call this script in a so-called standalone command task (i.e. a command task in a workflow which is neither a pre- nor a post-session command). And in this invocation you can hand over the email addresses for success / failure emails from some workflow variables, if you want; this way you can make these email addresses flexible and changeable without having to edit the workflow or the script.

                   

                  Does that help?

                   

                  Regards,

                  Nico

                  • 6. Re: Shell Script to move files from Infa_shared directory to another server
                    Asmita Kesarwani Guru

                    Hi HH,

                     

                    With regards to file watcher event, you may refer the below thread which talks about almost the same scenario:

                     

                    How to use event wait task if the input file comes with dynamic names?

                     

                    Once you get this created ,as Nico mentioned you can a shell script which FTP the file(s) and sends the respective email with the help of command task.

                     

                    Hope this helps.

                     

                    Thanks,

                    Asmita

                    • 7. Re: Shell Script to move files from Infa_shared directory to another server
                      Saurabh Shrivastava Guru

                      I have got the script that would send the mail alerts...
                      However, I am not able to find the command which will move the file from infa_shared to remote server..
                      Tried several commands suggested in different groups but none of those worked

                       

                      • 8. Re: Shell Script to move files from Infa_shared directory to another server
                        Saurabh Shrivastava Guru

                        Can someone please help ...???

                         

                        I have found the below command

                        sftp $MFT_USER@$MFT_HOST:REMOTE_DIR_PATH <<< $'put SOURCE_PATH' >> LOG_PATH 2>&1

                         

                        Upon executing this command via shell script from command task, I am getting error

                         

                        THIS IS A PRIVATE COMPUTER SYSTEM ---

                        USAGE MAY BE MONITORED AND UNAUTHORIZED ACCESS

                        OR USE MAY RESULT IN CRIMINAL OR CIVIL PROSECUTION

                         

                        Permission denied, please try again.

                        Received disconnect from MFT_HOST port 22:2: Too many authentication failures for MFT_USER

                        Authentication failed.

                        Couldn't read packet: Connection reset by peer

                         

                        I have defined below in my script itself

                        MFT_HOST='10.XX.X.XXX'

                        MFT_USER='XXXXX'

                        MFT_PSWD='*****'

                         

                        However, If I execute the above same sftp command in putty editor then it prompts me for password and upon giving the password, it copies the file at the remote server.

                         

                         

                        Below is the script that I am using:

                         

                        ### Parameters to Pass

                        # $1 - SOURCE  Path

                        # $2 - Remote Server Path

                        # $3 - PUT Logfile

                        # $4 - Email Subject & Body

                        #######################################################################################

                        MFT_HOST='10.XX.X.XXX'

                        MFT_USER='XXXXX'

                        MFT_PSWD='*****'

                        FAILURE_EMAIL='saurabh.s@xyz.com'

                        SUCCESS_EMAIL='saurabh.s@xyz.com'

                         

                        cd SCRIPT_PATH

                        rm -f LOG_PATH/$3

                         

                        ##SYNTAX

                        ##sftp {user}@{host}:{remote-path} <<< $'put {local-path}'

                        sftp $MFT_USER@$MFT_HOST:REMOTE_DIR_PATH <<< $'put SOURCE_PATH' >> LOG_PATH 2>&1

                         

                        ## I have used below command as well, even this didn't worked

                        ##sftp $MFT_USER:MFT_PSWD@$MFT_HOST:REMOTE_DIR_PATH <<< $'put SOURCE_PATH' >> LOG_PATH 2>&1

                         

                        export RC=$?

                        if [[ "$RC" > 0 ]];

                        then

                        echo `date +"%m-%d-%y_%T"` >> LOG_PATH/$3

                        echo "ERROR: $4 Put File to REMOTE Server Failed" >> LOG_PATH/$3

                        mail -s "$4 Put File to REMOTE Server Failed" $FAILURE_EMAIL<<EOF

                        "$4" File is NOT present in SOURCE PATH. Check log for details.

                        EOF

                        exit -1

                        else

                        echo `date +"%m-%d-%y_%T"` >> LOG_PATH/$3

                        echo "SUCCEEDED: $4 Put File to REMOTE Server Succeeded." >> LOG_PATH/$3

                        mail -s "$4 Put File to REMOTE Server Successfully Completed" $SUCCESS_EMAIL<<EOF

                        "$4" File Successfully Transferred to REMOTE Server. Check log for details.

                        EOF

                        fi

                        exit 0

                        • 9. Re: Shell Script to move files from Infa_shared directory to another server
                          Nico Heinze Guru

                          The I/O redirection probably is the culprit here.

                          You might want to check the command "expect", this is capable of redirecting input to commands with a more complex I/O structure like sftp with its warnings.

                           

                          Having written that, your script seems to be full of one particular error: in many cases you have no $ sign before a variable, and that means that those variable names will not be replaced by the content of the respective variable.

                          For example the sftp command: no $ sign in front of the REMOTE_DIR_PATH nor before the LOG_PATH.

                           

                          Another error: you should never use any exit code outside the range 0 - 127. In particular the -1 you've used is nothing else but another writing style for 255, and that's error code 127 plus some system-specific error flag.

                           

                          Regards,

                          Nico

                          • 10. Re: Shell Script to move files from Infa_shared directory to another server
                            Saurabh Shrivastava Guru

                            My bad, all the variables have $ symbol appended to it...
                            Its just that i have replaced it with actual path/values to these random ones in order to post in this forum(as i can't use show actual values here)...
                            This is what it is, even then it is giving that error

                             

                            cd $SCRIPT_PATH

                            rm -f $LOG_PATH/$3

                             

                            ##SYNTAX

                            ##sftp {user}@{host}:{remote-path} <<< $'put {local-path}'

                            sftp $MFT_USER@$MFT_HOST:$REMOTE_DIR_PATH <<< $'put $SOURCE_PATH' >> $LOG_PATH 2>&1

                             

                            export RC=$?

                            if [[ "$RC" > 0 ]];

                            then

                            echo `date +"%m-%d-%y_%T"` >> $LOG_PATH/$3

                            echo "ERROR: $4 Put File to REMOTE Server Failed" >> $LOG_PATH/$3

                            mail -s "$4 Put File to REMOTE Server Failed" $FAILURE_EMAIL<<EOF

                            "$4" File is NOT present in SOURCE PATH. Check log for details.

                            EOF

                            exit -1

                            else

                            echo `date +"%m-%d-%y_%T"` >> $LOG_PATH/$3

                            echo "SUCCEEDED: $4 Put File to REMOTE Server Succeeded." >> $LOG_PATH/$3

                            mail -s "$4 Put File to REMOTE Server Successfully Completed" $SUCCESS_EMAIL<<EOF

                            "$4" File Successfully Transferred to REMOTE Server. Check log for details.

                            EOF

                            fi

                            exit 0

                            • 11. Re: Shell Script to move files from Infa_shared directory to another server
                              Nico Heinze Guru

                              I can only repeat my previous suggestion, please take a look at the command "expect", I've never used it myself but I've read many posts during the past decades where this was recommended for exactly such cases.

                               

                              Regards,

                              Nico

                              • 12. Re: Shell Script to move files from Infa_shared directory to another server
                                user136699 Guru

                                You mentioned that when you run sftp from the command line, it asks for a password. This is because the ssh connectivity is not setup.

                                SFTP sits on top of ssh - so you should be able to ssh to the other host from this machine, without having to enter the password.

                                Refer to this link on setting up the keys: SSH login without password

                                 

                                Once this is setup correctly, you should be able to ssh to the other host without having to enter the password and so also sftp.

                                -Girish

                                • 13. Re: Shell Script to move files from Infa_shared directory to another server
                                  Saurabh Shrivastava Guru

                                  I used another way and was able to transfer the file to remote server via below lftp command in my script

                                  lftp sftp://$MFT_USER:$MFT_PSWD@$MFT_HOST -e "put DailyRenewals.txt; bye" >> $log_path/$3 2>&1

                                   

                                  The only challenge that now am facing is that the file name will change on a daily basis like DailyRenewalsYYYYMMDD.txt.

                                   

                                  In above command, I have given a hard coded filename so it worked. But I want the filename with date part appended to it should get transferred...

                                  • 14. Re: Shell Script to move files from Infa_shared directory to another server
                                    Nico Heinze Guru

                                    Create this whole command as a string in a mapping. This way you have the file name and the date at hand and can concatenate the target file name at will.

                                    Write this string as one single line of text to a flat file target.

                                    Execute this target file as a post-session success command.

                                     

                                    Regards,

                                    Nico

                                    1 2 Previous Next