I am new to Informatica and I have been tasked with understanding the company's IDQ installation so that it can be used with our data.
I have been trying to run profiles. For a long time profiles larger than 1-3 Million records would crash the server until a recent specs upgrade and now larger profiles run without issue. I tried to follow the recommended specs found here for individual IDQ use. The machine currently has a 4 core CPU, 30 GB of RAM, and a 1 TB drive (4x2 TB is recommended but that is too much, I think).
I was able to run a profile with 8.5 million records (12 columns and 3 rules) in and it took 2 hours and 48 mins. That is about 1.20 milliseconds per records, and it compares to smaller profiles I have run before. But when last night I ran 10 million records, it took 10 hours and 14 minutes, that is 3.68 ms per record, which is more than 3 times the time it took for a profile with 15% less records.
The data source is a logical data object that reads from a database, so I thought maybe the source DB is the issue, but I checked and the time it takes to reach "Read [#] rows, read  error rows for source table [table] instance name [instance]" for both profiles scales similarly to the scaling in record count.
The real difference is in the time between " Writer run started" and "Writer run completed" (on the logs, I mean). For the 8.5M case, the writer run takes 2 hours 14 mins, while for 10M it takes 9 hours 32 mins.
And last night was the first time that the 10M run even finish at all. Previously (with just 12 columns and no rules), when I tried 10M records, it would just get stuck and nothing would happen. The DIS temp directory would not grow in size after a while [stuck at a few MB]. For the profiles that I am refering now, the 8.5M case had a disTemp of over 300 GB, while for last night's 10M, disTemp grew to 400 GB.
Could this be related to the DIS properties/settings, or could this be something else like the profile warehouse DB not being adequate? Below are images with the current set up. I have also attached logs for both profile runs.
I apologize for the long thread message.
DIS process settings: