Sorry for posting the solution so late. But you know better late then never :).
As i mentioned in my earlier blog that i don't want to change the code, so i was looking for some configurational change which would solve the problem. I know that the system resouces like cpu utilization and memory utilization of the server was fairly low, so i was looking for the way to use the hardware resource to correct the issue.
The Solution:
I first tried with splitting the big csv file into smaller csv files, but it did not worked for long as system started thrashing a lot after an hour or so and finally become un-responsive. Now i know this solution will not work until i control/reduce the number of request coming to a server/process.
Now i have two ways to control/reduce the number of request per process:
- Either increase the number of server or
- Have multiple processes running on same server (Web Garden in IIS 6.0).
Since hardware resources on server are under utilized so i decided to have multiple processes running on each server. The request coming to each server is divided among different processes running on that server which reduces the among of request coming to each process.
This trick worked and the problem got resolved. Thanks to IIS 6.0 for providing Web Garden features. It really provided what i was looking for.
Please refer below article for configuring Web Garden in IIS 6.0
Be careful when deciding on using web garden as it is resource intensive and also have the risk of contention if you are accessing a shared resouce.
