feel free to email me at

Bottlenecks are good. Just cook them right.

Look at livejournal or blogger. Readers submit tons of data per second. Common way is to process it and put results into a database. Users got errors during high load. Stuff panics.

Hire bottleneck to do a good job for you.

Store input without processing, just an ultra-fast write down. With a separate task process it piece-by-piece. Limit cpu time for this. Bothering your ultra-fast write down is bad. Put it on a separate server if you have one.

Keep an eye on size of data to be processed, measure average time it spent unprocessed. Introduce an artificial bottleneck, limit amount of pieces per second processed. Avoid real bottleneck at cost of small delay.

You could apply this to data loading too.