Advoyant uses a lot of back end perl programming to accomplish the complex searching and reporting that happens each time a file is uploaded.

We use forks to spin off the searching so that several uploads can take place all at the same time.

The scripts then update the database as they move threw the record set.

Database schema
From there the website using AJAX makes calls back to the database every couple seconds to report on the status of the upload. When complete the perl scripts create a clean CSV file that is ready for download by the customer, and a sample list of matches found by the search process.