'ShopCircuit' is a price comparison engine that can read millions of products from suppliers' data feeds. It can handle dozens of product attributes. Using artificial intelligence, the engine tracks products' end-of-lifecycle to automatically remove products no longer available to consumers. Using compressed (gzipped) XML and CSV data as inputs, the engine can update a million product datasheets per hour on a single medium-end server.
We achieved high speeds without sacrificing scalability across multiple Memcached and MongoDB servers by moving feeds processing into RAM. We used Drupal as the datafeeds management backend. Apache Solr provided faceted search to end users. MongoDB carried the product data across several severs.
Built with: Drupal 7, Apache Solr, MongoDB, PHP, MySQL, Memcached