Refactoring for search

A few days ago I started implementing the search facilities for Xizi. Whilst doing this, I realised that I was going to have a huge problem with the performance of those searches unless I did something drastic to the way the data is stored.

The problem was that my naive strategy for search would be to whittle down the results from the database collection downwards, leading to the transfer of far too much data between MongoDB and the server process, and way too much of a memory footprint to handle the results. Furthermore, with the structure as it was, I could not move the query cost onto the server.

To tackle this, I have refactored all the storage mechanism so that data is stored as properly typed BsonDocuments, currently with String, Integer, DateTime and Decimal/Double (yeah yeah, I know) types supported. Although this is more fuss in the model-store to do this, previously all data was being stored as strings and that was far too simplistic. Now, I can create find queries which will execute within the database and return only the data we are interested in, instead of the mass that the server process would have to cull and filter.

The API for adding an Instance together with its Items has changed as a result, as each Item now specifies its type individually.