Some time later... I emerge from performing API re-write number 5. It's all @uatecuk's fault. He persuaded me to split up the databases into separate Mongo collections for each level down to Instances. Previously, I had managed to get search working with a Map-Reduce operation to cut out bits of Databases that I didn't need, but this was essentially running the query twice - once over all an application's Databases, and then again over the reduced set of data which had matching Items in order to reduce the data. Now, I have search working with just a Mongo query, and no Map-Reduce operation. This is a shame because I actually thought I was understanding Map-Reduce, slightly, though it is very likely I am mistaken as it was probably leading me into a false sense of confidence and security before koshing me on the back of the head.
A second thing... I have managed to write an initial Agent using the embedded IronPython script engine. This has access to restricted parts of the database (restricted by interface), and this has led me to produce a scheme for loading and saving state. This all appears to work so far and the fledgling Agent can save its current work, perhaps for processing later. I'm not sure yet what restrictions to place on the timing of Agent execution. Maybe none. Certainly, aspects of this will be rolled into the connected API events, which will have access to the same storage items.
Right. Must sleep. Off to Iceland for Eve Online Fanfest 2013 tomorrow. \o/
Wednesday, 24 April 2013
Sunday, 14 April 2013
API and Subscribed Events rewrite
In the middle of writing the API, I was simultaneously writing the subscribed events model while trying to multiplex the functions that the API needed to perform, as well as cope with both serialised objects coming in as well as field by field changes...
... Welp.
The more I looked at it, the more I was unhappy with it. So, I took a bit of time away from the keyboard and spent a couple of train journeys working on ideas on paper instead.
I started re-working the API yesterday and finished that work today. There is now a base class that encapsulates a bunch of preparation and validation work that was common across most of the API calls. Proper exception handling for invalid API calls has improved the composition of the classes, simplifying the logic and cutting down on screen real-estate. The API calls now support both serialised and non-serialised parameters in a way that is much less complex and messy than what went before. Very pleased with progress.
The MongoDB store has been split up into domain-specific stores based on the interface that they honour, while the conversions to and from BsonDocument objects have been implemented as a static extension class, further reducing the number of lines to cope with in each file.
Subscription event handling has been substantially re-written after some input from a friend about how I had essentially re-implemented condition-action behaviours but in a different way. With that in mind, and after a couple of panics over some bad ideas, a single behaviour has been added to the flow, hooked onto the "api-event" message.
Originally I was trying to add custom behaviours to the flow based on the database that is used, but I fell foul of not being able to modify the message bus's listener collection.
The new behaviour has a standard condition-action pattern, the condition being the usual plus whether the correct items for a subscription event are recorded in the control state. The action is to run the event engine as before. The conditions are still run in a separate way, I guess, but the framework is tidier. Since the event engine runs asynchronously, then I suppose there is no way to run the events as individual behaviours from within the message bus anyway.
... Welp.
The more I looked at it, the more I was unhappy with it. So, I took a bit of time away from the keyboard and spent a couple of train journeys working on ideas on paper instead.
I started re-working the API yesterday and finished that work today. There is now a base class that encapsulates a bunch of preparation and validation work that was common across most of the API calls. Proper exception handling for invalid API calls has improved the composition of the classes, simplifying the logic and cutting down on screen real-estate. The API calls now support both serialised and non-serialised parameters in a way that is much less complex and messy than what went before. Very pleased with progress.
The MongoDB store has been split up into domain-specific stores based on the interface that they honour, while the conversions to and from BsonDocument objects have been implemented as a static extension class, further reducing the number of lines to cope with in each file.
Subscription event handling has been substantially re-written after some input from a friend about how I had essentially re-implemented condition-action behaviours but in a different way. With that in mind, and after a couple of panics over some bad ideas, a single behaviour has been added to the flow, hooked onto the "api-event" message.
Originally I was trying to add custom behaviours to the flow based on the database that is used, but I fell foul of not being able to modify the message bus's listener collection.
The new behaviour has a standard condition-action pattern, the condition being the usual plus whether the correct items for a subscription event are recorded in the control state. The action is to run the event engine as before. The conditions are still run in a separate way, I guess, but the framework is tidier. Since the event engine runs asynchronously, then I suppose there is no way to run the events as individual behaviours from within the message bus anyway.
Sunday, 7 April 2013
Client updates, Scripting changes and Refactoring for profiling...
Busy few days and things are falling into place a bit more.
Some Asura toolkit changes which have made the Xizi.Client library a lot thinner. I was briefly using a separate DTO for transfer but after some soul-searching this has been scrapped and I have started using the native Xizi.Application.Model objects instead. To make that less traumatic (and the client much thinner), I have created a Xizi.Storage library and moved the implementation of the Mongo store and the various interfaces into it. Bits of the API have now been deprecated in preference for this passing of serialised objects. A next step for this is to make sure that the values can be passed safely in HTTP headers rather than on the query string; or at the very least to make the client completely agnostic to the underlying transport method.
I have rationalised the scripting, making the event scripting engine completely agnostic to the type of condition or action that it might be asked to consider. These are now behind common interfaces and the actual kind of provider loaded in each case is driven by an appropriate field in the EventDefinition object. Currently there is only an IronPython provider for conditions and actions, but I will certainly add a Json condition checker as it will be more suitably lightweight.
I did a little bit of experimentation with creating a dynamic Mongo query and so far I am pleased with the results. I'll try and get something official together for that next.
Finally, after getting an ANTS Performance Profiler licence at long last, I've had to do a huge bunch of re-engineering of the code in order for it to be properly profiled with the cheaper Standard edition. So, now there is a new Asura.Server library with a stand-alone HTTP server that can run Xizi, which can be profiled as a normal .exe application. Having to cope with two different types of incoming Http Context is such a massive pain; what were they thinking?
Some Asura toolkit changes which have made the Xizi.Client library a lot thinner. I was briefly using a separate DTO for transfer but after some soul-searching this has been scrapped and I have started using the native Xizi.Application.Model objects instead. To make that less traumatic (and the client much thinner), I have created a Xizi.Storage library and moved the implementation of the Mongo store and the various interfaces into it. Bits of the API have now been deprecated in preference for this passing of serialised objects. A next step for this is to make sure that the values can be passed safely in HTTP headers rather than on the query string; or at the very least to make the client completely agnostic to the underlying transport method.
I have rationalised the scripting, making the event scripting engine completely agnostic to the type of condition or action that it might be asked to consider. These are now behind common interfaces and the actual kind of provider loaded in each case is driven by an appropriate field in the EventDefinition object. Currently there is only an IronPython provider for conditions and actions, but I will certainly add a Json condition checker as it will be more suitably lightweight.
I did a little bit of experimentation with creating a dynamic Mongo query and so far I am pleased with the results. I'll try and get something official together for that next.
Finally, after getting an ANTS Performance Profiler licence at long last, I've had to do a huge bunch of re-engineering of the code in order for it to be properly profiled with the cheaper Standard edition. So, now there is a new Asura.Server library with a stand-alone HTTP server that can run Xizi, which can be profiled as a normal .exe application. Having to cope with two different types of incoming Http Context is such a massive pain; what were they thinking?
Monday, 1 April 2013
Refactoring for search
A few days ago I started implementing the search facilities for Xizi. Whilst doing this, I realised that I was going to have a huge problem with the performance of those searches unless I did something drastic to the way the data is stored.
The problem was that my naive strategy for search would be to whittle down the results from the database collection downwards, leading to the transfer of far too much data between MongoDB and the server process, and way too much of a memory footprint to handle the results. Furthermore, with the structure as it was, I could not move the query cost onto the server.
To tackle this, I have refactored all the storage mechanism so that data is stored as properly typed BsonDocuments, currently with String, Integer, DateTime and Decimal/Double (yeah yeah, I know) types supported. Although this is more fuss in the model-store to do this, previously all data was being stored as strings and that was far too simplistic. Now, I can create find queries which will execute within the database and return only the data we are interested in, instead of the mass that the server process would have to cull and filter.
The API for adding an Instance together with its Items has changed as a result, as each Item now specifies its type individually.
The problem was that my naive strategy for search would be to whittle down the results from the database collection downwards, leading to the transfer of far too much data between MongoDB and the server process, and way too much of a memory footprint to handle the results. Furthermore, with the structure as it was, I could not move the query cost onto the server.
To tackle this, I have refactored all the storage mechanism so that data is stored as properly typed BsonDocuments, currently with String, Integer, DateTime and Decimal/Double (yeah yeah, I know) types supported. Although this is more fuss in the model-store to do this, previously all data was being stored as strings and that was far too simplistic. Now, I can create find queries which will execute within the database and return only the data we are interested in, instead of the mass that the server process would have to cull and filter.
The API for adding an Instance together with its Items has changed as a result, as each Item now specifies its type individually.
Subscribe to:
Posts (Atom)