Just recently, my ex-boss Guy Murphy has released a candidate version of his behaviour-based micro-framework which comes from the same stable as Asura. As a porting exercise, I have decided to write the new version of Xizi using it as it favours functional decomposition in a way that Asura framework currently does not.
Guy's git-hub repository for Conclave can be found here: https://github.com/guy-murphy/Conclave
UPDATE:
Conclave has frozen and can be found at https://github.com/guy-murphy/conclave-public
Saturday 9 November 2013
Saturday 19 October 2013
API Separation and Everything After
Well, something else did crop up, and here I am again.
After far too much hand-wringing, I finally separated the Xizi API from the main hosted website and into its own silo.
But this is just treading water.
What I really want to do is two-fold;
One - Now that I know roughly what I want to achieve with Xizi, I feel that I should re-write this from scratch. The implementation is too complicated for what it actually should be doing.
Two - The more I work with the Asura framework, the more I wish that I'd made some different design decisions earlier on. The way I have implemented the behaviours does not lend itself well to functional decomposition and that's starting to drive me a bit mental as I have to change gears between different behavioural systems.
And there's not much point doing one without having done the other first for the sake of duplicated effort.
So, I'm going to re-write Asura.
Back in a bit.
[Gone coding]
After far too much hand-wringing, I finally separated the Xizi API from the main hosted website and into its own silo.
But this is just treading water.
What I really want to do is two-fold;
One - Now that I know roughly what I want to achieve with Xizi, I feel that I should re-write this from scratch. The implementation is too complicated for what it actually should be doing.
Two - The more I work with the Asura framework, the more I wish that I'd made some different design decisions earlier on. The way I have implemented the behaviours does not lend itself well to functional decomposition and that's starting to drive me a bit mental as I have to change gears between different behavioural systems.
And there's not much point doing one without having done the other first for the sake of duplicated effort.
So, I'm going to re-write Asura.
Back in a bit.
[Gone coding]
Sunday 14 July 2013
And... we're back
After a long stretch of working on a related side-project, I'm back and working on Xizi until something else rears its head. In truth, working on the other project has been incredibly useful in that it uses the Asura framework and this has given me experience of building a complete website with it, including interactive elements, asynchronous calls and discovering a new love ... Twitter Bootstrap.
Bootstrap, it feels like cheating. It's having as much of an effect on my view of front-end web development as learning model-store patterns influenced the way I architect application internals.
So, recent work has included improving the demo app, support for retrieving application details including auth tokens for users and available subscription events, support for themes in the template handler, default and user override parameters for an application for the condition engine, creating a user from an application, making Xizi.Client more robust by avoiding use of query strings where possible, making sure timeline events are spawned during test data setup, various bug fixes and remembering to add an update method for users :)
A whole bunch of things were ported in from my work on the other project, partly to do with front-end support - so themes, for example - and partly a big refactoring job which split the Context class into a basic base class and a web-related descendant class. The latter was due to requiring a job processor in the other project and I knew from recent experience that I could make that behaviour-based pretty easily.
Well, I'm stuck on train right now, so I'll get some other stuff done...
Bootstrap, it feels like cheating. It's having as much of an effect on my view of front-end web development as learning model-store patterns influenced the way I architect application internals.
So, recent work has included improving the demo app, support for retrieving application details including auth tokens for users and available subscription events, support for themes in the template handler, default and user override parameters for an application for the condition engine, creating a user from an application, making Xizi.Client more robust by avoiding use of query strings where possible, making sure timeline events are spawned during test data setup, various bug fixes and remembering to add an update method for users :)
A whole bunch of things were ported in from my work on the other project, partly to do with front-end support - so themes, for example - and partly a big refactoring job which split the Context class into a basic base class and a web-related descendant class. The latter was due to requiring a job processor in the other project and I knew from recent experience that I could make that behaviour-based pretty easily.
Well, I'm stuck on train right now, so I'll get some other stuff done...
Monday 6 May 2013
Progress report
It's amazing what a little bit of CSS can do to spruce things up a bit.
I've started making the Xizi web pages more presentable and it turned out that it wasn't so hard to do.
Here's the login page:
And a user's homepage:
I've started making the Xizi web pages more presentable and it turned out that it wasn't so hard to do.
Here's the login page:
 |
| (You'll never guess where I got the font from) |
 |
| (Placeholder graphics for now) |
You will perhaps notice that the FitnessApplication box shows a list of events that have occurred. This is an implementation of a timeline style history record that will be accessible if required. In this case, the timeline records are created by a Subscription Event that fires when an Instance is added. I wrote the event in a stored IronPython script, but it could just as well be in a .Net assembly too.
Along the way I fixed a few things and created a bunch more. Xizi.Client now transfers parameters via HTTP headers rather than on query string, as a date written in Universal Time was somehow getting a plus-sign stripped. As a part of that, all HTTP headers are now added in to the Asura.Web Context object as parameters automatically. I've also added a new Request behaviour - PreRender, which makes sure things like the User object are added to the Context.ControlState automatically and so will be available to the Razor templates.
Very pleased with progress and it's great to have something a bit more visual to work with.
Friday 3 May 2013
MongoDB search revisited
Well, Iceland was shit (ContextBot: chest infection, bad hotel location, misanthropy, expense.)
Anyway, moving on.
Re-written the Search function in the MongoDB store to split up the search criteria into separate ElemMatch queries after realising that I had failed to process multiple conditions correctly. By using this I now have a basic little form application for the FitnessApp idea that uses Xizi to pull back user's Instances that are within a 10% tolerance of entered values for newly entered running time and distance combinations.
I finished that in the departure lounge, while the plane was starting to board for London.
Most fun I'd had all week (*)
*: Apart from the tapas meal. That was awesome.
Anyway, moving on.
Re-written the Search function in the MongoDB store to split up the search criteria into separate ElemMatch queries after realising that I had failed to process multiple conditions correctly. By using this I now have a basic little form application for the FitnessApp idea that uses Xizi to pull back user's Instances that are within a 10% tolerance of entered values for newly entered running time and distance combinations.
I finished that in the departure lounge, while the plane was starting to board for London.
Most fun I'd had all week (*)
*: Apart from the tapas meal. That was awesome.
Wednesday 24 April 2013
API Rewrite #5 and Agents
Some time later... I emerge from performing API re-write number 5. It's all @uatecuk's fault. He persuaded me to split up the databases into separate Mongo collections for each level down to Instances. Previously, I had managed to get search working with a Map-Reduce operation to cut out bits of Databases that I didn't need, but this was essentially running the query twice - once over all an application's Databases, and then again over the reduced set of data which had matching Items in order to reduce the data. Now, I have search working with just a Mongo query, and no Map-Reduce operation. This is a shame because I actually thought I was understanding Map-Reduce, slightly, though it is very likely I am mistaken as it was probably leading me into a false sense of confidence and security before koshing me on the back of the head.
A second thing... I have managed to write an initial Agent using the embedded IronPython script engine. This has access to restricted parts of the database (restricted by interface), and this has led me to produce a scheme for loading and saving state. This all appears to work so far and the fledgling Agent can save its current work, perhaps for processing later. I'm not sure yet what restrictions to place on the timing of Agent execution. Maybe none. Certainly, aspects of this will be rolled into the connected API events, which will have access to the same storage items.
Right. Must sleep. Off to Iceland for Eve Online Fanfest 2013 tomorrow. \o/
A second thing... I have managed to write an initial Agent using the embedded IronPython script engine. This has access to restricted parts of the database (restricted by interface), and this has led me to produce a scheme for loading and saving state. This all appears to work so far and the fledgling Agent can save its current work, perhaps for processing later. I'm not sure yet what restrictions to place on the timing of Agent execution. Maybe none. Certainly, aspects of this will be rolled into the connected API events, which will have access to the same storage items.
Right. Must sleep. Off to Iceland for Eve Online Fanfest 2013 tomorrow. \o/
Sunday 14 April 2013
API and Subscribed Events rewrite
In the middle of writing the API, I was simultaneously writing the subscribed events model while trying to multiplex the functions that the API needed to perform, as well as cope with both serialised objects coming in as well as field by field changes...
... Welp.
The more I looked at it, the more I was unhappy with it. So, I took a bit of time away from the keyboard and spent a couple of train journeys working on ideas on paper instead.
I started re-working the API yesterday and finished that work today. There is now a base class that encapsulates a bunch of preparation and validation work that was common across most of the API calls. Proper exception handling for invalid API calls has improved the composition of the classes, simplifying the logic and cutting down on screen real-estate. The API calls now support both serialised and non-serialised parameters in a way that is much less complex and messy than what went before. Very pleased with progress.
The MongoDB store has been split up into domain-specific stores based on the interface that they honour, while the conversions to and from BsonDocument objects have been implemented as a static extension class, further reducing the number of lines to cope with in each file.
Subscription event handling has been substantially re-written after some input from a friend about how I had essentially re-implemented condition-action behaviours but in a different way. With that in mind, and after a couple of panics over some bad ideas, a single behaviour has been added to the flow, hooked onto the "api-event" message.
Originally I was trying to add custom behaviours to the flow based on the database that is used, but I fell foul of not being able to modify the message bus's listener collection.
The new behaviour has a standard condition-action pattern, the condition being the usual plus whether the correct items for a subscription event are recorded in the control state. The action is to run the event engine as before. The conditions are still run in a separate way, I guess, but the framework is tidier. Since the event engine runs asynchronously, then I suppose there is no way to run the events as individual behaviours from within the message bus anyway.
... Welp.
The more I looked at it, the more I was unhappy with it. So, I took a bit of time away from the keyboard and spent a couple of train journeys working on ideas on paper instead.
I started re-working the API yesterday and finished that work today. There is now a base class that encapsulates a bunch of preparation and validation work that was common across most of the API calls. Proper exception handling for invalid API calls has improved the composition of the classes, simplifying the logic and cutting down on screen real-estate. The API calls now support both serialised and non-serialised parameters in a way that is much less complex and messy than what went before. Very pleased with progress.
The MongoDB store has been split up into domain-specific stores based on the interface that they honour, while the conversions to and from BsonDocument objects have been implemented as a static extension class, further reducing the number of lines to cope with in each file.
Subscription event handling has been substantially re-written after some input from a friend about how I had essentially re-implemented condition-action behaviours but in a different way. With that in mind, and after a couple of panics over some bad ideas, a single behaviour has been added to the flow, hooked onto the "api-event" message.
Originally I was trying to add custom behaviours to the flow based on the database that is used, but I fell foul of not being able to modify the message bus's listener collection.
The new behaviour has a standard condition-action pattern, the condition being the usual plus whether the correct items for a subscription event are recorded in the control state. The action is to run the event engine as before. The conditions are still run in a separate way, I guess, but the framework is tidier. Since the event engine runs asynchronously, then I suppose there is no way to run the events as individual behaviours from within the message bus anyway.
Sunday 7 April 2013
Client updates, Scripting changes and Refactoring for profiling...
Busy few days and things are falling into place a bit more.
Some Asura toolkit changes which have made the Xizi.Client library a lot thinner. I was briefly using a separate DTO for transfer but after some soul-searching this has been scrapped and I have started using the native Xizi.Application.Model objects instead. To make that less traumatic (and the client much thinner), I have created a Xizi.Storage library and moved the implementation of the Mongo store and the various interfaces into it. Bits of the API have now been deprecated in preference for this passing of serialised objects. A next step for this is to make sure that the values can be passed safely in HTTP headers rather than on the query string; or at the very least to make the client completely agnostic to the underlying transport method.
I have rationalised the scripting, making the event scripting engine completely agnostic to the type of condition or action that it might be asked to consider. These are now behind common interfaces and the actual kind of provider loaded in each case is driven by an appropriate field in the EventDefinition object. Currently there is only an IronPython provider for conditions and actions, but I will certainly add a Json condition checker as it will be more suitably lightweight.
I did a little bit of experimentation with creating a dynamic Mongo query and so far I am pleased with the results. I'll try and get something official together for that next.
Finally, after getting an ANTS Performance Profiler licence at long last, I've had to do a huge bunch of re-engineering of the code in order for it to be properly profiled with the cheaper Standard edition. So, now there is a new Asura.Server library with a stand-alone HTTP server that can run Xizi, which can be profiled as a normal .exe application. Having to cope with two different types of incoming Http Context is such a massive pain; what were they thinking?
Some Asura toolkit changes which have made the Xizi.Client library a lot thinner. I was briefly using a separate DTO for transfer but after some soul-searching this has been scrapped and I have started using the native Xizi.Application.Model objects instead. To make that less traumatic (and the client much thinner), I have created a Xizi.Storage library and moved the implementation of the Mongo store and the various interfaces into it. Bits of the API have now been deprecated in preference for this passing of serialised objects. A next step for this is to make sure that the values can be passed safely in HTTP headers rather than on the query string; or at the very least to make the client completely agnostic to the underlying transport method.
I have rationalised the scripting, making the event scripting engine completely agnostic to the type of condition or action that it might be asked to consider. These are now behind common interfaces and the actual kind of provider loaded in each case is driven by an appropriate field in the EventDefinition object. Currently there is only an IronPython provider for conditions and actions, but I will certainly add a Json condition checker as it will be more suitably lightweight.
I did a little bit of experimentation with creating a dynamic Mongo query and so far I am pleased with the results. I'll try and get something official together for that next.
Finally, after getting an ANTS Performance Profiler licence at long last, I've had to do a huge bunch of re-engineering of the code in order for it to be properly profiled with the cheaper Standard edition. So, now there is a new Asura.Server library with a stand-alone HTTP server that can run Xizi, which can be profiled as a normal .exe application. Having to cope with two different types of incoming Http Context is such a massive pain; what were they thinking?
Monday 1 April 2013
Refactoring for search
A few days ago I started implementing the search facilities for Xizi. Whilst doing this, I realised that I was going to have a huge problem with the performance of those searches unless I did something drastic to the way the data is stored.
The problem was that my naive strategy for search would be to whittle down the results from the database collection downwards, leading to the transfer of far too much data between MongoDB and the server process, and way too much of a memory footprint to handle the results. Furthermore, with the structure as it was, I could not move the query cost onto the server.
To tackle this, I have refactored all the storage mechanism so that data is stored as properly typed BsonDocuments, currently with String, Integer, DateTime and Decimal/Double (yeah yeah, I know) types supported. Although this is more fuss in the model-store to do this, previously all data was being stored as strings and that was far too simplistic. Now, I can create find queries which will execute within the database and return only the data we are interested in, instead of the mass that the server process would have to cull and filter.
The API for adding an Instance together with its Items has changed as a result, as each Item now specifies its type individually.
The problem was that my naive strategy for search would be to whittle down the results from the database collection downwards, leading to the transfer of far too much data between MongoDB and the server process, and way too much of a memory footprint to handle the results. Furthermore, with the structure as it was, I could not move the query cost onto the server.
To tackle this, I have refactored all the storage mechanism so that data is stored as properly typed BsonDocuments, currently with String, Integer, DateTime and Decimal/Double (yeah yeah, I know) types supported. Although this is more fuss in the model-store to do this, previously all data was being stored as strings and that was far too simplistic. Now, I can create find queries which will execute within the database and return only the data we are interested in, instead of the mass that the server process would have to cull and filter.
The API for adding an Instance together with its Items has changed as a result, as each Item now specifies its type individually.
Friday 29 March 2013
Razor and Mono on Ubuntu
I just thought I'd write a little about the process of getting Xizi's new Razor-based template system working under Mono on Ubuntu.
I followed the guide at http://www.integratedwebsystems.com/2011/06/get-mvc3-razor-running-on-mono/#more-1156 by Nathan Bridgewater. This is immensely useful with lots of no-nonsense information on the process.
However, I did encounter a problem with how I'd set up my project in Visual Studio 2010. I had used the "Add Deployable Dependencies" feature which created a folder containing the various System and Microsoft namespace DLLs that would be required for the deployment. This configuration does NOT work under Mono, and rather frustratingly I wasted a good couple of hours trying to fix the problem rather than taking a step back and re-reading the instructions in Nathan's article.
The error that is produced complains about the version of DotNetOpenAuth that is linked to by the Microsoft.Web.WebPages.OAuth.dll file included in the dependencies. I tried following the thread of fixing this by trying to source a copy of DotNetOpenAuth, including the project it is from, of finding the right version for Mono and this version of .Net and the further I got then the more it unraveled.
In the end, I stopped, removed all that I had done and went back to the original deployment set up. After removing the Microsoft.Web.WebPages.OAuth.dll file and then receiving the error "System.Security.SecurityException: No access to the given key" I read about the same message in Nathan's article which gave a very basic instruction - "Delete Microsoft.Web.Infrastructure.dll from you bin directory" [sic]. The simple act of removing the Microsoft namespace DLLs from the included dependencies worked perfectly.
So, thanks to Nathan Bridgewater for compiling such a good guide; I'll try to read it more thoroughly next time :)
I followed the guide at http://www.integratedwebsystems.com/2011/06/get-mvc3-razor-running-on-mono/#more-1156 by Nathan Bridgewater. This is immensely useful with lots of no-nonsense information on the process.
However, I did encounter a problem with how I'd set up my project in Visual Studio 2010. I had used the "Add Deployable Dependencies" feature which created a folder containing the various System and Microsoft namespace DLLs that would be required for the deployment. This configuration does NOT work under Mono, and rather frustratingly I wasted a good couple of hours trying to fix the problem rather than taking a step back and re-reading the instructions in Nathan's article.
The error that is produced complains about the version of DotNetOpenAuth that is linked to by the Microsoft.Web.WebPages.OAuth.dll file included in the dependencies. I tried following the thread of fixing this by trying to source a copy of DotNetOpenAuth, including the project it is from, of finding the right version for Mono and this version of .Net and the further I got then the more it unraveled.
In the end, I stopped, removed all that I had done and went back to the original deployment set up. After removing the Microsoft.Web.WebPages.OAuth.dll file and then receiving the error "System.Security.SecurityException: No access to the given key" I read about the same message in Nathan's article which gave a very basic instruction - "Delete Microsoft.Web.Infrastructure.dll from you bin directory" [sic]. The simple act of removing the Microsoft namespace DLLs from the included dependencies worked perfectly.
So, thanks to Nathan Bridgewater for compiling such a good guide; I'll try to read it more thoroughly next time :)
Thursday 28 March 2013
Milestone: Events and Scripting
Big update today - added scripting and more efficient adding of instances.
Over the past few days I've been struggling with the composition of the event subscription model. I knew I wanted to have scripting in there somewhere, but I wasn't entirely sure where. However, I think I've reached the right kind of shape for it.
When Collections, Instances or Items are added to a Database, an event will be raised and broadcasted to any Subscribers that are listed in the Database. Each Subscriber is connected to an EventDefinition that describes both the trigger criteria and name of an action to be raised if that trigger fires.
An IronPython scripting engine has been added so that the trigger script of an EventDefinition can be interpreted as an expression yielding true or false. The event type, User, Database and the Collection, Instance, Item (as appropriate) are available to the IronPython script. At the moment, the test case is simply returning true if an Instance type matches a particular criteria.
The scripting engine is called asynchronously using a Task construct once the API has completed its operation. I've not used Tasks before so this is experimental for now and I'm not sure how they work under Mono yet. The engine is pluggable, so other scripting languages could also be made available in the future.
Over the past few days I've been struggling with the composition of the event subscription model. I knew I wanted to have scripting in there somewhere, but I wasn't entirely sure where. However, I think I've reached the right kind of shape for it.
When Collections, Instances or Items are added to a Database, an event will be raised and broadcasted to any Subscribers that are listed in the Database. Each Subscriber is connected to an EventDefinition that describes both the trigger criteria and name of an action to be raised if that trigger fires.
An IronPython scripting engine has been added so that the trigger script of an EventDefinition can be interpreted as an expression yielding true or false. The event type, User, Database and the Collection, Instance, Item (as appropriate) are available to the IronPython script. At the moment, the test case is simply returning true if an Instance type matches a particular criteria.
The scripting engine is called asynchronously using a Task construct once the API has completed its operation. I've not used Tasks before so this is experimental for now and I'm not sure how they work under Mono yet. The engine is pluggable, so other scripting languages could also be made available in the future.
Thursday 21 March 2013
XmlReader
Working with XmlReader is horrifying, but it's much, much faster than parsing with XmlDoc.
XiziClient has a much simpler test for operation success, now and ScrobblerApp has an optimised method for parsing the database xml it receives back from Xizi.
I'm a little bit concerned about what happens when these databases reach a certain size, but will press on for the moment.
XiziClient has a much simpler test for operation success, now and ScrobblerApp has an optimised method for parsing the database xml it receives back from Xizi.
I'm a little bit concerned about what happens when these databases reach a certain size, but will press on for the moment.
Tuesday 19 March 2013
API, Caching Razor and Session
In my haste to switch to using RazorEngine, I accidently the whole API. This has now been fixed.
Caching has been added to the Razor templates as well, based on [template-domain]-[template-name] keys.
Timings with ScrobblingApp are down to less than 2 seconds to read a database (must look into this) and saving a new instance and item in 50 milliseconds total. Not quite sure why those are taking so long.
Also, I've added API calls to the list of things that are excluded from having a session saved in the master cache, although it will be created and added to the context for the duration of the request.
Edit:
By setting RazorEngine to emit raw strings instead of html encoding, this has increased the performance of the database read api call - the compiled, cached version is much faster at processing. Time is now down to 0.5 seconds to read the database of 10 items. Still needs massive improvement but it's a start.
Caching has been added to the Razor templates as well, based on [template-domain]-[template-name] keys.
Timings with ScrobblingApp are down to less than 2 seconds to read a database (must look into this) and saving a new instance and item in 50 milliseconds total. Not quite sure why those are taking so long.
Also, I've added API calls to the list of things that are excluded from having a session saved in the master cache, although it will be created and added to the context for the duration of the request.
Edit:
By setting RazorEngine to emit raw strings instead of html encoding, this has increased the performance of the database read api call - the compiled, cached version is much faster at processing. Time is now down to 0.5 seconds to read the database of 10 items. Still needs massive improvement but it's a start.
Monday 18 March 2013
Migration to MongoDB
Have successfully migrated some of the core application model (XiziApplication, User) over to be stored in Mongo (or wherever) and not be in the Asura master cache anymore. This leaves just Templates and Session in the master cache, and only Session backed by Redis.
Sunday 17 March 2013
Razor
This weekend I have refitted the underlying toolkit to use Razor templates, using the very good RazorEngine.
Login and User pages now are in a basic working state after these changes.
Login and User pages now are in a basic working state after these changes.
Monday 11 March 2013
Milestone: Xizi staging server
After an evening of tinkering, I've managed to get the Xizi server working on my Ubuntu box, running under XSP4/Mono, with Redis and MongoDB all local. The machine hadn't been switched on since sometime last year so I took the opportunity to upgrade it to the current Ubuntu distribution while I was at it.
So, this box is essentially identical to the hosted testing server, which means it will be possible to just FTP the same code and config across and have it working there too.
Very pleased :)
So, this box is essentially identical to the hosted testing server, which means it will be possible to just FTP the same code and config across and have it working there too.
Very pleased :)
Sunday 10 March 2013
Friday 8 March 2013
Under construction
This will be the development blog for a project called Xizi.
Xizi is an application data storage framework, built using .Net and storage technologies like MongoDB and Redis.
More soon.
Xizi is an application data storage framework, built using .Net and storage technologies like MongoDB and Redis.
More soon.
Subscribe to:
Posts (Atom)