Requirements: Non-directive interviewsGregor-grams and Chappell-gramsHow the silverlight tag cloud works, including source codeTag cloud seeking harmonyUseful characters and keysThe Architectural Journal as an RSS feedWaiting for multiple threads ~ Figment Engine

Wednesday, July 1, 2009

Requirements: Non-directive interviews

Reading the Economist's article "The avatar will see you now" I came across:
"Meanwhile, a real person alongside the participant will answer any questions and, after the virtual visit is over, Dr Conboy-Hill will carry out “non-directive” interviews, a technique used by the police to encourage witnesses to choose what they want to talk about and to maximise recall"
What interested me was the snippet about "non-directive" interviews - it sounded like a technique that might be worthwhile for requirements gathering - I subsequently found "ACTIVE LISTENING IN THE NONDIRECTIVE INTERVIEW" which I think there are a few tricks worth picking up - especially rephrasing and the "brake shoes"...

of course I'm a sucker for any document that uses the word "interlocutor" ;-)

Tuesday, June 30, 2009

Gregor-grams and Chappell-grams

I've been doing a lot of work with an Enterprise Services Bus (ESB), and found Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions a very useful book - especially the diagram techniques "Gregor-grams".


These are useful for diagramming any message based system - I'm very keen on UML but it does not really address MOM systems in the large. Below is a Gregor-gram showing how a Sales Order Process might integrate SFdC, Zuora (Z-Billing) and Oracle Financials (a pattern that most businesses will need):

This diagram allows us to show how message move through the system and are transformed. There's also another good book is Chappell's Enterprise Service Bus which is focused on the ESB implementation of MOM (no surprise given the name!).

It has another diagramming techique, which at the moment I'm using for creating a static view of the ESB:
This is closer to a static UML view (deployment maybe), but it also allows you to express how the components connecting to an ESB exchange messages (HTTP/WCF etc). I feel that Gregor-grams are better for the dynamic view. The book never names these diagrams, so 've taken to calling them "Chappell-grams" given that they are meant to be an evolution of Gregor-grams.

Using Gregor-grams and Chappell-grams together allows us to document how a system works from both a dynamic and static view.

Thursday, June 25, 2009

How the silverlight tag cloud works, including source code

The TagCloud Silverlight application allows a user to see a visual representation of an RSS feed. It shows a visualization of the labels used within a feed, using size to indicate frequency and closeness to other tags showing frequency of co-occurrence.
This means that tags that occur often together (for example “Silverlight” and “Xaml”) will tend to appear next to each other, whilst tags that don’t occur together in the feed (for example “Silverlight” and “Credit Crunch”) will tend to be far apart.
Colour is also used to group the tag cloud, with the most frequently occurring tags taking a colour, and co-occurring tags inheriting that colour. This visually groups the cloud by colour as well.
Mouse-Over a tag shows lines to all other tags that it has occurred with, and clicking presents a popup menu of posts that mention that tag. Clicking on an item in the menu navigates to that post.
You can also click and drag a tag for fun, and watch all the other tags chase it..
Model
  • Cloud: the model of a RSS feed, separating the model from the UI. Processes an RSS feed and create subordinate Post and Tag with relationships. The cloud class is more of a management class that delegates most of the work to the Tag class.
  • Post: very simple class that represents a post/entry in an RSS feed. Used to allow navigation back to a particular post.
  • Tag: represents a tag (label/syndicationItem) in an RSS feed. Holds frequency count of how many times it has co-occurred with other tags. Has a SeekHarmony function that does the majority of the application “ah!” factor.
Approach
The application builds an internal model of the RSS feeds, critically in terms of tags and their co-occurrences. It then follows a simulation approach, where it tries to position each tag such that it is near tags it has co-occurred with, and far from tags it has not.
The algorithm for doing this does not pay attention to the overall cloud shape, but acts as an emergent system – the application of the simple harmony seeking behaviour between tags creating a global balance (generally a circle)
Code structure
Four parts
  • JavaScript library to make embedding the application less error prone
  • A server based proxy for getting RSS feeds to avoid cross domain issues
  • A silverlight library for math functions (mostly polar calculations and some extension methiods)
  • TagCloud silverlight application, with App and Page classes and three supporting classes (Cloud, Post and Tag)
Lifecycle
The silverlight application is embedded in a HTML page which passes a number of parameters via JavaScript to the Silverlight object creation code. The Silverlight application reads these settings and downloads the RSS feed via a server side proxy (to avoid cross-domain issues). The code then build an internal model of the RSS feed, with co-occurrence information about tags. On a timer basis the code then moves each tag such that it is nearer tags it co-occurred with, and further from ones it has not occurred with. Over time an order emerges which shows how the tags relate – tags that are nearer co-occur more.
Embedding the control in the page
For my blog I use JavaScript to embed the silverlight control
<div id='feCloud' style='text-align: center;'/>  
 <script src='http://www.figmentengine.com/tagCloud/feCloudv1.2.js' type='text/javascript'/>  
 <script type='text/javascript'>  
  var feCloudElementId = 'feCloud';  
  var feCloudFeedAddress = 'http://feedproxy.google.com/FigmentEngine';  
  var feCloudNavigateFormat = 'http://blog.figmentengine.com/search/label/{0}';  
  var feCloudSize = 400;  
  feTagCloudLoad(feCloudElementId, feCloudFeedAddress, feCloudNavigateFormat, feCloudSize, feCloudSize);  
 </script> 
 
The function in the feCloudv1.2.js is a wrapper for the Silverlight.js createObject call:
function feTagCloudLoad(elementId, feed, navigateFormat, width, height)
{
  var params = "";

  if (feed != null)
    params += "feedAddress=" + feed + ", ";
  if (navigateFormat != null)
    params += "navigateFormat=" + navigateFormat + ", ";

  var slWidth = 350;
  if (width)
   slWidth = width;
  params += "width=" + slWidth + ", "; 

  var slHeight = 350;
  if (height)
   slHeight = height;

  var container = document.getElementById(elementId);
  var cloudControl = document.createElement('object');
  
  cloudControl.setAttribute('data', 'data:application/x-silverlight-2,');
  cloudControl.setAttribute('type', 'application/x-silverlight-2');
  cloudControl.setAttribute('id', 'feCloudControl');
  container.appendChild(cloudControl);

  var host = "http://www.figmentengine.com/";
  var source = host + "tagCloud/TagCloudV1.2.xap";
  var parentElement = container;
  var callbackId = "feCloud";
  var properties = { width: slWidth, height: slHeight, version: "2.0.31005.0", enableHtmlAccess: "true" };
  var events = { };
  var initParams = params;
  Silverlight.createObject(source, parentElement,
   callbackId, properties, events, initParams);
}
I’ve put in bold the section that deals with passing the feed information via initParams. I also set enabledHtmlAccess to allow the Silverlight code to navigate the browser using HtmlPage.Window.Navigate.
Application startup
Reads the settings specified in the embedding JavaScript, most importantly the address of the RSS feed and how to navigate to page. It creates an instance of the Page and asks it to populate the TagCloud based on the feed address information.
Populating the cloud
Obtain the RSS feed: Due to issues in making requests to websites that the silverlight does not originate from I use the technique outlined at Franksworld (when your silverlight app needs to get data from another server that does not contain a crossdomain.xml then it proxies the call via the server)
protected void Page_Load(object sender, EventArgs e)
    {
        // Load the URI from the Query String
        string sourceUriString = Request.QueryString["Uri"];

        try
        {
            // Clear the output buffer
            Response.Clear();

            // Make new WebClient 
            WebClient webClientRequest = new WebClient();

            // Download data from URI
            byte[] requestByteArray = webClientRequest.DownloadData(sourceUriString);

            // Match the Mime Types
            string contentType = webClientRequest.ResponseHeaders["Content-type"].ToString();
            Response.ContentType = contentType;

            // Copy the Streams
            int requestByteArrayLength = requestByteArray.GetLength(0);
            Response.OutputStream.Write(requestByteArray, 0, requestByteArrayLength);
            Response.OutputStream.Close();

            // Exit the Page
            // see http://support.microsoft.com/kb/312629
            //Response.End();
            HttpContext.Current.ApplicationInstance.CompleteRequest();

        }
        catch(Exception ex)
        {
            // 5xx errors mean server error
            Response.StatusCode = 501;
            Response.StatusDescription = "Error encountered. Details: " + ex.Message;
        }
    }
Conversion of the RSS into the internal object model ignores most of the information in the feed, concentrating on posts, tags and co-occurrence.
Initializing the UI
At start-up the code creates a pool of connecter lines (to reduce the need to create them dynamically). It also create TextBlocks to represent each tag. It then start the timer to repeatedly move the tags around (seeking harmony)
Seeking Harmony
The code only works at a tag level, relying on emergent behaviour to get the UI effect. For each tag we do the following:
  • Each tag calculates how near it should be to all the other tags based on frequency of co-occurrence.
  • It then uses a polar conversion to work out the vector it would need to move in to get to this position.
  • We add up all these vectors for the tag, which gives us a vector that if it applied would be the ideal location for it.
  • We then down-scale the vector:
  1. So tags don’t jump massive distances
  2. Tags can react to where all the other tags have moved to on the next cycle 
Source code
Source code available as a zip

Tag cloud seeking harmony

I'm about to release the code for the tag cloud - here's a teaser of the code that each tag runs to seek harmony:

  internal void SeekHarmony(IEnumerable<Tag> valueCollection, double SIZE)
  {
   double maxOccurance = OccuredCount + 0.1d;

   // how we scale the whole cloud, 4 would be exactly the size of the area
   // 3 would be slightlight larger and 5 would be smaller
   const double ATTRACTION_SCALE = Math.PI;
   // how much tags that have not cooccurred repel each other
   // ϕ (golden ratio) gives us a nice aesthic
   const double REPEL_RATIO = 1.618d;
   // how much tags that cooccurred attract each other
   // it must be greater than the repel rate or everything
   // will fly away from each other
   const double ATTRACT_RATIO = Math.E;

   // assume we don't need to be moved
   Point offset = new Point(0, 0);

   // work out where the best place in 2d space to be
   // would be given where all the other tags are
   foreach (Tag otherTag in valueCollection)
   {
    // ignore ourself
    if (otherTag == this)
     continue;

    // work out vector bewteen us and the other tag
    Polar vector = Trigonometry.CartesianToPolar(this.Point, otherTag.Point);

    // how far apart?
    double distance = vector.Magnitude;

    // how much they repel by default
    double desiredDistance = maxOccurance * REPEL_RATIO;
    // did we co-occur with the other tag?
    int cooccurred = this[otherTag];
    if (cooccurred != 0)
     // we want to closer to the other tag based on frequency
     desiredDistance /= (cooccurred * (ATTRACTION_SCALE / ATTRACT_RATIO));

    // scale the distance based on the play area and how important
    // the other tag is compared to all the tags we co-occurred with
    distance = SIZE * (desiredDistance / (ATTRACTION_SCALE * maxOccurance));
    distance = (vector.Magnitude - distance); // towards not away

    // work out what vector needs to be applied to our
    // current position to move us closer
    Polar desiredVector = new Polar(vector.Direction, distance);
    Point offsetPoint = Trigonometry.PolarToCartesian(desiredVector);

    // accumlate all the proposed changes
    offset.X += offsetPoint.X;
    offset.Y -= offsetPoint.Y;
   }

   // update our new position, (scaled to fit within play area)
   // and also scaled to allow gradual movement
   double rateOfChange = SIZE / ATTRACTION_SCALE;
   point.X += offset.X / rateOfChange;
   point.Y += offset.Y / rateOfChange;
  }
hopefully I will post all the code and zip file tomorrow..

Friday, June 19, 2009

Useful characters and keys

Whilst writing code, comments or designs I often find these characters helpful, for example I often use "↑" to indicate "increase", so I can write "adding more servers ↑availability" or "homepage: ". They should all be unicode characters...

Useful characters
‹›«»
←↑→↓↔↕
$¢£¤¥€
½

‥ (two dot leader) … (ellipsis)
¡¿
§¶º†‡ (text markers)
– — ― (en, em, bar dashes)
™©®℅№
Numbers
¼ ½ ¾
⅓ ⅔
⅛ ⅜ ⅝ ⅞

° degree
Logic
≈ approx equal
≡, ≠ identity
¬, negation not
≤≥
∆ increament
Maths
∏ (product)
∑ (summation)
√ (root)
∂ (partial differential)
ⁿ (super n)
Bullets
■□▪▫○●◘◙◦
⌂ house! 

Thursday, June 18, 2009

The Architectural Journal as an RSS feed

I like the The Architectural Journal, but I find the "Architecture Journal Reader" not very helpful, mainly as I just want to read it in all my other feeds (as Brian on Software has said as well) - there is no way to get an RSS feed for it - which is annoying.

and you know what happens when you annoy a programmer..

I poked around the application directory and found that JournalReader.exe.config has a string "http://files.skyscrapr.net/users/JournalReader/data/master.xml" defined as DataFeedUri


this file contains sections for each language (English, Spanish etc) and links to the RSS feed - for english it's "en/toplevel.xml"
 
And guess what, toplevel.xml is an RSS feed!
though its not updated very quickly, and none of the links are correct ;-(
In addition toplevel.xml defines a set of sections (one for each edition of the journal) which you can then get the full xml (NITF format) for each article (I had to guess the location from looking at how it references images)

so an example full path is:

http://files.skyscrapr.net/users/JournalReader/data/en/Journal18/Jour18intro.doc.xml

note that in toplevel.xml the links all point to the same place which makes the feed unusable (clicking to read the full story takes to a page about Domain-Specific Modeling every time). The other problem is that the files don't get updated at the same time of a new edition - I guess someone has to remember to regenerate and upload them....

please MS. just make this available as a working RSS feed!

Monday, June 8, 2009

Waiting for multiple threads

Nice post on implementing a lock with timeout.

Though the code has two areas I would consider enhancing:
1) use WaitHandle.WaitAll to reflect the intent (and likelyhood) to aquire all locks or none [reason: performance]

2) There is a pathological case that two threads running at the same time will keep failing, if they hit the code for aquiring the locks in the for loop at the same time (since this is the case the code is written to handle it needs changing) - by implementing a variable amount of sleep time, thus ensuring that two threads that go to sleep at the same time don't wake up at the same time [reason: functional]