Choosing an Enterprise Data Modelling Tool

In my role as an Data Architect, I was recently asked to produce an enterprise data model to document the data flows and schemas for all of the data entites within our IT software infrastructure. It was clear that I was going to needed more features than Visio profides, which is after all just a diagramming tool.

I began investigating all of the modelling tools currently available in the market place to fine the most suitable tool for our medium sized organisation of around 2000 employees. The criteria was:

  • Ability to re-use the same objects in different views, e.g. current and future states
  • Ability to drilldown into views of varying levels of detail
  • Easily shared amongst colleagues
  • A flexible licensing model
  • Must be cost effective

Data Modelling Tools Assessment

Whilst the market leading tools such as CA ERWIN, Sybase PowerDesigner and IBM System Architect offer all the features I would need and a whole lot more, there was no way I was going to convince the decision makers that it would make sense financially to spend up to $25,000 USD for 4 or 5 licenses. The free tools such as ArchiMate and yEd were pretty good but didn’t really offer much more than Visio already does and were lacking in features such as import database and XML schemas. Sparx Enterprise Architect offered the best balance between functionality and cost so thats what I recommended using the following justifications;

Sparx Enterprise Architect provides a central, shared and joined-up representation of our information, analysis and system design at an affordable price.

Provides a rich data modelling environment to enable a consistent view of our information

  • Re-usable modelling framework allowing for views of varying detail for the same underlying objects
  • Drilldown from a high level view into the detail for a particular sub system

Provides an analysis environment to enable a connected view between business functional requirements and the required integration

  • Rich modelling for business, software and systems that reuses information and function definitions across diagrams
  • Business Analysts can use the UML capabilities and for defining and documenting requirements and connecting these with the information and functions on the designs
  • Can export models in PDF, Rich Text and HTML formats to enable sharing with the larger project team

Provides a functional environment to enable a connected view for our development and testing functions to ensure consistency across the project teams

  • Version Controlled Models to ensure the development and testing teams are working to the latest and correct design
  • Reverse engineering tools for XML, Databases and Code to help us visualise our exisiting systems and information
  • Full traceability from requirements to deployment

System.Transactions.TransactionException: The operation is not valid for the state of the transaction

When running large amounts of data into an MS SQL database via a composite WCF-SQL send port in BizTalk I would eventually get (after about 20mins) the error “System.Transactions.TransactionException: The operation is not valid for the state of the transaction”. Smaller amounts of data ran fine.

The solution was to turn off the transaction in the send port. Of course this means you lose the transactional capabilities but do you really want to lock a table on a live database for 20 mins anyway? It may be good practice to only turn this feature off when you know you are running a long transaction and then turn it back on again when you are finished.

Turn_Off_Transactions_In_Send_Port

Accessing Microsoft Dynamics CRM 4 Web Services from Windows 8 Modern UI App

Visual Studio 2012 is a big improvement overall but the ability to add a web reference has now disappeared completely, even from the “Advanced” button in the add service reference feature as per VS2010. Although most documentation seems to suggest adding an old asmx service as a service reference should work fine, it doesn’t always. The solution below works for the Microsoft Dynamics v4 web services but it is probably also valid for consuming other old style asmx services in Modern UI apps.

The 2 main problems I encountered were

  • I couldn’t add a custom header. In this case the CrmAuthenticationToken which defines which authentication protocol to use. Without this I’d simply get “The HTTP request is unauthorized with client authentication scheme ‘Negotiate’. The authentication header received from the server was ‘Negotiate,NTLM’.”
  • When you import the service reference, the autogenerated code in References.cd create the elements as XmlElementAttribute objects with an order by e.g. XmlElementAttribute(Order=1) which cause elements to mysteriously get set to null even though Fiddler proves that they are definitely not null. The reason is that the code doesn’t read them if they appear out of order in the response XML which in the Dynamics 4 API, they often are.

The solution to the first problem was mostly provided in this blog post, with just a few tweaks needed to make it work with the Windows Store App framework. It essentially creates a new EndPoint behavior which manually adds the CrmAuthenticationToken into the header. Here is the class in it’s entirety.

using MyCompany.Windows8.App.PersonHelper.cRM;
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Diagnostics;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.ServiceModel.Channels;
using System.ServiceModel.Description;
using System.ServiceModel.Dispatcher;
using System.Text;
using System.Threading.Tasks;
using System.Xml;

namespace MyCompany.Windows8.App.PersonHelper.ServiceCall
{

    public class CrmServiceConnectionParams
    {
        public String Scheme { get; set; }

        public String Url { get; set; }

        public CrmAuthenticationToken AuthenticationToken { get; set; }

        public BasicHttpSecurityMode SecurityMode { get; set; }

        private void setSecurityModeFromScheme()
        {
            switch (Scheme)
            {
                case "https":
                    SecurityMode = BasicHttpSecurityMode.Transport;
                    break;
                default:
                    SecurityMode = BasicHttpSecurityMode.None;
                    break;
            }
        }

        public CrmServiceConnectionParams(String url, CrmAuthenticationToken token)
        {
            if (url.Contains("://"))
            {
                string[] urlSplit = url.Split(new string[] { "://" }, StringSplitOptions.None);

                Scheme = urlSplit[0];
                Url = urlSplit[1];
            }
            else
                throw new ArgumentException("Failure creating CrmServiceConnectionParams instance. Invalid or missing URL scheme (e.g. 'http://').");

            AuthenticationToken = token;

            setSecurityModeFromScheme();
        }

        public CrmServiceConnectionParams(String scheme, String url, CrmAuthenticationToken token)
        {
            Scheme = scheme;
            Url = url;
            AuthenticationToken = token;

            setSecurityModeFromScheme();
        }

        public CrmServiceConnectionParams(String scheme, String url, CrmAuthenticationToken token, BasicHttpSecurityMode securityMode)
        {
            Scheme = scheme;
            Url = url;
            AuthenticationToken = token;
            SecurityMode = securityMode;
        }
    }

    public class CrmServiceInstance
    {
        private CrmServiceConnectionParams connectionParams;
        public CrmServiceConnectionParams ConnectionParams
        {
            get { return connectionParams; }
            set
            {
                connectionParams = value;
                spawnCrmService();
            }
        }

        private CrmServiceSoapClient crmService;
        public CrmServiceSoapClient CrmService
        {
            get { return crmService; }
            set
            {
                crmService = value;
                isCrmServiceReady = true;

                if (OnCrmServiceReady != null)
                    OnCrmServiceReady(this, new EventArgs());
            }
        }

        #region OnCrmServiceReady Event

        public event EventHandler<EventArgs> OnCrmServiceReady;

        private bool isCrmServiceReady;
        public bool IsCrmServiceReady
        {
            get { return isCrmServiceReady; }
        }

        #endregion

        public CrmServiceSoapClient CreateCrmService(String crmServiceUrl, CrmAuthenticationToken authToken, BasicHttpSecurityMode securityMode)
        {
            BasicHttpBinding httpBinding = new BasicHttpBinding(securityMode);
            httpBinding.MaxReceivedMessageSize = Int32.MaxValue;
            httpBinding.Security.Transport.ClientCredentialType = HttpClientCredentialType.Windows;

            EndpointAddress crmEndpoint = new EndpointAddress(crmServiceUrl);

            CrmServiceSoapClient crmService = new CrmServiceSoapClient(httpBinding, crmEndpoint);

            MessageHeader authTokenHeader = MessageHeader.CreateHeader("CrmAuthenticationToken",
            "http://schemas.microsoft.com/crm/2007/WebServices", string.Empty, new CrmAuthenticationTokenSerializer(authToken));

            crmService.ChannelFactory.Endpoint.EndpointBehaviors.Add(new CrmServiceBehavior(new CrmServiceMessageInspector(authTokenHeader)));
            crmService.ClientCredentials.Windows.AllowedImpersonationLevel = System.Security.Principal.TokenImpersonationLevel.Impersonation;
            crmService.ClientCredentials.Windows.ClientCredential = System.Net.CredentialCache.DefaultNetworkCredentials;

            return crmService;
        }

        private void spawnCrmService()
        {
            CrmService = CreateCrmService(
            ConnectionParams.Scheme + "://" + ConnectionParams.Url,
            ConnectionParams.AuthenticationToken,
            ConnectionParams.SecurityMode);
        }

        public CrmServiceInstance()
        {
            isCrmServiceReady = false;
        }

        private class CrmServiceMessageInspector : IClientMessageInspector
        {
            public MessageHeader ServiceHeader;

            #region IClientMessageInspector Members

            public void AfterReceiveReply(ref Message reply, object correlationState) { }

            public object BeforeSendRequest(ref Message request, IClientChannel channel)
            {
                request.Headers.Add(ServiceHeader);
                return null;
            }

            #endregion

            public CrmServiceMessageInspector(MessageHeader header)
            {
                ServiceHeader = header;
            }
        }

        private class CrmServiceBehavior : IEndpointBehavior
        {
            public CrmServiceMessageInspector ServiceInspector;

            #region IEndpointBehavior Members

            public void AddBindingParameters(ServiceEndpoint endpoint, BindingParameterCollection bindingParameters) { }

            public void ApplyClientBehavior(ServiceEndpoint endpoint, ClientRuntime clientRuntime)
            {
                clientRuntime.ClientMessageInspectors.Add(ServiceInspector);
            }

            public void ApplyDispatchBehavior(ServiceEndpoint endpoint, EndpointDispatcher dispatcher)
            {
                throw new NotImplementedException(); // Silverlight does not invoke this method.
            }

            public void Validate(ServiceEndpoint endpoint) { }

            #endregion

            public CrmServiceBehavior(CrmServiceMessageInspector inspector)
            {
                ServiceInspector = inspector;
            }
        }
    }

    public class CrmAuthenticationTokenSerializer : XmlObjectSerializer
    {
        #region CrmAuthenticationTokenSerializer Members

        private readonly string authType;
        private readonly string organizationName;
        private readonly string callerId;
        private readonly string crmTicket;

        public CrmAuthenticationTokenSerializer(CrmAuthenticationToken authToken)
        {
            callerId = Guid.Empty.ToString();
            authType = authToken.AuthenticationType.ToString();
            organizationName = authToken.OrganizationName;
            crmTicket = authToken.CrmTicket;
        }

        #endregion

        #region XmlObjectSerializer Members

        public override bool IsStartObject(XmlDictionaryReader reader)
        {
            return true;
        }

        public override object ReadObject(XmlDictionaryReader reader, bool verifyObjectName)
        {
            return null;
        }

        public override void WriteEndObject(XmlDictionaryWriter writer)
        {
            return;
        }

        public override void WriteObjectContent(XmlDictionaryWriter writer, object graph)
        {
            string tokenXmlLiteral = String.Empty;

            tokenXmlLiteral += "<AuthenticationType xmlns='http://schemas.microsoft.com/crm/2007/CoreTypes'>"
            + authType
            + "</AuthenticationType>";

            if (crmTicket != null && crmTicket != String.Empty)
                tokenXmlLiteral += "<CrmTicket xmlns='http://schemas.microsoft.com/crm/2007/CoreTypes'>"
                + crmTicket
                + "</CrmTicket>";

            tokenXmlLiteral += "<OrganizationName xmlns='http://schemas.microsoft.com/crm/2007/CoreTypes'>"
            + organizationName
            + "</OrganizationName>"
            + "<CallerId xmlns='http://schemas.microsoft.com/crm/2007/CoreTypes'>"
            + callerId
            + "</CallerId>";

            writer.WriteRaw(tokenXmlLiteral);
        }

        public override void WriteStartObject(XmlDictionaryWriter writer, object graph)
        {
            return;
        }

        #endregion
    }
}

So from the code that you want to make the call, you just create the client by using

CrmAuthenticationToken authToken = new CrmAuthenticationToken();
authToken.AuthenticationType = 0;
authToken.OrganizationName = "MyCompany";
authToken.CallerId = new Guid("00000000-0000-0000-0000-000000000000");

CrmServiceInstance instance = new CrmServiceInstance();
CrmServiceSoapClient client = instance.CreateCrmService("http://mycrmservername/MSCrmServices/2007/CrmService.asmx", authToken, BasicHttpSecurityMode.TransportCredentialOnly);

The solution to the 2nd problem was resolved in this blog post. Here are the steps I made to get it working.

  1. Click on PROJECT -> Show all files
    Show all files
  2. In the navigation pane browse to your service reference and open References.csOpen references file
  3. Open up Quick Find/Replace and search for “(\s*Order=\d+\s*\,?\s*)|(\,?\s*Order=\d+\s*)”, replace with nothing (i.e. blank) and make sure the replace all button is pressed. Then click Replace all. Note that if you ever re-import the service reference you will need to repeat thisFindReplace

Making Multiple Service Calls Asynchronously in a Windows 8 Modern UI App

Although it’s taken me a while to get my head around the asynchronous service calls with the await and async keywords, I think they are a great addition to the API. Simply adding the await keyword to an auto generated service proxy makes the service call run in a separate thread whilst the UI can continue it’s work as normal.

However, what if you load a page which makes multiple asynchronous calls to different services? Simply adding the await keyword to a method within the OnNavigatedTo or LoadState Page methods will mean that although the UI thread will still happily run in the background, each service call will be made sequentially which isn’t the most efficient way of doing it. Ideally we want all the service calls to be made at the same time to minimise the time it takes for the data to load.

The solution I found was to call the method without the await keyword and return the Task<> object. Then by using the ContinueWith method of the Task I could capture the results within a separate method as and when the service call finished. This way all the tasks are fired at the same time but the results are processed only when the method returned a result. Note that the results have to be brought back into the UI thread by using the Dispatcher object held within the page otherwise I got a message saying “the application called an interface that was marshalled for a different thread”.

protected override void LoadState(Object navigationParameter, Dictionary<String, Object> pageState)
{
	// Instantiate our helper class which makes all the service calls
	PersonServiceHelper helper = new PersonServiceHelper();
	// we could just simply run await() but because we are making multiple service calls, lets run them all asynchronously
	Task < MyCompany.Windows8.App.PersonHelper.PersonService.GetEmploymentDetailsResponse1 > t1 = 
		helper.getEmploymentDetails(PersonSearchResult.PACID);
	t1.ContinueWith(ReadEmploymentResults);
	
	Task<MyCompany.Windows8.App.PersonHelper.PersonService.GetWorkPhonesResponse1> t3 =
		helper.getWorkPhoneDetails(PersonSearchResult.PACID);
	t3.ContinueWith(ReadWorkPhonesResults);
	
	Task<MyCompany.Windows8.App.PersonHelper.PersonService.GetDirectoryDetailsResponse1> t4 =
		helper.getDirectoryDetails(PersonSearchResult.PACID);
	t4.ContinueWith(ReadDirectoryDetailsResults);
   
}

private async void ReadEmploymentResults(Task<MyCompany.Windows8.App.PersonHelper.PersonService.GetEmploymentDetailsResponse1> t)
{
	await this.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
	{
		this.DefaultViewModel["PersonEmployment"] = t.Result.GetEmploymentDetailsResponse.GetEmploymentDetailsResult;
	});
}


private async void ReadWorkPhonesResults(Task<MyCompany.Windows8.App.PersonHelper.PersonService.GetWorkPhonesResponse1> t)
{
	await this.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
	{
		this.DefaultViewModel["PersonWorkPhones"] = t.Result.GetWorkPhonesResponse.GetWorkPhonesResult.Phones;

	});
}

private async void ReadDirectoryDetailsResults(Task<MyCompany.Windows8.App.PersonHelper.PersonService.GetDirectoryDetailsResponse1> t)
{
	await this.Dispatcher.RunAsync(Windows.UI.Core.CoreDispatcherPriority.Normal, () =>
	{
		this.DefaultViewModel["PersonDirectoryDetails"] = t.Result.GetDirectoryDetailsResponse.GetDirectoryDetailsResult;
	});
}

Windows 8 Active Tiles not appearing when setting image as URL with Windows Authentication

I’d created Active Tiles using a Background task exactly as other tutorials said to and when using a text only active tile, the process worked fine. But whenever I specified an Image using a web URL the tiles simply did not appear. This had me stumped for ages and nothing useful was appearing in the event log other than “The message id for the desired message could not be found”. Useless!

After much scratching of heads I worked out that the Tile image URL does not work if the image sits on a web site which uses any kind of authentication mechanism such as Windows or Claims based. i.e. applications such as MS SharePoint or MS Dynamics CRM. The solution I cam up with was to download the image locally in the background task where you can specify client credentials and then reference it from the application temporary storage in the Tile.

Here is the code I used below. The Background task calls an async method to populate the tiles based on the results of an RSS feed.


using MyCompany.Windows8.App.MyCompanyNewsReaderHelper.DataModel;
using MyCompany.Windows8.App.MyCompanyNewsReaderHelper.Model;
using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading;
using System.Threading.Tasks;
using System.Xml;
using Windows.ApplicationModel.Background;
using Windows.Data.Xml.Dom;
using Windows.Storage;
using Windows.UI.Notifications;

namespace MyCompany.Windows8.App.MyCompanyNewsBackgroundTasks
{
    public sealed class MyCompanyNewsTileUpdater : IBackgroundTask
    {

        private List<FeedItem> _feedItems;
        private volatile object _lockFeedItemsList = new object();
        private IBackgroundTaskInstance taskInstance;
        private BackgroundTaskDeferral defferal;

        public void Run(IBackgroundTaskInstance taskInstance)
        {
            defferal = taskInstance.GetDeferral();
            try
            {
                RegisterDataSourceEvents();
                this.taskInstance = taskInstance;
                Debug.WriteLine("Trying to update MyCompanyNews Tile");
                FeedsDataSource.Instance.LoadFeedsParallel(true, false);

            }
            catch (Exception ex)
            {
                Debug.WriteLine("RSS Exception : " + ex.ToString());
            }
           
        }

        private void RegisterDataSourceEvents()
        {
            FeedsDataSource.Instance.FeedItemsAdded += OnFeedItemsAdded;
            FeedsDataSource.Instance.FeedItemsCleared += OnFeedItemsCleared;
            FeedsDataSource.Instance.FeedItemsLoadingCompleted += OnFeedItemsLoadingCompleted;
        }

        private async void OnFeedItemsLoadingCompleted(object sender, FeedItemsLoadingCompletedEventArgs feedItemsLoadingCompletedEventArgs)
        {
            Debug.WriteLine("Rss feed loading complete");
            _feedItems = FeedsDataSource.Instance.FeedItems;

           Debug.WriteLine(String.Format("Loaded {0} RSS items",_feedItems.Count.ToString()));
           
           var updater = TileUpdateManager.CreateTileUpdaterForApplication();
           updater.EnableNotificationQueue(true);
           updater.Clear();

           int i = 0;
           foreach (FeedItem item in _feedItems)
           {
               Debug.WriteLine("trying to load tile for post " + item.Title);
               if (!String.IsNullOrEmpty(item.mainImageUrl))
               {
                   // Unfortunately it seems that when the tile displays the image from a direct URL
                   // Windows authentication is supported. But it is here so we will download the image to application
                   Uri imgLocation = await GetLocalImageAsync(item.mainImageUrl, item.UniqueId);
                   var tileWide = TileUpdateManager.GetTemplateContent(TileTemplateType.TileWideSmallImageAndText03);
                   var tileWideVisual = tileWide.GetElementsByTagName("visual");
                   ((XmlElement)tileWideVisual[0]).SetAttribute("branding", "logo");

                   var textNodes = tileWide.GetElementsByTagName("text");
                   textNodes[0].AppendChild(tileWide.CreateTextNode(item.Title));
                   var tileImageAttributes = tileWide.GetElementsByTagName("image");
                   Debug.WriteLine("image URL is " + item.thumbnailImageUrl);
                   ((XmlElement)tileImageAttributes[0]).SetAttribute("src", imgLocation.AbsoluteUri);
                   ((XmlElement)tileImageAttributes[0]).SetAttribute("alt", "red graphic");


                   Debug.WriteLine("XML is " + tileWide.GetXml());
                   updater.Update(new TileNotification(tileWide));

                   i++;
               }

               // We can only store 4 tiles at a time
               if (i > 4)
                   break;
               
           }

           Debug.WriteLine("Finished updating tiles");
           defferal.Complete();
        }

        private void OnFeedItemsAdded(object sender, FeedItemsAddedEventArgs e)
        {
            Debug.WriteLine("Feed item added");
            _feedItems = FeedsDataSource.Instance.FeedItems;
        }

        private void OnFeedItemsCleared(object sender, FeedItemsClearedEventArgs feedItemsClearedEventArgs)
        {
            Debug.WriteLine("Feed items cleared");
                _feedItems = null;
        }

        private async Task<Uri> GetLocalImageAsync(string internetUri, string uniqueName)
        {
            if (string.IsNullOrEmpty(internetUri))
            {
                return null;
            }

            HttpWebRequest theRequest = HttpWebRequest.CreateHttp(internetUri);
            theRequest.UseDefaultCredentials = true;

            using (var response = await theRequest.GetResponseAsync())
            {
                using (var stream = response.GetResponseStream())
                {
                    var desiredName = string.Format("{0}.jpg", uniqueName);
                    var file = await ApplicationData.Current.LocalFolder.CreateFileAsync(desiredName, CreationCollisionOption.ReplaceExisting);
                    

                    
                    List<Byte> allBytes = new List<byte>();
                    using (Stream imageStream = response.GetResponseStream())
                    {
                        byte[] buffer = new byte[4000];
                        int bytesRead = 0;
                        while ((bytesRead = await imageStream.ReadAsync(buffer, 0, 4000)) > 0)
                        {
                            allBytes.AddRange(buffer.Take(bytesRead));
                        }
                    }
                    
                    await FileIO.WriteBytesAsync(file, allBytes.ToArray()); 

                    
                    return new Uri(string.Format("ms-appdata:///local/{0}.jpg", uniqueName), UriKind.Absolute);
                    
                }
            }
        }
    }
}