Microsoft Lead Gartner 2019 Magic Quadrant for Analytics and BI

By | Data Visualisation | No Comments

Hot BI news is the recently announced 2019 Gartner Magic Quadrant for Analytics and BI and Microsoft was the clear leader, driven in part without doubt by the explosive growth of PowerBI .

2019 Gartner Magic Quadrant BI & Analytics

2019 Gartner Magic Quadrant BI & Analytics

PowerBI is the story

PowerBI is advancing at a rate which its key competitors – Qlik & Tableau – are struggling to match. Further underpinning Microsoft’s lead is its ongoing investment in the underlying Azure Data & Analytics Platforms which give it an edge that competitors just can’t match.

One thing I am frequently hearing from the market is that other platforms are struggling to match PowerBI’s compelling price point – some might cynically say Microsoft are using their deep pockets to undercut everyone else. However in a recent conversation I had with a long standing Cognos customer, once they understood what the product could do – and how much cheaper and faster it would be – it drove them to reconsider their strategy.

I freely admit was initially cynical about Self Service BI a few years ago as it rapidly became transparent that for all the slickness of creating great looking reports the tools were still beholden to a clean set of well managed data. Now modern data platforms are reducing the cost and complexity of providing this data, I am now holding the position that self service BI can really deliver on its value rather than just provide a final polish to a Data Warehouse – as long as it is paired with such a platform.

I continue to maintain my (unpopular in some circles) position that Qlik has nothing unique to offer any more and is doomed to irrelevance unless it innovates or at the very least catches up with its competitors. Tableau remains a solid tool for self service analytics, but the absence of an underlying data platform is going to start hurting it before too long. I would expect it’s longevity to be tied to being acquired by someone suitably huge.

I also note the absence of any serious competition from the other two cloud megavendors. Google offers Data Studio & Amazon has Quicksight – but neither rate a mention. I would watch this space carefully as the pace of innovation by both companies is fierce and Google in particular has strong AI / ML capabilities. Both are also ramping up their own data platform services.

Outside the Big 3

If you are on any of the legacy on premise tools in the 2019 Gartner Magic Quadrant for Analytics and BI such as Cognos/IBM, Oracle or SAS then i’d be seriously considering where you go next. The pace of innovation in the cloud is hard to ignore and users risk lagging behind their competitors if they cling to these.

SAP and Salesforce have a strong story within their own source but have their weakness in using data from outside the native platform. Doing anything in SAP is horribly expensive, and my conversations with Salesforce BI users have not left a very positive impression of the tools’ capabilities.

The remainder either are strong in their niches and / or have minimal presence in Australia (Microstrategy is basically unsupported here as there’s no people doing it) so i’ll not pass comment on them.

Where to from here?

If you have reviewed the 2019 Gartner Magic Quadrant for Analytics and BI and decided you want to know more about PowerBI and the Microsoft Data Platform, we can help.

Full disclosure – FTS Data & AI are a Microsoft Gold Partner so this post is a bit biased. However if you are not using PowerBI and are looking at migrating to a more cost effective platform, want to understand how cloud capabilities are transforming data and analytics – or work for Qlik and want to lure me into a dark alley – please contact us.

FTS Data & AI are Microsoft Gold Partners Data Analytics

Microsoft Gold Partners Data Analytics

 

SSIS Integration Runtime Connectivity Testing

By | Data Platform | No Comments

SSIS Integration Runtime Connectivity Testing is hard as there is no physical Azure VM to log in to as part of the Azure Data Factory (ADF). While behind the scenes there is effectively a VM spun up there is no way to access it.

The scenario our Data Platform team faced was reasonably simple – we needed to connect to a 4D database that sat behind the Storman application that our customer used so that we could extract data for their various workloads. Because 4D is not supported by the Generic ODBC source in Azure Data Factory, we needed to use the 4D ODBC driver. This meant using SSIS to leverage the driver.

The client is well managed in terms of security so the target system can only be accessed within their network. Their Azure network was connected to theirs and properly secured, so part of the setup of the SSIS Integration Runtime in Azure Data Factory is to ensure that it is joined to the virtual network.

Houston, we have a problem

SSIS Azure Data Factory

SSIS Azure Data Factory

However, despite all this – we couldn’t get the ODBC connection to work when deployed. Due to stability issues our first suspect was the driver – after all it frequently crashed Visual Studio 2017 / SSDT and configuration was a pain. Also, initially we couldn’t connect on our dev machines as we weren’t on the clients VPN (easily fixed, fortunately). Then we had the wrong target server (again easily fixed).

Once we got on to ADF of course our debugging options got more limited as we now were having to do SSIS Integration Runtime Connectivity Testing without all the tools available on our desktops . Initially we struggled because the runtime was very slow at sharing its metadata (package paths, connection managers, etc.) so we weren’t initially sure it was even able to work with the driver. Eventually we got enough metadata to start playing with the JSON of the task to configure it. However we continued to get errors in ADF that were’t really helping.

Our breakthrough came when we remembered we could just connect to the more familiar and mature environment of the SSIS catalog that is deployed alongside the runtime. We configured the package correctly, ran it and got a more manageable ODBC error – “Cannot reach Destination server”. A quick ping from our desktops proved the server could be pinged, so as a test we used a simple package with just a script task to ping the server. This worked just fine on our desktop, but when deployed the script task reported failure.

So a quick connectivity test helped pin it down to probable network config issue. Now it’s in the Infrastructure teams hands to ensure everything is configured correctly, but at least we have (for now at least) got SSIS & the ODBC driver off the list of probable causes of the issue. It’s also taught us a few things about SSIS Integration Runtime Connectivity Testing.

Confusion Matrix showing True Positives, True Negatives, False Positives and False Negatives

False Negatives: Evaluating Impact in Machine Learning

By | AI & ML | No Comments

Recently, I had the opportunity to build a regression model for one of FTS Data & AI‘s customers in the medical domain. Medical data poses an interesting challenge for machine learning experiments. In most cases when running algorithms for binary classification, the expected result in the training set will contain a large percentage of negatives. For example the goal of an experiment might be to predict if – based on a set of known clinical test results – a patient has a certain medical condition. The percentage of positive results in such a set, if it is a generic dataset for a vast number of medical conditions will most likely be very low. As a result a machine learning model when initially tested using a small set of chosen features will most likely come up with a high number of false negatives.

The latter however is a big problem in experiments involving clinical data, i.e. categorising that a patient does not have a certain medical condition incorrectly could have disastrous consequences. Once a confusion matrix is built, the model’s effectiveness is measured using indicators such as area under curve, accuracy, precision, recall and F1 score. In medical datasets, recall plays a big role as it measures the impact of false negatives. It can therefore hold significant weight in determining the most appropriate model for a given experiment.

The definition of recall is –

Recall = (True Positives) / (True Positives + False Negatives)

In the confusion matrix, the denominator in this equation makes up the total actual positives. So, recall therefore is effectively measuring the correct positive predictions over the actual number of positives in the dataset. If there were no false negatives, recall would be at the ideal score of 1, however if a large number of actual positives were predicated as negatives (i.e. false negatives), recall would be much lower.

As the model evolves and more relevant features are chosen for prediction, recall should start improving. In domains such as medicine where false negative predictions can have dire consequences, the recall score should play a vital role in choosing the most optimum model.

Microsoft Gold Partners Data Analytics

By | Uncategorised | No Comments

We are excited to announce that FTS Data & AI have now become Microsoft Gold Partners in Data Analytics! This is an important development in our ongoing partnership with Microsoft in Australia. It shows we have reached the high standards expected of partners that qualify for the program. The Data Analytics competency is one that shows we have a deep expertise in the associated technologies – SQL Server, PowerBI and other Azure capabilities such as Azure Data Factory.

FTS Data & AI are Microsoft Gold Partners Data Analytics

Microsoft Gold Partners Data Analytics

Microsoft are a key partnership for us as they provide an industry leading platform for delivering Data Analytics solutions to our customers. Us achieving this partnership level is a demonstration of the capabilities of our technical team. It is also a recognition of the customer successes we have helped drive.

What does Microsoft Gold Partners in Data Analytics mean for customers?

Becoming Microsoft Gold Partners in Data Analytics has customer benefits in terms of confidence in our expertise and access to the latest technology.

For our customers it demonstrates that they are working with a technology partner that has certified personnel, with access to the latest in training and product updates. We have to maintain a number of certified people on our team who are qualified in the Data Analytics platforms that Microsoft provide. These certifications cover on premise and cloud technologies and range from traditional Business Intelligence through to Big Data solutions. Microsoft Gold Partners also get access to cutting edge product training as we are encouraged to upskill on the latest platforms.

Gold Partners also benefit from product information & roadmaps and industry insights direct from the Microsoft Data Platform team. As the product suite is updated we receive access to product news and information. We are also given Azure Credits so we can trial the new capabilities as part of our commitment to keeping our customers informed and aware of the latest developments.

D365 FinOps Reporting options

By | Data Visualisation | No Comments

The D365 FinOps Reporting landscape is a bit tricky to navigate. The documentation has not keeping up with the pace of product development. This makes the technical complexities difficult to navigate. In this quick post we provide an overview of what options you have. Dynamics 365 Finance and Operations (aka D365 FinOps) is a powerful ERP that can significantly improve business process efficiency, but for us at FTS Data & AI, we also see that there is a lot of additional value to be obtained from the data it captures.

What are the D365 FinOps Reporting options?

The three key tools within the D365 FinOps Reporting suite are:

  • Financial Reporting (aka: FRW)
  • PowerBI
  • SQL Server Reporting Services or SSRS (aka: Document Reporting Services)

There is also a fourth way – which is to pull the data out (aka BYOD or Bring Your Own Database) and report on it with whatever you like, but that’s for another post.

Financial Reporting

The simplest option – Financial Reporting (previously Management Reporting) – has a designer built into the D365 environment. It’s purpose is to provide Management Financial Reporting – i.e, Balance Sheet, Cash Flow, P&L – with some ability to customise your financial reports to your organisations viewpoint, with some filtering available for operational segments.

However its key limitation is that it is Financial in nature only and geared towards providing fairly static, statutory type reporting. Add to that the design and build of reports is very manual and it quickly reaches its limits. A typical example of such a report is shown below – you can see it is very much geared towards line item summaries.

D365 Finance Report Writer

D365 Finance Report Writer

PowerBI

PowerBI is planned to be the key reporting tool for D365 and may well replace the other options over time. A significant number of the embedded reports are already built in PowerBI. Part of any D365 Subscription is a PowerBI Embedded instance. All of the PowerBI reports in D365 are managed through this. Typically the generic reports do not suit the individual business needs and thus they need customising or extending. This is possible – complex to get started – but simple enough to execute once initiated, for example leveraging our Data Visualisation team who have deep experience doing this. Below is a “Before & After” set of examples to show the difference between out of the box reports and ones that have been enhanced by our team.

D365 FinOps PowerBI Report

D365 FinOps PowerBI Report – Before

 

D365 FinOps PowerBI Report

D365 FinOps PowerBI Report – After

A key restriction is that the embedded reports have to run off of the data in the Entity Store. This is a subset of the data in D365, with predefined and scheduled aggregations. To add content to the Entity Store requires a D365 developer, which adds cost to the report development process.

SQL Server Reporting Services (SSRS)

In every module in D365 there is a substantial set of SSRS reports that are included under the name of “Document Reporting Services“. These are embedded into D365 and present views of data with very simple parameters. You can view the technical details of included reports here and there are at least a thousand of them.

As with the PowerBI option it is possible to customise, brand and extend these reports. In this case the data source is any query that can be written against the Application Object Tree (AOT). Importantly this can show transactional as well as aggregate views. This allows for the production of customer facing content such as invoices.

Summary

So there are three options, each with their quirks:

  • FRW for detailed Financial Reporting
  • PowerBI for Aggregate Analytical Reporting
  • SSRS for highly customised reports

It’s not the clearest or most well documented landscape to navigate, but it is generally possible to produce the content you need to meet your organisations reporting requirements.