Agile BI Development Done Right

Business intelligence (BI) has developed to such advanced degree that not only corporations but also many small to medium sized businesses can’t fathom their lives without it.

BI solutions provide accurate analysis of an organization’s data in a manner that allows managers to make quick decisions based on company performance. Based on real-time visualized data about their marketing and sales initiatives, marketing and sales departments can become better aligned.

As more and more Chief Information Officers (CIOs) understand the need to adapt to their customers’ constantly evolving needs, they find that their business intelligence tools need to be able to respond accordingly.

So how can organizations best incorporate agile BI development into their projects to deliver these valuable business insights to their customers?

An Iterative, Non-Linear Approach to BI Projects

What often happens in BI development is that the business environment changes before the solution can be delivered. One simple explanation for this is the reliance on the traditional “waterfall” development approach.

First used in the late 70s in scholarly articles and later as the standard the US Department of Defense developed for working with software development contractors, this method is focused on the long-term results and relies on standards, stability, and centric control.

The idea is that no stage of development should be repeated since that would be inefficient and costly. This linear approach is particularly useful when the project is clearly defined and end users don’t need to change its scope after it has begun, which is not the case for most BI development projects.

The BI development life cycle instead consists of a number of different stages, each of which can evolve and change through the project’s lifetime:

• Gathering of Requirements
• Data Source Identification
• Data Integration (ETL) Development
• Data Warehouse Development
• Reporting and Analytics Development
• BI Solution Testing, Deployment, and Implementation

To view the DW/BI Lifecycle click here

Agile development, in contrast to the waterfall method, allows teams to prioritize, plan, perform, and review the work with the goal of delivering solutions for users in usable chunks. According to the 9th Global Project Management Survey by PMI, 71% of organizations use Agile on their projects sometimes, often or always.

Through the Agile BI development model, BI tools deliver valuable information and data to users, along with business insights, team collaboration, and most importantly, the ability to adjust to change as it occurs.

Scrum and extreme programming are both different approaches to Agile, but this article will focus on the Scrum method. Scrum projects are divided into sprints, with planning, review and retrospective sessions. Sprints have different categories with work items that can be classified according to the following:

• Requirement Analysis
• Data Presentation and Visualization
• Data Integration and Modelling
• Data Augmentation
• Testing and Validation
• System Configuration

Each of these work items can be combined, leaving the question of how to combine the work items into each sprint. Agile BI works on the assumption that user feedback and requirement refinement are vital to decide which items to prioritize. In particular, most BI teams expect the most need to iterate the first three categories.

Work items with less value to the user would be pushed to a later sprint, helping to facilitate a more valuable BI solution to the user in less time.

Incorporate Insights from Business Experts Early On

Sprints aren’t the only way to ensure successful BI development projects. Selecting the right team is also essential.

Too often, the focus for developing many business tools is confined to technical experts, such as data architects and software engineers. But these technical experts don’t necessarily have the understanding of what the end-user is looking for in terms of a BI tool.

The task of the product owner for any BI development project should be to ensure the project delivers value to the business. The right product owner should not only be a subject matter expert but also have a deep understanding of the entire business.

Since Agile BI development’s end goal is to provide data to the user – often in a visual form in order for them to take immediate action – this business understanding is essential.

The process of choosing the right product manager and incorporating business insights, however, requires buy-in from senior management, who would give him or her the necessary amount of time and level of authority needed to work on and take responsibility for the project’s success.

Once the right product manager is found, the iterative model approach is essential for ensuring business value is incorporated from the earliest stages of development.

Since the goal of Agile development is close collaboration between team members, it is also important that the entire team is well suited to the project. The Scrum Master, for example, must manage the scope of the project and ensure enough time is given for different prioritized tasks.

In the quick and changing environment of scrum projects, it is important to understand that many members of the development team have multiple skills and can oversee one another and provide support when necessary to improve team productivity and capacity.

Listen to Stakeholders at Every Step of the Way

One of the most frequent complaints of businesses is that by the time IT has implemented the reporting and analytics features, the issue is no longer relevant. Since Agile shortens the development cycle and provides a quicker time to market, it can positively affect how your users perceive the value of your BI services.

But Agile cannot be implemented without the active participation from stakeholders to provide their requirements and work with the team. After all, they are the ones affected the most by the outcome of the project. Stakeholders can be anyone from team members to product owners, to the marketing and sales department head or the head of customer service. They could be internal or external.

The key to Agile BI development success is the ability to incorporate the feedback from these multiple and diverse stakeholders at every step of your project or sprints. And as business needs change and evolve to constantly suit customer demands, Agile BI development projects will only continue to drive value by keeping these stakeholders in mind.

Many thanks to Vitaly Prus for his contribution.  He is a certified Scrum Master and Head of the Agile Testing Department at A1QA software testing company. Vitaly manages the team of 60 QA Engineers who have successfully completed over 30 projects.