When one thinks of Agile, they typically think of the practice as it applies to software development, enabling companies to iteratively release systems in short blocks of time. However, the same strategy can be applied to business intelligence (BI) solutions as well.
If there's one member of the AtoBI team who knows everything there is about Agile project management, it's Head of Operations Jamie Supple. The BI veteran spoke with us about how Agile enables project stakeholders to glean new insights from their data and allocate resources to which they're most valued.
"Projects are occurring too quickly nowadays for Waterfall to be sustainable."
1. Do you feel as if Waterfall or sequential services have become outdated? Why?
I think for BI, Waterfall development has become outdated, purely because business is moving too quickly these days. By the time you've established the Scope of a Waterfall project – before you've done any development the requirements are often outdated and the business has moved on.
So if you actually see a Waterfall project through to full completion (typically a year to a year and a half), you have something that, if the project leaders had the choice, they wouldn't actually want delivered because it's no longer relevant to their needs.
BI projects are occurring too quickly nowadays for Waterfall to be sustainable. Not to mention, stakeholders change their minds, especially when data they didn't foresee arising enlightens them to new realities.
2. Agile development typically involves applying iterative adjustments to software based on continuous user feedback. How does AtoBI integrate these practices into client services?
What we do is follow a modified Scrum/Kanban approach to Agile. That involves using artefacts such as Stories and Cards, which we put on a Wall detailing what tasks the team has to complete. At the start of each day, we have a 'Stand-up'. During a Stand-up, we discuss what was completed yesterday, what we need to get done the day of the meeting and address any factors that may prevent us from finishing tasks (blockers).
We try to get as many high-level decision makers to attend these Stand-ups as we can – preferably the sponsor of the enterprise BI solution. The more people you can get to those daily Stand-ups, the more feedback you get on whether or not the stories are relevant and ready to go. In addition, when the leading stakeholders attend, they can get a sense of how quickly the project is moving, enabling them to decide what's best for the business now and in the near future.
3. What makes Agile ideal for sustaining customer relationships?
This is where the Stand-ups come into play. Face-to-face communication allows us to discuss any relevant problems facing the project.
Often, managers make decisions once and then wait until the results of those decisions generate an answer of some kind.
However, when they're involved in Stand-ups, they have the opportunity to communicate with developers and business users. These interactions allow them to see just how quickly the team can implement their ideas and decisions.
A manager may have a suggestion and change the entire focus of a particular aspect of the project. For us, that's a huge benefit because it demonstrates our ability to deliver meaningful BI.
4. How does Agile allow you to remain productive when resource issues arise?
When running an Agile project, we typically have a backlog, which is the full gamut of the user Stories. In addition, we have a ready-for-development column which has Stories that have been properly qualified and estimated by us. All of our Stories have acceptance criteria and possess all the supporting documentation. Until those stipulations are met, we don't send something for development.
While we generally have as many cards ready for development as possible, if it looks like someone is being dragged off a project, for example, what we'll do is ramp up the ready-for-dev cards so that development can continue.
If we run out of ready-for-dev cards, then we can go straight into the backlog ourselves, but there have been instances where we've stopped a Sprint mid-project if there is a key resource that is absolutely required. But that's only happened once.
5. Can you describe a time when Agile enabled AtoBI's experts to resolve a client issue?
Well, it happens all the time. Data issues are probably the biggest hurdle you'll encounter in the BI landscape because people assume they have good data, and that's what they want to report on. However, whenever we've applied solutions like QlikView and Qlik Sense across data, they expose all the flaws within that information.
"Agile is all about what makes sense in regards to the project."
Whenever this has occurred, we have discovered the need for additional resources or processes to accompany certain data to verify its quality. In a non-Agile world, that would entail us stopping the project, allocating a large chunk of time to assessing all of the information and then coming back to the project once we've finished. Instead, all we do is create another Card to clean up that data and assign people to complete it. We can even prioritise that Card if we think it's imperative to address.
Agile is all about what makes sense in regards to the project. The aim is to deliver as much as you possibly can within each iteration. The goal isn't to deliver everything because 'everything' at the start of the project is certainly not 'everything' by the time you finish. Why? Because either we or our clients will discover things they want to add in, change or remove.
Bottom line: The project scope will change, particularly in the BI world. The reason why we choose Agile is because our clients will see their data in a way they've never seen it before. Remember these are smart people, and they need the freedom to be able to act on their "what if we did this?" moments as projects progress. Agile essentially allows stakeholders to change their minds as new insights arise from the delivered Stories.
6. Although it's not the same, Agile is closely tied to DevOps development practices. Do you think Agile encourages a closer relationship between yourself and your clients' operations teams? How?
I've actually worked with DevOps teams before, and they've been highly integrated. Because of the way we deploy the Qlik architecture, there is a requirement for close collaboration with the operations team. However, in a Qlik environment, the developers on the client's side are often a part of the operations team anyway.
When you get Qlik implementations working really well, what you have is the business developing the front end through data visualisations. So you might get a financial analyst who has crafted a particular analysis for a cost structure, and they'll spend a week iterating it themselves using Qlik. Then what they'll do is speak to the operational team, which is comprised of the back-end developers. The analyst will say to them "I have created this object, could you publish this in the document for the rest of the uses?"
Typically, the developers handle the more difficult front-end tasks as well as create the data architectures and those sorts of things. There's typically a lot of communication between the business users and the developers, which produces a lot of shared knowledge.
7. Agile encourages evolution, in some respects. Do you think the data market inadvertently causing the Agile methodology to take a new form? If so, in what way?
I wouldn't necessarily say that the data/BI industry is causing Agile itself to evolve because all Agile projects are inherently different. Therefore, each Agile initiation will undergo its own evolution.
What I have seen in the BI world – and this may have an impact on how organisations employ Agile projects – is that BI endeavours typically only have one developer. In contrast, software development projects will have a team of such professionals, all of which can complete and pick up new cards on an ongoing basis.
In BI, what Agile becomes is more of a free-form client relationship management tool rather than a "dividing up tasks" and "work stream" management tool. So, what it enables you to do is be quicker and integrate client changes at a visual level, as well as allow those changes to be referred to and enacted.
8. Are there any challenges associated with Agile?
One of the biggest problems we have with Agile is that, sometimes, clients assume that once you put something into a Sprint, that 'something' will be delivered. Sure, you may allocate 10 stories to one Sprint, but every company has a different velocity. Company X may have an inferior infrastructure, which means it may take longer to deliver on that Sprint. In contrast, Company Y may have a SQL guy who can provide data at a faster rate than the leaders originally thought.
The thing is, we can't judge a company's velocity until we're in the thick of it so to speak. Once we've completed enough cards, we'll be able to say "Ok, this is when the project will be completed."