23/02/2015
By James Heywood, Technical Consultant, Amido
Big data has been the technology buzzword for the last few years, with no sign that interest is waning. It seems that anyone who is anyone is utilising big data to make a difference to their business. But what exactly is big data and more importantly, why and how should your business be looking at this?
Why Big Data?
In the past organisations had only a few key statistics and pieces of data with which to try and draw a picture of, and understand their customers, such as web analytics, order histories and basic profile and account information.
Skip forward to today and we now find that there are numerous new and deep sources of data available, thanks to the proliferation of smart phones, wearable devices and online services that we now take for granted yet were almost unthinkable only a few years ago. So the challenge has become identifying what is useful information and acting upon it quickly and in a way which differentiates you from your competitors.
The uses and benefits of big data are numerous and depend on the nature of your business. For retailers and other customer facing organisations the obvious benefit is to gain a deeper understanding of your customers and gain insights into the way they shop and the products they like, enabling you to target them in a more personalised way.
The types of data and uses of this mentioned above are merely scratching the surface. The most interesting uses of big data yet are undoubtedly being developed right now and will surface over the next few years.
How to begin?
Before embarking on a potential big data project, the following questions should be asked:
1.Why do you need to collect, store and analyse a massive amount of information? What is the value to your business in doing so?
If you cannot clearly describe what the purpose of the project is then you are probably just hoovering up data which won't necessarily benefit your organisation. You should be able to quickly describe the reason for the project in the form of an elevator pitch. If you can do this then you probably have a good reason for the use of big data within your organisation.
2. What data are you actually going to utilise? Is this data generated by your own business, is it external, or a combination of both?
There is an ever-expanding array of data that can be used in a big data project and before you start, you need to understand the pros and cons associated with each type of data you are working with.
Some of the benefits of working with a variety of data is that you can combine these in interesting ways. If you had purchase information and location information for example you could identify when and where customers spend their hard earned money and target products accordingly.
As with all things there must be balance and the downside of working with a wide variety of data is that it will often be held in multiple places in a range of different formats. To be able to begin to use this data then someone needs to collate and manipulate it into a suitable form.
3. How are you going to collect and store this data? How are you going to analyse it and what are you looking for in the data?
You may well need to put systems in place to provide the data you need which in itself could be a whole separate project and may require changes to the way you work as well as the tools and software systems you use.
Once you have access to the required data, organisations must also consider how it is going to be analysed. You may need to buy in software and expertise around this or choose to work with your own analysis tools.
4. Where are you going to put all of this data?
Another consideration is whether to invest in hardware to store and host this internally to your business. If so, where? Your office, a dedicated location or data centre? If you plan on storing this data in the cloud, have you considered which suppliers you will use?
Further to the cost implication, the location of the data will impact on the way that it is managed and maintained, which will be reflected in the roles and services that you either have to acquire from other members of your organisation, or from external suppliers, which leads onto the next question.
5. Who in your organisation is going to be responsible for setting this up and maintaining it moving forwards?
Do you have any in house skills and/or indeed anyone willing to take this role on even if they do have the necessary skills? Alternatively you may need to recruit a person or a whole team to set this up and manage the project.
Key roles often overlooked in big data projects include those of data curation and maintenance, you may not have a role of this kind in your business, nor an individual or team who have the skills necessary to perform this.
As well as specific new roles such as the ones described above it is important to identify all stakeholders for a project of this kind, especially if you need time from key teams or individuals.
In summary
While there are many challenges that you will need to address when considering a project of this scale, the benefits of putting big data at the heart of your organisation are numerous and valuable. Hopefully thinking about this up front along the lines discussed above will help you to identify real value for your business as well as ensuring success for the big data projects you have in mind.
