Asia-Pacific businesses have made significant progress in data democratisation, but challenges ahead may still hinder its progress, according to cloud computing company Snowflake.
Data democratisation, which promises that a wide range of non-technical users can consistently use data to make data-informed decisions, is an imperative for today’s data-driven enterprises, said Sanjay Deshmukh, Vice President, ASEAN and India at Snowflake.
This empowers organisations to use their data to develop hyper-personalised services for higher revenue and growth in today’s highly competitive business landscape.
However, there are impediments to making data freely available, according to Snowflake. These include Asia’s unique regulatory and compliance requirements, the persistence of data silos that make it difficult to access data, insufficient data collaboration, and the ability of users to adopt modern data tools.
Another key hurdle is the skills gap. Earlier this year, research firm IDC found that 53 per cent of organisations in the Asia-Pacific are taking 3 to 4 months longer compared to a year ago to fill technology roles, and the roles for data management professionals and data scientists and data analysts are the most difficult roles to fill.
Breaking down silos and promoting cross-team collaboration are essential steps for organisations embarking on this journey, said Deshmukh.
On a positive note, advances in generative AI are helping to enable data democratisation within enterprises, as it reduces the need for additional technical skills.
“By reducing the need for additional technical skills, generative AI enables controllers, marketers, and other non-technical users to easily access and leverage data for informed decision-making,” said Deshmukh.
“This accessibility of data empowers organisations with AI capabilities, facilitating more data science work for end users, and ultimately driving business benefits,” he added.
Large language models (LLMs) can help with data democratisation by providing comprehensible information and expertise from many different fields.
However, business organisations cannot just apply a large open-source LLM to their data and expect dependable outcomes.
“Simply applying a huge open source LLM to data and hoping for reliable results will be difficult, unreliable and expensive to run,” said Deshmukh.
Implementing a large open-source LLM can be challenging, unreliable, and expensive. In Asia-Pacific, organisations that use LLM need to consider training in non-English languages and collaborating with open-source software providers in the region.
Some other challenges include governance policies that heavily influence data trends, and conflicts may arise when data is dispersed across multiple locations due to differing governance policies. In addition, hosting LLMs on external services increases the risk of exposing proprietary data.
Potential risks can be mitigated by implementing robust machine-learning practices, having clear guidelines for application, governed data, result validation techniques, audits, and a defined solution scope.
A first step is to have a clear ROI of the problem being solved, and to optimise the LLM using well-managed data that can be tracked and validated.
“Organisations should develop strategies to bring LLMs to their data, rather than the other way around, while maintaining consistent governance,” said Deshmukh.
