Chinese telecommunications company Huawei has for some time now expanded its horizons beyond the physical manufacturing of networking equipment. The firm’s operational (and consulting) services have spread its ‘intelligence’ to the point where it now actively contributes to a number of open source projects. Given the firm’s breadth and influence, the growth (or failure) of these projects may be influencing the way the telecommunications structures in many business will now be affected.
The mighty Wah!
The company that likes to be pronounced "wah-way" has this month opened up its ‘Project Astro’ -- a Spark SQL on HBase package. So what is it?
Apache Spark itself is ‘cluster computing’ software that enables ‘parallel’ processing in data analytics applications where many events happen concurrently. Spark SQL allows us to query structured data as a ‘distributed dataset’ (i.e. from a variety of sources) in Spark. Apache HBase is a ‘key-value’ data store (i.e. the unique identifier that denotes what a chunk of data is) that runs on top of the Hadoop Distributed File System (HDFS) -- in other words, it’s a database that’s really good at being updated really fast and scaled outwards really fast with billions of rows and millions of columns.
The idea behind ‘Project Astro’ is to fuse, couple and integrate the best database bits of HBase with the parallel processing big data analytics of Spark and Spark SQL.
“Dubbed Astro, the package provides powerful online query and analytics capability for large scale data processing in vertical enterprises. Huawei has embedded Spark into its big data solution, FusionInsight, which is used by the world’s largest bank, Industrial and Commercial Bank of China -- and communications service provider, China Mobile,” said the company, in a press statement.
Data terms you need to know: pruning & pushdowns
Huawei will also launch Spark as a cloud service in China on Huawei Public Cloud this year. The technology has ‘handling’ features here that include data pruning (just as it sounds, getting rid of troublesome information data parts), intelligent scan (not just looking for data, but having meta-type information about the information and being able to validate it too) and so-called ‘pushdowns’ (directing database logic controls to go and perform a defined job) like custom filters and coprocessor controls.
The Huawei team has contributed new features to Spark SQL, machine learning technologies and Spark R to help enrich the standard libraries in Spark.
President of Huawei Central Software Wang Chenglu insists that open source is in the firm’s blood from core business networking (where the firm helped drive the network openness as founding member of OPNFV), to cloud computing and the IoT (where the firm open sourced LiteOS -- a lightweight IoT Operating System).
“Spark, as the most influential community among Apache projects and as a preeminent framework for data processing and analytics, has been our focus and top priority. We believe that putting Spark as the core of our big data solution can bring significant benefits,” he said.
Another happy 'Huawei intelligent retail' customer, delighted with backend parallel processing data... [+]
It’s good news, basically
Spark rose to prominence within the big data world, as it provides a robust programming framework, rich set of APIs and libraries and a pretty vibrant ecosystem. As Forbes writer Paul Miller puts it, this is a good news story and the technology here is on the ascendancy.
“Spark certainly has a lot to recommend it. It’s fast, it supports a number of popular development languages (Java, Scala, Python, R) and it includes easy access to a growing set of commonly used algorithms in data science. It is, to all intents and purposes, a great way to start playing around with data; whether that’s a few megabytes on your laptop or a petabyte or two in a cluster,” writes Miller.
For those new to Huawei and its approach to open source, there is nothing here to doubt the company’s wider intentions. When it says, “Huawei is deeply committed to Spark and intends to participate extensively in joint community and industry efforts,” it probably means it. It would be nice to see the firm put some of its innovation into context in terms of where these technologies sit on a practical and developmental basis globally, but that is what we have tried to do here.
