22 July 2017
Apache Hive and Apache Hadoop both had releases this week, and there a number of great articles on Presto, Airflow, Kafka, and more. There are a bit more vendor posts than normal—please keep sending me links to help keep content diverse!
Good read on one team's path to adopting Presto. They describe several ways that they evaluated Presto (and other options), including performance, security, testing with users (their analysts). There are also some takeaways based on their production rollout.
http://allegro.tech/2017/06/presto-small-step-for-devops-engineer-big-step-for-big-data-analyst.html
The recently released Hue 4 has a new interface and some impressive new features like autocomplete for SQL (including proposing joins based on popularity) and a data wiki/governance tool.
http://gethue.com/the-hue-4-user-interface-in-detail/
http://gethue.com/hue-4-sql-editor-improvements/
Recent versions of Apache Kafka have added backwards and forwards compatibility of Kafka Clients. This post describes how it was added to the protocol, how to enable the feature, and what to expect when using a client that is a different version from the brokers.
https://www.confluent.io/blog/upgrading-apache-kafka-clients-just-got-easier/
Apache Airflow is gaining popularity as a workflow engine for big data, so it's no surprise that the Databricks team has built an integration. This is a good overview of what it looks like to get started with Airflow (including key concepts), using Databricks integration as the example.
https://databricks.com/blog/2017/07/19/integrating-apache-airflow-with-databricks.html
An interesting look at a real world use case, this post describes how Rabobank has moved their customer alerts system from the mainframe to Apache Kafka (built with a multi-data center deploy and Kafka Streams). The code, which is just under 75 lines of Java, is concise and easy to follow. The new system generates alerts in under 5 seconds, versus minutes or hours on the old one.
https://www.confluent.io/blog/real-time-financial-alerts-rabobank-apache-kafkas-streams-api/
Amazon EMR has supported using S3 as the data store for HBase clusters for a while now, and with the latest release (more details below), HBase Read Replicas can be created using HBase data in S3. This tutorial gives a walkthrough of HBase Read Replicas using Amazon EMR.
https://aws.amazon.com/blogs/big-data/setting-up-read-replica-clusters-with-hbase-on-amazon-s3/
This post has a good overview of an architecture that supports real-time processing of sensor data to detect anomalies. It uses StreamSets for ingesting, H2O (there's sample code on github) for building a model, and Spark Streaming for evaluating the model at scale.
https://mapr.com/blog/real-time-anomaly-detection-1/
This post describes many of the major features of Apache Kafka including its ordering/durability guarantees, key compaction, and schema evolution. In addition, there are diagrams to explain each of the major concepts/features.
https://www.confluent.io/blog/apache-kafka-for-service-architectures/
Amazon Redshift Spectrum is a service for querying data in S3 using the same underlying engine as classic Redshift. This post gives details on Spectrum's architecture and what its advantages are (including eliminating ETL and allowing to scale compute and storage independently).
Qubole has written about some of the challenges (and fixes) they've come across as they try to run Apache Airflow at scale.
https://www.qubole.com/blog/creating-apache-airflow-as-a-service/
DataEngConf NYC, which takes place on October 30 and 31, has announced the Call for Proposals. Talks are 30 minutes, and the CFP closes on September 1st.
http://info.dataengconf.com/data-science-engineering-call-for-proposals-decnyc17
Dremio, who is a force behind the Apache Arrow in-memory format, has announced their first product, a BI query engine that sits as a gateway between a BI tool and a backend stores (including RDBMS, Hive, HBase, S3, and more).
Amazon EMR 5.7.0 was released. The major new feature in this release is the ability to create a read-replica HBase cluster when the cluster is storing data in S3. Other updates include support for Apache Flink 1.3.0 and new versions of Apache Zeppelin and APache Phoenix.
Cask has announced support for Microsoft Azure in addition to Amazon Web Services for running their CDAP Cloud Sandbox.
http://blog.cask.co/2017/07/introducing-cdap-cloud-sandbox-for-microsoft-azure/
Implyr is a new package that enables querying of Impala from R using dplyr. This post has a brief introduction to the integration.
http://blog.cloudera.com/blog/2017/07/implyr-r-interface-for-apache-impala/
Apache Hive 2.3.0 was announced. It's a large release, containing a large number of bug fixes and improvements, and a few new features (including support for set operations and materialized views).
Apache Hadoop 2.8.1 was released with critical security fixes atop the 2.8.0 release. The 2.8.2 release, which will have more fixes, is currently targeted for an August release.
Curated by Datadog ( http://www.datadog.com )
Self-Service Data Integration, Kodiak Data, and Kubernetes! (Palo Alto) - Wednesday, July 19
https://www.meetup.com/BigDataApps/events/239890782/
Sensors, Spark, and Kafka: Applied Machine Learning (Minnetonka) - Wednesday, July 19
https://www.meetup.com/TwinCities-Bigdata-Analytics/events/241333229/
Data Streaming Panel Event (Chicago) - Thursday, July 20
https://www.meetup.com/TechinMotionChicago/events/240293492/
SQL on Hadoop and Modern Analytic Databases with Ian Cook (Chapel Hill) - Tuesday, July 18
https://www.meetup.com/Research-Triangle-Analysts/events/237582880/
DATA Rave Party: Spark Edition (Sao Paulo) - Thursday, July 20
https://www.meetup.com/Cloudera-Brasil-Meetup/events/241374445/
Applied Data Engineering #1 (London) - Wednesday, July 19
https://www.meetup.com/Applied-Data-Engineering-London/events/240932091/
The State of Spark and Hive in the Cloud, by Nico Poggi (Barcelona) - Thursday, July 20
https://www.meetup.com/Spark-Barcelona/events/241275453/
July Meetup: Scala, Spark, Docker, Elasticsearch (Bucharest) - Thursday, July 20
https://www.meetup.com/Bucharest-Big-Data-Meetup/events/240990369/
Big Data Processing on AWS (Tel Aviv-Yafo) - Tuesday, July 18
https://www.meetup.com/israelclouds/events/241304553/
Kafka Operations (Herzeliyya) - Wednesday, July 19
https://www.meetup.com/ops-for-developers/events/241304477/
If you didn't receive this email directly, and you'd like to subscribe to weekly emails please visit https://hadoopweekly.com