Just a normal evening in Portugal

Connector Builder by Airbyte

Airbyte users can now create ELT connectors without code in 10 minutes, using the Connector Builder. Address your team’s long tail of connector needs. It handles authentication, pagination, rate limiting, schema handling, transformation, response decoding...

Link

Data Engineering Patterns: Write-Audit-Publish (WAP)

Write-Audit-Publish (WAP) is a pattern in data engineering that gives teams greater control over data quality. It was popularized by Netflix back in 2017 in a talk by Michelle Winters at the DataWorks Summit called “Whoops the Numbers are wrong! Scaling Data Quality @ Netflix.”

Link

How do we Continuously Grow and Evolve in our roles

Event by Kamran Ali

Link

Microsoft launches Fabric, a new end-to-end data and analytics platform

Microsoft today launched Microsoft Fabric, a new end-to-end data and analytics platform (not to be confused with Azure Service Fabric).

Link

Databricks Releases Keynote Lineup and Generation AI Programming for 2023 Data + AI Summit

SAN FRANCISCO - June 1, 2023 – Databricks, the data and AI company, revealed the full agenda and lineup of featured speakers for the upcoming Data + AI Summit, a global event for the data community. On June 26-29, tens of thousands of data leaders, open source enthusiasts, and Databricks customers and partners will come together in person in San Francisco and virtually from around the world to learn about large language models (LLMs), the lakehouse paradigm, and Databricks’ latest product innovations and contributions to the open source community. Tickets are still available to join the data community in San Francisco; register here to attend the live event. 

Link

Test data quality during CI/CD development

Use this guide to install and set up Soda to test the quality of your data during your development lifecycle. Catch data quality issues in a GitHub pull request before merging data management changes, such as transformations, into production.

Link

Mastering Monitoring

A Step-by-Step Guide to Using Prometheus and Grafana with CrateDB Cloud 

In this interactive live demo, learn, step-by-step, how to use Prometheus and Grafana with CrateDB Cloud. 

CrateDB Cloud users can implement their own monitoring. With the guidance of Walter Behmann, Engineer at CrateDB Cloud, you will learn how to monitor a cloud service, specifically CrateDB, using Prometheus and Grafana. 

By the end of this webinar, you will be equipped to level up your monitoring game. 

Link

Welcome Okera: Adopting an AI-centric approach to governance

 Okera, the world's first AI-centric data governance platform. Okera solves data privacy and governance challenges across the spectrum of data and AI

Thought for the day

“I am comfortable here” has killed more dreams of software engineer than anything else.

Subscribe now

Share Just Enough Data

Leave a comment

Live with Astronomer: Simplified DAG authoring with the Astro SDK

In this “Live with Astronomer” session, we’ll give an overview of the Astro SDK and show how it makes writing ELT pipelines easy and efficient. We’ll cover everything you need to get started using this open source tool in just a few minutes.

When: May 2, 2023 at 2pm EDT

Link

DATA+AI Summit Generation AI

Large Language Models (LLM) are taking AI mainstream. Join the premier event for the global data community to understand their potential and shape the future of your industry with data and AI.

San Francisco, Moscone Center
June 26 - 29, 2023

Link

Horizontally scaling Kafka consumers with rendezvous hashing

How we used rendezvous hashing to horizontally scale Kafka consumers to support hundreds of concurrent topics with fewer connections, thus lowering our infrastructure costs.

Link

Spark Connect Available in Apache Spark 3.4

In Apache Spark 3.4, Spark Connect introduced a decoupled client-server architecture that allows remote connectivity to Spark clusters using the DataFrame API and unresolved logical plans as the protocol. The separation between client and server allows Spark and its open ecosystem to be leveraged from everywhere. It can be embedded in modern data applications, in IDEs, Notebooks and programming languages.

Link

Databricks and Hugging Face integrate Apache Spark for faster AI model building

Databricks and Hugging Face have collaborated to introduce a new feature that allows users to create a Hugging Face dataset from an Apache Spark data frame. This new integration provides a more straightforward method of loading and transforming data for artificial intelligence (AI) model training and fine-tuning. Users can now map their Spark data frame into a Hugging Face dataset for integration into training pipelines.

Link

ChatGPT Prompt Engineering for Developers

In ChatGPT Prompt Engineering for Developers, you will learn how to use a large language model (LLM) to quickly build new and powerful applications.  Using the OpenAI API, you’ll be able to quickly build capabilities that learn to innovate and create value in ways that were cost-prohibitive, highly technical, or simply impossible before now.

Link

Mastering AI-Powered Product Development: Introducing Promptimize for Test-Driven Prompt Engineering

 Promptimize, a Python toolkit designed to measure success and outcomes while doing prompt engineering. 

Link

LEARNING IS THE ONLY WAY TO GROW.

Ajith Shetty

Bigdata Engineer — Bigdata, Analytics, Cloud and Infrastructure.

Medium Subscribe✉️ ||More blogs📝||LinkedIn📊||Profile Page📚||Git Repo👓

Subscribe now

Hello Everyone!

We are in the middle of the crisis and the job market is very difficult. With many layoffs happening all around the world I want to do my bit for the community.

Recently I have come across one of the coolest Interview guide to prepare yourself for your next challenge.

software-engineering-interview-guide

lakeFS 💛 DuckDB: Why is DuckDB all the rage in the Data Community?

https://www.meetup.com/lets-talk-data-sf/events/292111441/

DuckDB is coming to SF! Join us as we explore the possibilities of how lakeFS and DuckDB can improve your database management!

--------
🤝Organizer : DuckDB & lakeFS
📍Location: Trellis Co-working & Events (981 Mission St, San Francisco)
🍕Catering: Pizza & Drinks

Different choices To Orchestrate your Pipelines

You would have seen a lot of discussions happening around w.r.t the Alternative solution for the Airflow.

Calling the topic to be Alternative solution for Airflow would be wrong. Every business has its own requirements and business goals. We cant always fit a same solution for all the problems.

With that we can say that we can have multiple solutions for the same problem.

So have you heard about Mage.

Here is my medium blog talking about it.

Batch Processing vs Stream Processing

You heard it right. The discussion of Batch VS Stream is still not over.

One of the most important decisions organizations make when it comes to data processing is whether to use stream or batch processing. Stream processing is quickly becoming the go-to option for many companies because of its ability to provide real-time insights and immediate actionable results. With the right stream processing platform, companies can easily unlock the value of their data and use it to gain a competitive edge.This article will explore why stream processing is taking over, including its advantages over batch processing, such as its scalability, cost-effectiveness, and flexibility. 

And here is the blog which talks more about it.

Medium Blog

The View Below The Waterline Of Apache Iceberg And How It Fits In Your Data Lakehouse

Cloud data warehouses have unlocked a massive amount of innovation and investment in data applications, but they are still inherently limiting. Because of their complete ownership of your data they constrain the possibilities of what data you can store and how it can be used. Projects like Apache Iceberg provide a viable alternative in the form of data lakehouses that provide the scalability and flexibility of data lakes, combined with the ease of use and performance of data warehouses. Ryan Blue helped create the Iceberg project, and in this episode he rejoins the show to discuss how it has evolved and what he is doing in his new business Tabular to make it even easier to implement and maintain.

Link

Introducing Quix Streams, an open source library for telemetry data streaming

Lightweight, powerful, no JVM and no need for separate clusters of orchestrators. Here’s a look at our next-gen streaming library for C# and Python developers including feature summaries, code samples, and a sneak peek into our roadmap.

Link

Spark On AWS Lambda

Apache Spark on AWS Lambda is a standalone installation of Spark running on AWS Lambda. The Spark is packaged in a Docker container, and AWS Lambda is used to execute the image along with the PySpark script. Currently, heavier engines like Amazon EMR, AWS Glue, or Amazon EMR serverless are required for event driven or streaming smaller files to use Apache Spark. When processing smaller files under 10 MB in size per payload, these engines incur resource overhead costs and operate more slowly(slower than pandas). This container-based strategy lowers overhead costs associated with spinning up numerous nodes while processing the data on a single node. Use Apache Spark on AWS Lambda for event-based pipelines with smaller files if you're seeking for a less expensive choice, according to customers.

Link

Serverless Spark CI:CD on AWS with GitHub Actions

Interested in combining Amazon EMR Serverless and GitHub Actions to continuously test and deploy your PySpark code to AWS? In this session, Damon will talk about how to configure unit tests when a push is made, run quick data quality checks when a pull request is opened, and automatically package and deploy tagged releases to Amazon S3 for use in your Spark jobs.

github

Twitter has open-sourced tweet recommendation Algorithm

As repeatedly promised by Twitter CEO Elon Musk, Twitter has opened a portion of its source code to public inspection, including the algorithm it uses to recommend tweets in users’ timelines.

Link

github

twitter blog

You are the only one who would know your worth. you do not need a validation from anyone.

Ajith Shetty

Bigdata Engineer — Bigdata, Analytics, Cloud and Infrastructure.

Medium Subscribe✉️ ||More blogs📝||LinkedIn📊||Profile Page📚||Git Repo👓

Leave a comment

Share

Share Just Enough Data

Hey Everyone,

Apologies, I could not post the weekly newsletter last week. But I promise, not to miss it in the future and to keep you always posted with the Data Engineering news weekly going forward.

Do share your thoughts in the comment section if you have an feedback :)

How to Create a Machine Learning Framework

With only 1 in 10 models making their way into production, deploying models and generating business value is already a challenge that many organisations face. However as you scale your machine learning (ML) operation, this challenge catalyses, infrastructure can become unwieldy, and in turn, ML becomes less effective.

Event by Seldon

Online

Tue, Nov 30, 2021, 4:30 PM - 5:30 PM (your local time)

Link

Data Observability Learning Summit 2021

At the first Data Observability Learning Summit, startup founders, senior executives, and engineering thought leaders shared challenges related to DataOps (Data Management, lineage, Data Quality, Timeliness), MLOps/AIOps (ML Monitoring, Trusted AI), and the need to better connect the two. We are pleased to share these recordings with the wider audience to share knowledge and build community in the space of Data Observability.

Link

Data Virtualization seems promising. But does it scale with your data and BI needs?

Data engineers and data architects strive to provide data consumers with the data and analytics user experience they need. Data virtualization can look very promising — meet all my BI goals and not have to move data? Yes please!

However, as your data, user, and application scale grows, old and new problems arise that put you back where you started — or worse. Data lakes combined with Dremio can address BI needs at any scale.

Link

Good to great series: Reverse ETL success stories for data teams

It’s no secret: Data teams are the heart and soul of modern, data-led businesses. They’re responsible for powering the company’s day-to-day operations with vital, fresh insights modeled for each specific use case their business teams need.

Link

Introducing Analytics at Meta

The Facebook company is now Meta, and we couldn’t be more excited about the vision Mark laid out in his keynote at Connect. While our company name may have changed, our company mission remains the same: to give people the power to build community and connect the world. We are the Analytics Team at Meta, and we’re excited to play a role in seeing that mission realized.

Link

Airflow 2.2.2, 2021-11-15 is out

• Add a Docker Taskflow decorator (#15330, #18739)

• Add Airflow Standalone command (#15826)

• Display alert messages on dashboard from local settings (#18284)

• Advanced Params using json-schema (#17100)

• Ability to test connections from UI or API (#15795, #18750)

• Add Next Run to UI (#17732)

Link

Will Apache Arrow Flight SQL replace ODBC and JDBC for Analytics/BI workloads?

Most popular BI tools rely on ODBC or JDBC to bring data in from where it resides. For example, Microsoft Power BI relies mostly on ODBC, Tableau relies on both, Looker relies on JDBC, and Qlik relies on both depending on the product. This has worked well enough, mostly because both standards have been around for a long time and have inertia behind them. In many ways, however, these standards are tolerated rather than embraced by developers. There’s a reason ODBC has been referred to as Other Developers Buggy Code.

Link

The Rise of Data Observability

Architecting the future of data trust

Lior Gavish, CTO and Co-Founder of Monte Carlo

What is data observability and how can you apply it to your data stack? Lior Gavish, co-founder of Monte Carlo and creator of the data observability category, discusses his vision for the future of end-to-end data trust at scale.

Link

Evolution of the SQL language at Databricks: ANSI standard by default and easier migrations from data warehouses

by Bilal Aslam, Serge Rielau, Shant Hovsepian and Reynold Xin Posted in PLATFORM BLOGNovember 16, 2021

Today, we are excited to announce that Databricks SQL will use the ANSI standard SQL dialect by default. This follows the announcement earlier this month about Databricks SQL’s record-setting performance and marks a major milestone in our quest to support open standards. This blog post discusses how this update makes it easier to migrate your data warehousing workloads to Databricks lakehouse platform. Moreover, we are happy to announce improvements in our SQL s

Link

At the end of the day you will see success. For some it would be days and for others it could be Years.

Ajith Shetty

Bigdata Engineer — Bigdata, Analytics, Cloud and Infrastructure.

Medium Subscribe✉️ ||More blogs📝||LinkedIn📊||Profile Page📚||Git Repo👓

What is a Streaming SQL Database?Learn How to Use SQL Streaming Databases

Streaming SQL databases can simplify how organisations work with streaming data sources like Apache Kafka. A complete SQL streaming database provides the powerful offering to developers of asking questions about streaming data, and then getting correct answers with millisecond latency, even when the underlying data changes rapidly - commonly called incremental view maintenance.

Link

Data + AI World Tour: Discover Lakehouse | Live meetup!

Join Databricks and the Data + AI World Tour event for a live meetup to close out the Data + AI World Tour EMEA event! We have two talks for you, one panel discussion and a tech talk. Come join in and ask your questions.

*Please note this is a EMEA timezone friendly meetup. This meetup is part of the Livecast Europe, Middle East and Africa on 16 November 2021 | 9:00 GMT, 10:00 CET Register here on the World Tour event page for free: https://dbricks.co/31tthg0 to check out all the other awesome sessions!

Link

Coalese

Sessions are organized by category: Culture & Teams, Data Futures, dbt-led Sessions, Frameworks & Best Practices, Premier Partner sessions, Tool deep dives, and Workshops.

Link

How to Choose the Right Structure for Your Data Team

Best practices for building a data team at a hyper-growth startup, from hiring your first data engineer to IPO.

Link

Supercharging Apache Superset

How Airbnb customised Superset for business intelligence at scale

Link

What is an A/B Test?

An A/B test is a simple controlled experiment. Let’s say — this is a hypothetical! — we want to learn if a new product experience that flips all of the boxart upside down in the TV UI is good for our members.

Link

Data lakehouse by Databricks

A data lakehouse is a new, open data management architecture that combines the flexibility, cost-efficiency, and scale of data lakes with the data management and ACID transactions of data warehouses, enabling business intelligence (BI) and machine learning (ML) on all data.

Link

Trust me there is always a solution for every problem. You just have to look for it.

Technical Webinar with Demo - Apache Iceberg + Dremio on ADLS

Data lakes are designed to store and seamlessly make available vast amounts of data to users across the organization. However, many organizations run into challenges like effective use of data and analytics directly on the lake.

Data lakes need to hide the complexity of underlying data structures and physical storage from end users in order to maximize value.

Link

Lightning-fast SQL Queries + Transactions directly on the Data Lake

SCHEDULE (times in PST, in the San Francisco bay area):
6:50 Join Zoom, register here to receive zoom link: https://www.aicamp.ai/event/eventdetails/W2021102519

7:00 SFbayACM intro, upcoming events, introduce the speaker

7:10 presentation starts (~60 min with Q&A)
8:10 - 8:30 wrap up

TALK DESCRIPTION:
Data Lakes have been built with a desire to democratize data - to allow more and more people, tools, and applications to make use of more and more data. A key capability needed to enable more users is the ability to hide the complexity of underlying data structures and physical data storage. The de-facto standard has been the Hive table format, released by Facebook in 2009 that addresses some of these problems, but falls short at data, user, and application scale. So what is the answer? Apache Iceberg. Apache Iceberg table format is now in use and contributed to by many leading tech companies like Netflix, Apple, Airbnb, LinkedIn, Dremio, Expedia, and AWS.

Link

All about Azure from the community experts!

Link

Casting the spell: Druid in practice

Thursday, October 28, 2021, 8:40 AM - 9:10 AM PDT

In this talk, we will also share some of the best practices and tips we’ve gathered over the years. We will cover the following topics: * Data modeling * Ingestion * Retention and deletion * Query optimization

Link

Data Observability 101: Everything You Need to Know to Get Started

What is data observability and does it make sense for your stack? Here’s your go-to guide to starting on the path towards data trust at scale.

One of the most common questions I get from customers is: “How do I get started with data observability?” And for good reason. 🙂 

Link

Scheduling and Timetables in Airflow

One of the most fundamental features of Apache Airflow is the ability to schedule jobs. Historically, Airflow users could schedule their DAGs by specifying a schedule_interval with a cron expression, a timedelta, or a preset Airflow schedule.
Link

Democratizing the Data Stack—Airflow for Business Workflows

Learn how Hightouch drives action in marketing & sales teams with Reverse ETL, SQL, and Apache Airflow

Link

DataHub Features Overview

DataHub is a modern data catalog built to enable end-to-end data discovery, data observability, and data governance. This extensible metadata platform is built for developers to tame the complexity of their rapidly evolving data ecosystems, and for data practitioners to leverage the full value of data within their organization.

Here’s an overview of DataHub’s current functionality. Curious about what’s to come? Check out our roadmap.

Link

Production is called as production for obvious reasons. So be certain before you apply any patch.

To keep you alert: https://www.blef.fr/data-deleted-from-production/

Data & AI Monetization

Link

Generating Synthetic Data With GANs

Do we have enough data? Are our datasets imbalanced? How can we accelerate research while avoiding data leakage? Generative Adversarial Networks (GANs) are a promising AI solution to these questions. With GANs, we can generate more and better data that is more fair and generalizable, which can be used to improve ML models and algorithm testing. GANs are also promising in the field of data privacy, since they could break barriers to data sharing, allowing companies and institutions to accelerate research findings.

Link

Decision Making at Netflix

This introduction is the first in a multi-part series on how Netflix uses A/B tests to make decisions that continuously improve our products, so we can deliver more joy and satisfaction to our members. Subsequent posts will cover the basic statistical concepts underpinning A/B tests, the role of experimentation across Netflix, how Netflix has invested in infrastructure to support and scale experimentation, and the importance of the culture of experimentation within Netflix.

Link

Accelerating Early SaaS Growth While Building a Sustainable Business

Link

6 Surprising AWS Charges You Should Monitor Closely

Every company should actively monitor its cloud costs, which can drive up the overall cloud bill. However, if you handle them properly, this can be avoided. Here are some AWS charges you should keep an eye on.

Link

Accelerating Into the U.S. Grand Prix With McLaren

If you thought this week couldn’t get more thrilling, I’ve got news for you: We’re shifting into overdrive. On the heels of our action-packed customer and user conference, .conf21, we’re heading to Austin, Texas this weekend to cheer on our data-driven partners at McLaren Racing for the Formula 1 U.S. Grand Prix. 

Link

The Gartner Magic Quadrant for Metadata Management was just scrapped. Here’s everything you need to know

Metadata management started out as an IT discipline. As we embraced the internet, and as data types and formats exploded, IT teams were put in charge of creating an “inventory of data.”

Link

Case Study: eGoGames Esports Platform Uses Rockset for Real-Time Analytics on Gaming Data

From business communications and financial transactions to trip planning and activity tracking, much of our lives run through smartphones today. eGoGames will help you add competitive esports to that list.

The Future of the Data Engineer

One of the first data engineers at Facebook and Airbnb, he wrote and open sourced the wildly popular orchestrator, Apache Airflow, followed shortly thereafter by Apache Superset, a data exploration tool that’s taking the data viz landscape by storm. Currently, Maxime is CEO and co-founder of Preset, a fast-growing startup that’s paving the way forward for AI-enabled data visualization for modern companies. 

Link

Monitors as Code: A New Way to Deploy Custom Data Quality Monitors From Your CI/CD Workflow

Monte Carlo releases Monitors as Code, a new feature that allows data engineers to easily configure new data quality monitors as part of their daily workflow.

Link

When you are in tough situation, just take a step back and relax. Give yourself a time, you will figure out a way :)

SCALING ML MODELS, DATA, ALGORITHMS, & INFRASTRUCTURE

The creators of The Transformer, TensorFlow, Kubernetes, Apache Spark, Tesla Autopilot, Keras, Apache Arrow, MLPerf, Matroid, Turing Award Winners, astronauts, and others will present about running and scaling machine learning algorithms on a variety of computing platforms, such as GPUs, CPUs, FPGAs, TPUs, & the nascent AI chip industry.

Link

Dremio Cloud Live Demo

See Dremio Cloud in action and ask our product experts your questions

Join our live weekly demo where our product experts will showcase Dremio Cloud and answer your most pressing data questions.

Dremio Cloud is a true SaaS SQL lakehouse platform that eliminates the need to install any software or manage infrastructure. You’ll learn how Dremio enables you to run high-performance queries and dashboards directly on cloud data lake storage, without having to copy data into proprietary data warehouses.

Link

Cloud Data Pipelines (RO, online, free)

This month we're hosting a special Cloud Native Bucharest meetup, as part of Big Data Week Bucharest. That's why it's not on Thursday :)
Do note this event is free, you don't need a conference ticket.

Link

What Is Azure Data Lake Storage (ADLS)?

Microsoft Azure Data Lake Storage (ADLS) is a fully managed, elastic, scalable and secure file system that supports HDFS semantics and works with the Apache Hadoop ecosystem. It provides industry-standard reliability, enterprise-grade security and unlimited storage that is suitable for storing a large variety of data. It is built for running large-scale analytics systems that require large computing capacity to process and analyze large amounts of data. Data stored in ADLS can easily be analyzed using Hadoop frameworks like MapReduce and Hive.

Link

The subsurface podcast

Testing pipeline to join 30billion rows of data quickly.

Link

Doordash: As an orchestration engine, Apache Airflow

let us quickly build pipelines in our data infrastructure. However, as our business grew to 2 billion orders delivered, scalability became an issue. Our solution came from a new Airflow version which let us pair it with Kubenetes, ensuring that our data infrastructure could keep up with our business growth.

Link

Google launches serverless Spark, AI workbench, new data offerings at Cloud Next

While the cloud has been great for data and analytics -- given its limitless storage and compute capacity  -- it has also caused a real regression in productivity for data professionals. The reason for this, simply put, is that the major cloud providers have hurled numerous data platforms at the market and left it to customers to pick the right combination of services, then integrate them. Say what you will about the old guard enterprise software behemoths, but they did spare their customers much of the "assembly required" experience that the cloud hyperscalers impose today.

Link

No matter how much you extend your working hours, your work will never be over.

Remember, Personal life is also as important as professional.

Hey Everyone! welcome to week 8 edition of Just enough data.

How C.H. Robinson accelerated business time to value by modernizing its legacy data infrastructure

Thursday  |  Oct 7th

9:00 am PT  |  11:00 am CT  |  12:00 pm ET 

Join Snowflake, HVR, and dbt for a fireside chat with C.H. Robinson to learn the power of a modern data architecture.

During this session, Chris Fitzharris, Director of Data and Analytics at C.H. Robinson, a global third-party logistics provider, will share how they successfully implemented an efficient and cost-effective solution to modernize their legacy data infrastructure. 

Link

Live Rockset Product Demo

Tuesdays at 10:00 AM PST

Learn how to easily and instantly connect and analyze data with massive scalability within Rockset. Join our weekly demo where product experts showcase key Rockset features and answer questions live from the audience.

Link

Discover the latest technologies, best practices, use cases for putting ML models into production environments.

OCT 12-13, 2021 | 8:30 AM - 4:00 PM PT

Link

[Webinar] Everything you Need to Know About Airflow 2.2

In this informative webinar we will cover everything you need to know about Airflow 2.2. We'll go through all of the new features large and small, as well as show you how to leverage all of the new features and how you can get cleaner and more efficient DAGs as a result.

Tue, Oct 5, 2021, 7:30 PM - 8:30 PM IST

Link

ACCESS EMERGING TECH ON YOUR COFFEE BREAK

Rise & IGNITE connects enterprise companies with next-gen technology solutions addressing the most current challenges organizations are facing today.

October 12, 2021 3:00pm–3:30pm EDT 12:00pm-12:30pm PDT

Link

What is metadata?

And why you need to start leveraging yours

Link

DREMIO MODERNIZATION WORKSHOP

The intent of this workshop is to educate users on the features of Dremio and how to accelerate Time-to-Insight using Dremio on AWS. A Background in AWS Glue and other AWS Analytics services is not required, but they may come in handy.

Link

How to Run Apache Airflow on Kubernetes at Scale

As an orchestration engine, Apache Airflow let us quickly build pipelines in our data infrastructure. However, as our business grew to 2 billion orders delivered, scalability became an issue. Our solution came from a new Airflow version which let us pair it with Kubenetes, ensuring that our data infrastructure could keep up with our business growth.

Link

Databricks buys 8080 Labs for its low-code data exploration tools

Databricks announced today it’s acquiring the Germany-based startup 8080 Labs GmbH, maker of a popular user interface-based data science tool called bamboolib that helps people without coding skills explore and transform data quickly and easily. Financial terms of the deal were not disclosed.

Link

3 Steps for a Successful Data Migration

Migrations are one of the most common initiatives that data teams own. But as everyone knows, they are complex beasts. They always take longer than they need to, many different teams are involved, and they’re super visible — not just to users, but execs, too. Everyone is anxious when a migration is underway. And, there’s always one ongoing or about to start.

Link

No matter how resilient pipeline you build there will be one edge case which will make it go haywire. Review and discussion should be the most basic step before you build anything.

From Future Data

Boss Talk: Future Data Edition

In this Future Data edition of their engaging Clubhouse series, a16z Co-Founder Ben Horowitz and Databricks Co-founder and CEO Ali Ghodsi join Sisu Founder and CEO Peter Bailis for a discussion on leadership, management, and more.

Link

Databricks: The data lakehouse comes to you

Data + AI Summit is the event for the data community and Data + AI World Tour brings the energy of Summit to a whole new audience. With content, customers and speakers tailored to each region, the World Tour showcases why lakehouse is quickly becoming the global standard for data architecture.

Link

5 Best Practices to Completely Eliminate Costly Data Copies With a SQL Lakehouse Platform

Your data teams and business users depend on mission-critical dashboards, BI tools and ad hoc queries to gain insight into your organizational data. You should be able to make data readily accessible directly from your cloud data lake storage with the speed and performance your business users are looking for, without creating optimized data copies. This whitepaper shares the 5 best practices that will help you completely eliminate expensive data copies with a SQL lakehouse platform.

Download Whitepaper

Apache NiFi vs. Apache Airflow

Apache Airflow and Apache NiFi are, in fact, two whistles to a somewhat different tune. Still, you may be wondering which one is better suited for your expectations and goals. By the end of this article, you will no longer have doubts. 

Link

Github Backend Data

Ever wondered where does the data get stored when you commit to Github.

Here is the coolest post:Partitioning GitHub’s relational databases to handle scale

Link

Lineage Metadata Where You Need It - Datafold’s GraphQL API

Column-level lineage lets you see where your data is coming from and going to in seconds. Zoom in to trace the flow and truly understand where the values in a column are coming from without digging through SQL code or spending hours hunting through PRs. This can be particularly helpful when you need to understand the downstream impact of a change in your pipeline or track where PII data is being used.

Link

Data Governance 101: What it is, and Why it’s Critical to Your Business

In an increasingly data-driven world, organizations use data to remain competitive, find new opportunities, and provide better service to their customers. While using this data, organizations must comply with laws and regulations applicable to data privacy, storage, and processing and extract valuable insights. 

Link

No matter how proficient you are in your field. Having the basics clear is the main key.

Trino Summit

OCTOBER 21ST & 22ND, 2021

Virtually join the community of Trino users and developers on October 21st & 22nd, where we'll share the latest on Trino and learn how some of the most innovative companies are using this technology to power their analytics platforms.

Link

Oracle Cloud Infrastructure Virtual Thought Leadership Event

October 22 at 4 PM IST

Register now for exclusive insights into our big data services trends and innovation journey. We can’t wait to share how our advances can help customers and employees alike, unlock data’s fundamental value and learn to extract and use it.

Link

The Docker Everything Bagel™ – Spin Up A Local Data Stack

An important part of developing an open source project like lakeFS is assisting and advising our users. When they run into an issue and feel pain, we want to feel that pain, too. Quite literally.

This means recreating the environment, running the same code, and raising the same error.

In complex, modern data stacks this is easier said than done. Developed from experience over the past year, we have a setup that helps us in this pursuit. Affectionately, it is referred to as the Everything Bagel.

Link

Create Powerful Data Pipelines by Mastering Sensors

• Configure your Sensors to avoid freezing your Airflow instance

• Dive into the ExternalTaskSensor

• Smart Sensors, what are they? Should you use them?

Link

Datafold: Why to Conduct Data Quality Post-mortems: Lessons learned from GitLab, Google, and Facebook

Organizations lose an average of $15 million per year due to poor data quality, according to Gartner Research. Poor data quality— inaccuracy, inconsistency, incompleteness, and unreliability— can have serious negative consequences for a data-driven company. Given the cost of these consequences, organizations have a high incentive to spot, fix, and understand them so that they don’t reoccur. That’s the goal of post-mortems.

Link

Feature Engineering at Scale

A reference implementation of the design patterns is provided in the attached notebooks to demonstrate how first-class design patterns can simplify the feature engineering process and facilitate efficiency across the silos in your organization. The approach can be integrated with the recently-launched Databricks Feature Store, the first of its kind co-designed with an MLOps and data platform, and can leverage the storage and MLOps capabilities of Delta Lake and MLFlow.

Link

How Istio, Tempo, and Loki speed up debugging for microservices

The architect decided to develop that system based on microservices. Hundreds of them! You, as a developer, think why? Why does the architect hate me so much? And then, the main question of the moment: How am I supposed to debug this?

Of course, we all understand the benefits of a microservice architecture. But we also hate the downsides. One of those is the process of debugging or running a postmortem analysis across hundreds of services. It is tedious and frustrating.

Link

Friday’s are not meant for production deployments. I repeat Abort!!

Webinars

Superset Meetup: Apache Superset 1.3

Apache Superset 1.3 was just released and brings a large suite of improvements. In this Meetup, we will showcase improvements to Dashboard Native Filters, new charts that were added, new supported data sources, and the numerous user experience improvements.

Time

Sep 22, 2021 10:00 AM Pacific Time (US and Canada)

Register

Pulsar Virtual Summit Europe 2021

The Pulsar Summit is the global event for engineers, architects, data scientists, and technical leaders interested in Pulsar and the messaging and streaming ecosystem. It is a unique opportunity to network and learn about Pulsar project updates, ecosystem developments, best practices, and adoption stories.

October 6th, 2021

Register

Future Data Driven

is a marquee, online event focusing on the comprehensive Microsoft Data Platform. Our mission is to inform and update attendees with Data & AI, DevOps & DataOps, PowerBI & Visualization, Integration & Automation and cloud infrastructure. We are excited to welcome you to participate at our free first ever Future Data Driven online event.

WHEN

29 September, 2021

Link

Open Source Data Stack Conference

Building the modern stack with open source data solutions.

SEPT. 28 - 30

Register

Manage Dependencies Between Airflow Deployments, DAGs, and Tasks

If you have missed the live session by Astronomer here is the recording.

Link

Logging, Monitoring, and Observability in Google Cloud PartI

Rishi Singhal

Customer EngineerGoogle Cloud

Rishi is a Customer Engineer for Google Cloud Platform (GCP). As part of his role he works with various Startups/Enterprises to solve the problems & challenges they are currently facing. He also helps them in visualising their NEXT journey with the power of GCP. Rishi has more than 16 years of IT experience with expertise in Application development and Data processing. He also has done startup consulting in Transport, Healthcare & Stock Trading domains.

Link

Datafold: Data Quality Management According to Lyft, Shopify, and Thumbtack

Given the complexity of real-world data quality management, it’s helpful to look at actual teams that have managed to establish and scale strong management practices. This article explores the data challenges faced by Shopify, Lyft, and Thumbtack and the systems they built to conquer them.

Link

Dremio: What Is a Data Pipeline?

Moving data between systems requires many steps: from copying data, to moving it from an on-premises location into the cloud, to reformatting it or joining it with other data sources. Each of these steps needs to be done, and usually requires separate software.

A data pipeline is the sum of all these steps, and its job is to ensure that these steps happen reliably to all data. These processes should be automated, but most organizations need at least one or two engineers to maintain the systems, repair failures and update according to the changing needs of the business.

Link

Templating in Airflow

Templating is a powerful concept in Airflow to pass dynamic information into task instances at runtime. For example, say you want to print the day of the week every time you run a task.

Link

The Modern Data Stack: Open-source Edition

Most open-source products listed below are in fact open-core, i.e. primarily maintained and developed by teams that make money off consulting about, hosting, and offering “Enterprise” features for those technologies. In this post, we are not taking into account the features that are only available through the SaaS/Enterprise versions, thereby comparing only openly available solutions.

Link

The importance of automating production ops

While running database services for AWS, Anurag Gupta, now Founder & CEO of Shoreline.io, learned the importance of going beyond monitoring and incident response to implement incident automation to automatically resolve common (and uncommon) production issues. In this episode, he talks about the importance of automating production ops, why just using cloud hosting or containers doesn’t fully solve the problem and how to think about building out an effective SRE team. 

Link

Always start from small, no matter how big is the problem. Keeping the base clear is the key.

Subscribe now

Share Just Enough Data

Newsletter Update: If you are wondering if the newsletter Name has changed, then you are right! It is Just Enough Data anymore(isn’t it cool 😎)

Webinar alert 1:

Airflow * MLops meetup

Thursday Sep2

In this virtual meetup, we will cover how to leverage Airflow in complex machine learning pipelines, where users need model reproducibility, the ability to run efficiently at large scale, and the power to run pipelines dynamically with changing inputs, parameters and algorithms.

After attending the session, Airflow engineers will know techniques to establish robust MLOps including how to use Airflow to automate all phases of ML model lifecycle.

Link

Webinar alert 2:

Online event focusing on the comprehensive Microsoft Data Platform

Our mission is to inform and update attendees with Data & AI, DevOps & DataOps, PowerBI & Visualization, Integration & Automation and cloud infrastructure. We are excited to welcome you to participate at our free first ever Future Data Driven online event.

WHERE

Virtual Event

WHEN

29 September, 2

Link

Dow Inc. uses AI to create new products

Dow Inc., one of the three largest materials science companies in the world, is an American multinational corporation that delivers a broad range of science-based products and solutions. Dow is undergoing a digital transformation using the Microsoft Azure, Office 365, and Dynamics 365 clouds. By combining Dow’s decades of chemistry experience with Microsoft Azure AI and machine learning, the company is shortening the discovery process for developing new products from months of exploratory laboratory work to seconds, speeding time to value for both Dow and its customers. Looking ahead, the company plans to harness this same AI-driven process to predict the products customers will want before they want them.

Link

The Docker Everything Bagel™ – Spin Up A Local Data Stack

An important part of developing an open source project like lakeFS is assisting and advising our users. When they run into an issue and feel pain, we want to feel that pain, too. Quite literally.

This means recreating the environment, running the same code, and raising the same error.

In complex, modern data stacks this is easier said than done. Developed from experience over the past year, we have a setup that helps us in this pursuit. Affectionately, it is referred to as the Everything Bagel.

Link

Defining Data Ownership

"You lost me at the first sentence. Data Engineers can't be data owners."

Link

Create Powerful Data Pipelines by Mastering Sensors

Do you use Sensors in your data pipelines? Do you need to wait for a file before executing the next step? Are you looking to execute your task after a task completes in another DAG? Would you like to wait for an import in your SQL table before executing the next task? The answer? Airflow Sensors! Sensors are pretty basic in their usage. You wait for a condition to be true. It’s as simple as that. But... That’s not all! Not only are there some arguments which allow you to make your sensors more reliable, but there are also a large number of sensors covering a wide range of use cases. By the end of this webinar, from basic to advanced, Sensors won’t hold any more secrets for you.

Link

The State of Data Quality in 2021

We ran a survey of data producers (data/analytics engineers) and consumers (product managers, data scientists, analysts, and other roles) to explore trends in tools, pain points, and goals. The survey had 231 respondents, mostly in SaaS, Finance, Consumer Internet, AI, Cloud, and Retail, and mostly from mid-to-large sized companies.

Link

Coolest way to meet Data Engineers and Scientists around the world

- Tooling, and data stacks
- Use cases and new approaches,
- PR request, issues
- and more! Let us give you the first warm welcome to this new community 

https://lnkd.in/eiKbZUps

Link

How to Build a Modern Data Stack

Breaking down what a modern data stack means in practice. We discuss four core components, five reasons to set it up, and how to orchestrate it.

Link

Frequently Asked Questions About the Data Lakehouse

In short, a Data Lakehouse is an architecture that enables efficient and secure Artificial Intelligence (AI) and Business Intelligence (BI) directly on vast amounts of data stored in Data Lakes.

Link

RudderStack:Here is a quick guide on how we leverage the functionality of the RudderStack application right here at RudderStack.

Link

Thought for the week:

Coding/Debugging is your forte until you can find everything in StackOverflow :P

Subscribe now

Share

Webinar: Take your ML projects from idea to production

AWS Dev Day Machine Learning Edition dives deep into the latest developments in machine learning (ML) and demonstrates how you can take your ML projects from idea to production. We’ll introduce Transformers architecture and discuss practical applications in Computer Vision and Protein Sequence Classification. We’ll also share best practices for the implementation of distributed model training and model pruning to optimize your models for production. Key concepts will be conveyed via demos.

Agenda

Live Q&A Session
Our experts will be available to answer all your Machine Learning related question via live Q&A broadcast!
September 30th, 10:00 - 12:00 GTM | 11:00 - 13:00 BST | 12:00 - 14:00 CEST

Link

Data engineering failure — Why is it almost impossible to meet deadlines?

Failure. I would like to write about failure. Why is it so hard for data engineers to evaluate projects duration but also to meet deadlines?

Over my last 7 years of experience I had the opportunity to work with multiple data engineering teams. A common pattern I noticed projects after projects was the difficulty to define and meet milestones for engineers.

Link

tropos.io: Our take on Apache SuperSet

Our current take on Apache Superset; and why we think it is a viable alternative to Tableau at scale.

Link

Decoding the Data Mesh

Zhamak Dehghani, founder of the data mesh, dispels common misunderstandings around this the data mesh, an increasingly popular approach to building a distributed data architecture, and shares how some of the best teams are getting started. 

Link

What I look for in open core data infrastructure companies

There has never been a better time to build a data infrastructure company. Despite all the buzz over the last decade about the need for businesses to be data-driven, the reality is that the infrastructure to enable meaningful data work at scale has lagged the hype. That is changing now. Why?

Link

Databricks Raises $1.6 Billion Series H Investment at $38 Billion Valuation

Databricks, the Data and AI company, today announced a $1.6 Billion round of funding to accelerate innovation and adoption of the data lakehouse, as the data architecture’s popularity across data-driven organizations continues to grow at a rapid pace. 

Link

Complex tasks orchestration at Hurb with Apache Airflow

How Apache Airflow is helping Hurb’s Data Engineering team to create and orchestrate workflows in an elegant way

Link

Data Lineage at Slack

How we automate building and maintaining data lineage at Slack.

Reinventing how the world does work inevitably creates a lot of data. Each year, Slack’s scale has increased and the volume of data ingested and stored has kept pace. To make it possible to understand relationships within our data, we’ve invested heavily in an automated data lineage framework. This facilitates producer/consumer coordination, improves risk mitigation, impact analysis, and better execution of data programs here at Slack.

Link

firebolt.io: Snowflake vs Databricks vs Firebolt

how do you compare Snowflake and Databricks?

Link

Thought for the week :)

Don’t unravel a warning in your code on Friday, you never know its just a tip of the iceberg!

Ajith Shetty

BigData Engineer — Love for Bigdata, Analytics, Cloud and Infrastructure.

Subscribe✉️ ||More blogs📝||Linked In📊||Profile Page📚||Git Repo👓

Webinar Data Mesh 101: What It Is & Why You Need It

Sept 9th | 10am - 11am EST | 3pm-4pm BST

What is Data Mesh? Join Starburst for a 30 minute introduction into this modern approach to managing analytics at scale. This is the first installment of a series on Data Mesh. Coined by Zhamak Dehghani, Principal Technology Consultant at ThoughtWorks, Data Mesh embraces decentralization over-centralization, meaning it allows companies to become more efficient in accessing and exploiting data as a core architectural approach. Data Mesh addresses the flaws in monolithic data warehouses models. 

LINK

Salesforce announces first integrations with Slack after closing $28B sale

LINK

Adobe buys collaborative video editing software firm Frame.io for $1.275B

LINK

Building a data discovery solution with Amundsen and Amazon Neptune

LINK

Data warehousing has problems. A data mesh could be the solution

LINK

Data Quality Meetup brings together Data leaders and practitioners to share and learn the best practices for data quality, governance & productivity.

LINK

Zero to PoC in 2 Weeks: Jumpstart your data and AI projects with Databricks Industry Solution Accelerators

LINK

Let Starburst Light Up Your World

At Starburst, we want to free our customers to see the invisible and achieve the impossible. Seeing the invisible means allowing you to have access to data you never had access to before. Achieving the impossible means unlocking new insights through greater access to all of your enterprise information. We do that by turning the data warehousing model upside down. 

LINK

Apache Arrow DataFusion 5.0.0 Release

LINK

Data Mesh Behind the Walls – Saxo Bank; Data Mesh Learning Meetup #009

The CEO of $28 billion AI startup Databricks says 'sky computing' is the future, with Amazon, Microsoft, and Google becoming more open cloud platform

LINK

My name is Ajith and I'm a Bigdata engineer, an aspiring writer, Love spending time in learning new technologies and writing the blogs. Thanks for visiting and I'd love to connect!

Here are my medium blogs.

What's inside DELTA LAKE

Who, What and Why DAGSTER!!

Spark Bucketing is not as simple as it looks

Lesser Known Facts/Short cuts in Spark(PART2)

Let’s talk about Spark (Un)Cache/(Un)Persist in Table/View/DataFrame, In depth

Spark, Parallelising the Parallel Jobs

Predicate Pushdown is for real??

Lesser Known Facts/Short cuts in Spark(PART1)

What’s the buzz about Parquet File format?

Subscribe✉️ ||Linked In📊||Git Repo👓

https://iconiqgrowth.com/monte-carlo-data-reliability-in-every-scenario/

DataHub Roadmap Q3 and Q4

https://datahubproject.io/docs/roadmap/

[Webinar] Creating Powerful Data Pipelines by Mastering Sensors

Event by Astronomer Online

Tue, Aug 24, 2021, 7:00 PM - 8:00 PM IST

https://www.linkedin.com/events/webinar-creatingpowerfuldatapip6831511707458785280/

Astronomer Weekly Demo

Event by Astronomer

Online

Thu, Aug 26, 2021, 6:00 PM - 7:00 PM IST

https://www.linkedin.com/events/astronomerweeklydemo6830789669744066560/

Delta -IO Roadmap 2021 H2 (discussion) #748

https://github.com/delta-io/delta/issues/748

Why we invested in Firebolt: Snowflake catapulted the data warehouse into the cloud. Now Firebolt is making it fly

https://medium.com/dawn-capital/snowflake-catapulted-the-data-warehouse-into-the-cloud-f3332448d3c

Subscribe now

Leave a comment

Share

Sign up now so you don’t miss the first issue.

Subscribe now

In the meantime, tell your friends!