20210427 azure lille_meetup_azure_data_stack

premiseo.com dataredkite.com
26/02/2021 Meetup Azure Lille 1
Azure Data

dataredkite.com
premiseo.com
Who are we ?
26/02/2021 2
I'm a data and cloud Architect and Spark lover.
I worked many years as an Oracle consultant and
expert, and now I work with Cloud solutions devoted to
solve complex problems with high volumes of data.
I am a Data Analyst & Solution Architect indepedent -
☁️ MCSE, Cosmos DB & Delta lover.
I developed my skills through various clients' projects,
teaching at the University and personal proof of
concepts.
I’m also the Co-Founder of DataRedkite, a product which
can quickly give to its user a good management of data
in Microsoft Azure DataLake.
Laurent Leturgez Alexandre Bergere
Meetup Azure Lille

dataredkite.com
premiseo.com
Summary
Relational Databases NoSQL Databases Big Data Storage
Data Big Data Streaming
Storage :
Compute :

Storage
4

Relational Databases
Managed relational SQL Database
as a service
Azure SQL Database
Managed MariaDB database
service for app developers
Azure Database for MariaDB
Managed MySQL database service
for app developers
Azure Database for MySQL
Managed Postgres database
Azure Database for PostGres

as a service
Azure SQL Database
for app developers
Azure Database for PostGres

as a service
Azure SQL Database Azure SQL Database

dataredkite.com
premiseo.com
Azure SQL Database
• Azure SQL
• SQL Server Paas service
• Managed upgrades, patches, backups and monitoring
• Latest Stable version of SQL Server
• 99,99% availability
• Deployment model
• Single Database : database runs on non shared resources
• Elastic Pool : database runs with a collection of databases that share set of resources at a
predictable price

dataredkite.com
premiseo.com
Azure SQL Database
• Azure SQL
• Purchasing model
• DTU (Database Transaction Unit) : https://docs.microsoft.com/en-us/azure/azure-sql/database/service-tiers-
dtu
• Basic tier
• Standard Tier
• Premium Tier
• vCore model
• Serverless
• Service Tier
• General Purpose (vCore) / Standard (DTU) : Common workloads
• Business Critical (vCore) / Premium (DTU) : High transaction and availability / low latency IO
• HyperScale (vCore) :
• Up to 100Tb Database
• Rapid Scale up (compute resources)
• Rapid Scale out (read only nodes : read workload / hot-standby)

dataredkite.com
premiseo.com
Azure SQL Database
• Azure SQL Managed Instance
• Features
• Paas platform for lift and shift at scale
• Broadest SQL Server engine compatibility (network integration, features etc.)
• With perservation of all Paas capabilities (patching, updates, backups, HA etc.)
• vCore purchase model only
• BYOL available
• SQL Virtual Machine
• SQL Server deployment on VM (Linux and Windows)
• Can choice SQL Server version
• From 2008 R2
• Up to 2019

dataredkite.com
premiseo.com
Azure SQL Database
Azure SQL Database
Managed Instance
Instance scoped model with
high compatibility to SQL Server
Best for modernisation at scale
with low cost effort (lift & shift)
Single
Standalone managed database
for predictable and stable
workloads
Elastic Pool
Shared resources model :
multitenant

as a service
Azure SQL Database
for app developers
Azure Database for PostGre

dataredkite.com
premiseo.com
Azure Database for PostgreSQL
• Paas Service for PostgreSQL
• Runs on Windows
• Single Server
• v9.5 to 11
• Up to 64 vCores depending on SKU (https://docs.microsoft.com/en-us/azure/postgresql/concepts-pricing-
tiers)
• Up to 2 for Basic SKU
• Up to 64 for General Purpose SKU
• Up to 32 for Memory Optimized SKU
• Bunch of PG Extensions available
• Automated Backup (retention up to 35days)
• Backup frequency and backup types depend on database size
• Geo-redundant backup option (General Purpose & Memory Optimized)

dataredkite.com
premiseo.com
• HyperScale (Citus)
• High performance and analytical workloads beyond 100Gb
• Hyperscale delivers
• Horizontal scaling across multiple machine (with Sharding)
• Query parallelization across these servers
• High performance for analytics
• Based on server groups
• Design approach required for table distribution and performance
• Distributed tables (based on distribution column)
• Reference tables (content concentrated into a single shard replicated on every worker node)
• Local tables (ordinary unsharded tables. Perfect for small tables not involded into joins)
• Automated backup through storage snapshots

dataredkite.com
premiseo.com
• Flexible Server (Preview)
• Automated patching
• Automatic backups
• Performance adjustment in three switchable compute tiers : Burstable, GP, Memory Optimized
High Availability Zone Redundant HA (Optional)

as a service
Azure SQL Database
for app developers

dataredkite.com
premiseo.com
• Paas Service for MariaDB
• Runs on Windows
• Single Server
• V10.2 and 10.3
• Up to 64 vCores depending on SKU (https://docs.microsoft.com/en-us/azure/mariadb/concepts-pricing-tiers)

as a service
Azure SQL Database
for app developers

dataredkite.com
premiseo.com
• Paas Service for MySQL
• Runs on Windows
• Single Server
• V5.6, 5.7, and 8.0
• Up to 64 vCores depending on SKU (https://docs.microsoft.com/en-us/azure/mysql/concepts-pricing-tiers)

dataredkite.com
premiseo.com
Flexible Server (Preview)
• V5.7
• Automated patching
• Automatic backups
• Performance adjustment in three switchable compute tiers : Burstable, GP, Memory Optimized
• Network Isolation
• Private Access through Vnet integration
• Public Access

dataredkite.com
premiseo.com
Flexible Server (Preview)
High Availability Zone Redundant HA (Optional)

NOSQL Databases
Globally distributed, multi-model
database for any scale
Azure Cosmos DB

dataredkite.com
premiseo.com
Azure Cosmos DB
26/02/2021 23
A globally distributed, massively scalable, multi-model database service
Azure Cosmos DB
o SQL API
o MongoDB API
o Cassandra API
o Gremlin API
o Table API

dataredkite.com
premiseo.com
Azure Cosmos DB
26/02/2021 24
Throughput
Cosmic Notes

Big Data
Storage
REST-based object storage for
unstructured data
Storage Account
Massively scalable, secure data
lake functionality built on Azure
Blob Storage
Azure Data Lake Storage

Big Data
Storage
unstructured data
Storage Account
Blob Storage

dataredkite.com
premiseo.com
Storage Account
26/02/2021 27
o Azure Blobs : A scalable object store for text and binary data
o Azure Files : Managed file shares for cloud or on-premises deployments
o Azure Queues : A messaging store for reliable messaging between application components
o Azure Tables : A NoSQL store for no-schema storage of structured data
Azure Storage accounts are the base storage type within Azure. Azure Storage offers a very scalable object store for data
objects and file system services in the cloud. It can also provide a messaging store for reliable messaging, or it can act as a
NoSQL store.
Azure selected four of these data services and placed them together under the name Azure Storage. The four services are
Azure Blobs, Azure Files, Azure Queues, and Azure Tables. The following illustration shows the elements of Azure Storage

dataredkite.com
premiseo.com
Storage Account
26/02/2021 28
Type of Storage Account
Storage account type Services Redundancy options
General-purpose V2 Basic storage account type for blobs, files, queues, and tables. Recommended
for most scenarios using Azure Storage.
LRS, GRS, RA-GRS, ZRS, GZRS,
RA-GZRS
General-purpose V1 Legacy account type for blobs, files, queues, and tables. Use general-purpose
v2 accounts instead when possible.
LRS, GRS, RA-GRS
BlockBlobStorage Storage accounts with premium performance characteristics for block blobs
and append blobs. Recommended for scenarios with high transactions rates, or
scenarios that use smaller objects or require consistently low storage latency.
LRS, ZRS
FileStorage Files-only storage accounts with premium performance characteristics.
Recommended for enterprise or high performance scale applications.
LRS, ZRS
BlobStorage Legacy Blob-only storage accounts. Use general-purpose v2 accounts instead
when possible.
LRS, GRS, RA-GRS

dataredkite.com
premiseo.com
Replication Options
27/04/2021 29

dataredkite.com
premiseo.com
Replication Strategy
27/04/2021 30

Big Data
Storage
unstructured data
Storage Account
Blob Storage

dataredkite.com
premiseo.com
Azure Datalake Store
26/02/2021 32
Azure Data Lake Storage is a Hadoop-compatible data repository that can store any size or type of data. This storage
service is available as Generation 1 (Gen1) or Generation 2 (Gen2).
Key features of Data Lake Storage:
o Unlimited scalability
o Hadoop compatibility
o Security support for both access control lists (ACLs) & RBAC (for Gen 2 only)
o POSIX compliance
o An optimized Azure Blob File System (ABFS) driver that's designed for big-data analytics
o Zone-redundant storage
o Geo-redundant storage
Azure Datalake Gen 1 Azure Datalake Gen 2

dataredkite.com
premiseo.com
Choose a storage solution on Azure
26/02/2021 33
Data classification Operations Latency & throughput Transactional support Recommended service
Product catalog data Semi-structured because of
the need to extend or modify
the schema for new products
o Customers require a high
number of read operations,
with the ability to query on
many fields within the
database.
o The business requires a
high number of write
operations to track the
constantly changing
inventory.
High throughput and low
latency
Required Azure Cosmos DB
Photos and videos Unstructured o Only need to be retrieved
by ID.
o Customers require a high
number of read operations
with low latency.
o Creates and updates will be
somewhat infrequent and
can have higher latency
than read operations.
Retrievals by ID need to
support low latency and high
throughput. Creates and
updates can have higher
latency than read operations.
Not required Azure Blob storage
Business data Structured Read-only, complex analytical
queries across multiple
databases
Some latency in the results is
expected based on the
complex nature of the queries
Required Azure SQL Database

Compute
34

Data
Compute
Managed data-integration solution
Data Factory
Process events with serverless
code
Azure Functions

Data
Compute
Data Factory
code
Azure Functions

dataredkite.com
premiseo.com
Azure Function
37
Azure Functions is the serverless compute service from Microsoft. Functions are event-driven: each function defines a
trigger — the exact definition of the event source, for instance, the name of a storage queue.
Uses cases:
If you want to... then...
Build a web API Implement an endpoint for your web applications using the HTTP trigger
Process file uploads Run code when a file is uploaded or changed in blob storage
Build a serverless workflow Chain a series of functions together using durable functions
Respond to database changes Run custom logic when a document is created or updated in Cosmos DB
Run scheduled tasks Execute code at set times
Create reliable message queue systems Process message queues using Queue Storage, Service Bus, or Event Hubs

dataredkite.com
premiseo.com
Azure Function
38
Consumption Plan Functions
Consumption Plan (B1, B2, B3, S1, S2, S3
Scale automatically and only pay for compute resources when your functions are running. On
the Consumption plan, instances of the Functions host will be dynamically added and
removed based on the number of incoming events.
Premium plan (P1v2, P2v2, P3v3)
While automatically scaling based on demand, use prewarmed workers to run applications
with no delay after being idle, run on more powerful instances and connect to VNETs.
Azure App Service plan
Run Functions within an App Service plan at regular App Service plan rates. Good fit for long-
running operations, as well as when more predictive scaling and costs are required.
Azure Functions hosting options : Azure Plan

dataredkite.com
premiseo.com
27/04/2021 39
Durable Functions is a library that brings workflow orchestration abstractions to Azure Functions. It introduces a number of idioms and tools
to define stateful, potentially long-running operations, and manages a lot of mechanics of reliable communication and state management
behind the scenes.
Log of events in the course of orchestrator
progression
3 steps of a workflow executed in sequence
https://medium.com/hackernoon/making-sense-of-azure-durable-functions-
645ecb3c1d58
Azure Function
Azure Durable Functions

Data
Compute
Data Factory
code
Azure Functions

dataredkite.com
premiseo.com
Azure Data Factory
• Serverless Data Integration service
• Data Pipeline : logical group of activities
• Data Flow : Data Transformation activity
• Data Copy : Data Transfer activity
• SSIS Integration
• Git integration

dataredkite.com
premiseo.com
Azure Data Factory
• Job scheduling
• Automatically through internal Scheduler
• Manually
• SDK : .NET, Python
• REST API
• PowerShell

dataredkite.com
premiseo.com
Azure Data Factory
• Integration runtime
• Compute infrastructure used by ADF to provide data integration
• Azure : Serverless
• Self Hosted : Onprem or Azure Virtual Machine (Windows)
• SSIS
Activity Features
Azure Data Flow
Data Copy
Dispatch Activity (HDI, Databricks, SQL …)
Cloud to Cloud data transfer/flows
Self-Hosted Data Flow
Data Copy
Dispatch Activity (HDI, Databricks, SQL …)
OnPrem or Virtual Machine deployment (Windows)
OnPrem <-> Cloud data transfer/flows
When connectors are not available
SSIS SSIS Package execution Private or public Network

Big Data
Compute
Fast, easy, and collaborative
Apache Spark-based analytics
platform
Azure Databricks
HDInsight supports the latest open
source projects from the Apache
Hadoop and Spark ecosystems.
Azure HDInsight
Managed Enterprise
Datawarehouse and BigData
Analytics service
Azure Synapse Analytics

Big Data
Compute
platform
Azure Databricks
Azure HDInsight
Managed Enterprise
Analytics service

dataredkite.com
premiseo.com
Azure Databricks
26/02/2021 46

dataredkite.com
premiseo.com
Azure Databricks
26/02/2021 47
Azure Databricks is a data analytics platform optimized for the Microsoft Azure cloud services platform. Azure Databricks
offers two environments for developing data intensive applications:
o Azure Databricks Workspace: provides an interactive workspace that enables collaboration between data engineers,
data scientists, and machine learning engineers.
o Azure Databricks SQL Analytics: provides an easy-to-use platform for analysts who want to run SQL queries on their
data lake, create multiple visualization types to explore query results from different perspectives, and build and share
dashboards.

Big Data
Compute
platform
Azure Databricks
Azure HDInsight
Managed Enterprise
Analytics service

dataredkite.com
premiseo.com
Azure HDInsights
• Managed Hadoop distribution for Azure
• Based on Cloudera Hortonworks hadoop distribution
• Comes in various flavours / shapes (VM shapes and number)
• Hadoop : General purpose (HDFS, Yarn, MapReduce, Hive, Pig, Sqoop, Oozie)
• Spark
• Kafka
• HBase
• Hive / LLAP (Interactive Query)
• Storm (Stream processing)
• ML Services with R

dataredkite.com
premiseo.com
Azure HDInsights
• At least one Storage account mandatory (for libs and binaries)
• External Metastores available for Ambari, Hive and Oozie
• HDInsights architecture

Big Data
Compute
platform
Azure Databricks
Azure HDInsigth
Managed Enterprise
Analytics service

dataredkite.com
premiseo.com
Stream Analytics
Data Factory
Data Lake
Modern Analytics
MPP
Datawarehouse

dataredkite.com
premiseo.com
MPP
Datawarehou
se
Choice of language (T-SQL, Spark
SQL, Python, Scala, .Net)
Analytics ready (Analysis Services,
Power BI)
Data Science and AI Ready (Azure
Machine Learning integration)

dataredkite.com
premiseo.com
Synapse Analytics
• Sample Use Case : Pure Business Intelligence !

dataredkite.com
premiseo.com
• Not for small database (Usually > 1Tb)
• Cost Model
• Synapse Provisioned
• T-SQL Pool with DWU (Datawarehouse Units)
• Storage (Geo redundant option)
• Synapse Serverless
• Spark Pools
• Synapse Pipeline

dataredkite.com
premiseo.com
• Architecture
• DMS (Data Movement Service)
• Used for Data Colocation
• Key point: Data Partitioning
and Data Distribution

dataredkite.com
premiseo.com
• Hash distributed table • Replicated Table • Round Robin
distributed Table
• Example
• Dimension to Fact table join

Big Data
Streaming
Real-time data stream processing
from millions of IoT devices
Azure Stream Analytics
Connect, monitor and manage
billions of IoT assets
Azure IoT Hub
Real-time data stream with Kafka
Azure HDInsigth & Kafka
Use Spark Streaming with
Databricks
Spark Streaming with Databricks

Big Data
Streaming
Azure IoT Hub
Databricks

dataredkite.com
premiseo.com
Azure Streaming Analytics
26/02/2021 60

dataredkite.com
premiseo.com
Azure Streaming Analytics
26/02/2021 61
o Azure Stream Analytics supports user-defined functions (UDF) or user-defined aggregates (UDA) in JavaScript for cloud jobs and C# for IoT
Edge jobs
UDFs, UDAs, and custom deserializers:
o Analyze real-time telemetry streams from IoT devices
o Web logs/clickstream analytics
o Geospatial analytics for fleet management and driverless vehicles
o Remote monitoring and predictive maintenance of high value assets
o Real-time analytics on Point of Sale data for inventory control and anomaly detection
Examples scenarios:

Big Data
Streaming
Azure IoT Hub
Databricks

dataredkite.com
premiseo.com
Azure Iot Hub
63
Azure IoT Hub :
o The cloud gateway that connects IoT devices to gather data and drive business insights and automation.
o The big data streaming service of Azure. It is designed for high throughput data streaming scenarios where customers
may send billions of requests per day.
o Bi-directional communication capabilities

dataredkite.com
premiseo.com
Iot Hub or Event Hubs
64
IoT Hub was developed to address the unique requirements of connecting IoT devices to the Azure cloud while Event Hubs
was designed for big data streaming. Microsoft recommends using Azure IoT Hub to connect IoT devices to Azure.
IoT Capability IoT Hub standard tier IoT Hub basic tier Event Hubs
Device-to-cloud messaging
Protocols: HTTPS, AMQP, AMQP over webSockets
Protocols: MQTT, MQTT over webSockets
Per-device identity
File upload from devices
Device Provisioning Service
Cloud-to-device messaging
Device twin and device management
Device streams (preview)
IoT Edge

Big Data
Streaming
Azure IoT Hub

dataredkite.com
premiseo.com
Apache Kafka on HDInsight architecture

Big Data
Streaming
Azure IoT Hub
Databricks

dataredkite.com
premiseo.com
Azure Databricks
26/02/2021 68
o Apache Spark Streaming is a scalable fault-tolerant streaming processing system that natively supports both batch and
streaming workloads.
o Spark Streaming is an extension of the core Spark API

Data Tools

dataredkite.com
premiseo.com
Azure Data Studio
26/02/2021 70
Azure Data Studio is a cross-platform database tool that you can run on Windows, macOS, and Linux. You'll use it to
connect to SQL Data Warehouse and Azure SQL Database.
Previously released under the preview name SQL Operations Studio, Azure Data Studio offers a modern editor experience
with IntelliSense, code snippets, source control integration, and an integrated terminal. It is engineered with the data
platform user in mind, with built in charting of query result sets and customizable dashboards.

dataredkite.com
premiseo.com
Storage Explorer
26/02/2021 71
Begin by downloading and installing Storage Explorer. You can use Storage Explorer to do several operations against data
in your Azure Storage account and data lake:
o Upload files or folders from your local computer into Azure Storage.
o Download cloud-based data to your local computer.
o Copy or move files and folders around in the storage account.
o Delete data from the storage account.

dataredkite.com
premiseo.com
Visual Studio Code
26/02/2021 72
Visual Studio Code is a lightweight source code editor which runs on your desktop and is available for Windows, macOS
and Linux. It comes with built-in support for JavaScript, TypeScript and Node.js and has a rich ecosystem of extensions for
other languages (such as C++, C#, Java, Python, PHP, Go) and runtimes (such as .NET and Unity).

Data Migration Tools

dataredkite.com
premiseo.com
Summary
26/02/2021 74
Scenario Some recommended solutions
Disaster Recovery Azure geo-redundant backups
Read Scale Use read-only replicas to load balance read-only query
workloads (preview)
ETL (OLTP to OLAP) Azure Data Factory or SQL Server Integration Services or
Databricks
Migration from on-premises SQL Server to Azure SQL
Database
Azure Database Migration Service
Kept up-to-date across several Azure SQL databases or SQL
Server database
Azure SQL Data Sync
Detecting compatibility issues that can impact database
functionality in your new version of SQL Server or Azure SQL
Database
Data Migration Assistant (DMA)

Resources

dataredkite.com
premiseo.com
Azure charts
26/02/2021 76
https://azurecharts.com/

26/02/2021 77
Just few sources in Microsoft Learn:
o Azure for the Data Engineer
o Store data in Azure
o Work with relational data in Azure
o Large Scale Data Processing with Azure Data Lake Storage Gen2
o Implement a Data Streaming Solution with Azure Streaming Analytics
o Implement a Data Warehouse with Azure SQL Data Warehouse
Sources

dataredkite.com
premiseo.com
Fill the form
78
https://forms.office.com/Pages/ResponsePage.as
px?id=M3s0akU8nUyLePs4Zpn6Tp_2uFsS8cJJsHCS
wweCY5JUNVlMMllQNU4yRUVVWjFEOU5GVVc2S
VU3Si4u
Your turn !

platform
Azure Databricks
Next Session: Azure
Databricks

dataredkite.com
premiseo.com
Thank you
26/02/2021 80
Meetup Azure Lille
dataredkite.com
https://premiseo.com/

20210427 azure lille_meetup_azure_data_stack

More Related Content

What's hot

What's hot (20)

Similar to 20210427 azure lille_meetup_azure_data_stack

Similar to 20210427 azure lille_meetup_azure_data_stack (20)

More from Alexandre BERGERE

More from Alexandre BERGERE (6)

Recently uploaded

Recently uploaded (20)

20210427 azure lille_meetup_azure_data_stack