Mongo db world 2014 nyc mongodb on azure - tips tricks and examplesBrian Benz
This document discusses MongoDB options on Azure, including running MongoDB on virtual machines or using hosted MongoDB services. It provides an example of how to provision a MongoDB replica set across multiple Azure virtual machines and configure endpoints. Automation support is available for deploying and configuring MongoDB on Azure virtual machines. Links are also included for tutorials and other resources on MongoDB and Azure.
The document discusses SQL Server 2014's in-memory OLTP feature. It begins by explaining the need for an in-memory architecture due to hardware trends. It then covers how the in-memory tables store and access data via optimized structures and algorithms. Native compiled stored procedures are also discussed. The benefits are high performance for hot datasets that fit entirely in memory, while limitations include unsupported data types and inability to partially store tables.
This document discusses AlwaysOn availability technologies in SQL Server 2017, including Failover Cluster Instances (FCI) and Availability Groups (AG). It provides an overview of how FCI and AG work to provide high availability and disaster recovery, including capabilities like multi-subnet support. The document also summarizes new features and enhancements to FCI and AG in SQL Server 2014, 2016 and 2017.
- Distributed Replay allows replaying a captured workload from multiple client computers to better simulate production loads.
- A controller coordinates the replay across clients to reproduce the original query rates or run in stress test mode faster than original rates.
- It improves on SQL Server Profiler for application compatibility testing, performance debugging, capacity planning, and benchmarking.
- Events are replayed in synchronization mode to match original order, or unsynchronized to stress test without timing constraints.
Session delivered to SQL Relay 2015 in Nottingham.
In this session we look at some of the fundamental elements that you need to understand in order to build an IaaS solution that will meet the requirements to be covered by the Microsoft availability SLA. Additionally we will look at building a Microsoft Azure IaaS Solution.
Demo Code available at: http://1drv.ms/1PC8707
*Note if you want to use the demo code you will need a Microsoft Azure subscription.
Microsoft released SQL Azure more than two years ago - that's enough time for testing (I hope!). So, are you ready to move your data to the Cloud? If you’re considering a business (i.e. a production environment) in the Cloud, you need to think about methods for backing up your data, a backup plan for your data and, eventually, restoring with Red Gate Cloud Services. In this session, you’ll see the differences, functionality, restrictions, and opportunities in SQL Azure and On-Premise SQL Server 2008/2008 R2/2012. We’ll consider topics such as how to be prepared for backup and restore, and which parts of a cloud environment are most important: keys, triggers, indexes, prices, security, service level agreements, etc.
SQL Server 2014 New Features (Sql Server 2014 Yenilikleri)BT Akademi
The document summarizes several topics discussed by Ismail Adar including buffer pool extension, resource governor for I/O, delayed durability, DMV sys.dm_exec_query_profiles, and selecting into parallel. Buffer pool extension allows using SSD storage to increase the amount of memory available for the buffer pool. Resource governor for I/O provides I/O level isolation between workloads. Delayed durability controls the durability of transactions. The DMV sys.dm_exec_query_profiles profiles query execution. Selecting into allows inserting results of a query in parallel into a table.
SQL Server Source Control for Beginners is a presentation about using source control for SQL Server databases. It discusses:
- Common source control systems like SVN, TFS, and Mercurial that can be used to track database changes.
- How RedGate SQL Source Control allows committing database objects and changes directly from SQL Server Management Studio.
- Key features like rolling back changes, maintaining referential integrity, and locking objects while working.
- Demonstrates setting up a database in source control and committing changes.
- Additional tools that can deploy databases directly from source control like SQL Compare and SQL Data Compare.
Upgrade to MySQL 5.7 and latest news planned for MySQL 8Ted Wennmark
The document discusses upgrading to MySQL 5.7 from previous versions. It provides an agenda that covers MySQL 5.7, upgrading to MySQL 5.7, and MySQL 8. It then discusses reasons to upgrade including performance/scalability improvements in MySQL 5.7, new features in 5.7 like JSON support and optimizer improvements, staying on a fully supported release, and security improvements in 5.7. Benchmarks show MySQL 5.7 is up to 6x faster than previous versions on OLTP workloads.
SQL Server 2017 will bring SQL Server to Linux for the first time. This presentation covers the scope, schedule, and architecture as well as a background on why Microsoft is making SQL Server available on Linux.
PASS VC: SQL Server Performance Monitoring and BaseliningPARIKSHIT SAVJANI
When managing large scale deployment of SQL Server instances, it is important for DBAs to setup proactive monitoring & establishing performance baselines which helps in performance tuning, capacity planning & identifying workload patterns. Attend this session to learn what data should a DBAs collect & how, to monitor & establish performance baseline in SQL Server.
KoprowskiT - SQLBITS X - 2am a disaster just beganTobias Koprowski
A document outlines best practices for surviving a disaster involving SQL Server infrastructure. It recommends being well prepared with regular backups stored offsite, documented restore procedures, clear roles and responsibilities, and service level agreements defining acceptable downtimes. Key aspects of preparation include backups, restore testing, documentation, contact lists, hardware and software inventory, passwords, encryption keys, defined teams, and keeping management informed. The overall message is that with proper planning, a disaster can be survived by following the best practice of being prepared.
MySQL 8.0 is the latest Generally Available version of MySQL. This session will give a brief introduction to MySQL 8.0 and help you upgrade from older versions, understand what utilities are available to make the process smoother and also understand what you need to bear in mind with the new version and considerations for possible behaviour changes and solutions. It really is a simple process.
This talk is about covering MySQL as Open Source Database. We will be talking about the past, now and future of MySQL.
What are the best use cases of MySQL and edge Cases of MySQL? Time permitting mention on war stories, real-life examples. We will also cover important topics like scalability and proxy wars.
On my first session I would to introduce everyone to formerly known SQL Azure (actually Windows Azure SQL Database). Under Tips and Trick session I will show which points, features, compatibility and non-compatibility for SQL Azure are important for DBA's. I will cover functionalities, performance, cost, and sla and security aspects.
After break I will show how we can work with our data in the Cloud using SQL Azure, Blob Storage, what functionality of backup, restore, encryption and availability are available for us, how we can implement hybrid environment and when an why it is (or not) good practice.
And finally I hope we will find few minutes for discussion about Future of the DBA (not only in AD 2016)
2° Ciclo Microsoft CRUI 3° Sessione: l'evoluzione delle piattaforme tecnologi...Jürgen Ambrosi
L’obiettivo è quello di fare una panoramica dello stato dell’arte sulle tecnologie a supporto dei database. Alcuni esempi sono la tecnologia in-memory integrata con le funzionalità di analisi operative in tempo reale e della tecnologia Always Encrypted per la protezione dei dati utilizzati in locale o durante gli spostamenti. La tecnologia in-memory consente di migliorare di 30 volte le performance delle transazioni utilizzando hardware standard di settore. Inoltre i Big Data e l'analisi sono diventati un importante fattore di differenziazione competitivo, ma la gestione delle enormi quantità di dati correlate a un tempo di attività 24 ore su 24 continua a essere una sfida per l'IT. Oggi è più importante che mai soddisfare a livello aziendale l'esigenza di prestazioni, disponibilità e sicurezza efficace per gestire carichi di lavoro mission-critical a un costo contenuto. Le soluzioni Microsoft fissano un nuovo standard nelle performance mission-critical.
This document provides an overview of In-Memory OLTP and other SQL Server 2016 features such as Stretch Database, Always Encrypted, Dynamic Data Masking, and Query Store. It discusses how In-Memory OLTP can significantly improve database application performance through its memory-optimized tables and natively compiled stored procedures. It also summarizes capabilities for several high availability and security features introduced in SQL Server 2016.
The document provides an overview of new features and enhancements in SQL Server 2008 including:
- .NET Framework integration and new data types
- Database engine improvements like partitioning and failover clustering
- Management tools like SQL Server Management Studio and SQLCMD
- Performance tuning tools like the Database Tuning Advisor
- Analytics capabilities including Analysis Services and Reporting Services
- Replication, reporting, and integration with other Microsoft technologies
It also discusses best practices for upgrading from previous versions of SQL Server to version 2008.
Maintenance Plans for Beginners (but not only) | Each of experienced administrators used (to some extent) what is called Maintenance Plans - Plans of Conservation. During this session, I'd like to discuss what can be useful for us to provide functionality when we use them and what to look out for. Session at 200 times the forward-300, with the opening of the discussion.
Enhancements that will make your sql database roar sp1 edition sql bits 2017Bob Ward
This document provides information about various SQL Server features and editions. It includes a list of features available in each edition like row-level security, dynamic data masking, and in-memory OLTP. It also includes memory limits, MAXDOP settings, and pushdown capabilities for different editions. The document discusses lightweight query profiling improvements in SQL Server 2016 SP1 and provides details on predicate pushdown indicators in showplans.
Introduction to microsoft sql server 2008 r2Eduardo Castro
In this presentation we review the new features in SQL 2008 R2.
Regards,
Ing. Eduardo Castro Martinez, PhD
http://comunidadwindows.org
http://ecastrom.blogspot.com
Dans cette session nous vous présenterons les différentes manières d'utiliser SQL Server dans une infrastructure Cloud (Microsoft Azure). Seront présentés des scénarios hybrides, de migration, de backup, et d'hébergement de bases de données SQL Server en mode IaaS ou PaaS.
This document summarizes the key points from a presentation on SQL Server 2016. It discusses in-memory and columnstore features, including performance gains from processing data in memory instead of on disk. New capabilities for real-time operational analytics are presented that allow analytics queries to run concurrently with OLTP workloads using the same data schema. Maintaining a columnstore index for analytics queries is suggested to improve performance.
Kellyn Pot'Vin-Gorman presented on copy data management and virtualization for DBAs. She discussed how virtualization can be used to provision databases more quickly and easily for tasks like patching and testing without needing to copy large amounts of physical data. She also covered how command line interfaces can be used to automate some of these processes.
SSIS provides capabilities for ETL operations using a control flow and data flow engine. It allows importing and exporting data, integrating heterogeneous data sources, and supporting BI solutions. Key concepts include packages, control flow, data flow, variables, and event handlers. SSIS can be optimized for scalability through techniques like parallelism, avoiding blocking transformations, and leveraging SQL for aggregations. Performance can be monitored using tools like SQL Server logs, WMI, and MOM. SSIS is interoperable with data sources like Oracle, Excel, and flat files.
Microsoft released SQL Azure more than two years ago - that's enough time for testing (I hope!). So, are you ready to move your data to the Cloud? If you’re considering a business (i.e. a production environment) in the Cloud, you need to think about methods for backing up your data, a backup plan for your data and, eventually, restoring with Red Gate Cloud Services (and not only). In this session, you’ll see the differences, functionality, restrictions, and opportunities in SQL Azure and On-Premise SQL Server 2008/2008 R2/2012. We’ll consider topics such as how to be prepared for backup and restore, and which parts of a cloud environment are most important: keys, triggers, indexes, prices, security, service level agreements, etc.
SQL Server 2017 - Mejoras Impulsadas por la ComunidadJavier Villegas
Con SQL Sever 2017 Microsoft incorpora nuevas funcionalidades propuestas por la comunidad de profesionales de Data Platform, hablaremos acerca de las mismas así como también sobre las funcionalidades más destacadas de la próxima versión de SQL
Automatic upgrade and new error logging in my sql 8.0 octStåle Deraas
Upgrading to from MySQL 5.7 to MySQL 8.0, and discussion on improvements that are done to the MySQL upgrade process. Discussion on the new implementation of error logging in MySQL 8.0
This document provides an introduction to Azure SQL Database. It describes Azure SQL Database as a fully managed relational database service. It notes that Azure SQL Database differs from SQL Server in some ways, such as not supporting certain T-SQL constructs and commands. The document also discusses server provisioning, database deployment, monitoring, and new service tiers for Azure SQL Database that offer different levels of scalability, performance, and business continuity features.
Multitenancy con SQL Server e Azure SQL DatabaseGianluca Hotz
In questa sessione faremo il punto sulle soluzioni disponibili per venire incontro agli scenari di implementazione multitenancy con SQL Server e Azure SQL Database.
Microsoft SQL Server PaaS (Platform as a Service)Gianluca Hotz
SQL Server in modalità gestita (Platform as a Service) offre un servizio di database relazionale con una copertura funzionale praticamente identica a quella del Database Engine di un SQL Server tradizionale con una riduzione dei costi di implementazione e manutenzione e, in alcuni casi, il beneficio di nuove funzionalità tipicamente introdotte in anticipo rispetto alle versioni On-Premises. In questa sessione faremo una panoramica sullo stato dell'arte delle offerte Micorsoft, con Azure SQL Database, in tutte le sue declinazioni, con particolare attenzione alle Managed Instance, e di quella Amazon con RDS for SQL Server.
SQL Server 2022 Programmability & PerformanceGianluca Hotz
SQL Server 2022 has introduced many new features across all areas of the product. In this session, we will focus on the news regarding programmability and performance improvements.
SQL Server 2022 includes several new features to improve performance, security, and availability. Some key enhancements include buffer pool parallel scans to improve scalability on large memory machines, "hands-free" temporary database improvements to reduce latch contention, and query store hints to shape query plans without code changes. The document provides links to documentation on using new features like resumable add table constraints, XML compression, and hybrid buffer pool direct writes.
Data Integrity with SQL Database LedgerGianluca Hotz
Gianluca Hotz presented on SQL Database Ledger, which enables data integrity in SQL databases through cryptography. SQL Database Ledger captures database state changes in blockchain-inspired data structures called ledgers to provide a tamper-evident record. It makes SQL database data tamper-evident through cryptography while providing the same SQL users already know across Azure and on-premises. SQL Database Ledger offers both append-only and updatable ledger tables to suit different use cases.
Best Practices for Running Microsoft SQL Server on AWSGianluca Hotz
The document discusses best practices for running Microsoft SQL Server on AWS. It provides an overview of options for deploying SQL Server on AWS, including using Amazon RDS or Amazon EC2. When using RDS, AWS manages the SQL Server instance and provides features like automated backups and read replicas. When using EC2, the user has more control but must manage SQL Server, backups, and high availability. The document discusses considerations and techniques for optimizing SQL Server performance on EC2, including storage options and configuration.
Come utilizzare AWS Database Migration Service per migrare SQL Server ad Amaz...Gianluca Hotz
In questo appuntamento affronteremo l'argomento migrazione SQL Server su cloud e come AWS Database Migration Service (DMS) può aiutarci. Per scoprire diversi modi per migrare un database SQL Server su AWS cloud. Per imparare come usare DMS per migrare un database SQL Server su AWS cloud- Per scoprire i vantaggi dell'utilizzo di DMS.
Sempre più aziende guardano con interesse a registri distribuiti e tecnologie Blockchain per non dover dipendere da entità centralizzate di controllo e verifica. Tuttavia, il settore è ancora in continua evoluzione e l’adozione di questo tipo di approccio potrebbe essere eccessivo per molti sistemi che sono ancora per loro natura centralizzati, e la cui riscrittura risulterebbe in investimenti importanti. In questa sessione vedremo una tecnologia antimanomissione dei dati introdotta recentemente in Azure SQL Database.
Youtube: https://www.youtube.com/watch?v=jh3CJ1ns0JQ
Il Query Processor è uno dei componenti più sofisticati di un RDBMS, quello di SQL Server non fa eccezione e sono state introdotte molte novità per risolvere le Query in modo più efficiente. In questa sessione affronteremo l'argomento ripercorrendo le varie funzionalità a partire dal nuovo modello del "Cardinality Estimator", introdotto nella versione 2014, per arrivare a tutto ciò che ricade sotto il nome di "Intelligent Query Processor" tra cui le funzionalità di "Adaptive Query Processing", introdotte nella versione 2017, e le novità introdotte nella versione 2019. Il tutto senza dimenticare le funzionalità per aiutare a gestire eventuali problematiche di regressione e coadiuvato da dimostrazioni pratiche.
Youtube: https://www.youtube.com/watch?v=3hpPpK-qUM0
In questa sessione vedremo una panoramica delle soluzioni SQL Server IaaS e PaaS disponibili in AWS e come affrontare al meglio una migrazione verso tali ambienti.
SQL Server Failover Cluster Instances con Azure Managed DisksGianluca Hotz
Youtube: https://www.youtube.com/watch?v=dgyXkN3FVb4
Come implementare un cluster di SQL Server AlwaysOn Failover Cluster Instances (FCI) con Azure Managed Disks.
SQL Server Failover Cluster Instances con Amazon FSx in AWSGianluca Hotz
Implementare un cluster di SQL Server in modalità AlwaysOn Failover Cluster Instances (FCI) con Amazon Web Services (AWS). In particolare, utilizzando il servizio Amazon EC2 per l’esecuzione delle istanze SQL Server, e il servizio Amazon FSx for Windows File Server per gestire lo storage condiviso, ed implementare una architettura distribuita multi-AZ.
SQL Server Data Virtualization with polybaseGianluca Hotz
Demos: https://github.com/ghotz/Presentations/tree/master/SQL%20Start%202020/Demos
Event site: https://www.sqlstart.it/2020
Vimeo: https://vimeo.com/ugiss/polybase
Polybase è la tecnologia introdotta con SQL Server 2016 per eseguire query distribuite in ambienti eterogenei. Inizialmente con un supporto di sistemi esterni limitato, in SQL Server 2019 è stato esteso per supportare non solo Hadoop e SQL Server in tutte le declinazioni, tra cui Azure SQL Data Warehouse, ma anche Teradata, Oracle, MongoDB e sistemi generici accessibili via ODBC. In questa sessione vedremo come funziona il meccanismo di integrazione dal punto di vista tecnico con qualche esempio pratico.
Solution Manual for First Course in Abstract Algebra A, 8th Edition by John B...rightmanforbloodline
Solution Manual for First Course in Abstract Algebra A, 8th Edition by John B. Fraleigh, Verified Chapters 1 - 56,.pdf
Solution Manual for First Course in Abstract Algebra A, 8th Edition by John B. Fraleigh, Verified Chapters 1 - 56,.pdf
Combined supervised and unsupervised neural networks for pulse shape discrimi...Samuel Jackson
Our methodology for pulse shape discrimination is split into two steps. Firstly, we learn a model to discriminate between pulses using "clean" low-rate examples by removing pile-up & saturated events. In addition to traditional tail sum discrimination, we investigate three different choices for discrimination between γ-pulses, fast, thermal neutrons. We consider clustering the pulses directly using Gaussian Mixture Modelling (GMM), using variational autoencoders to learn a representation of the pulses and then clustering the learned representation (VAE+GMM) and using density ratio estimation to discriminate between a mixed (γ + neutron) and pure (γ only) sources using a multi-layer perceptron (MLP) as a supervised learning problem.
Secondly, we aim to classify and recover pile-up events in the < 150 ns regime by training a single unified multi-label MLP. To frame the problem as a multi-label supervised learning method, we first simulate pile-up events with known components. Then, using the simulated data and combining it with single event data, we train a final multi-label MLP to output a binary code indicating both how many and which type of events are present within an event window.
How AI is Revolutionizing Data Collection.pdfPromptCloud
Artificial Intelligence (AI) is transforming the landscape of data collection, making it more efficient, accurate, and insightful than ever before. With AI, businesses can automate the extraction of vast amounts of data from diverse sources, analyze patterns in real-time, and gain deeper insights with minimal human intervention. This revolution in data collection enables companies to make faster, data-driven decisions, enhance their competitive edge, and unlock new opportunities for growth.
AI-powered tools can handle complex and dynamic web content, adapt to changes in website structures, and even understand the context of data through natural language processing. This means that data collection is not only faster but also more precise, reducing the time and effort required for manual data extraction. Furthermore, AI can process unstructured data, such as social media posts and customer reviews, providing valuable insights into customer sentiment and market trends.
Embrace the future of data collection with AI and stay ahead of the curve. Learn more about how PromptCloud’s AI-driven web scraping solutions can transform your data strategy. https://www.promptcloud.com/contact/
Data analytics is a powerful tool that can transform business decision-making across industries. Contact District 11 Solutions, which specializes in data analytics, to make informed decisions and achieve your business goals.
Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion dataSamuel Jackson
We present our work to improve data accessibility and performance for data-intensive tasks within the fusion research community. Our primary goal is to develop services that facilitate efficient access for data-intensive applications while ensuring compliance with FAIR principles [1], as well as adoption of interoperable tools, methods and standards.
The major outcome of our work is the successful creation and deployment of a data service for the MAST (Mega Ampere Spherical Tokamak) experiment [2], leading to substantial enhancements in data discoverability, accessibility, and overall data retrieval performance, particularly in scenarios involving large-scale data access. Our work follows the principles of Analysis-Ready, Cloud Optimised (ARCO) data [3] by using cloud optimised data formats for fusion data.
Our system consists of a query-able metadata catalogue, complemented with an object storage system for publicly serving data from the MAST experiment. We will show how our solution integrates with the Pandata stack [4] to enable data analysis and processing at scales that would have previously been intractable, paving the way for data-intensive workflows running routinely with minimal pre-processing on the part of the researcher. By using a cloud-optimised file format such as zarr [5] we can enable interactive data analysis and visualisation while avoiding large data transfers. Our solution integrates with common python data analysis libraries for large, complex scientific data such as xarray [6] for complex data structures and dask [7] for parallel computation and lazily working with larger that memory datasets.
The incorporation of these technologies is vital for advancing simulation, design, and enabling emerging technologies like machine learning and foundation models, all of which rely on efficient access to extensive repositories of high-quality data. Relying on the FAIR guiding principles for data stewardship not only enhances data findability, accessibility, and reusability, but also fosters international cooperation on the interoperability of data and tools, driving fusion research into new realms and ensuring its relevance in an era characterised by advanced technologies in data science.
[1] Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016) https://doi.org/10.1038/sdata.2016.18
[2] M Cox, The Mega Amp Spherical Tokamak, Fusion Engineering and Design, Volume 46, Issues 2–4, 1999, Pages 397-404, ISSN 0920-3796, https://doi.org/10.1016/S0920-3796(99)00031-9
[3] Stern, Charles, et al. "Pangeo forge: crowdsourcing analysis-ready, cloud optimized data production." Frontiers in Climate 3 (2022): 782909.
[4] Bednar, James A., and Martin Durant. "The Pandata Scalable Open-Source Analysis Stack." (2023).
[5] Alistair Miles (2024) ‘zarr-developers/zarr-python: v2.17.1’. Zenodo. doi: 10.5281/zenodo.10790679
[6] Hoyer, S. & Hamman, J., (20
Annex K RBF's The World Game pdf documentSteven McGee
Signals & Telemetry Annex K for RBF's The World Game / Trade Federations / USPTO 13/573,002 Heart Beacon Cycle Time - Space Time Chain meters, metrics, standards. Adaptive Procedural template framework structured data derived from DoD / NATO's system of systems engineering tech framework
Overview of Statistical software such as ODK, surveyCTO,and CSPro
2. Software installation(for computer, and tablet or mobile devices)
3. Create a data entry application
4. Create the data dictionary
5. Create the data entry forms
6. Enter data
7. Add Edits to the Data Entry Application
8. CAPI questions and texts
The Rise of Python in Finance,Automating Trading Strategies: _.pdfRiya Sen
In the dynamic realm of finance, where every second counts, the integration of technology has become indispensable. Aspiring traders and seasoned investors alike are turning to coding as a powerful tool to unlock new avenues of financial success. In this blog, we delve into the world of Python live trading strategies, exploring how coding can be the key to navigating the complexities of the market and securing your path to prosperity.
Introduction to Data Science
1.1 What is Data Science, importance of data science,
1.2 Big data and data Science, the current Scenario,
1.3 Industry Perspective Types of Data: Structured vs. Unstructured Data,
1.4 Quantitative vs. Categorical Data,
1.5 Big Data vs. Little Data, Data science process
1.6 Role of Data Scientist
4. #sqlsat871
Who am I?
Gianluca Hotz | @glhotz | ghotz@ugiss.org
Independent Consultant, Founder and Mentor SolidQ
20+ years on SQL Server (from 4.21 in 1996)
Database modeling and development, sizing and administration, upgrade and migration,
performance tuning
Interests
Relational model, DBMS architecture, Security, High Availability and Disaster Recovery
Community
20 years Microsoft MVP SQL Server (from 1998)
Founder and President UGISS
User Group Italiano SQL Server (PASS Chapter)
6. #sqlsat871
Installation
Most new things in 2016/2017
Separate downloads
IFI and tempdb configuration
Linux
R and Python integration
Integration Services Scale-out
Polybase
8. #sqlsat871
SQL Server on Linux
Replication support
Snapshot, Transactional and Merge
Support for Microsoft Distributed Transaction Coordinator (MSDTC)
Always On Availability Group on Docker containers with Kubernetes
Kubernetes operator deploys StatefulSet including container with mssql-server
container and health monitor
OpenLDAP support for third-party AD providers
Machine Learning Services (In-Database) on Linux
New container registry
10. #sqlsat871
Polybase
Data Virtualization
External Data Source
External Tables
SQL Server 2016+
Azure Blob Storage, Hadoop
SQL Server 2019
SQL Server (all flavors), Oracle, Teradata, MongoDB, CosmosDB
(Mongo API)
ODBC Generic Type (e.g. DB2, SAP Hana, Excel, …)
11. #sqlsat871
External Tables vs Linked Servers
External Tables
Database Scoped
ODBC Drivers
Read-only*
Scale out queries with push-down
Failover with AG
Basic authentication
Distributed Transactions not supported
Linked Servers
Instance Scoped
OLEDB Providers
Read/write
Single threaded queries with push-down
Requires separate config from AG
Basic and integrated authentication
Distributed Transactions supported
13. #sqlsat871
SQL Server 2019
Big Data Clusters (Preview)
Compute pool
SQL Compute
Node
SQL Compute
Node
SQL Compute
Node
…
Compute pool
SQL Compute
Node
IoT data
Directly
read from
HDFS
Persistent storage
…
Storage pool
SQL
Serve
r
Spark
HDFS Data Node
SQL
Serve
r
Spark
HDFS Data Node
SQL
Serve
r
Spark
HDFS Data Node
Kubernetes
pod
Analytics
Custom
apps BI
SQL Server
master instance
Node Node Node Node Node Node Node
SQL
Data mart
SQL Data
Node
SQL Data
Node
Compute pool
SQL Compute
Node
Storage Storage
14. #sqlsat871
Polybase in Big Data Cluster vs Stand-alone
Feature
Big data
cluster
Stand
alone
Create external data source for SQL Server, Oracle,
Teradata, and Mongo DB
X X
Create external data source using a compatible third-
party ODBC Driver
X
Create external data source for HADOOP data source X X
Create external data source for Azure Blob Storage X X
Create external table on a SQL Server data pool X
Create external table on a SQL Server storage pool X
Scale-out query execution X X
15. #sqlsat871
Other Services
Master Data Services
Silverlight controls replaced with HTML
SQL Server Machine Learning Services
Windows Server Failover Cluster support
Partition-Based modeling
18. #sqlsat871
Resumable online index operations
Resume after failure (e.g. out of disk space)
Pause and resume later (e.g. free temporarily resources)
Manage large indexes using less log an shorter transactions
Fit rebuild operations into limited maintenance windows
REBUILD
WITH (ONLINE = ON, RESUMABLE = ON, MAX_DURATION = 30 MINUTES);
SQL Server 2017+ Resumable Index Rebuild
SQL Server 2019+ Resumable Index Create
21. #sqlsat871
DBCC CLONEDATABASE
Instantaneous schema-only copy of a database for
troubleshooting
No data, full-schema, statistics and Query Store
Non-blocking
Read-only by default (can be changed)
Optionally NO_STATISTICS, NO_QUERYSTORE
SQL Server 2012 SP4, 2014 SP2 CU3, 2016 SP1, 2017
New in SQL Server 2019
Columnstore Statistics
22. #sqlsat871
Accelerated Database Recovery
Benefits
Fast and consistent database recovery
Number/size of active transactions don’t impact recovery time
Instantaneous transaction rollback
Active time and number of updates don’t impact rollback time
Aggressive Log Truncation
Even with long running transactions, prevents growing out of control
High level
Versioning all physical database modifications
Only logical operations undone (are limited and can be undone instantly)
Active transactions at crash time are marked as aborted
Any versions generated aborted transactions can be ignored user queries
Currently available in Preview in Azure SQL Database and CTP2.3+!
23. #sqlsat871
ADR Components
Persisted Version Store (PVS)
New version store, stored in the database instead of tempdb
Enable also resource isolation
Logical Revert
Asynchronous process performing row level version based undo
Keeps track of all aborted transactions
Performs rollback using PVS
Releases all locks immediately after transaction abort
sLog
Secondary log stream storing log records for non versioned operations
Low volume and in-memory
Serialized on disk during CHECKPOINT
Enables aggressive transaction log truncation
Cleaner
Asynchronous process that cleans page versions
26. #sqlsat871
Improved diagnostic data for stats blocking
Query Waiting for synchronous update operations
Now sys.dm_exec_requests shows SELECT (STATMAN)
New WAIT_ON_SYNC_STATISTICS_REFRESH wait stat
27. #sqlsat871
Availability Groups Enhancements
More synchronous replicas
SQL Server 2012 4 replicas, 2 synchronous
SQL Server 2014 8 replicas, 2 synchronous
SQL Server 2017 8 replicas, 3 synchronous
SQL Server 2019 8 replicas, 5 synchronous
Secondary-to-primary read/write redirection
READ_WRITE_ROUTING_URL and ApplicationIntent=ReadWrite (default)
Killer feature to replace Listener
Cluster technology not offering listener-like features
Multi-subnet scenarios too complex to setup/maintain (e.g. Pacemaker)
Read scale-out or DR with cluster type NONE
28. #sqlsat871
Storage Class Memory / PMEM
Allows low latency I/O
memory-mapped memcpy-like operations in user mode
SQL Server 2016 SP1
NVDIMM-N for tail of the log caching
SQL Server 2019
PMEM devices Linux
support for data, log In-Memory OLTP checkpoint files placement
Hybrid Buffer Pool
Clean pages direct referenced on PMEM devices without copy
Dirty pages still kept in DRAM
29. #sqlsat871
Estimating Data Compression savings
sp_estimate_data_compression_savings
Returns specified object’s current size and estimates
SQL Server < 2019
ROW and PAGE compression
SQL Server 2019+
Adds COLUMNSTORE and COLUMNSTORE_ARCHIVE
compression
Object type determines Columnstore type
E.g. Heap -> Clustered, Clustered index -> Clustered
30. #sqlsat871
Query Profiling Infrastructure
Run-time information on query execution plans
E.g. live query statistics
Lightweight profiling
introduced in SQL Server 2014 SP2, 2016 SP1+
2% expected CPU overhead vs. 75%
SQL Server 2019+
Enabled by default
XEvents query_post_execution_plan_profile (vs query_post_execution_showplan)
DMF sys.dm_exec_query_plan_stats() last known *actual* execution plan
31. #sqlsat871
Internal database pages information
Undocumented DBCC PAGE
New in SQL Server 2019
sys.dm_db_page_info(DatabaseId, FileId, PageId, Mode)
page_resource column in sys.dm_exec_requests and sys.sysprocesses
sys.fn_PageResCracker(page_resource) to get db_id, file_id, page_id
34. #sqlsat871
Truncation error messages
Error message 8152 too generic
String or binary data would be truncated
SQL Server 2019 introduces message 2628
String or binary data would be truncated in table '%.*ls',
column '%.*ls'. Truncated value: '%.*ls’
Enabled with Trace Flag 460
Opt-in required to avoid breaking parsing applications
35. #sqlsat871
Extensibility Framework
Secure external script execution
Scale/optimization opportunities
SQL Server integration (e.g. store procedures, PREDICT)
Language Support
SQL Server 2016+ Support for R
SQL Server 2017+ Support for Python
SQL Server 2019+ Support for Java
36. #sqlsat871
Extensibility Framework Components
Launchpad Service
One per SQL Server instance (with Machine Learning Services)
Provides security isolation
BxlServer
Manage communications between SQL Server and external processes
Binary Exchange Language data format
SQLSatellite
Extensibility API used by BxlServer
I/O data/arguments , error handling
37. #sqlsat871
Java Language extension
Leverages Extensibility Framework
Through sp_execute_external_script
Current support
On Windows version 1.10 (JRE 10, JDK 10)
On Linux version 1.8 (JRE 8, JDK 8)
38. #sqlsat871
UTF-8 Support
Full support for import/export, collations, replication, …
Still not for Linked Servers, In-Memory OLTP, External Table
(Polybase)
CHAR and VARCHAR support (Windows collations only)
UTF8 in collation names
E.g. LATIN1_GENERAL_100_CI_AS_SC_UTF8
Can provide storage savings
E.g. up to 50% from NCHAR(10) to CHAR(10) with UTF8 vs UTF16
39. #sqlsat871
SQL Graph enhancements
Derived tables and view support in MATCH queries
Set of nodes/edges using UNION ALL
Useful for heterogeneous entities or connections between them
MATCH support in MERGE
Edge Constraints
CONNECTION constraint
42. #sqlsat871
Certificate Management
Extended in SQL Server Configuration manager
View and validate certificates installed
View certificates close to expiration
Deploy Certificates across machine in Availability Groups
Deploy Certificates across machine in Failover Cluster Instances
43. #sqlsat871
SQL Data Discovery and Classification
SQL Server management Studio Tool (V17.5)
Discovery & Recommendations, Labeling, Reporting
Metadata con be persisted and queried
Based on Extended Properties
sys_information_type_name, sys_sensitivity_label_name
Support for SQL Server 2008+ and Azure SQL Database
44. #sqlsat871
SQL Server Sensitivity Classification
SQL Server 2019+
(already available in Azure SQL Database)
T-SQL command ADD|DROP SENSITIVITY CLASSIFICATION
applies to tables, columns
LABEL, LAtBEL_ID, INFORMATION_TYPE, INFORMATION_TYPE_ID
Metadata stored in sys.sensitivity_classifications
SQL Server Audit add column data_sensitivity_information
46. #sqlsat871
Always Encrypted with Secure Enclaves
Basic architecture as SQL Server 2016+ implementation
Now allows server-side computation on encrypted columns
In-Place Encryption (ALTER TABLE for initial encryption)
Rich computations (e.g. range comparisons, LIKE predicates, …)
Inside secure enclaves
Virtualization-based Security (VBS) secure memory enclaves
also known as Virtual Secure Mode(VSM) enclaves
Operation on plaintexts cannot be disclosed outside enclave
Column Master Keys sent over secure channel by client driver
Still some limitations (no indexing)
Performance optimizations pending…
47. #sqlsat871
Data Masking
Dynamic Data Masking
On the original database
Original data intact
On-the-fly at query time
Based on user permissions
Static Data Masking
On a copy of the database
Original data not retrievable
At storage level
Masked for everyone
48. #sqlsat871
Static Data Masking
Component of SQL Server Management Studio V18 Preview5+
Define per-column masking configuration
NULL, Single-Value, Shuffle, Group Shuffle, String Composite
Can save and load it
It’s basically a backup/restore and modify data according to config
No automation yet
49. #sqlsat871
Static Data Masking Limitations
No temporal and memory-optimzed tables
No computed and identity columns
No geometry and geography types
Azure SQL Database Hyperscale service tier not supported
Statistics not updated
No cleanup in case of error
can leave sensitive data copies (backupset)
Data and log files may contain sensitive data
retrievable with hex editor
52. #sqlsat871
Intelligent Query Processing
Intelligent QP
Adaptive QP
Adaptive
Joins
Batch Mode
Interleaved
Execution
Memory
Grant
Feedback
Row Mode Batch Mode
Approximate
QP
Approximate
Count
Distinct
Table
Variable
Deferred
Compilation
Batch Mode
on Row Store
Scalar UD
Inlining
53. #sqlsat871
Execution Modes
Row Mode
Execution tree iterators consume 1 row at a time
Traditional execution mode for Rowstore
Batch Mode
Execution tree iterators consume a batch of rows at a time
Optimal with large scan operations (e.g. large table aggregates or joins)
SQL Server 2012 introduced to leverage Columnstore Indexes
SQL Server 2016/2017 extended usage scenarios for CI
SQL Server 2019 extended usage scenarios to Rowstore
54. #sqlsat871
Batch Mode on RowStore
Help reducing CPU Consumption
Columnstore still a better choice
for OLAP workload that is I/O bound
can’t always create it (e.g. impact on OLTP, features not supported)
Limitations
In-Memory tables not supported (only heaps & disk-based b-trees)
Not used when fetching/filtering LOB columns
(including sparse columns sets & XML)
55. #sqlsat871
Batch Mode on Rowstore Control
SQL Server < 2019
Some scenarios covered with tricks… (article part1, part2, part3)
SQL Server 2019+
Scenarios supported directly by Query Processor
On by default with database compatibility level 150+
ALTER DATABASE SCOPED CONFIGURATION
SET BATCH_MODE_ON_ROWSTORE = ON|OFF
OPTION (USE HINT ('ALLOW_BATCH_MODE’));
OPTION (USE HINT (‘DISALLOW_BATCH_MODE'));
56. #sqlsat871
Memory Grant
Excessive Grant
Too much memory allocated vs. memory used
Impact: blocking, out-of-memory, reduced concurrency
Poor Grant
Not enough memory allocated resulting in data spill to tempdb
Impact: slow query, excessive disk usage (tempdb)
Grant increase
dynamic grants increase allocation too much
impact: server instability, unpredictable performance
57. #sqlsat871
Memory Grant Feedback
Post-execution evaluation
Updates grant value for cached plan
E.g. more memory if spilled, less if excessive grant
Version support
SQL Server 2017+ Batch Mode
SQL Server 2019+ Row Mode
Plan caching
Not persistent (i.e. not save in Query Store)
OPTION(RECOMPILE) prevents caching and memory grant feedback
58. #sqlsat871
Memory Grant Feedback Control
Batch Mode
On by default with database compatibility level 140+
ALTER DATABASE SCOPED CONFIGURATION
SET BATCH_MODE_MEMORY_GRANT_FEEDBACK = ON|OFF
OPTION (USE HINT('DISABLE_BATCH_MODE_MEMORY_GRANT_FEEDBACK'));
Row Mode
On by default with database compatibility level 150+
ALTER DATABASE SCOPED CONFIGURATION
SET ROW_MODE_MEMORY_GRANT_FEEDBACK = ON|OFF
OPTION (USE HINT ('DISABLE_ROW_MODE_MEMORY_GRANT_FEEDBACK'));
59. #sqlsat871
Troubleshooting Memory Grant Feedback
Parameter sensitive scenarios
Some queries requires different plans with different grants
Memory grant feedback will disable itself when unstable
Extended Events to monitor changes
SQL Server 2017+ memory_grant_feedback_loop_disabled
SQL Server 2019+ memory_grant_updated_by_feedback
SQL Server 2019+ execution plan attributes
IsMemoryGrantFeedbackAdjusted
No: First Execution, Accurate Grant, Feedback disabled
Yes: Adjusting, Stable
LastRequestedMemory
60. #sqlsat871
Batch mode adaptive joins
Scenario
Nested loop algorithm better for small build join inputs
Hash algorithm better for bigger inputs
Adaptive joins defer choice after first input scanned
61. #sqlsat871
Interleaved Execution
Problem with multi-statement table valued functions
(MSTVFs)
SQL Server <= 2012 optimize with cardinality = 1
SQL Server 2014 & 2016 optimize with cardinality = 100
SQL Server >= 2017
Start optimization
Pause and executes MSTVFs if candidate
Resume optimization with correct cardinality
62. #sqlsat871
Table Variable vs Temporary Tables
Area Temporary Tables Table Variables
Manual statistics creation and update Yes No
Indexes Yes Only inline index definitions allowed
Constraints Yes Only PRIMARY KEY, UNIQUE and CHECK
Automatic statistics creation Yes No
Creating and using a temporary object in
a single batch
Compilation of a statement that
references a temporary table that doesn’t
exist is deferred until the first execution of
the statement
A statement that references a table
variable is compiled along with all other
statements before any statement that
populates the Table Variable is executed,
so compilation sees it as 1
63. #sqlsat871
Table Variable Deferred Compilation
Before SQL Server 2019
Statement referencing TV compiled before population
Number of row estimate fixed at 1
Starting with SQL Server 2019
Behaves like Temporary Tables
Statement referencing non existing TV is deferred until first execution
Number of row estimate much better
Control
On by default with database compatibility level 150+
ALTER DATABASE SCOPED CONFIGURATION
SET DEFERRED_COMPILATION_TV = ON|OFF
OPTION (USE HINT ('DISABLE_DEFERRED_COMPILATION_TV'));
64. #sqlsat871
Scalar UDF inlining
T-SQL user defined functions that returns a single data value
Performance problems
Iterative invocation
once per row, context switching especially with query execution
Lack of costing
before, only relational operators were costed, assumption to be cheap…
Interpreted execution
each statement executes in isolation, no cross-statement optimizations
Serial execution
Intra-query parallelism not allowed
65. #sqlsat871
Scalar UDF Automatic inlining
In SQL Server 2019 Scalar UDF automatically transformed into
Scalar Expressions
Scalar Subqueries
Optimize the whole plan (UDFs no longer visible)
Control
On by default with database compatibility level 150+
ALTER DATABASE SCOPED CONFIGURATION
SET TSQL_SCALAR_UDF_INLINING = ON|OFF
OPTION (USE HINT ('DISABLE_TSQL_SCALAR_UDF_INLINING'));
CREATE FUNCTION … WITH INLINE = ON | OFF
66. #sqlsat871
Scalar UDF inlining example
Query without UDF Query with UDF (no inlining) Query with UDF (inlining)
Execution time 1.6 seconds 29 minutes 11 seconds 1.6 seconds
CREATE FUNCTION dbo.discount_price(@price DECIMAL(12,2), @discount DECIMAL(12,2))
RETURNS DECIMAL (12,2) AS BEGIN RETURN @price * (1 - @discount); END
SELECT L_SHIPDATE, O_SHIPPRIORITY
, SUM(dbo.discount_price(L_EXTENDEDPRICE, L_DISCOUNT))
FROM LINEITEM, ORDERS
WHERE O_ORDERKEY = L_ORDERKEY
GROUP BY L_SHIPDATE, O_SHIPPRIORITY
ORDER BY L_SHIPDATE
10GB CCI compressed TPC-H Schema, 2 x CPUs (12 cores), 96GB RAM, SSD storage
67. #sqlsat871
Scalar UDF inlining requirements
Written using the following constructs
DECLARE, SET (var declaration/assignments)
SELECT (single/multiple var assignments)
IF/ELSE (arbitrary nesting levels)
RETURN (single or multiple)
UDF nested/recursive function calls
Relational operations like EXISTS, ISNULL
No invocation of functions that are
time-dependent (GETDATE())
has side effects (NEWSEQUENTIALID())
Uses EXECUTE AS CALLER (default)
No table variables references
No table-valued parameters references
No user-defined types references
Not natively compiled
interop supported
Not a partition function
Not referenced in
GROUP BY clauses
computed columns
check constraints
No signatures added to it
68. #sqlsat871
Scalar UDF inlining troubleshoting
Column is_inlineable in sys.sql_modules
Doesn’t imply it will always be inlined! (e.g. 1000s lines of code)
Execution Plan
If inlined successfully, xml node <UserDefinedFunction> will be missing
Extended Events
tsql_scalar_udf_not_inlineable
69. #sqlsat871
APPROX_COUNT_DISTINCT
Returns approximate number of unique non-null values in groups
HyperLogLog algorithm guarantees
up to 2% error rate within 97% probability
Fast data exploration with low memory footprint
E.g. dashboards, trend analysis, feature selection, etc.
Think 10 billion rows,
1 user using 1,5GB memory vs 100 users using 12MBs
Tradeoff: precision, only scenarios where exact values are not necessary!