This document discusses thread reuse using the RELEASE(DEALLOCATE) bind option in DB2, considerations for lock avoidance, and lessons learned on DB2 locking. It provides primers on thread reuse, the RELEASE bind option, lock avoidance techniques like commit log sequence numbers and possibly uncommitted bits, and the ramifications of lock avoidance for SQL. It recommends using programming techniques to avoid data currency exposures when using lock avoidance, and outlines how to identify packages that can safely be rebound with CURRENTDATA(NO).
Educational seminar lessons learned from customer db2 for z os health check...John Campbell
This presentation presented at the Polish DB2 User Group introduces and discusses the most common issues uncovered by the DB2 for z/OS Development SWAT Team from 360 Degree DB2 for z/OS Continuous Availability Assessment (DB2 360) Studies.
This slide contains all the basic concepts of ISPF. It's giving the simple and easy step to get the knowledge of Interactive system productivity facility. If u like it then give me feedback on email anilbharti85@gmail.com Thanks v much.
A K Bharti
TSA provides automatic monitoring and availability management of resources configured for high availability in a cluster domain. It monitors DB2 HADR resources and DB2 instance resources, and can start, stop, and fail over these resources between nodes when failures occur. The document provides examples of how DB2 HADR and instance resources are defined and monitored by TSA using the IBM.Application resource type.
The document discusses configuring Oracle's network environment. It describes using tools like Enterprise Manager and tnsping to manage listeners, configure net service aliases, and test connectivity. It also covers establishing connections, naming methods, and using shared vs dedicated server processes.
The document discusses DB2's use of storage on the mainframe. It notes that DB2 uses VSAM data sets to store tablespaces, indexes, and other objects. These data sets can be managed by DB2 storage groups or SMS. Storage groups are lists of volumes where data sets are placed. The document recommends letting DB2 manage data sets using storage groups for less administrative work, but with less control, or defining your own data sets for more control but more work. It also provides details on where to find storage-related information in the DB2 catalog.
A First Look at the DB2 10 DSNZPARM ChangesWillie Favero
This document discusses changes to DB2 subsystem parameter module (DSNZPARM) in DB2 10. It provides information on DSNZPARM macros, how parameters can be changed through installation panels or dynamically using -SET SYSPARM command, and differences between hidden, opaque and visible parameters. The document also introduces new documentation for opaque parameters and explains how to display current DSNZPARM settings using sample program DSN8ED7.
Running MariaDB in multiple data centersMariaDB plc
The document discusses running MariaDB across multiple data centers. It begins by outlining the need for multi-datacenter database architectures to provide high availability, disaster recovery, and continuous operation. It then describes topology choices for different use cases, including traditional disaster recovery, geo-synchronous distributed architectures, and how technologies like MariaDB Master/Slave and Galera Cluster work. The rest of the document discusses answering key questions when designing a multi-datacenter topology, trade-offs to consider, architecture technologies, and pros and cons of different approaches.
Intro to MongoDB
Get a jumpstart on MongoDB, use cases, and next steps for building your first app with Buzz Moschetti, MongoDB Enterprise Architect.
@BuzzMoschetti
This document discusses Oracle database backup and recovery. It covers the need for backups, different types of backups including full, incremental, physical and logical. It describes user-managed backups and RMAN-managed backups. For recovery, it discusses restoring from backups and applying redo logs to recover the database to a point in time. Flashback recovery is also mentioned.
Implementation and Use of Generic VTAM Resources with Parallel SYSPLEX FeaturesCA Technologies
The document discusses how to implement Generic VTAM Resources and Parallel Sysplex Features in CA IDMS systems. Generic VTAM Resources allow multiple IDMS systems to share a common connection resource, improving availability. Parallel Sysplex Features enable data sharing across multiple update systems using a shared cache in a coupling facility, improving performance and high availability. Key steps include defining Generic VTAM resources, XCF groups, and shared cache structures, and configuring the IDMS systems to use these resources and participate in the data sharing group.
Archaic database technologies just don't scale under the always on, distributed demands of modern IOT, mobile and web applications. We'll start this Intro to Cassandra by discussing how its approach is different and why so many awesome companies have migrated from the cold clutches of the relational world into the warm embrace of peer to peer architecture. After this high-level opening discussion, we'll briefly unpack the following:
• Cassandra's internal architecture and distribution model
• Cassandra's Data Model
• Reads and Writes
DB2 for z/OS - Starter's guide to memory monitoring and controlFlorence Dubois
DB2 for z/OS makes more and more use of REAL memory to improve performance and reduce cost. But if you don't carefully budget and monitor the use of REAL memory on your system, you could be putting your applications at risk. This presentation will go back to the basics and answer the most common questions about REAL memory management including: how does DB2 uses virtual and REAL memory? how to build a budget based on system settings and buffer pool sizes? how to size the LFAREA? what are the key performance indicators and how do I know I am running 'safely'? what can be done to protect the system?
BlueStore: a new, faster storage backend for CephSage Weil
BlueStore is a new storage backend for Ceph that provides faster performance compared to the existing FileStore backend. BlueStore stores metadata in RocksDB and data directly on block devices, avoiding double writes and improving transaction performance. It supports multiple storage tiers by allowing different components like the RocksDB WAL, database and object data to be placed on SSDs, HDDs or NVRAM as appropriate.
An Oracle database consists of objects like tables, views, and programs owned by user accounts. SQL is used to perform operations on database objects like creating, modifying, viewing, and deleting them. There are two main types of SQL commands: DDL for defining objects and DML for manipulating data. Users have privileges like creating tables or inserting data that are assigned by the database administrator. Database objects must follow naming conventions and can be created and modified using SQL commands in tools like SQL*Plus.
Apache Cassandra is a free, distributed, open source, and highly scalable NoSQL database that is designed to handle large amounts of data across many commodity servers. It provides high availability with no single point of failure, linear scalability, and tunable consistency. Cassandra's architecture allows it to spread data across a cluster of servers and replicate across multiple data centers for fault tolerance. It is used by many large companies for applications that require high performance, scalability, and availability.
This document discusses relational database design and normalization. It introduces normalization as a theory to design relational schemas without redundancy. Well-normalized tables avoid data inconsistencies. The document then discusses advantages of normalization like avoiding data redundancy and update anomalies. It provides examples of different types of update anomalies like modification, deletion, and insertion anomalies. It also discusses different normal forms like BCNF and how determinants relate to normal forms.
Ben Prusinski is presenting on Oracle R12 E-Business Suite performance tuning. He will cover methodology, best practices, and techniques from basic to advanced. The presentation includes tuning at the infrastructure, application, and database levels with a focus on a holistic approach. Specific areas that will be discussed are concurrent manager tuning including queue size, sleep cycle, cache size, and number of processes.
Deep Dive: a technical insider's view of NetBackup 8.1 and NetBackup AppliancesVeritas Technologies LLC
Together, NetBackup 8.0 and 8.1 are perhaps the two most significant consecutive releases in NetBackup history. Attend this session to learn how the newly released NetBackup 8.1 builds on version 8.0 to deliver the promise of modern data protection and advanced information management like never before. This session will feature a detailed technical overview of the new security architecture in NetBackup 8.1 that keeps data secure across any network, new dedupe to the cloud capabilities that deliver industry-leading performance, instant recovery for Oracle, added support for virtual and next-gen workloads, faster and easier deployments, and many other new features and capabilities.
Top 10 Mistakes When Migrating From Oracle to PostgreSQLJim Mlodgenski
As more and more people are moving to PostgreSQL from Oracle, a pattern of mistakes is emerging. They can be caused by the tools being used or just not understanding how PostgreSQL is different than Oracle. In this talk we will discuss the top mistakes people generally make when moving to PostgreSQL from Oracle and what the correct course of action.
DB2 11 for z/OS Migration Planning and Early Customer ExperiencesJohn Campbell
This extensive presentation provides help and guidance to help DB2 for z/OS customer migrate as quickly as possible, but safely to V11. The material will provide additional planning information, share customer customer experiences and best practices.
Contains information about the DB2 DSNZPARM that forms the DB2 configuration parameters. All about the different types of zPARMs. A way to update them dynamically.
This document discusses various DB2 database objects and utilities. It provides descriptions of storage groups, databases, tablespaces, tables, indexes, views, and the utilities for unload, load, reorganization, running statistics, and copy. It includes examples of creating and using these objects and utilities.
Planning and executing a DB2 11 for z/OS Migration by Ian Cook Surekha Parekh
This document discusses planning and executing a migration from DB2 10 to DB2 11 for z/OS. It begins with an overview of the DB2 11 Early Support Program (ESP) feedback, which was positive regarding performance, quality, and reliability. The presentation then covers key aspects of developing a migration project plan, including assembling a project team, identifying technical considerations, and creating a test plan. It emphasizes early elimination of risks and issues. Sample project frameworks are provided to help structure planning and testing across sandbox, development, and production environments. Attendees are advised to contact software vendors to coordinate DB2 version requirements.
DB2 10 & 11 for z/OS System Performance Monitoring and OptimisationJohn Campbell
This is a "One day Seminar -ODS " . The objectives of this ODS are to focus on key areas
• System address space CPU, EDM pools, data set activity, logging, lock/latch contention, DBM1
virtual and real storage, buffer pools and GBP,…
• Identify the key performance indicators to be monitored
• Provide rules-of-thumb to be applied
• Typically expressed in a range, e.g. < X-Y
• If <x,>Y, need further investigation and tuning - RED
• Boundary condition if in between - AMBER
• Investigate with more detailed tracing and analysis when time available
• Provide tuning advice for common problems
IBM DB2 Analytics Accelerator Trends & Directions by Namik Hrle Surekha Parekh
IBM DB2 Analytics Accelerator has drawn lots of attention from DB2 for z/OS users. In many respects it presents itself as just another DB2 access path (but what a powerful one!) and its deep integration into DB2 as well as application transparency makes it one of the most exciting DB2 enhancements in years. The IBM DB2 Analytics Accelerator complements DB2 by adding industry leading data intensive complex query performance thanks to being powered by the Netezza engine and enhances DB2 to the ultimate database management system that delivers the best of both worlds: transactional as well as analytical workloads. This presentation brings the latest news from the IDAA development and shows the trends and directions in which this technology develops.
Key Note Session IDUG DB2 Seminar, 16th April London - Julian Stuhler .Trito...Surekha Parekh
This document discusses technology themes for DB2 in 2014 and beyond, including cost reduction, high availability, in-memory computing, skills availability, database commoditization, and big data. It outlines current capabilities and future directions for DB2 on both z/OS and LUW platforms, emphasizing ongoing focus on reducing costs while improving availability, performance and analytics capabilities through techniques like in-memory computing and integration with big data technologies. The future of DB2 skills and the changing IT landscape are also addressed.
This presentation shortly describes key features of Apache Cassandra. It was held at the Apache Cassandra Meetup in Vienna in January 2014. You can access the meetup here: http://www.meetup.com/Vienna-Cassandra-Users/
MongoDB is an open-source, document-oriented database that provides high performance and horizontal scalability. It uses a document-model where data is organized in flexible, JSON-like documents rather than rigidly defined rows and tables. Documents can contain multiple types of nested objects and arrays. MongoDB is best suited for applications that need to store large amounts of unstructured or semi-structured data and benefit from horizontal scalability and high performance.
Introduction to memcached, a caching service designed for optimizing performance and scaling in the web stack, seen from perspective of MySQL/PHP users. Given for 2nd year students of professional bachelor in ICT at Kaho St. Lieven, Gent.
This document introduces HBase, an open-source, non-relational, distributed database modeled after Google's BigTable. It describes what HBase is, how it can be used, and when it is applicable. Key points include that HBase stores data in columns and rows accessed by row keys, integrates with Hadoop for MapReduce jobs, and is well-suited for large datasets, fast random access, and write-heavy applications. Common use cases involve log analytics, real-time analytics, and messages-centered systems.
Redis is an in-memory key-value store that is often used as a database, cache, and message broker. It supports various data structures like strings, hashes, lists, sets, and sorted sets. While data is stored in memory for fast access, Redis can also persist data to disk. It is widely used by companies like GitHub, Craigslist, and Engine Yard to power applications with high performance needs.
Final project report on max life insurance 2Student
This document is a project report submitted by Rohit Kumar Pandey for his MBA degree. The project focuses on product promotion of Max Life Insurance. It includes an introduction to life insurance and Max Life Insurance, research methodology, data analysis, findings, and recommendations. The objective is to identify areas to improve Max Life's performance and study consumer perception of the company. The document provides background information on life insurance classification, policies, and the insurance industry in India before discussing the research conducted.
Hpverticacertificationguide 150322232921-conversion-gate01Anvith S. Upadhyaya
Here are the key points about projection segmentation in Vertica:
- Projection segmentation splits large projections into multiple segments and distributes those segments across database nodes for improved parallelism and high availability.
- The segmentation process randomly distributes rows of a projection across all available nodes using a hash function. This random distribution balances the load evenly.
- Segmented projections allow Vertica to parallelize queries by enabling each node to work independently on its portion of the data.
- If a node fails, its segments can be recovered from the duplicate segments stored on other live nodes, ensuring the data remains available.
- Segmentation is determined automatically by Vertica based on projection size and number of nodes. The system monitors segment
Here are the key points about projection segmentation in Vertica:
- Projection segmentation splits large projections into multiple segments and distributes those segments across database nodes for improved parallelism and high availability.
- The segmentation process randomly distributes rows of data across all available nodes using a hash function. This random distribution helps optimize query performance.
- Segmentation allows Vertica to parallelize queries by enabling each node to work independently on its portion of the data.
- It also provides high availability because if a node fails, its data segments are available on other nodes, avoiding data loss.
- During recovery, the replacement node can retrieve missing segments from the live segments on other nodes.
- Administrators can control
IDUG NA 2014 / 11 tips for DB2 11 for z/OSCuneyt Goksu
DB2 11 includes several new features such as global variables, the ability to alter partition keys online without impacting availability, selecting data from directory tables, dropping columns, auto-mapping of tables during reorganization, transparent archiving of data, enhancements to RUNSTATS utilities, and deprecated functionality. Some highlights include global variables that can be shared across SQL statements, altering partition limits online which sets partitions to AREOR status until reorganization, and dropping columns in tables without taking them offline.
The document summarizes new features in Oracle Database 12c from Oracle 11g that would help a DBA currently using 11g. It lists and briefly describes features such as the READ privilege, temporary undo, online data file move, DDL logging, and many others. The objectives are to make the DBA aware of useful 12c features when working with a 12c database and to discuss each feature at a high level within 90 seconds.
Maaz Anjum - IOUG Collaborate 2013 - An Insight into Space Realization on ODA...Maaz Anjum
The document provides an overview of Maaz Anjum, a solutions architect specializing in Oracle products like OEM12c, Golden Gate, and Engineered Systems. It lists his email, blog, and experience using Oracle products since 2001. It also provides details about Bias Corporation, the company he works for, including its founding date, certifications, expertise, customers, and implementations.
The document provides an overview of NewSQL databases. It discusses why NewSQL databases were created, including the need to handle extreme amounts of data and traffic. It describes some key characteristics of NewSQL databases, such as providing scalability like NoSQL databases while also supporting SQL and ACID transactions. Finally, it reviews some examples of NewSQL database products, like VoltDB and Google Spanner, and their architectures.
Christian Johannsen presents on evaluating Apache Cassandra as a cloud database. Cassandra is optimized for cloud infrastructure with features like transparent elasticity, scalability, high availability, easy data distribution and redundancy. It supports multiple data types, is easy to manage, low cost, supports multiple infrastructures and has security features. A demo of DataStax OpsCenter and Apache Spark on Cassandra is shown.
VoltDB is an in-memory SQL database designed for high throughput transactional workloads. It uses a partitioned architecture that scales horizontally across commodity servers. Transactions in VoltDB are executed through stored procedures written in Java against a single or multiple partitions. VoltDB is optimized for high performance through its asynchronous client communication, single-threaded execution model within partitions, and full ACID transaction support through snapshots and K-safety. Migrating applications to VoltDB may require changes to take advantage of its partitioning and programming model centered around stored procedures.
Trace flags are used to temporarily change SQL Server's behavior for debugging or diagnosing issues. This document discusses several trace flags including:
TF 652, 661, 834, 836 which disable certain SQL Server processes or enable large page allocations.
TF 1211, 1224 which avoid lock escalation. TF 1117 forces data files to auto grow equally. TF 1204, 1205, 1222 provide more information on deadlocks.
TF 1118 addresses tempdb contention. TFs 3226, 3014, 3004 provide more backup/restore details. TF 4199 enables query processor fixes. TF 3502 prints checkpoint messages.
The document provides explanations of these trace flags
OracleStore: A Highly Performant RawStore Implementation for Hive MetastoreDataWorks Summit
Today, Yahoo! uses Hive in many different spaces, from ETL pipelines to adhoc user queries. Increasingly, we are investigating the practicality of applying Hive to real-time queries, such as those generated by interactive BI reporting systems. In order for Hive to succeed in this space, it must be performant in all aspects of query execution, from query compilation to job execution. One such component is the interaction with the underlying database at the core of the Metastore.
As an alternative to ObjectStore, we created OracleStore as a proof-of-concept. Freed of the restrictions imposed by DataNucleus, we were able to design a more performant database schema that better met our needs. Then, we implemented OracleStore with specific goals built-in from the start, such as ensuring the deduplication of data.
In this talk we will discuss the details behind OracleStore and the gains that were realized with this alternative implementation. These include a reduction of 97%+ in the storage footprint of multiple tables, as well as query performance that is 13x faster than ObjectStore with DirectSQL and 46x faster than ObjectStore without DirectSQL.
The document provides an overview of 14 topics related to Oracle Autonomous Database. It begins with how to get started with the Autonomous Database free tier and Oracle Machine Learning. It then discusses cross region data guard, exporting data as JSON to object storage, wallet rotation, partitions with external tables in cloud, set patch level when cloning, performance monitoring, data safe audit retention time increase, change concurrency limits via console, SQL monitor report, ASH analytics in performance hub, workload metrics on performance hub, and customer managed keys.
Oracle Database 12c includes over 500 new features. Some key new features include:
- Oracle Database 12c Express (EM Express) which replaces Database Control and has less features than Database Control but does not require Java or an app server.
- New online capabilities like online DDL operations with no DDL locking, online move of partitions with no impact to queries, and online statistics gathering for bulk loads.
- Adaptive SQL Plan Management which allows the optimizer to select a more optimal plan at execution time based on current statistics.
- Multitenant architecture which allows consolidation of multiple databases into one container database with pluggable databases.
The document discusses JDBC best practices for DB2 programmers, including using prepared statements instead of statements to improve performance through dynamic caching, closing JDBC objects in finally blocks to avoid resource leaks, consolidating SQL string formation for readability and performance, and specifying column names for maintainability. It also covers limiting platform-specific features and using only java.sql classes for portability. Future directions discussed include the maturing JDBC specification and increasing use of object-relational mapping tools.
DB2 Express-C is a free version of the DB2 database server from IBM. It has no usage or deployment limits and can run on Windows, Linux, Mac OS X, and Solaris operating systems. Minimum requirements are 256MB of RAM but it is recommended to have at least 1GB. DB2 Express-C provides basic database functionality and sits below the paid DB2 Workgroup and Enterprise editions in terms of features. It uses concurrency controls like locking and transactions to allow for multi-user access to the database.
Cisco at v mworld 2015 vmworld - cisco mds and emc xtrem_io-v2ldangelo0772
The document discusses the synergy between Cisco MDS 9000 switches and EMC XtremIO flash arrays. It highlights that the combination provides industry-leading storage networking and flash array technologies, features, and performance. Specifically, Cisco MDS switches offer high port density, scalability, reliability, and virtualization capabilities. EMC XtremIO arrays deliver flash-optimized storage with high performance, data reduction, snapshots, and simplified management. The integrated solution delivers the benefits of both best-of-breed products to provide superior storage infrastructure for enterprise data centers.
Lessons learned from Isbank - A Story of a DB2 for z/OS InitiativeCuneyt Goksu
Isbank initiated a DB2 for z/OS project in 2007 with two System z9 EC machines running z/OS 1.7 and DB2 V8. They installed DB2 V8 with Turkish codepage support, enabled one-way and two-way data sharing, attended training, and explored DB2 functionality. They developed a test environment with 5 data sharing groups and 4 members each and a production environment with 1 data sharing group and 4 members. They implemented a new core banking Java application using DB2 and explored performance monitoring and tuning techniques.
Best practices for Data warehousing with Amazon Redshift - AWS PS Summit Canb...Amazon Web Services
Get a look under the hood: Understand how to take advantage of Amazon Redshift's columnar technology and parallel processing capabilities to improve your delivery of queries and improve overall database performance. You’ll also hear about how the University of Technology Sydney (UTS) are using Redshift. The University of Technology Sydney will describe how utilizing Amazon Redshift enabled agility in dealing with Data Quality, a capacity to scale when required, and optimizing development processes through rapid provisioning of Data Warehouse environments.
Speaker: Ganesh Raja, Solutions Architect, Amazon Web Services with Susan Gibson, Manager, Data and Business Intelligence, UTS
Level: 300
Kellyn Pot'Vin-Gorman presented on copy data management and virtualization for DBAs. She discussed how virtualization can be used to provision databases more quickly and easily for tasks like patching and testing without needing to copy large amounts of physical data. She also covered how command line interfaces can be used to automate some of these processes.
iland Internet Solutions: Leveraging Cassandra for real-time multi-datacenter...DataStax Academy
iland has built a global data warehouse across multiple data centers, collecting and aggregating data from core cloud services including compute, storage and network as well as chargeback and compliance. iland's warehouse brings actionable intelligence that customers can use to manipulate resources, analyze trends, define alerts and share information.
In this session, we would like to present the lessons learned around Cassandra, both at the development and operations level, but also the technology and architecture we put in action on top of Cassandra such as Redis, syslog-ng, RabbitMQ, Java EE, etc.
Finally, we would like to share insights on how we are currently extending our platform with Spark and Kafka and what our motivations are.
Leveraging Cassandra for real-time multi-datacenter public cloud analyticsJulien Anguenot
iland has built a global data warehouse across multiple data centers, collecting and aggregating data from core cloud services including compute, storage and network as well as chargeback and compliance. iland's warehouse brings actionable intelligence that customers can use to manipulate resources, analyze trends, define alerts and share information.
In this session, we would like to present the lessons learned around Cassandra, both at the development and operations level, but also the technology and architecture we put in action on top of Cassandra such as Redis, syslog-ng, RabbitMQ, Java EE, etc.
Finally, we would like to share insights on how we are currently extending our platform with Spark and Kafka and what our motivations are.
Similar to Using Release(deallocate) and Painful Lessons to be learned on DB2 locking (20)
Getting Started with Interactive Brokers API and Python.pdfRiya Sen
In the fast-paced world of finance, automation is key to staying ahead of the curve. Traders and investors are increasingly turning to programming languages like Python to streamline their strategies and enhance their decision-making processes. In this blog post, we will delve into the integration of Python with Interactive Brokers, one of the leading brokerage platforms, and explore how this dynamic duo can revolutionize your trading experience.
Introduction to Data Science
1.1 What is Data Science, importance of data science,
1.2 Big data and data Science, the current Scenario,
1.3 Industry Perspective Types of Data: Structured vs. Unstructured Data,
1.4 Quantitative vs. Categorical Data,
1.5 Big Data vs. Little Data, Data science process
1.6 Role of Data Scientist
DESIGN AND DEVELOPMENT OF AUTO OXYGEN CONCENTRATOR WITH SOS ALERT FOR HIKING ...JeevanKp7
Long-term oxygen therapy (LTOT) and novel techniques of evaluating treatment efficacy have enhanced the quality of life and decreased healthcare expenses for COPD patients.
The cost of a pulmonary blood gas test is comparable to the cost of two days of oxygen therapy and the cost of a hospital stay is equivalent to the cost of one month of oxygen therapy, long-term oxygen therapy (LTOT) is a cost-effective technique of treating this disease.
A small number of clinical investigations on LTOT have shown that it improves the quality of life of COPD patients by reducing the loss of their respiratory capacity. A study of 8487 Danish patients found that LTOT for 1524 hours per day extended life expectancy from 1.07 to 1.40 years.
The Rise of Python in Finance,Automating Trading Strategies: _.pdfRiya Sen
In the dynamic realm of finance, where every second counts, the integration of technology has become indispensable. Aspiring traders and seasoned investors alike are turning to coding as a powerful tool to unlock new avenues of financial success. In this blog, we delve into the world of Python live trading strategies, exploring how coding can be the key to navigating the complexities of the market and securing your path to prosperity.
Harnessing Wild and Untamed (Publicly Available) Data for the Cost efficient ...weiwchu
We recently discovered that models trained with large-scale speech datasets sourced from the web could achieve superior accuracy and potentially lower cost than traditionally human-labeled or simulated speech datasets. We developed a customizable AI-driven data labeling system. It infers word-level transcriptions with confidence scores, enabling supervised ASR training. It also robustly generates phone-level timestamps even in the presence of transcription or recognition errors, facilitating the training of TTS models. Moreover, It automatically assigns labels such as scenario, accent, language, and topic tags to the data, enabling the selection of task-specific data for training a model tailored to that particular task. We assessed the effectiveness of the datasets by fine-tuning open-source large speech models such as Whisper and SeamlessM4T and analyzing the resulting metrics. In addition to openly-available data, our data handling system can also be tailored to provide reliable labels for proprietary data from certain vertical domains. This customization enables supervised training of domain-specific models without the need for human labelers, eliminating data breach risks and significantly reducing data labeling cost.
Annex K RBF's The World Game pdf documentSteven McGee
Signals & Telemetry Annex K for RBF's The World Game / Trade Federations / USPTO 13/573,002 Heart Beacon Cycle Time - Space Time Chain meters, metrics, standards. Adaptive Procedural template framework structured data derived from DoD / NATO's system of systems engineering tech framework
Towards an Analysis-Ready, Cloud-Optimised service for FAIR fusion dataSamuel Jackson
We present our work to improve data accessibility and performance for data-intensive tasks within the fusion research community. Our primary goal is to develop services that facilitate efficient access for data-intensive applications while ensuring compliance with FAIR principles [1], as well as adoption of interoperable tools, methods and standards.
The major outcome of our work is the successful creation and deployment of a data service for the MAST (Mega Ampere Spherical Tokamak) experiment [2], leading to substantial enhancements in data discoverability, accessibility, and overall data retrieval performance, particularly in scenarios involving large-scale data access. Our work follows the principles of Analysis-Ready, Cloud Optimised (ARCO) data [3] by using cloud optimised data formats for fusion data.
Our system consists of a query-able metadata catalogue, complemented with an object storage system for publicly serving data from the MAST experiment. We will show how our solution integrates with the Pandata stack [4] to enable data analysis and processing at scales that would have previously been intractable, paving the way for data-intensive workflows running routinely with minimal pre-processing on the part of the researcher. By using a cloud-optimised file format such as zarr [5] we can enable interactive data analysis and visualisation while avoiding large data transfers. Our solution integrates with common python data analysis libraries for large, complex scientific data such as xarray [6] for complex data structures and dask [7] for parallel computation and lazily working with larger that memory datasets.
The incorporation of these technologies is vital for advancing simulation, design, and enabling emerging technologies like machine learning and foundation models, all of which rely on efficient access to extensive repositories of high-quality data. Relying on the FAIR guiding principles for data stewardship not only enhances data findability, accessibility, and reusability, but also fosters international cooperation on the interoperability of data and tools, driving fusion research into new realms and ensuring its relevance in an era characterised by advanced technologies in data science.
[1] Wilkinson, M., Dumontier, M., Aalbersberg, I. et al. The FAIR Guiding Principles for scientific data management and stewardship. Sci Data 3, 160018 (2016) https://doi.org/10.1038/sdata.2016.18
[2] M Cox, The Mega Amp Spherical Tokamak, Fusion Engineering and Design, Volume 46, Issues 2–4, 1999, Pages 397-404, ISSN 0920-3796, https://doi.org/10.1016/S0920-3796(99)00031-9
[3] Stern, Charles, et al. "Pangeo forge: crowdsourcing analysis-ready, cloud optimized data production." Frontiers in Climate 3 (2022): 782909.
[4] Bednar, James A., and Martin Durant. "The Pandata Scalable Open-Source Analysis Stack." (2023).
[5] Alistair Miles (2024) ‘zarr-developers/zarr-python: v2.17.1’. Zenodo. doi: 10.5281/zenodo.10790679
[6] Hoyer, S. & Hamman, J., (20
2. #IDUG
2
Agenda
• Thread reuse using RELEASE(DEALLOCATE)
• Primer on thread reuse with RELEASE bind option
• Considerations and limitations
• Painful Lessons to be Learned on DB2 Locking
• Primer on lock avoidance
• CURRENTDATA(YES) versus CURRENTDATA(NO)
• Ramifications of lock avoidance for SQL
• What should an application programmer do
• Finding packages to safely rebind with CURRENTDATA(NO)
3. #IDUG
3
“Resources”: Static SQL
• Static statements
• Packages and statements
• Parent locks
• Index look-aside buffer, dynamic prefetch tracking via sequential detection
4. #IDUG
4
RELEASE - BIND and REBIND Option
• Determines when to release the resources that a program uses
• RELEASE(COMMIT) - Releases resources at commit point
• RELEASE(DEALLOCATE) - Releases resources when thread terminates (child
page/row locks still released at commit)
• RELEASE(INHERITFROMPLAN) – Support added by APAR PM07087
• Default behavior
• BIND PLAN - COMMIT
• BIND PACKAGE - plan value
• REBIND PLAN/PACKAGE existing value
• DB2Binder Utility for JDBC and SQLJ
• DEALLOCATE is default in DB2 10
• COMMIT is default in DB2 9 and earlier releases
5. #IDUG
Package Allocation
Program A :
“Insert into table 1”
Thread allocation
Sign on, authorization checking
Locate SKCT and SKPT
Allocate CT/PT and STMT1
Start Insert STMT1
Lock table space 1
Lock data page
End STMT1
“insert into table 2”
Allocate STMT2
Start Insert STMT2
Lock table space 2
Lock data page
End STMT2
COMMIT ;
Program A
Stmt 1: Insert into table1
Stmt 2: Insert into table2
SPT01
EDM_SKELTON_POOL
SKCT/SKPT
V10 Thread storage above the bar
V9 EDM thread pool below the bar
CT
PT
SQL Stmt1
SQL Stmt2
I/O
6. #IDUG
6
Package Allocation and Commit
Program A
Stmt 1: Insert into table1
Stmt 2: Insert into table2
Commit ;
SPT01
EDM_SKELTON_POOL
SKCT/SKPT
V10 Thread storage above the bar
V9 EDM thread pool below the bar
CT
PT
SQL Stmt1
SQL Stmt2
I/O
Expensive Operation!
Lock TS 1
Lock table1 data page
Lock TS 2
Lock table2 data page
RELEASE PT, statements
Unlock data pages
Unlock TS1, TS2
COMMIT
7. #IDUG
7
Thread Reuse and RELEASE(DEALLOCATE)
• Thread reuse eliminates CPU cost of DB2 thread allocation and deallocation
• CICS
• Protected ENTRYs
• Organic reuse of ENTRYs (or POOL)
• IMS/TM
• Fast Path (IFP) regions
• Wait-For-Input (WFI) regions
• Pseudo Wait-For-Input (PWFI) regions
• DDF
• High Performance DBATs
• WebSphere Type 2 local connections
• Batch with intermediate commits
• Use of RELEASE(DEALLOCATE) coupled with effective thread reuse i.e., thread
persistence
• Further reduces the CPU cost with potential for significant savings (up to 10% plus)
• Resources are held until thread deallocation
• Without thread reuse RELEASE(COMMIT) vs. RELEASE(DEALLOCATE) is a moot point for
discussion
8. #IDUG
8
Limitations and Considerations
• Virtual and real storage
• DB2 9 and earlier - DBM1 virtual storage below the 2G bar and real storage
• Package information is stored in EDM pool below the bar
• DB2 10 after REBIND - real storage usage
• Package information is stored in thread storage above the bar in DB2 9 and earlier
releases
• Accumulated DB2 object control blocks
• Virtual, real, potentially CPU cost for scanning the objects built up under the thread
• Recommendations
• Design for thread reuse for
• High volume simple transactions
• Complex transactions at a reasonably high rate
• Selectively use RELEASE(DEALLOCATE) on high use packages – use % of Total Acctg Class
7 CPU
• DBM1 31-bit virtual storage constraint (DB2 9)
• Real storage constraint (DB2 10)
• Use CICS or WebSphere parameter to periodically clean up and rejuvenate threads
(thread deallocation)
9. #IDUG
9
Considerations for Clean Up
• REUSELIMIT (0-10,000) in CICS TS V4R2 - default 1000
• Number of times a thread can be reused before it is terminated
• Use default and monitor DB2 storage usage and adjust the number if needed
• WebSphere Type 2 connection Aged Timeout - default 0
• The interval in seconds before a physical connection is discarded
• Consider setting WAS “aged timeout” to less than 5 min, recommend using 120
secs to reduce exposure of long living threads
• DB2 10 High Performance DBATs (threads)
• Thread will go inactive every 200 commits
• No user control for this value
• DB2 11 optimizes RELEASE(DEALLOCATE) execution so that it is consistently
better performing than RELEASE(COMMIT)
10. #IDUG
10
Considerations - Concurrency
• More persistent threads with RELEASE(DEALLOCATE) is also trade off with
concurrency
• BIND/REBIND
• SQL DDL
• Online REORG which invalidates packages
• For RELEASE(DEALLOCATE) some locks are held beyond commit until thread
termination
• Mass delete locks (SQL DELETE without WHERE clause)
• Gross level lock acquired on behalf of a SQL LOCK TABLE
• Table space defined with LOCKSIZE TABLESPACE|TABLE
• Note: no longer a problem for gross level lock acquired by lock escalation
• DO YOUR HOMEWORK BEFORE USING PERSISTENT THREADS WITH
RELEASE(DEALLOCATE) BIND OPTION
11. #IDUG
11
Primer on lock avoidance
• Combination of techniques used by DB2 to try to avoid taking a S page/row
locks when processing for read only SQL whilst preventing the retrieval of
uncommitted data by the application
• (1) Page latching (and page p-lock in data sharing) controlled by DB2 to ensure
physical consistency of the page
• (2) Commit log sequence number (CLSN) – at the page level
• DB2 tracks "time" of last update to page (on page) (A)
• DB2 tracks "time" of oldest uncommitted activity on every pageset/partition (B)
• Non Data Sharing
• CLSN = lowest uncommitted RBA for all active transactions for a given pageset
• Data Sharing
• For non-GBP-dependent page sets, each member uses a local CLSN = lowest uncommitted LRSN for all active
transactions for a given pageset
• For GBP-dependent page sets, a Global CLSN value is maintained for the entire data sharing group = lowest CLSN value
across all members across all page sets (GBP-dependent or not)
• If (A) < (B) everything on the page is guaranteed to be committed
• Else, check Possibly UNCommitted bits (PUNC bits)
12. #IDUG
12
Primer on lock avoidance …
• Combination of techniques to prevent retrieval of uncommitted data …
• (3) Possibly UNCommitted bits (PUNC bits) – at the row level
• On each row, a PUNC bit is set when the row is updated
• PUNC bits are periodically reset
• If successful CLSN check and more than 25% of the rows have the PUNC bit ON
• RR scanner
• REORG TABLESPACE
• If the PUNC bit is not ON, the row/key is guaranteed to be committed
13. #IDUG
13
CURRENTDATA(YES) versus CURRENTDATA(NO) …
• Plans and packages have a better chance for lock avoidance if they are bound
with ISOLATION(CS) and CURRENTDATA(NO)
• What is CURRENTDATA?
• Helps to determine the currency and stability of data returned to an application
cursor
• Only applies to applications bound with ISOLATION(CS)
• What is isolation CURSOR STABILITY(CS)?
• Data fetched by a cursor is committed, but if the application process returns to the
same page, another application might have since updated, deleted, or inserted
qualifying rows
• If the cursor is defined as FOR UPDATE OF
• Data returned by the cursor is stable and it may not be updated by another
transaction while the updatable cursor is positioned on it
• If the cursor is defined as FOR READ|FETCH ONLY, or it is implicitly read only (or it is
ambiguous)
• ISOLATION(CS) ensures that the data returned is committed and the stability of
the cursor is determined by the CURRENTDATA option
14. #IDUG
14
Primer on lock avoidance …
• Benefits of lock avoidance
• Improved concurrency through less lock collisions
• Decrease in lock and unlock activity requests, with an associated decrease in CPU
resource consumption and data sharing overhead
• V8 improvements
• Lock avoidance for non-cursor ‘singleton’ SELECT
• In V7, ISOLATION(CS) CURRENTDATA(YES) acquires S page/row lock on the
qualified row
• In V8, DB2 no longer acquires and hold S page/row lock on the qualified row for
ISOLATION(CS) CURRENTDATA(YES or NO)
• Overflow lock avoidance when the update of a variable length row in a data page
results in a new row that cannot fit in that page i.e., indirect reference
• In V7, no lock avoidance on both pointer and overflow
• In V8, lock on pointer only
• Need to distinguish carefully between eligibility for lock avoidance and
actually achieving it
15. #IDUG
15
Primer on lock avoidance …
• BIND option ISOLATION(CS) with CURRENTDATA(NO) could
reduce # Lock/Unlock requests dramatically
• High Unlock requests/commit could also be possible from
• Large number of relocated rows after update of compressed or VL
row
• Lock/Unlock of pointer record (or page)
• Large number of pseudo-deleted entries in unique indexes
• Lock/Unlock of data (page or row) in insert to unique index when
pseudo-deleted entries exist
• Both can be eliminated by REORG
Field Name Description
QTXALOCK LOCK REQUESTS
QTXAUNLK UNLOCK REQUESTS
LOCKING ACTIVITY QUANTITY /SECOND /THREAD /COMMIT
------------------------ -------- ------- ------- -------
...
LOCK REQUESTS 521.0M 24.2K 3134.34 1050.75
UNLOCK REQUESTS 478.1M 22.2K 2876.06 964.16
Lock avoidance may not be working effectively if
Unlock requests/commit is high, e.g. >5
ROT
16. #IDUG
16
Primer on lock avoidance …
• Effective lock avoidance is very important in data sharing
• Global locks propagated beyond IRLM to Coupling Facility are relatively expensive
• Global management of lock contention/resolution is very expensive
• Effective Lock avoidance is critical to achieving good performance and lower data
sharing overhead
• Any long-running UR(s) can reduce the effectiveness of lock avoidance by stopping
the Global CLSN value from moving forward
• Recommendation: Aggressively monitor long-running URs
• 'First cut' ROTs:
• URs running for a long time without committing: zparm URCHKTH<=5
• Message DSNR035I
• URs performing massive update activity without committing: zparm
URLGWTH=10(K)
• Message DSNJ031I
• Need management ownership and process for getting rogue applications fixed up
so that they commit frequently based on
• Elapsed time and/or
• CPU time (no. of SQL update statements)
17. #IDUG
17
CURRENTDATA(YES) versus CURRENTDATA(NO) …
• Switching from CURRENTDATA(YES) to CURRENTDATA(NO)
• In general, can be done without any adverse effect to applications
• But consider the following scenario:
1. Program A bound with ISOLATION(CS) CURRENTDATA(YES) fetches a row from
a cursor with ORDER BY – and the row is read from the base table and not from
a work file
2. Program B tries to update the row just read by Program A
• The access is not allowed because the CURRENTDATA(YES) option for Program A caused a lock to
be acquired on the page/row in the base table
3. Program A then issues a searched UPDATE of the row just fetched from step 1.
• In the above example CURRENTDATA(YES) protects the integrity of the data read by
Program A in between the FETCH and the searched UPDATE
• If Program A was to be rebound with CURRENTDATA(NO) then
1. Program B may (should) be able to access the row just fetched by Program A
• This assumes correct timing and that lock avoidance is effective
2. When Program A issues the searched UPDATE, it could wipe out the changes
just made by Program B
18. #IDUG
18
CURRENTDATA(YES) versus CURRENTDATA(NO) …
• Switching from CURRENTDATA(YES) to CURRENTDATA(NO) …
• Now reconsider the same scenario:
1. Program A bound with ISOLATION(CS) CURRENTDATA(YES) fetches a row from
a cursor with ORDER BY, but the row is now read from a work file
2. Program B tries to update the row just read by Program A
1. Access is now allowed as there is no lock on the page/row in the base table
3. Program A then issues a searched UPDATE of the row just fetched from step 1
• In the above example CURRENTDATA(YES) does NOT protects the integrity of the
data read by Program A in between the FETCH and the searched UPDATE
19. #IDUG
19
Ramifications of lock avoidance for SQL
• Data may be returned without a lock
• All data returned will be committed at the time it is taken from page
• But may change by the time the application sees it
• DB2 only attempts lock avoidance for ISOLATION(CS) transactions when BOTH
of the following are true:
• The plan/package is bound CURRENTDATA(NO)
• The cursor (if cursor-based FETCH) is declared FOR READ|FETCH ONLY, or has some
other attribute (e.g., ORDER BY) that makes it a read-only cursor
• Applications that depend on the stability of what is under a read only cursor,
should use CURRENTDATA(YES)
• Default is CURRENTDATA(NO)
• Now lets look at another example
20. #IDUG
20
Ramifications of lock avoidance for SQL …
• Example
DECLARE CURSOR AS ...
SELECT THIRD_COL ... WHERE CLUST_COL = 'ABC'
AND ANOTHER_COL = 'DEF'
ORDER BY CLUST_COL; <--- makes cursor read only
FETCH ... <--- say this returns row ABC.DEF.GHI without taking a lock
UPDATE SET … WHERE CLUST_COL = 'ABC'
AND ANOTHER_COL = 'DEF'
AND THIRD_COL = 'GHI‘
• What is the issue?
• Row may no longer be ABC.DEF.GHI by the time update executes
• Another transaction may have gotten in and updated it to ABC.DEF.XYZ
• So update will receive "row not found"
21. #IDUG
21
What should an application programmer do
• Use programming techniques which avoid possible data currency exposures and access path
dependencies (these sometimes occur regardless of the CURRENTDATA option)
• Define cursors with their intended use
• Easier to read application code
• Number of ambiguous cursors can be reduced
• Some queries will not allow the use of a FOR UPDATE cursor due to possible work file usage e.g.,
with ORDER BY
• Use predicates on searched updates to enforce data currency
• When a FOR UPDATE cursor cannot be used with a query it is common for application
programmers to use a read only with a subsequent searched UPDATE or DELETE
• It is important to use WHERE predicates to ensure that data has not changed since your cursor
first selected the data
• Include all columns which logically determine if the update is necessary, instead of updating
based solely on the key
• This is an exposure regardless of the CURRENTDATA option, but the use of CURRENTDATA(NO)
can increase the exposure opportunity
• Consider the use of a timestamp or a version number
• Add a timestamp or version number column to a table to record the last update
• Select the timestamp or version number column with the cursor
• Use it as a predicate on the searched UPDATE or DELETE to ensure that an application is
updating the same row
22. #IDUG
22
Finding packages to safely rebind with CURRENTDATA(NO)
• Must recognise that access path dependencies will inevitably occur
• CURRENTDATA(YES) will not stop access path dependency and associated
exposures
• Access path independence and CURRENTDATA(NO) should be the goal for all
applications
• Can safely rebind packages that use only true read-only cursors with
CURRENTDATA(NO)
24. #IDUG
John Campbell
DB2 for z/OS Development
campbelj@uk.ibm.com
Session 6007
Using RELEASE(DEALLOCATE) and Painful Lessons to be Learned on DB2 Locking