A review of I_O behavior on Oracle database in ASM
•
0 likes•43 views
In this document I test and investigate i/o processes and try to achieve a deep understanding of I/O behavior in Oracle ASM environment.
Report
Share
![Test and Results:
100M tablespace is created and oracle dbw0 process of database instance is traced with strace
tool.
[root@ tmp]# ps -ef|grep dbw
oracle 10131 1 0 15:53 ? 00:00:00 ora_dbw0_DB2
...
$strace -af -p 10131 -o dbw.txt &
In a new sqlplus session (database instance), tablespace is created.
sqlplus / as sysdba
sql> create tablespace test1 data
fi
le '+DG1' size 100M;
Same test is repeated for each possible value to enable and disable asynchronous I/O with the
parameters and the following results are found:
Con
fi
g 1:
FILESYSTEMIO_OPTIONS=SETALL
DISK_ASYNCH_IO=TRUE
times(NULL) = 532400079
times(NULL) = 532400079
io_submit(5623808, 2, {...}) = 2
gettimeofday({1227275694, 483102}, NULL) = 0
io_getevents(5623808, 1, 1024, {...}{600, 0}) = 1
times(NULL) = 532400079
io_getevents(5623808, 1, 1024, {...}{600, 0}) = 1
times(NULL) = 532400079
When both parameters are activated to use asynchronous I/O, dbw0 process performs the
I/O asynchronously. The result is expected.
Con
fi
g 2:
FILESYSTEMIO_OPTIONS=NONE
DISK_ASYNCH_IO=TRUE
gettimeofday({1227275539, 15789}, NULL) = 0
times(NULL) = 532384532
io_submit(13086720, 2, {...}) = 2
gettimeofday({1227275539, 16181}, NULL) = 0
io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1
times(NULL) = 532384532
io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1
FILESYSTEMIO_OPTIONS is set to NONE.
This should normally disable the DISK_ASYNCH_IO when data
fi
les are located on a regular
fi
lesystem (non-ASM).](https://cdn.statically.io/img/www.slideshare.net/data:image/gif;base64,R0lGODlhAQABAIAAAAAAAP///yH5BAEAAAAALAAAAAABAAEAAAIBRAA7)
More Related Content
Similar to A review of I_O behavior on Oracle database in ASM
Similar to A review of I_O behavior on Oracle database in ASM (20)
More from Alireza Kamrani
More from Alireza Kamrani (20)
Recently uploaded
Recently uploaded (20)
A review of I_O behavior on Oracle database in ASM
- 1. ASM Inherently Performs Asynchronous I/O Regardless of fi lesystemio_options Parameter A complete review of I/O behavior in Oracle database in ASM and OS con fi guration plus recommendations The purpose of this document is to explain the behaviour of I/O calls depending database parameters FILESYSTEMIO_OPTIONS and DISK_ASYNCH_IO when the Oracle RDBMS is located on ASM diskgroups. FILESYSTEMIO_OPTIONS controls the behaviour of I/O options given by operating system but it has no a ff ect when ASM is used. FILESYSTEMIO_OPTIONS enables asynchronous I/O (AIO) when the value is set to SETALL/ ASYNC. 📍 However, ASM bypasses the fi lesystem layer in this case because ASM I/O is entirely controlled by DISK_ASYNCH_IO parameter. AIO needs to be enabled/disabled by setting disk_asynch_io to parameter values TRUE/FALSE. In order to test this behaviour database writer process (DBW0) is traced with strace during a tablespace creation. Use strace commnad in linux : An example strace of dbw0 process when AIO is used shows io_submit/io_getevents calls. gettimeofday({1227275539, 15789}, NULL) = 0 times(NULL) = 532384532 io_submit(13086720, 2, {...}) = 2 gettimeofday({1227275539, 16181}, NULL) = 0 io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1 times(NULL) = 532384532 io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1 and the strace of dbw0 process with AIO disabled shows pwrite64: gettimeofday({1227274991, 462322}, NULL) = 0 times(NULL) = 532329776 pwrite64(22, "362420030@230233v00014VB00t000t0000000"..., 8192, 6153068544) = 8192 times(NULL) = 532329776 times(NULL) = 532329776 pwrite64(24, "362420030@230233v00014VB00t000t0000000"..., 8192, 6199205888) = 8192 times(NULL) = 532329776
- 2. Test and Results: 100M tablespace is created and oracle dbw0 process of database instance is traced with strace tool. [root@ tmp]# ps -ef|grep dbw oracle 10131 1 0 15:53 ? 00:00:00 ora_dbw0_DB2 ... $strace -af -p 10131 -o dbw.txt & In a new sqlplus session (database instance), tablespace is created. sqlplus / as sysdba sql> create tablespace test1 data fi le '+DG1' size 100M; Same test is repeated for each possible value to enable and disable asynchronous I/O with the parameters and the following results are found: Con fi g 1: FILESYSTEMIO_OPTIONS=SETALL DISK_ASYNCH_IO=TRUE times(NULL) = 532400079 times(NULL) = 532400079 io_submit(5623808, 2, {...}) = 2 gettimeofday({1227275694, 483102}, NULL) = 0 io_getevents(5623808, 1, 1024, {...}{600, 0}) = 1 times(NULL) = 532400079 io_getevents(5623808, 1, 1024, {...}{600, 0}) = 1 times(NULL) = 532400079 When both parameters are activated to use asynchronous I/O, dbw0 process performs the I/O asynchronously. The result is expected. Con fi g 2: FILESYSTEMIO_OPTIONS=NONE DISK_ASYNCH_IO=TRUE gettimeofday({1227275539, 15789}, NULL) = 0 times(NULL) = 532384532 io_submit(13086720, 2, {...}) = 2 gettimeofday({1227275539, 16181}, NULL) = 0 io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1 times(NULL) = 532384532 io_getevents(13086720, 1, 1024, {...}{600, 0}) = 1 FILESYSTEMIO_OPTIONS is set to NONE. This should normally disable the DISK_ASYNCH_IO when data fi les are located on a regular fi lesystem (non-ASM).
- 3. However, since data fi les are located on ASM, FILESYSTEMIO_OPTIONS has no a ff ect in this case. I/O is asynchronous. Con fi g 3: FILESYSTEMIO_OPTIONS=NONE DISK_ASYNCH_IO=FALSE gettimeofday({1227274991, 462322}, NULL) = 0 times(NULL) = 532329776 pwrite64(22, "362420030@230233v00014VB00t000t0000000"..., 8192, 6153068544) = 8192 times(NULL) = 532329776 times(NULL) = 532329776 pwrite64(24, "362420030@230233v00014VB00t000t0000000"..., 8192, 6199205888) = 8192 times(NULL) = 532329776 Con fi g 4: FILESYSTEMIO_OPTIONS=SETALL DISK_ASYNCH_IO=FALSE gettimeofday({1227275223, 67935}, NULL) = 0 times(NULL) = 532352937 pwrite64(21, "362420030@125kv00024ZA005000t0000000"..., 8192, 5628780544) = 8192 times(NULL) = 532352937 times(NULL) = 532352937 pwrite64(22, "362420030@125kv00024ZA005000t0000000"..., 8192, 5565865984) = 8192 times(NULL) = 532352937 In the last two cases, whatever the fi lesystemio_options is set, if DISK_ASYNCH_IO is false, IO is done synchronously. ASMLIB behaves similarly but with an exception in strace output. 🚩 When ASMLIB is used, ASMLIB performs AIO depending on how the I/O is passed to it. And It makes no decisions at all. This is entirely up to ASM layer. As the above tests also implies and ASM is entirely controlled by the disk_asynch_io parameter. ⛳ So, you can also control the ASMLIB by changing the value for disk_asynch_io parameter. When using ASMlib, the result show read calls that look like this in the strace output: read(16, "MSA02070P00000003024521516000003603472670000"..., 80) = 80
- 4. So this is asynchronous i/o. DISK_ASYNCH_IO In 23c DISK_ASYNCH_IO controls whether I/O to data fi les, control fi les, and log fi les is asynchronous (that is, whether parallel server processes can overlap I/O requests with CPU processing during table scans). If your platform supports asynchronous I/O to disk, Oracle recommends that you leave this parameter set to its default value. However, if the asynchronous I/O implementation is not stable, you can set this parameter to false to disable asynchronous I/O. If your platform does not support asynchronous I/O to disk, this parameter has no e ff ect. If you set DISK_ASYNCH_IO to false, then you can increase DB_WRITER_PROCESSES or use DBWR_IO_SLAVES to simulate asynchronous I/O In 19c By default, the DISK_ASYNCH_IO initialization parameter in the parameter fi le is set to TRUE. To enable asynchronous Input-Output on fi le system fi les: 1. Ensure that all Oracle Database fi les are located on fi le systems that support asynchronous Input-Output. 2. Set the FILESYSTEMIO_OPTIONS initialization parameter in the parameter fi le to ASYNCH or SETALL. Note:If the fi le system fi les are managed through ODM library interface or Direct NFS Client, asynchronous Input-Output is enabled by default. There is no need to set FILESYSTEMIO_OPTIONS to enable asynchronous Input-Output in these environments. Asynchronous Input-Output Support Note: On Linux, Automatic Storage Management uses asynchronous Input-Output by default. Asynchronous Input-Output is not supported for database fi les stored on Network File Systems. Oracle Database supports kernel asynchronous Input-Output. This feature is disabled by default. By default, the DISK_ASYNCH_IO initialization parameter in the parameter fi le is set to TRUE to enable asynchronous I/O on raw devices. Property Description Parameter type Boolean Default value true Modi fi able No Modi fi able in a PDB No Range of values true | false Basic No
- 5. Enabling Direct Input-Output Support Direct Input-Output support is available and supported on Linux. To enable direct Input-Output support: • Set the FILESYSTEMIO_OPTIONS initialization parameter to DIRECTIO. • Set the FILESYSTEMIO_OPTIONS initialization parameter in the parameter fi le to SETALL, which will enable both asynchronous Input-Output and direct Input-Output. DBWR_IO_SLAVES DBWR_IO_SLAVES speci fi es the number of I/O server processes used by the DBW0 process. DBWR_IO_SLAVES is relevant only on systems with only one database writer process (DBW0). The DBW0 process and its server processes always write to disk. By default, the value is 0 and I/O server processes are not used. If you set DBWR_IO_SLAVES to a nonzero value, the number of I/O server processes used by the ARCH and LGWR processes is set to 4. However, the number of I/O server processes used by Recovery Manager is set to 4 only if asynchronous I/O is disabled (either your platform does not support asynchronous I/O or disk_asynch_io is set to false). Typically, I/O server processes are used to simulate asynchronous I/O on platforms that do not support asynchronous I/O or that implement it ine ffi ciently. However, you can use I/O server processes even when asynchronous I/O is being used. In that case the I/O server processes will use asynchronous I/O. I/O server processes are also useful in database environments with very large I/O throughput, even if asynchronous I/O is enabled. Another method to checking i/o status: Many times there is a requirement to check if Asynchronous I/O is working on Linux Platform, so we can try to use it for our data fi les access inside database. SOLUTION The slabinfo maintains statistics about objects in memory. Some of the structs used by Asynchronous I/O are treated as objects in the virtual memory, so we can look for those structs on slabinfo. The ones related to AIO are named kio*. $ cat /proc/slabinfo | grep kio Since Oracle Linux provides di ff erent kernels (including RedHat Compatiable Kernel, Oracle UEK), the output could be slightly di ff erent. Property Description Parameter type Integer Default value 0 Modi fi able No Modi fi able in a PDB No Range of values 0 to operating system-dependent Basic No
- 6. If Async I/O is enabled: $ cat /proc/slabinfo | grep kio kioctx 270 270 128 9 9 1 : 252 126 kiocb 66080 66080 96 1652 1652 1 : 252 126 kiobuf 236 236 64 4 4 1 : 252 126 or kioctx 518 670 384 10 1 : tunables 54 27 8 : slabdata 67 67 0 kiocb 595 720 256 15 1 : tunables 120 60 8 : slabdata 48 48 60 and if Async I/O is disabled or not in using: $ cat /proc/slabinfo | grep kio kioctx 0 0 128 0 0 1 : 252 126 kiocb 0 0 96 0 0 1 : 252 126 kiobuf 0 0 64 0 0 1 : 252 126 or kioctx 0 0 384 10 1 : tunables 54 27 8 : slabdata 0 0 0 kiocb 0 0 256 15 1 : tunables 120 60 8 : slabdata 0 0 0 In UEK4 the slabinfo entries for aio are no longer available. The caches used by the aio code are merged with other caches: $ ls -l /sys/kernel/slab | grep kio lrwxrwxrwx. 1 root root 0 Dec 15 15:53 aio_kiocb -> :t-0000128 lrwxrwxrwx. 1 root root 0 Dec 15 15:53 kioctx -> :t-0000640 In the SLAB allocator there are three di ff erent caches involved. The kioctx and kiocb are Async I/O data structures that are de fi ned in aio.h header fi le. If it shows a non zero value that means async io is enabled. If you have the source code loaded, you can review it at fi le aio.h. This fi le is located under: /usr/src/linux-<version>/include/linux/aio.h These data structures are using to track the I/O requests, and are allocated as part of the __init_aio_setup() call in aio.c. Example strace of dbw0 process with AIO enabled (init.ora parameter fi lesystemio_options = asynch) shows: ... io_submit(3071864832, 1, {{0xb7302e34, 0, 1, 0, 21}}) = 1 gettimeofday({1176916625, 58882}, NULL) = 0 io_getevents(-1223102464, 1, 1024, {{0xb7302e34, 0xb7302e34, 8192, 0}}, {600, 0}) = 1 ... Example strace of dbw0 process with AIO disabled ( fi lesystemio_options = none): ... pwrite64(21, "624200004213000220B24300016207357001"..., 8192, 36077568) = 8192 times(NULL) = 1775653082
- 7. times(NULL) = 1775653082 pwrite64(21, "624200<213000220B2430001625400020*"..., 8192, 36143104) = 8192 ... Also you can use OSWatcher tools to achieve more details and information. Caveat for ASMLib If Oracle ASMLib (see http://oss.oracle.com/projects/oracleasm ) is deployed, the kiocb structs are not used. ASMLib does not use the POSIX aio_*() functions. You will never see any kioctx or kiocb structures from ASMLib. It is far lower level than that. In fact, ASMLib does AIO or SyncIO depending on how the I/O is passed to it, It makes no decisions at all. This is entirely up to kfk and the layers above it, kfk is entirely controlled by the disk_asynch_io parameter. So, we can check whether ASMLib is doing AIO by SQL*Plus command "show param disk_asynch_io". (You can disable AIO by setting disk_asynch_io=false) With ASMLib, AIO is done via ioctl() calls (2.4 kernel), or read() calls (2.6 kernel) on the ASM device. Whether ASMLib uses aio depends on whether oracle is con fi gured to do aio, In oracle 10g, if ASMLib is in use, the i/o is asynchronous, because oracle 10g enables aio by default. The strace when using ASMlib will show read calls that look like this: ... read(16, "MSA020100P000222377377377@3133735000"..., 80) = 80 ... The fi rst 3 characters, byte-swapped, are ASM, indicating an ASMLib I/O command structure. Note: Recommend "strace -c" which doesn't cause too much overhead. I/O operations in UNIX and Linux systems typically go through the fi le system cache. Although this doesn't represent a problem in itself, this extra processing does require resources. Bypassing the fi le system cache reduces CPU requirements, and frees up the fi le system cache for other non-database fi le operations. Operations against raw devices automatically bypass the fi le system cache. When a synchronous I/O request is submitted to the operating system, the writing process blocks until the write is complete before continuing processing. With asynchronous I/O, processing continues while the I/O request is submitted and processed. This allows asynchronous I/O to bypass some of the performance bottlenecks associated with I/O operations. Oracle can take advantage of direct I/O and asynchronous I/O on supported platforms using the FILESYSTEMIO_OPTIONS parameter, whose possible values are listed below. • ASYNCH - Enabled asynchronous I/O where possible. • DIRECTIO- Enabled direct I/O where possible. • SETALL- Enabled both direct I/O and asynchronous I/O where possible.
- 8. • NONE - Disabled both direct I/O and asynchronous I/O. Most of the performance tuning issues can be related to I/O in any database. Oracle provides only two main parameters to control I/O behaviour these are fi lesystemio_options and disk_asynch_io Oracle recommends to set parameter fi lesystemio_options to value 'setall' but it is not always good practise especially when SGA is small. setting it to setall lets your Oracle DB perform I/O operations without going to fi le system cache and it saves overhead of double caching but if SGA is smaller and DB host machine has large free memory then it is not good to set this parameter to value setall. In this case you should increase DB_CACHE_SIZE and only then set fi lesystemio_options to setall. As example if you setting up Oracle database to handle large I/O of new Dataware house project. And set fi lesystemio_options to setall. This maybe speeded up load operations aproximately by 5% whose caching in fi le system was not useful as these were insert statements and not repeated. At the same time increased SGA to quite high to cache I/O as much as possible so that repetitive select queries can bene fi t from SGA as in this case when fi lesystemio_options was set to setall then fi le system cache was not available. So summary is be prudent when setting fi lesystemio_options to setall to enable direct read/write and asynchronous operations. Review dbwriter_processes a ff ect on i/o: Other parameters to a ff ect write (as well as read) is dbwriter_processes. When asynchronous I/O operations are slower in operating system in comparison to synchronous I/O then turn o ff asynchronous I/O by setting disk_asynch_io to false and set multiple db writer processes by increasing dbwriter_processes values from 1 to 2,3 or 4 suitable value to your system. Alternate is incrase dbwr_io_slaves from 0 to 2,3,4 suitable value. You would be keen to disable asynchronous I/O when you see high average_wait on event db_ fi le_parallel_wait. Other reason for turning it o ff will be synchronous I/O is more reliable. SQL> select event,average_wait from v$system_event where event like 'db fi le parallel write'; EVENT AVERAGE_WAIT ----------------- ------------ db fi le parallel write 28.2 [ centi seconds] This is not a good ASYNCH I/O. Try Synchronous I/O and check. Note 1: Asynchronous I/O operations are more prone to block corruptions than synchronous operations so many DBAs disable it and follow practice as mentioned in above paragraph. So if you do not have standby databases and oracle 11g and newer then which autoamatically recovers corrupted block on primary then you would not want asynchronous I/O. Note 2: From 11g R2 for tuning purpose, the “db fi le async I/O submit” should be treated as “db fi le parallel write” in previous releases.
- 9. db fi le parallel write This event occurs in the DBWR. It indicates the time that DBWR spends waiting for I/O completion. If asynchronous I/O is available, then the db fi le asynch I/O submit wait event captures any time spent in submitting I/Os to the underlying storage. When asynchronous I/O is not available, db fi le parallel write captures the time spent during submit and reap. Wait Time: While there are outstanding I/Os, DBWR waits for some of the writes to complete. DBWR does not wait for all of the outstanding I/Os to complete. Some Oracle shops are plagued with slow I/O intensive databases, and this tip is for anyone whose STATSPACK top-5 timed events shows disk I/O as a major event. Example Statspack output: Top 5 Timed Events % Total Event Waits Time (s) Ela Time ------------ ------------ ----------- ----------- db fi le sequential read 2,598 7,146 48.54 db fi le scattered read 25,519 3,246 22.04 library cache load lock 673 1,363 9.26 CPU time 2,154 934 7.83 log fi le parallel write 19,157 837 5.68 This tip is important to you if you have reads waits in your top-5 timed events. Remember, if disk I/O is not your bottleneck then making it faster WILL NOT improve performance. Also, this is a OS-level solution, and often I/O-bound Oracle databases can be fi xed by tuning the SQL to reduce unnecessary large-table full-table scans. Monitoring fi le I/O using the stats$ fi lestatxs view: For optimal disk performance, Oracle should always use direct I/O to its data fi les, bypassing any caching at the OS layer. This must be enabled both in Oracle and in the operating system. The debate: Oracle supports multiple blocksized tablespaces in a single database including 2k, 4k, 8k, 16k and 32k blocksizes. The debate is about the bene fi t of di ff erent blocksizes (e.g. 8k, 32k) for reducing Oracle physical disk I/O and about whether the SA is likely to con fi gure the Oracle server to use direct I/O for Oracle. Some claim that placing large-related things (i.e. indexes) in a large blocksize results in a single physical fetch (less disk I/O), but others say this is NOT true because the OS blocksize (and the JFS cache) result in multiple OS I/O. According to this web page on direct I/O by Steve Adams: http://www.ixora.com.au/q+a/io.htm "It also avoids performance problems associated with using database block sizes that do not match the fi le system bu ff er size exactly. (more). Whether direct I/O is available, and the way in which it can be enabled, are dependent on the operating system and fi le system type in use. Direct I/O can sometimes be enabled for an entire fi le system, using a fi le system mount option. Application programs may also be able to request direct I/O for a particular fi le by setting a fl ag when opening the fi le." Adams goes on to note that direct I/O can increase the size of the RAM available for the working set in the Oracle data bu ff ers: "By switching to raw or direct I/O and by giving Oracle the memory that would otherwise be used by the operating system to cache Oracle data, a much larger working set of data can be cached, and a much higher cache hit rate can be sustained with obvious performance bene fi ts."
- 10. Oracle controls direct I/O with a parameter named fi lesystemio_options. According to this page by Steve Adams the fi lesystemio_options parameter must be set in order for Oracle to read data blocks directly from disk: "Databases that use a combination of say raw log fi les and raw temp fi les with fi le system based data fi les may wish to use kernelized asynchronous I/O against the raw fi les, but to avoid ine ffi cient threaded asynchronous I/O against the data fi les. This can be done by allowing disk_asynch_io to default to TRUE, but setting fi lesystemio_options to either none or directIO." Checking your Server Methods for con fi guring the OS will vary depending on the operating system and fi le system in use. Here are some examples of quick checks that anyone can perform to ensure that you are using direct I/O: • Solaris - Look for a "forcedirectio" option. Oracle DBAs claim this option makes a huge di ff erence in I/O speed for Sun servers. Here is the Sun documentation: http://docs.sun.com/ db/doc/816-0211/6m6nc6713?a=view • AIX - Look for a "dio" option. Here is a great link for AIX direct I/O: http://www-106.ibm.com/ developerworks/eserver/articles/DirectIO.html • Veritas VxFS - (including HP-UX, Solaris and AIX), look for "convosync=direct". It is also possible to enable direct I/O on a per- fi le basis using Veritas QIO; refer to the "qiostat" command and corresponding man page for hints. For HPUX, see Oracle on HP-UX - Best Practices. • Linux - Linux systems support direct I/O on a per- fi lehandle basis (which is much more fl exible), and I believe Oracle enables this feature automatically. Someone should verify at what release Oracle started to support this feature (it is called O_DIRECT). See Kernel Asynchronous I/O (AIO) Support for Linux and this great OTN article: Talking Linux: OCFS Update. I have found that the culprit in a few databases is tables that need purging or better indexing of columns. The memory/IO hogs can be seen with the following query. When I create the indexes or purge rows, I fi nd my db block hit ratio jumping and the physical reads per hour dropping and the meg of memory in the following query dropping: SELECT COUNT(*)*8192/1024/1024 meg_in_memory, o.OBJECT_NAME Object_in_Memory FROM DBA_OBJECTS o, V$BH bh WHERE o.DATA_OBJECT_ID = bh.OBJD AND o.OWNER != 'SYS' GROUP BY o.OBJECT_NAME having count(*)>0 ORDER BY COUNT(*); Regards, Alireza Kamrani.