As we all know , DB poller is used to retrieve different kind of data from the Database.
Consider a scenario , where in you are polling a table , which is rapidly updated by another process. So now every polling cycle , the eligible records for the DB poller , will be huge in number.
And in cases where the data retrieved is metadata related to some other information like Batch id or OrderId , which are used in return to retrieve more information , the server will not be able to handle such amount of data, even in case it handles there will be a huge performance impact.
How do we restrict this kind of behaviour ?
During configuring our DB poller , we have few performance related parameter which we can use to restrict this kind of behaviour.
Database rows per XML document - MaxRaiseSize
On read (inbound) you can set maxRaiseSize = 0 (unbounded), meaning that if you read 1000 rows, you will create one XML with 1000 elements, which is passed through a single Oracle BPEL Process Manager instance. A merge on the outbound side can then take all 1000 in one group and write them all at once with batch writing
Database rows per Transactions - MaxTransactionSize
Assume that there are 10,000 rows at the start of a polling interval and that maxTransactionSize is 100. In standalone mode, a cursor is used to iteratively read and process 100 rows at a time until all 10,000 have been processed, dividing the work into 10,000 / 100 = 100 sequential transactional units. In a distributed environment, a cursor is also used to read and process the first 100 rows. However, the adapter instance will release the cursor, leaving 9,900 unprocessed rows (or 99 transactional units) for the next polling interval or another adapter instance. For load balancing purposes, it is dangerous to set the maxTransactionSize too low in a distributed environment (where it becomes a speed limit). It is best to set the maxTransactionSize close to the per CPU throughput of the entire business process. This way, load balancing occurs only when you need it
Throttling - RowsPerPollingInterval
DB throttling is the mechanism to control the number of database records processed by the SOA engine in a particular interval through DB Adapter. Throttling also can be used to control the number of records send to the end systems. If the throttling is not defined, the end systems may flood with number of messages that will affect the functioning of the end systems. Throttling parameters should be configured based on the end systems capacity to process the incoming messages. As of Oracle Adapters release 11.1.1.6.0 we can set the inbound DBAdapter property RowsPerPollingInterval to control the throttling. It acts as a limit on the number of records which can be processed in one polling interval. The default value is unlimited. The Patch 12881289 should be applied to enable this for SOA 11.1.1.5.0 and earlier versions.
The maximum rows processed per second are: Number of active nodes in SOA cluster x NumberOfThreads x RowsPerPollingInterval / PollingInterval.
MaxTransactionSize can be thought of as RowsPerDatabaseTransaction or DatabaseFetchSize that is how many records will be fetched to DB Adapter engine from the database for each transaction. It does not affect how many rows can be processed in one polling interval period.
For eg:
If you want to pick only 10 rows at a time i.e., only 10 rows per one instance , you need to give the value 10 for the property called "database rows per XML document"
"database rows per transaction" value should be greater than or equal to the "database rows per XML document" value.
whatever you give the polling frequency, when the database adapter looks for records in the table and it has lets say 100 records to process...and if you give database rows per XML document as 10 and database rows per transaction as 20 this is how it behaves at runtime...
first the database adapter gets 20 rows from database and now 2 instances will be created with 10 records in each...
then the next 20 records will be retrieved and creates 2 instances with 10 records each....
and this process continues until all records are processed with the appropriate read value...
In addition to all these properties , there is one more property called MaxRows.
We can use this property to handle the maximum number of rows to be picked up at the JDBC source level.
eg. If we have 10 records in a table. The DB poller query is matching all 10 records , normally it should pick all 10 records but if we set maxRows to some value 'n', only those n records will be picked and processed.
Hope this helps..
Please write to us in case you need any more clarifications.
Consider a scenario , where in you are polling a table , which is rapidly updated by another process. So now every polling cycle , the eligible records for the DB poller , will be huge in number.
And in cases where the data retrieved is metadata related to some other information like Batch id or OrderId , which are used in return to retrieve more information , the server will not be able to handle such amount of data, even in case it handles there will be a huge performance impact.
How do we restrict this kind of behaviour ?
During configuring our DB poller , we have few performance related parameter which we can use to restrict this kind of behaviour.
Database rows per XML document - MaxRaiseSize
On read (inbound) you can set maxRaiseSize = 0 (unbounded), meaning that if you read 1000 rows, you will create one XML with 1000 elements, which is passed through a single Oracle BPEL Process Manager instance. A merge on the outbound side can then take all 1000 in one group and write them all at once with batch writing
Database rows per Transactions - MaxTransactionSize
Assume that there are 10,000 rows at the start of a polling interval and that maxTransactionSize is 100. In standalone mode, a cursor is used to iteratively read and process 100 rows at a time until all 10,000 have been processed, dividing the work into 10,000 / 100 = 100 sequential transactional units. In a distributed environment, a cursor is also used to read and process the first 100 rows. However, the adapter instance will release the cursor, leaving 9,900 unprocessed rows (or 99 transactional units) for the next polling interval or another adapter instance. For load balancing purposes, it is dangerous to set the maxTransactionSize too low in a distributed environment (where it becomes a speed limit). It is best to set the maxTransactionSize close to the per CPU throughput of the entire business process. This way, load balancing occurs only when you need it
Throttling - RowsPerPollingInterval
DB throttling is the mechanism to control the number of database records processed by the SOA engine in a particular interval through DB Adapter. Throttling also can be used to control the number of records send to the end systems. If the throttling is not defined, the end systems may flood with number of messages that will affect the functioning of the end systems. Throttling parameters should be configured based on the end systems capacity to process the incoming messages. As of Oracle Adapters release 11.1.1.6.0 we can set the inbound DBAdapter property RowsPerPollingInterval to control the throttling. It acts as a limit on the number of records which can be processed in one polling interval. The default value is unlimited. The Patch 12881289 should be applied to enable this for SOA 11.1.1.5.0 and earlier versions.
The maximum rows processed per second are: Number of active nodes in SOA cluster x NumberOfThreads x RowsPerPollingInterval / PollingInterval.
MaxTransactionSize can be thought of as RowsPerDatabaseTransaction or DatabaseFetchSize that is how many records will be fetched to DB Adapter engine from the database for each transaction. It does not affect how many rows can be processed in one polling interval period.
For eg:
If you want to pick only 10 rows at a time i.e., only 10 rows per one instance , you need to give the value 10 for the property called "database rows per XML document"
"database rows per transaction" value should be greater than or equal to the "database rows per XML document" value.
whatever you give the polling frequency, when the database adapter looks for records in the table and it has lets say 100 records to process...and if you give database rows per XML document as 10 and database rows per transaction as 20 this is how it behaves at runtime...
first the database adapter gets 20 rows from database and now 2 instances will be created with 10 records in each...
then the next 20 records will be retrieved and creates 2 instances with 10 records each....
and this process continues until all records are processed with the appropriate read value...
In addition to all these properties , there is one more property called MaxRows.
We can use this property to handle the maximum number of rows to be picked up at the JDBC source level.
eg. If we have 10 records in a table. The DB poller query is matching all 10 records , normally it should pick all 10 records but if we set maxRows to some value 'n', only those n records will be picked and processed.
Hope this helps..
Please write to us in case you need any more clarifications.
No comments:
Post a Comment