Fw: DatabaseSequenceFilter is very inefficient with lots of tables

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Fw: DatabaseSequenceFilter is very inefficient with lots of tables

Neil Swartz
My MySQL database has about 100 tables with a lot of FK references. The  DatabaseSequenceFilter takes about 20 seconds to construct from a connection.
I am using it in jUnit and cache it for the tests in a suite, but each suite takes 20-30 seconds to run.When I turned on general_log in MySQL I can see that it is using "show tables", "desc <table>"  and "desc %"  very frequently. It really could cache the results of this and it would be a lot faster.
I was also looking for ways to save & restore the object since it rarely changes. Maybe generate it from a connection once in a build cycle.
I am using 2.4.8 with Arquillian.
I think I have a solution. See attached file. AbstractMetaDataBasedSearchCallback.java
 
This is a proposed fix. It caches the requests to the DB for metadata. I threw this together quickly. It is not clean, not thread-safe, etc.
Needs more work, but it functions well for unit tests (single threaded)
It assumes that your DB is not changing as you run.

Is there a better way to deal with this? If not, maybe this technique could be included conditionally. (Some flag to determine if it should use caching)
 
Neil Swartz
[hidden email]


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
dbunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dbunit-user

AbstractMetaDataBasedSearchCallback.java (17K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

DatabaseSequenceFilter is very inefficient with lots of tables

Neil Swartz
<Resending since the last email got garbled>

My MySQL database has about 100 tables with a lot of FK references. The  DatabaseSequenceFilter takes about 20 seconds to construct from a connection.I am using it in jUnit and cache it for the tests in a suite, but each suite takes 20-30 seconds to run.When I turned on general_log in MySQL I can see that it is using "show tables", "desc <table>"  and "desc %"  very frequently. It really could cache the results of this and it would be a lot faster.

I was also looking for ways to save & restore the object since it rarely changes. Maybe generate it from a connection once in a build cycle.
I am using 2.4.8 with Arquillian.
I think I have a solution. See attached file. AbstractMetaDataBasedSearchCallback.java 
This is a proposed fix. It caches the requests to the DB for metadata. I threw this together quickly. It is not clean, not thread-safe, etc.
Needs more work, but it functions well for unit tests (single threaded)
It assumes that your DB is not changing as you run.

Is there a better way to deal with this? If not, maybe this technique could be included conditionally. (Some flag to determine if it should use caching)
 
Neil Swartz


------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
dbunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dbunit-user

AbstractMetaDataBasedSearchCallback.java (17K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Fw: DatabaseSequenceFilter is very inefficient with lots of tables

Neil Swartz

<Resending since the last email got garbled>


My MySQL database has about 100 tables with a lot of FK references. The  DatabaseSequenceFilter takes about 20 seconds to construct from a connection.I am using it in jUnit and cache it for the tests in a suite, but each suite takes 20-30 seconds to run.When I turned on general_log in MySQL I can see that it is using "show tables", "desc <table>"  and "desc %"  very frequently. It really could cache the results of this and it would be a lot faster.


I was also looking for ways to save & restore the object since it rarely changes. Maybe generate it from a connection once in a build cycle.
I am using 2.4.8 with Arquillian.
I think I have a solution. See attached file. AbstractMetaDataBasedSearchCallback.java 
This is a proposed fix. It caches the requests to the DB for metadata. I threw this together quickly. It is not clean, not thread-safe, etc.
Needs more work, but it functions well for unit tests (single threaded)
It assumes that your DB is not changing as you run.


Is there a better way to deal with this? If not, maybe this technique could be included conditionally. (Some flag to determine if it should use caching)
 
Neil Swartz
[hidden email]
------------------------------------------------------------------------------
See everything from the browser to the database with AppDynamics
Get end-to-end visibility with application monitoring from AppDynamics
Isolate bottlenecks and diagnose root cause in seconds.
Start your free trial of AppDynamics Pro today!
http://pubads.g.doubleclick.net/gampad/clk?id=48808831&iu=/4140/ostg.clktrk
_______________________________________________
dbunit-user mailing list
[hidden email]
https://lists.sourceforge.net/lists/listinfo/dbunit-user

AbstractMetaDataBasedSearchCallback.java (17K) Download Attachment