[SQLObject] Disabling sqlite autocommits to speed up inserts

Top Page
Author: Matt Doran
Date:  
To: sqlobject-discuss
Subject: [SQLObject] Disabling sqlite autocommits to speed up inserts
Hi there,

I'm very new to SQLObject, but I really like it's simplicity. However
I've encountered an issue that I'm struggling to resolve.

I'm using the latest stable build - sqlobject-0.7.1dev_r1684-py2.4, and
also pysqlite 2.2.0.

However I've found the performance is very slow. I can do about 10
inserts a second. This is similar performance to what I can achieve
with pysqlite directly if I commit after every insert. However, if I
commit after after a batch insert I can insert 1000 rows per second with
pysqlite directly.

It appears that SQLObject is performing a commit after every operation
(i.e. auto-commit), however I have not found a way to disable this
behaviour. I've tried "connection.autoCommit = False" and tried
explicitly using transactions, but neither seemed to work.

I've attached a script showing what I've been trying. Here's a summary:

    connection = connectionForURI(connection_string)
    connection.autoCommit = False
    sqlhub.processConnection = connection

    trans = connection.transaction()

    # ... do inserts...

    trans.commit()



Any advice would be greatly appreciated!

Cheers,
Matt