[ Content | View menu ]

ADOdb Cache Write Error

Mark Mzyk | October 23, 2008

On a project I’ve been working on, I’ve recently seen several ADOdb cache write errors being thrown. When searching Google for a possible cause I didn’t find anything, so I thought I’d put this bread crumb out there for any others who might be experiencing the same problem.

ADOdb is a database abstraction library for PHP and Python. I’m working in PHP in this case.

The cache write error is being thrown when the project is under high load. After much investigation, it appears that this is a race condition within ADOdb when it is being used concurrently and the cache is enabled.

Here’s the scenario:

The cache expires. Then two users attempt to access the service which makes a call to ADOdb to access the database. Both users make a request that executes the same SQL query with the same variables set. Two calls to the ADOdb cache execute function are made. Since the cache is expired, the cache execute function then calls the ADOdb cache write function. The cache write function uses the SQL query to generate a hash that determines the proper cache file to write to. Since both requests generated the same SQL query, each request attempts to write to the same file. The cache write function uses flock(), a PHP file locking function , to lock the cache file. One request will win the lock and successfully write the cache file. The other will fail to win the lock. This will cause the cache write function to return false. On seeing the return value of false, the cache execute function will then throw the cache write error for the user that made the request that failed to acquire the lock. This scenario can potentially play out every time the cache expires and two users make the same request concurrently.

I have not been able to reproduce the error during testing, but I do believe this is what is happening in this instance.

I recommend that if you want to examine this further and you’re using ADOdb with PHP, that you read the php.net page on flock(), as there is a lot of information on how flock() works and what other users have experienced.

I haven’t yet developed a good solution to prevent these cache write errors, so I’m open to suggestions. Although it maybe something that doesn’t need a solution and can safely be ignored.

Filed in: Programming.

4 Comments

  1. Comment by W. Andrew Loe III:

    This generally called ‘dog-piling’.

    There are a variety of solutions depending on your tolerance for serving ‘stale’ data.

    http://highscalability.com/strategy-break-memcache-dog-pile
    http://blog.kovyrin.net/2008/03/10/dog-pile-effect-and-how-to-avoid-it-with-ruby-on-rails-memcache-client-patch/

    October 27, 2008 @ 12:34
  2. Comment by Mark:

    I didn’t realize it was called dog-piling. I’ll read the links and see what I can learn. Thanks.

    October 27, 2008 @ 13:00
  3. Comment by Andy:

    Hey, Mark.. I took a stab at this one yesterday, with no luck. For what it’s worth, this post is now Google’s #1 result for the query “cache write error memcache adodb”. Cool for you, a little depressing for me :)

    Did your investigations ever lead you to the idea it was a problem internal to memcached?

    March 21, 2009 @ 08:47
  4. Comment by Mark:

    Hey Andy,

    It’s been a while since I looked at this, but I remember that what I found was that there could be a potential problem with the code in the PHP implementation of memcached, although I’m not certain on that. It is also possible this is caused by the way we’re using memcached and isn’t a coding problem but instead a utilization problem. Unfortunately, it’s hard to tell without setting up tests to look at the issue more closely, which is something I’ve never done and hasn’t been a pressing issue (for me).

    March 22, 2009 @ 21:14