python peewee操作sqlite锁表的问题分析

2023-08-21 08:26:03 作者：金色旭光

Peewee是一种简单而小的ORM,在使用python orm 框架 peewee 操作数据库时时常会抛出以一个异常,下面我们就来分享一下具体的原因以及解决办法吧

在使用python orm 框架 peewee 操作数据库时时常会抛出以一个异常，具体的报错就是 database is locked

初步了解是因为sqlite锁的颗粒度比较大，是库锁。当一个连接在写数据库时，另一个连接在想要写任意一张表都会报错。

为了解决这个问题，做如下的实验分析问题

1.理论分析

SQLite 是一个软件库，实现了自给自足的、无服务器的、零配置的、事务性的 SQL 数据库引擎。

SQLite允许多个进程/线程同时进行读操作，但在同一时刻只允许一个线程进行写操作。SQLite在进行写操作时，数据库文件会被锁定，此时任何其他的读/写操作都会被阻塞，如果阻塞超过5秒钟，就会抛出描述为“database is locked”的异常。

出现上述现象的原因是SQLite只支持库级锁，不支持并发执行写操作，即使是不同的表，同一时刻也只能进行一个写操作。
例如，事务T1在表A新插入一条数据，事务T2在表B中更新一条已存在的数据，这两个操作是不能同时进行的，只能顺序进行。

2.建表

import datetime
from peewee import AutoField, DateTimeField, Model, SqliteDatabase, TextField, IntegerField
db = SqliteDatabase('my_app.db', pragmas={'journal_mode': 'wal'})
class BaseModel(Model):
    """A base model that will use our Sqlite database."""
    id = AutoField()
    update_time = DateTimeField(default=datetime.datetime.now)
    class Meta:
        database = db
class User(BaseModel):
    name = TextField()
    age = IntegerField()
    class Meta:
        table_name = "user"
if __name__ == "__main__":
    db.connect()
    db.create_tables([User])
    User.create(name="ljk", age=29)
    res = User.select()
    for i in res:
        print(i.name, i.age)

3.串行写操作不会锁库

串行执行不会锁表，同时也说明事务完成之后锁立即释放

import time
import threading
from peewee_demo import User
def write_sql(num):
    user = User.get_by_id(1)
    print(f"传入数值：{num}")
    print("睡眠10s, 开始")
    time.sleep(10)
    print("睡眠10s, 结束")
    user.age = num
    user.save()
write_sql(100)
write_sql(300)

传入数值：100
睡眠10s, 开始
睡眠10s, 结束
传入数值：300
睡眠10s, 开始
睡眠10s, 结束

4.两个线程同时写会锁表

import time
import random
import threading
from peewee_demo import User
def write_sql(index):
    users = User.select()
    for user in users:
        user.age = random.randint(100, 200)
        print(f"in {index} , now is {time.time()}")
        user.save()
if __name__ == "__main__":
    p1 = threading.Thread(target=write_sql, args=(1, ))
    p2 = threading.Thread(target=write_sql, args=(2, ))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

(idt_dev) ➜ peewee_sqlite python main.py
in 1 , now is 1691136403.4496074
in 2 , now is 1691136403.4499302
Exception in thread Thread-2:
Traceback (most recent call last):
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3246, in execute_sql
cursor.execute(sql, params or ())
sqlite3.OperationalError: database is locked
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
self.run()
File "/usr/local/lib/python3.8/threading.py", line 870, in run
self._target(*self._args, **self._kwargs)
File "main.py", line 13, in write_sql
user.save()
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 6785, in save
rows = self.update(**field_dict).where(self._pk_expr()).execute()
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 1966, in inner
return method(self, database, *args, **kwargs)
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 2037, in execute
return self._execute(database)
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 2555, in _execute
cursor = database.execute(self)
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3254, in execute
return self.execute_sql(sql, params)
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3246, in execute_sql
cursor.execute(sql, params or ())
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3014, in __exit__
reraise(new_type, new_type(exc_value, *exc_args), traceback)
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 192, in reraise
raise value.with_traceback(tb)
File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3246, in execute_sql
cursor.execute(sql, params or ())
peewee.OperationalError: database is locked
in 1 , now is 1691136403.4617224
in 1 , now is 1691136403.467874
in 1 , now is 1691136403.475302
in 1 , now is 1691136403.4822652
in 1 , now is 1691136403.489331
in 1 , now is 1691136403.4965873
in 1 , now is 1691136403.5043068
in 1 , now is 1691136403.5117881
in 1 , now is 1691136403.5194569
in 1 , now is 1691136403.5266187
in 1 , now is 1691136403.5337832
in 1 , now is 1691136403.5410187
in 1 , now is 1691136403.5481625
in 1 , now is 1691136403.555381
in 1 , now is 1691136403.5625844
in 1 , now is 1691136403.569803
in 1 , now is 1691136403.5772254
in 1 , now is 1691136403.5843408
in 1 , now is 1691136403.5914726

5.同时一个读+一个写不会锁表

import time
import random
import threading
from peewee_demo import User
def write_sql(index):
    users = User.select()
    for user in users:
        user.age = random.randint(100, 200)
        print(f"in write {index} , now is {time.time()}")
        user.save()
def read_sql(index):
    users = User.select()
    for user in users:
        print(f"in read {index}, now is {time.time()}, name: {user.name}")
if __name__ == "__main__":
    p1 = threading.Thread(target=write_sql, args=(1, ))
    p2 = threading.Thread(target=read_sql, args=(2, ))
    p1.start()
    p2.start()
    p1.join()
    p2.join()

in write 1 , now is 1691136578.3930526
in read 2, now is 1691136578.3933816, name: person_P0
in read 2, now is 1691136578.3934226, name: person_P1
in read 2, now is 1691136578.3934548, name: person_P2
in read 2, now is 1691136578.3934836, name: person_P3
in read 2, now is 1691136578.3935122, name: person_P4
in read 2, now is 1691136578.3935406, name: person_P5
in read 2, now is 1691136578.3935676, name: person_P6
in read 2, now is 1691136578.393595, name: person_P7
in read 2, now is 1691136578.3936222, name: person_P8
in read 2, now is 1691136578.3936503, name: person_P9
in read 2, now is 1691136578.3936775, name: person_P10
in read 2, now is 1691136578.393705, name: person_P11
in read 2, now is 1691136578.3937323, name: person_P12
in read 2, now is 1691136578.3937595, name: person_P13
in read 2, now is 1691136578.3937871, name: person_P14
in read 2, now is 1691136578.3938174, name: person_P15
in read 2, now is 1691136578.3938463, name: person_P16
in read 2, now is 1691136578.3938737, name: person_P17
in read 2, now is 1691136578.393901, name: person_P18
in read 2, now is 1691136578.3939342, name: person_P19
in write 1 , now is 1691136578.4051046
in write 1 , now is 1691136578.4108906
in write 1 , now is 1691136578.4169016
in write 1 , now is 1691136578.4225135
in write 1 , now is 1691136578.4282284
in write 1 , now is 1691136578.4340622
in write 1 , now is 1691136578.4397743
in write 1 , now is 1691136578.4456632
in write 1 , now is 1691136578.451795
in write 1 , now is 1691136578.4575145
in write 1 , now is 1691136578.463979
in write 1 , now is 1691136578.471128
in write 1 , now is 1691136578.4781554
in write 1 , now is 1691136578.4851305
in write 1 , now is 1691136578.4925086
in write 1 , now is 1691136578.4996982
in write 1 , now is 1691136578.5068758
in write 1 , now is 1691136578.5138164
in write 1 , now is 1691136578.520577

6.加锁

加锁和数据库设置：

不管加什么锁，都不能解决lock的问题
是否设置读写模式都不影响读写操作

db = SqliteDatabase('my_app.db', pragmas={'journal_mode': 'wal'})

def write_sql(index):
    users = User.select()
    # with db.atomic("IMMEDIATE"):
    with db.atomic("EXCLUSIVE"):
        print("user")
        for user in users:
            try:
                user.age = random.randint(100, 200)
                time.sleep(1)
                print(f"in write {index} , now is {time.time()}")
                user.save()
            except Exception as e:
                print(e)
in write 10 , now is 1691142036.4625945
in write 10 , now is 1691142037.464804
in write 10 , now is 1691142038.467277
in write 10 , now is 1691142039.4688525
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3246, in execute_sql
    cursor.execute(sql, params or ())
sqlite3.OperationalError: database is locked
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/threading.py", line 932, in _bootstrap_inner
    self.run()
  File "/usr/local/lib/python3.8/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "main.py", line 11, in write_sql
in write 10 , now is 1691142040.4720113
    with db.atomic("EXCLUSIVE"):
  File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 4363, in __enter__
    return self._helper.__enter__()
  File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 4398, in __enter__
    self._begin()
  File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 4384, in _begin
    self.db.begin(*args, **kwargs)
  File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3765, in begin
    self.execute_sql(statement)
  File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3246, in execute_sql
    cursor.execute(sql, params or ())
  File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3014, in __exit__
    reraise(new_type, new_type(exc_value, *exc_args), traceback)
  File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 192, in reraise
    raise value.with_traceback(tb)
  File "/home/ljk/.virtualenvs/idt_dev/lib/python3.8/site-packages/peewee.py", line 3246, in execute_sql
    cursor.execute(sql, params or ())
peewee.OperationalError: database is locked
in write 10 , now is 1691142041.4745347
in write 10 , now is 1691142042.4767966
in write 10 , now is 1691142043.4779344
in write 10 , now is 1691142044.4796853
in write 10 , now is 1691142045.482223
in write 10 , now is 1691142046.4840803
in write 10 , now is 1691142047.4864902
in write 10 , now is 1691142048.4888134
in write 10 , now is 1691142049.491353
in write 10 , now is 1691142050.4932055
in write 10 , now is 1691142051.4950705
in write 10 , now is 1691142052.496692
in write 10 , now is 1691142053.4988236
in write 10 , now is 1691142054.500759
in write 10 , now is 1691142055.5022364

7.解决方案

from gpt3.5

SQLite 是一种嵌入式数据库，它默认情况下不支持多个进程同时写入。然而，有几种方法可以解决这个问题：

串行化访问：通过确保只有一个进程在任何给定时间写入数据库，可以使用互斥锁或信号量来实现串行化访问。这种方法可以保证数据的一致性，但可能会影响性能。
读写锁：SQLite 提供了一种读写锁机制，可以允许多个进程同时读取数据库，但只允许一个进程写入。这种方式可以提高并发性能，但需要在应用程序中正确实现读写锁的使用。
延迟写：可以通过将写操作延迟到合适的时机来避免同时写的问题。例如，可以将写操作缓冲到内存中，然后在合适的时机一起写入数据库。这种方式可以提高性能，但需要考虑数据一致性和恢复的问题。
使用独立的数据库服务器：如果应用程序需要支持大规模并发写入，可以考虑使用独立的数据库服务器，如MySQL或PostgreSQL。这样可以通过连接池和并发控制机制来实现并发写入。
选择哪种解决方案取决于应用程序的具体需求和性能要求。需要权衡数据一致性、并发性能和开发复杂性，并根据实际情况选择最适合的方法。

8.串行化访问

使用全局锁，当进行写操作之前获取锁，写操作完成释放锁。没有获取到锁抛出异常，让页面展示出来

import time
import random
import threading
from base_model import User, db
Lock = False
def write_sql(index):
    time.sleep(random.randint(1, 4))
    global Lock
    if Lock:
        print(f"i am {index}, 数据库被lock，退出执行")
        return
    else:
        print(f"i am {index}, 数据库可以使用")
        Lock = True
    user = User.get_by_id(10)
    user.age = 200
    user.save()
    Lock = False
if __name__ == "__main__":
    data = []
    for i in range(20):
        p = threading.Thread(target=write_sql, args=(i, ))
        data.append(p)
    for i in data:
        i.start()
    for i in data:
        i.join()

(dev) ➜ peewee_sqlite python main.py
i am 6, 数据库可以使用
i am 4, 数据库可以使用
i am 8, 数据库可以使用
i am 19, 数据库可以使用
i am 16, 数据库可以使用
i am 1, 数据库可以使用
i am 0, 数据库可以使用
i am 10, 数据库可以使用
i am 2, 数据库可以使用
i am 11, 数据库可以使用
i am 7, 数据库可以使用
i am 9, 数据库可以使用
i am 3, 数据库可以使用
i am 17, 数据库可以使用
i am 12, 数据库可以使用
i am 14, 数据库可以使用
i am 5, 数据库可以使用
i am 15, 数据库可以使用
i am 13, 数据库可以使用
i am 18, 数据库被lock，退出执行

9.总结

sqlite多线程无法同时写的特性并没有解决，只能通过业务层面规避这个问题。具体来说就是在需要写入的地方判断一下是否有其他写入任务，没有则获取全局写入标识，执行写操作；有其他写入任务则返回特定状态码，告诉用户其他业务逻辑正在使用数据库。虽然不优雅，but是当下最优解。

不要问为什么不用mysql，上面有人不让用~

到此这篇关于python peewee操作sqlite锁表的问题分析的文章就介绍到这了,更多相关eewee操作sqlite内容请搜索脚本之家以前的文章或继续浏览下面的相关文章希望大家以后多多支持脚本之家！