Python中運(yùn)行并行任務(wù)技巧
示例
標(biāo)準(zhǔn)線(xiàn)程多進(jìn)程,生產(chǎn)者/消費(fèi)者示例:
Worker越多,問(wèn)題越大
# -*- coding: utf8 -*-
import os
import time
import Queue
import threading
from PIL import Image
def create_thumbnail(filename, size=(128, 128)):
try:
fp, fmt = filename.rsplit('.', 1)
im = Image.open(filename)
im.thumbnail(size, Image.ANTIALIAS)
im.save((fp + '_'+'x'.join(str(i) for i in size) + '.'+fmt), im.format)
return '%s thumbnail success!' % filename
except Exception:
return '%s thumbnail failed!' % filename
def get_image_paths(folder):
return [os.path.join(folder, f) for f in os.listdir(folder) if 'png' in f]
class Consumer(threading.Thread):
def __init__(self, queue):
threading.Thread.__init__(self)
self._queue = queue
def run(self):
while True:
content = self._queue.get()
if isinstance(content, str) and content == 'quit':
break
respone = create_thumbnail(content)
print 'Bye bye!'
def Producer():
filenames = get_image_paths('images')
queue = Queue.Queue()
worker_threads = build_worker_pool(queue, 4)
start_time = time.time()
for filename in filenames:
queue.put(filename)
for worker in worker_threads:
queue.put('quit')
for worker in worker_threads:
worker.join()
print time.time() - start_time
def build_worker_pool(queue, size):
workers = []
for _ in range(size):
worker = Consumer(queue)
worker.start()
workers.append(worker)
return workers
if __name__ == '__main__':
Producer()
map
Map能夠處理集合按順序遍歷,最終將調(diào)用產(chǎn)生的結(jié)果保存在一個(gè)簡(jiǎn)單的集合當(dāng)中。

# -*- coding: utf8 -*-
import os
import time
from multiprocessing import Pool
from PIL import Image
def create_thumbnail(filename, size=(128, 128)):
try:
fp, fmt = filename.rsplit('.', 1)
im = Image.open(filename)
im.thumbnail(size, Image.ANTIALIAS)
im.save((fp + '_'+'x'.join(str(i) for i in size) + '.'+fmt), im.format)
return '%s thumbnail success!' % filename
except Exception:
return '%s thumbnail failed!' % filename
def get_image_paths(folder):
return [os.path.join(folder, f) for f in os.listdir(folder) if 'png' in f]
def main():
filenames = get_image_paths('images')
start_time = time.time()
pool = Pool(4)
pool.map(create_thumbnail, filenames)
pool.close()
pool.join()
print time.time() - start_time
if __name__ == '__main__':
main()
相關(guān)文章
教你用Python尋找重復(fù)文件并刪除的腳本寫(xiě)法
這篇文章主要介紹了如何用Python尋找重復(fù)文件并刪除,該腳本主要包括diskwalk,chechsum,find_dupes,delete模塊,其中diskwalk模塊是遍歷文件的,給定路徑,遍歷輸出該路徑下的所有文件,需要的朋友可以參考下2022-01-01
Pyside6 安裝和簡(jiǎn)單界面開(kāi)發(fā)過(guò)程詳細(xì)介紹
PySide是跨平臺(tái)應(yīng)用程序框架Qt的Python綁定,Qt是跨平臺(tái)C++圖形可視化界面應(yīng)用開(kāi)發(fā)框架,自推出以來(lái)深受業(yè)界盛贊,Pyside6是利用Python語(yǔ)言進(jìn)行開(kāi)發(fā)的GUI,所以在使用Pyside6前要先安裝Python環(huán)境,本文給大家介紹Pyside6 安裝和簡(jiǎn)單界面開(kāi)發(fā)過(guò)程,一起看看吧2023-10-10
Python單鏈表的簡(jiǎn)單實(shí)現(xiàn)方法
這篇文章主要介紹了Python單鏈表的簡(jiǎn)單實(shí)現(xiàn)方法,包括定義所需的字段及具體實(shí)現(xiàn)代碼的分析,需要的朋友可以參考下2014-09-09
Python??MkDocs優(yōu)雅地編寫(xiě)文檔
在軟件開(kāi)發(fā)過(guò)程中,編寫(xiě)文檔是非常重要的一環(huán),文檔不僅可以幫助用戶(hù)理解和使用你的軟件,還可以提高團(tuán)隊(duì)協(xié)作效率,然傳統(tǒng)的文檔寫(xiě)作方式往往繁瑣而復(fù)雜,不易于維護(hù)更新,MkDocs工具以簡(jiǎn)潔、優(yōu)雅的方式編寫(xiě)文檔,并且能夠輕松生成漂亮的靜態(tài)網(wǎng)站2024-01-01
python multiprocessing 多進(jìn)程并行計(jì)算的操作
這篇文章主要介紹了python multiprocessing 多進(jìn)程并行計(jì)算的操作,具有很好的參考價(jià)值,希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧2021-03-03
python3 cvs將數(shù)據(jù)讀取為字典的方法
今天小編就為大家分享一篇python3 cvs將數(shù)據(jù)讀取為字典的方法,具有很好的參考價(jià)值,希望對(duì)大家有所幫助。一起跟隨小編過(guò)來(lái)看看吧2018-12-12
Python3非對(duì)稱(chēng)加密算法RSA實(shí)例詳解
這篇文章主要介紹了Python3非對(duì)稱(chēng)加密算法RSA,結(jié)合實(shí)例形式分析了Python3 RSA加密相關(guān)模塊安裝及使用操作技巧,需要的朋友可以參考下2018-12-12

