分叉/多螺纹工艺| 巴什

我想使我的一段代码更有效率。 我正在考虑把它分成多个进程，让它们一次执行50/100次，而不是一次。

例如（伪）：

for line in file; do foo; foo2; foo3; done 

我想这个for循环运行多次。 我知道这可以用分叉来完成。 它看起来像这样吗？

 while(x <= 50) parent(child pid) { fork child() } child { do foo; foo2; foo3; done return child_pid() } 

还是我想这是错误的方式？

谢谢！

Answer 1

在默认的bash脚本（非交互式）中，JOB CONTROL被禁用，所以你不能执行命令：job，fg和bg。

这对我来说很好：

 #!/bin/sh set -m # Enable Job Control for i in `seq 30`; do # start 30 jobs in parallel sleep 3 & done # Wait for all parallel jobs to finish while [ 1 ]; do fg 2> /dev/null; [ $? == 1 ] && break; done 

最后一行使用“fg”将后台作业放到前台。 它在一个循环中完成，直到fg返回1（$？== 1），当不再有任何后台作业时，它会执行此操作。

Answer 2

我不知道在bash中有任何明确的fork调用。 你可能想要做的是追加&到你想要在后台运行的命令。 你也可以使用&在你在bash脚本中定义的函数：

 do_something_with_line() { line=$1 foo foo2 foo3 } for line in file do do_something_with_line $line & done 

编辑 ：要限制同时后台进程的数量，你可以尝试这样的事情：

 for line in file do while [`jobs | wc -l` -ge 50 ] do sleep 5 done do_something_with_line $line & done 

Answer 3

使用GNU Parallel，您可以执行以下操作：

 cat file | parallel 'foo {}; foo2 {}; foo3 {}' 

这将在每个CPU核心上运行一个作业。 跑50做：

 cat file | parallel -j 50 'foo {}; foo2 {}; foo3 {}' 

观看介绍video以了解更多信息：

http://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Answer 4

我不喜欢使用wait因为它被阻塞，直到进程退出，这是不理想的，当有多个进程等待，因为我不能得到一个状态更新，直到当前进程完成。 我宁愿使用kill -0和sleep的组合。

给定一个等待的pids数组，我使用下面的waitPids()函数来获得持续的反馈，说明哪些pid还有待完成。

 declare -a pids waitPids() { while [ ${#pids[@]} -ne 0 ]; do echo "Waiting for pids: ${pids[@]}" local range=$(eval echo {0..$((${#pids[@]}-1))}) local i for i in $range; do if ! kill -0 ${pids[$i]} 2> /dev/null; then echo "Done -- ${pids[$i]}" unset pids[$i] fi done pids=("${pids[@]}") # Expunge nulls created by unset. sleep 1 done echo "Done!" } 

当我在后台启动一个进程时，我使用下面的这个实用程序函数把它的PID立即添加到pids数组中：

 addPid() { desc=$1 pid=$2 echo "$desc -- $pid" pids=(${pids[@]} $pid) } 

这是一个示例，显示如何使用：

 for i in {2..5}; do sleep $i & addPid "Sleep for $i" $! done waitPids 

这里是反馈看起来如何：

 Sleep for 2 -- 36271 Sleep for 3 -- 36272 Sleep for 4 -- 36273 Sleep for 5 -- 36274 Waiting for pids: 36271 36272 36273 36274 Waiting for pids: 36271 36272 36273 36274 Waiting for pids: 36271 36272 36273 36274 Done -- 36271 Waiting for pids: 36272 36273 36274 Done -- 36272 Waiting for pids: 36273 36274 Done -- 36273 Waiting for pids: 36274 Done -- 36274 Done! 

Answer 5

让我试试例

 for x in 1 2 3 ; do { echo a $x ; sleep 1 ; echo b $x ; } & done ; sleep 10 

并使用jobs来查看正在运行的内容。

Answer 6

根据你们所有人分享的内容，我可以把它们放在一起：

 #!/usr/bin/env bash VAR1="192.168.1.20 192.168.1.126 192.168.1.36" for a in $VAR1; do { ssh -t -t $a -l Administrator "sudo softwareupdate -l"; } & done; WAITPIDS="$WAITPIDS "$!;...; wait $WAITPIDS echo "Script has finished" Exit 1 

这一次列出了三台机器上的Mac上的所有更新。 后来，我用它来执行所有机器的软件更新，当我CAT我的ipaddress.txt

Answer 7

这是我的线程控制function：

 #!/bin/bash # This function just checks jobs in background, don't do more things. # if jobs number is lower than MAX, then return to get more jobs; # if jobs number is greater or equal to MAX, then wait, until someone finished. # Usage: # thread_max 8 # thread_max 0 # wait, until all jobs completed thread_max() { local CHECK_INTERVAL="3s" local CUR_THREADS= local MAX= [[ $1 ]] && MAX=$1 || return 127 # reset MAX value, 0 is easy to remember [ $MAX -eq 0 ] && { MAX=1 DEBUG "waiting for all tasks finish" } while true; do CUR_THREADS=`jobs -p | wc -w` # workaround about jobs bug. If don't execute it explicitily, # CUR_THREADS will stick at 1, even no jobs running anymore. jobs &>/dev/null DEBUG "current thread amount: $CUR_THREADS" if [ $CUR_THREADS -ge $MAX ]; then sleep $CHECK_INTERVAL else return 0 fi done } 

Answer 8

haridsv的方法非常好，它可以灵活地运行处理器插槽设置，其中一些进程可以在作业完成后提交新作业的情况下继续运行，从而保持整体负载。 这里是我的模块haridsv的代码ng格式“作业”的网格的n槽处理器（我用它模拟模型的网格）随后的testing输出为8个作业3，一次运行总计运行，提交，完成和剩余

 #!/bin/bash ######################################################################## # see haridsv on forking-multi-threaded-processes-bash # loop over grid, submitting jobs in the background. # As jobs complete new ones are set going to keep the number running # up to n as much as possible, until it tapers off at the end. # # 8 jobs ngrid=8 # 3 at a time n=3 # running counts running=0 completed=0 # previous values prunning=0 pcompleted=0 # ######################################################################## # process monitoring functions # declare -a pids # function checkPids() { echo ${#pids[@]} if [ ${#pids[@]} -ne 0 ] then echo "Checking for pids: ${pids[@]}" local range=$(eval echo {0..$((${#pids[@]}-1))}) local i for i in $range; do if ! kill -0 ${pids[$i]} 2> /dev/null; then echo "Done -- ${pids[$i]}" unset pids[$i] completed=$(expr $completed + 1) fi done pids=("${pids[@]}") # Expunge nulls created by unset. running=$((${#pids[@]})) echo "#PIDS :"$running fi } # function addPid() { desc=$1 pid=$2 echo " ${desc} - "$pid pids=(${pids[@]} $pid) } ######################################################################## # # Loop and report when job changes happen, # keep going until all are completed. # idx=0 while [ $completed -lt ${ngrid} ] do # if [ $running -lt $n ] && [ $idx -lt ${ngrid} ] then #################################################################### # # submit a new process if less than n # are running and we haven't finished... # # get desc for process # name="job_"${idx} # background execution sleep 3 & addPid $name $! idx=$(expr $idx + 1) # #################################################################### # fi # checkPids # if something changes... if [ ${running} -gt ${prunning} ] || \ [ ${completed} -gt ${pcompleted} ] then remain=$(expr $ngrid - $completed) echo " Running: "${running}" Submitted: "${idx}\ " Completed: "$completed" Remaining: "$remain fi # save counts to prev values prunning=${running} pcompleted=${completed} # sleep 1 # done # ######################################################################## 

testing输出：

  job_0 - 75257 1 Checking for pids: 75257 #PIDS :1 Running: 1 Submitted: 1 Completed: 0 Remaining: 8 job_1 - 75262 2 Checking for pids: 75257 75262 #PIDS :2 Running: 2 Submitted: 2 Completed: 0 Remaining: 8 job_2 - 75267 3 Checking for pids: 75257 75262 75267 #PIDS :3 Running: 3 Submitted: 3 Completed: 0 Remaining: 8 3 Checking for pids: 75257 75262 75267 Done -- 75257 #PIDS :2 Running: 2 Submitted: 3 Completed: 1 Remaining: 7 job_3 - 75277 3 Checking for pids: 75262 75267 75277 Done -- 75262 #PIDS :2 Running: 2 Submitted: 4 Completed: 2 Remaining: 6 job_4 - 75283 3 Checking for pids: 75267 75277 75283 Done -- 75267 #PIDS :2 Running: 2 Submitted: 5 Completed: 3 Remaining: 5 job_5 - 75289 3 Checking for pids: 75277 75283 75289 #PIDS :3 Running: 3 Submitted: 6 Completed: 3 Remaining: 5 3 Checking for pids: 75277 75283 75289 Done -- 75277 #PIDS :2 Running: 2 Submitted: 6 Completed: 4 Remaining: 4 job_6 - 75298 3 Checking for pids: 75283 75289 75298 Done -- 75283 #PIDS :2 Running: 2 Submitted: 7 Completed: 5 Remaining: 3 job_7 - 75304 3 Checking for pids: 75289 75298 75304 Done -- 75289 #PIDS :2 Running: 2 Submitted: 8 Completed: 6 Remaining: 2 2 Checking for pids: 75298 75304 #PIDS :2 2 Checking for pids: 75298 75304 Done -- 75298 #PIDS :1 Running: 1 Submitted: 8 Completed: 7 Remaining: 1 1 Checking for pids: 75304 Done -- 75304 #PIDS :0 Running: 0 Submitted: 8 Completed: 8 Remaining: 0