How to Start Background Process in Python

Python is a powerful and popular language that provides tons of useful features. It is often used as a scripting language to perform critical processes as well as running high traffic websites. One of the best features of python is its ability to easily work with other languages such as C/C++ as well as shell scripting. Sometimes you may need to run background process from your python script. Here python’s ability to interact with Linux shell is very useful. In this article, we will learn how to start background process in Python.

Why Run Background Process in Python

Many software developers use shell scripts to easily perform various tasks such as data backup, log parsing, file processing and more. You can run these scripts as background tasks in terminal. This way they run even when you close terminal session. It is very convenient to be able to do all this from within your Python script or application. Therefore, it is essential to learn how to run a background process in Python. It allows you to quickly offload many processes to the background, that continue running even after your Python script stops executing.

How to Start Background Process in Python

Let us say you want to run the following shell command from Python.

$ rm -r /path/to/directory

The above command ‘rm -r’ recursively deletes all contents of a directory.

There are a couple of ways to run background tasks from within Python script – using os module and using subprocess module. We will learn both these solutions one by one.

1. Using Subprocess

Subprocess module allows you to create new processes in Python. It helps you connect to input, output and error pipes. You can use it to run external commands and functions from Python environment. It returns the exit code of these commands that you can use for further processing.

Open your python script e.g. test.py in a text editor.

$ vi test.py

Add the following code to set execution environment.

#!/usr/bin/env

Next, import subprocess module.

import subprocess

This module features Popen() function that allows you to run shell commands from python. It returns a stream pointer that can be used to read/write to the pipe created between calling and called application. It is commonly used to run background processes.

Here is how to run a shell command to remove a file using rm command. You need to split the background command by space character, into a list of strings and input this list of strings in Popen() function. Please ensure to provide full path to file to avoid errors.

handle=subprocess.Popen(["rm","-r","/path/to/directory"])

We have stored the result of subprocess.Popen() function in a handle variable. This is perhaps the most useful feature of Popen() function. It allows you to control and terminate the process midway before completion.

Please note, Popen() is non-blocking. So your Python script will continue with its execution immediately after Popen() is called. If you want to block Python execution till Popen() completes, you need to call communicate() function.

handle.communicate()

Let us look at some variants of the above command.

You may also need to use sudo command to overcome ‘permission denied’ errors.

handle=subprocess.Popen(["sudo","rm","-r","/path/to/directory"])

You might also want to add ‘&’ character after the command in case you want to run it as background process in shell itself.

handle=subprocess.Popen(["sudo","rm","-r","/path/to/file","&"])

If you want to continue running the command, even after the spawned shell process terminates, then include a nohup command at the beginning.

handle=subprocess.Popen(["nohup","sudo","rm","-r","/path/to/file","&"])

Save and close the file. Make it an executable with the chmod command.

$ sudo chmod +x test.py

Run the python script with following command.

$ python test.py

If you need to terminate this shell script from Python, call the terminate() function on the stream pointer returned by Popen().

handle.terminate()

2. Using os module

os module is a time-tested Python module used for interacting with operating system from Python. It allows you to easily manage files, directories and processes from Python.

It provides spawnl() function that can be used to run shell commands from Python script. Here is the command to run our ‘rm -r’ command using this function.

import os
os.spawnl(os.P_DETACH, 'rm -r /path/to/file')

Please note, unlike subprocess, you do not need to split the shell command. You can directly use it with spawnl() function.

os module also provides another function system() which can also be used to run shell commands.

import os
os.system('rm -r /path/to/file')

Both spawnl() and system() return only the error value, unlike Popen() function that gives you a process pointer to work with.

Conclusion

In this article, we have learnt how to run background process in Python. You can use it to run Linux commands, shell scripts and even other python scripts from a given python script. It is very handy for Python developers and system administrators to run critical tasks and perform system maintenance without disrupting their application.

We have learnt two different ways to run background tasks in Python – using subprocess module as well as os module. According to Python documentation, using subprocess module is the new and recommended way to run background tasks in Python. This is because it provides easier ways to spawn new processes and get their results.

Also read:

How to Prevent NGINX from Serving .git directory
How to Prevent Apache from Serving .git directory
How to Check if String is Substring of List Items
How to Check if Column is Empty or Null in MySQL
How to Modify MySQL Column to Allow Null