!!!This is still in progress, need to add links to the actual code!!!

There are countless tutorials and explanations for the multiprocessing in Python, but most of them are examples with limited or no real application. I’ll show you a real task for a real company with it’s challenges and my solutions. Having in mind that I’m self learned Python enthusiast most likely there are better approaches and I’ll be glad to learn about them.

What will you learn:

• What is „Cosine similarity“ for product suggestions? In short it is to represent your products as vectors and mathematically calculate the cosine of the angle between each two products. The bigger the cosine the closer the products are. Sounds easy and it is simple and at the same time it isn’t.
• When you have billions of calculations to do how to parallelize them using Python.
  • Multiprocessing with it’s problems with data sharing.
  • Fork and spawn processes.
  • Queue, SimpleQueue, manager().Queue() – why so many queues?
  • Garbage collector and the queues.
• Few other interesting moments:
  • How big is one integer? Why is it not 8 bytes?
  • Decorators and context managers
  • How important is to use native functions vs own code and why. TODO
  • What is lru_cache?
  • List, tuple, set, frozenset, dictionary – collections TODO

Be welcome to join this journey with me 🙂

Introduction to the problem

I had to develop a recommendation system for e-market site. We have a lot of products and a lot of customers. How to show you more relevant products to the one you are looking at?

There are generally two approaches:

• profile the user and compare him to other users and show what the others were interested in – you record his browsing history, orders, likes etc. The most important – you compare user <=> user.
• Item-to-item approach – you build a „profile“ for each product and compare each two of them to see if their profiles are alike. This way you compare item <=> item

I read about the two and decided the latter one is better. The Amazon’s – Item-to-Item Collaborative Filtering (link to pdf) drew my attention the most and I decided to go with it.

Cosine similarity

So what is „Item-to-Item Collaborative Filtering“? Each products is represented as mathematical vector and we are comparing the cosine between each two vectors. WTF is vector? I forgot?

Crash course in school geometry – Vector = line segment with certain length and direction. If the start of the vector is the beginning of the coordinate system it is often skipped. For example vector in two-dimensional space from point (0,0) to point (5,5) will be represented just with (5,5). We will be using exactly the same vectors – always starting from the center. In general Amazon are looking at the items as vectors(the coordinates of these vectors are the customers who have some relation with this product). If two vectors are parallel they are maximally similar, if two vectors are perpendicular they are maximally different( this is like cosine). The comparison between two products is the cosine of the angle between two vectors in n-dimensional space. In this case we are comparing the direction of the vectors, not their length. The result (cosine) will be between 0 and 1, where 1 means maximum similarity.

In general:

1) For every product P and for every customer you record the customer engagement with this product

2) You compute the similarity between every two products (Px, Py)

Mathematical calculation of cosine similarity:

https://en.wikipedia.org/wiki/Cosine_similarity

To be honest I don’t really understand this – I’ve forgotten this math, but there are good people who have shown how to calculate it.

In general if the dimensions of the first vectors are x and the dimensions of the second vector are y the cosine is:

sumxy / sqrt(sumxx * sumyy)

I think the theory is too much. Let’s show some examples.

In our website we have 3 items and 15 customers. For simplicity the engagement of the customer to the item will be expressed only by purchases.

I = Item

C = Customer

Each of the items was bought by the following customers:

I1 = [C1, C5, C7, C8]

I2 = [C2, C5, C8, C10]

I3 = [C2, C5, C10, C12, C15]

We have 15 customers, so the vector for each item will have 15 dimensions. If the customer purchased the item we put 1, otherwise 0. We get the following vectors:

I1 = (1,0,0,0,1,0,1,1,0,0,0,0,0,0,0)

I2 = (0,1,0,0,1,0,0,1,0,1,0,0,0,0,0)

I3 = (0,1,0,0,1,0,0,0,0,1,0,1,0,0,1)

When we calculate the cosine we get the following results:

I1 vs I2 = 0.5
I1 vs I3 = 0.22360679774997896
I2 vs I3 = 0.6708203932499369

Items I2 and I3 are most similar and I1 and I3 are least similar. 

Everything seams easy – and it is. Calculate the customer engagement, generate the vectors and calculate the simple „sumxy / sqrt(sumxx * sumyy)“

The programming problem

What is the hard part? Well our site is not small, not huge – 170 000 products, 400 000 customer profiles. This means we will have 170k vectors, each with 400k dimensions. Further more with 170 000 products we have 14 449 915 000 couples. Yes! 14+ billion couples to calculate, each couple is two vectors with 400 000 dimensions. As you can imagine the increase of couples to calculate increases in geometric progression.

Not only the number of calculations is frightening but also the memory needed – For 170k vectors with 400k dimensions each you have to keep 68 000 000 000 (68 billion) integers, each integer is 4 bytes in C (in Python it is 28 bytes, but more about this later). I’ll leave the computation of the needed memory for you.

And as real good guys we want our program to work not only today, but at least in the next few years. You can do the calculations for 250 000 products and 600 000 customers if you want.

The business problem

The numbers and time may be big, but you can invest some more money and you will get what you need. After all you don’t do it with pencil and paper, the computer does it. Start it, wait as long as needed and the results will come. The problem actually is how to calculate the customer engagement. There are many, many factors which you can use – purchases, page visits, likes, add to basket, sex, age, location, race are just the first that come to my mind. Each business is unique more or less and the combination of the factors can be different and also the weight of the different factors can be different. For example if you are specialized site with little traffic (high conversion rate) for something very specific, the page views can be important for you because you know your visitors know what they are looking for and they don’t browse just like this. On the other hand if you have a lot of traffic and your conversion rate is low the page visits most likely will not be so important to you.

Because of all these factors there is no simple approach to find the correct combination for you. You can’t simply hire someone to do it for you.

Programming

Now that we know what we are facing we can start with our actual task. We will focus in the programming problem and leave open options for future changes for the business problem.

Few additional requirements:

1. We should be able to define „groups“ of categories for which to calculate the comparisons, in case some day we decide to limit the calculations or just compare product within the group not the whole site
2. We want to keep only top X items
3. We want to keep top X items for each level of relation between two products. If the two products have the same category they have relation 0 (closest). If two products are in categories which have common parent category the relation is 1. If two products are in categories which have common parent-parent the relations is 2 and so on.
4. We should have easy way to modify in future the calculation of the relation customer<=>product

Analyze the data you are working with

I’ve always considered that knowing the content of your data is as important as it’s structure. So before I start with anything I decided to analyze a bit my data. With the data I have it turns out that very few customers have relation to each product. In fact 99% of the products have less than 1000 customers with some relation. Which means that these products have 399 000 dimensions with zeroes! Very good news – we don’t have to keep all these zeroes, because anything multiplied by zero is zero 🙂

Plan the structure based on the flow of the data

What we know:

1. We will have 170 000 vectors, each with 400 000 dimensions
2. Billions of computations to compare couples of products
3. The computations does not rely on one another. It means that computation 3 doesn’t have to wait 2 to finish. So we can calculate these in parallel
4. We will have billions of results, but we want to keep only few of them

We can put the computations of couples in parallel, but we need single process to prepare the data and define the work and we need single process to gather the results and keep only the ones we need.

Get the data

Initially to calculate the customer engagement we will use the page visits and the orders. So first we need to get data from our tables:

1. products – obvious why
2. orders – to get the purchases
3. user page view per product – to get the page visits
4. categories – because of the requirement for grouping

Our code will be able to use groups but for the sake of performance tests we will have only one group – all the products.

We need to create vector for each product having the commitment of each user. This means list with 400 000 element, but remember 99% of the products have less than 1000 customers with some relation.

The memory problem – can we keep all the values in the vectors?

Every developer knows that the standard integer is 32 bits or 4 bytes. In reality it is much bigger:

>>> import sys
>>> x = 100
>>> sys.getsizeof(x)
28

4 bytes is only the size of the data. In reality an integer variable in Python is whole object from class int which beside the data has methods (like int.bit_length()) and so on. This is not Python specific, in PHP for example the integer is 32 bytes.

So list with 400k python integers for each product is not an option. What are the other options? We can use Python’s array type which is more memory efficient and the type of the array is defined at creation. Does this look familiar? Like the arrays in C. We know that everything in Python is eventually executed in C(more about this later). If so can we use something to do the calculation for us in C? NumPy is first thing that comes to my mind.

NumPy is for sure faster for such calculations and has memory efficient arrays. In order to calculate the cosine between two vectors in NumPy you can use :

from numpy import dot, zeros
from numpy.linalg import norm
cosine = dot(v1, v2)/(norm(v1)*norm(v2))  # v1 and v2 are the vectors

I don’t know what does the formula mean, neither I care, but the result is correct.

But even if we use NumPy arrays our memory will be off the roof – the 399k zeros are just wasted memory.

I was curious and decided to find how much it will take exactly. I ended up with VM with 200GB RAM, 190GB used. This is for 170k vectors with 400k dimensions each. And what about the next year when we will have more products and more users? There must be another way.

We can use dictionary and keep only the positions in which we have non-zero values. We can then calcuate the cosine with simple loop:

def cosine_similarity(v1, v2):
    sumxx, sumxy, sumyy = 0, 0, 0
    # we want v1 to be the longer vector
    if len(v2) > len(v1):
        v1, v2 = v2, v1

    # calc the longer vector and x*y
    for v, x in v1.items():
        sumxx += x * x
        if v in v2:
            y = v2[v]  # in future this may not be 1
            sumxy += x * y
    for v, y in v2.items():
        sumyy += y * y
    return sumxy / sqrt(sumxx * sumyy)

This way we have dictionary with 1000 or less values, which is way better than 400k.

We will see later how we can further optimize this.

The parallel computations – how to calculate all the results faster

Good, now we know how to make single calculation in an efficient way. But we have billions of calculations which are independent from one another. It means that calculation of cosine similarity between product1 and product3 does not need to know the result of cosine similarity between product1 and product2. Yes, at the end we need to know all the results in order to get only the top ones, but as for calculating the similarity itself the calculations are independent. The solution is doing the calculations in parallel! This way depending on how many processes we use we can shorten the time as many times. For example if we use 10 processes the whole process will be approximately 10 times faster.

There are two more things to take care of:

1. We need to ensure that each calculation is done only once and no two processes do the same job
2. We need to collect all the results in order to get only top ones

When calculating similarity between two products it doesn’t matter if it s (p1, p2) or (p2, p1). If we have either one calculated the other calculation is not needed. For us this means that when we calculate the similarity for product1 (p1) with all others then when calculating the similarities for the others we don’t need to compare them to p1. So

• for p1 we will calculate from p2..p170000, 169 999 couples
• for p2 we will calculate from p3..p170000, 169 998 couples
• for p3 we will calculate from p4..p170000, 169 997 couples
• …
• for p169999 we will calculate only with p170000, 1 couple
• for p170000 we won’t make any calculation because we have calculated all its combinations already

In order to ensure that each couple is calculated only once we will create FIFO(First In First Out) queue with the indexes of the products. We will give this queue to the processes and each process will pick one index at time. The process will calculate the couples for this index. The couples will be only with products with higher index. Then the process will get the next available index.

This means that the processes will not do exactly the same amount of work – the process doing the calculations for p1 will have to process 169 999 couples(p2-p170000), while the process for p2 will have to process 169 998 couples (p3-p170000). Such a small difference is insignificant and trying to make the processes do exactly the same amount of work will be waste of time. Yes, towards the end one process will make 10 calculations, while the next will make 9, which is 10% difference, but still it is acceptable.

The next thing is to actually get the top results. Unfortunately this cannot be done in parallel because you need all values for a given product. Yes, it can be done in parallel on product level, but it would be hard to organize so many queues (one for each calculation process) to keep track of each product. Further more because we are doing parallel computations getting top X items from a list with Y values is much, much faster than doing the calculations for cosine similarity for Y couples.

In order to keep top X items we don’t need to wait for the computation of all results. We can keep only X results and when new result come we compare it to the smallest from the ones we keep – if the new result is bigger then we pop the smallest result and we use the new one. For example with top5:

• at certain moment we have [0.95, 0.70, 0.86, 0.58, 0.75]
• 0.73 is the next to come, the lowest is 0.58 < 0.73
• the list becomes [0.95, 0.70, 0.86, 0.73, 0.75]
• 0.50 is the next to come, the lowest is 0.70, the list remains unchanged

We can further optimize this by rearranging the list every time we make some replacement and then we will have to compare only with the last item from the list, because it will be the smallest one, so we will make only once comparison instead of X.

Multiprocessing in Python – processing with data sharing between the processes

In order to achieve all from the above we need:

1. Queue to organize the „work“ for the processes. This queue will have „only“ 170 000 items (the indexes of the products) and only one process will put items into this queue – this is important, as we will see in a bit.
2. Queue to gather the results so topX items can be calculated. This queue will have billions of items – the result of each couple. Many (all the calculation) processes will put items into this queue.
3. Start processes. We have to decide whether to „fork“ or „spawn“ the processes
4. Share the data for all the vectors with all processes. We can either „pass“ the data to the processes as parameter or create a „shared memory“, share one and the same object with all processes.

Before we continue we need to clarify the difference between multithreading and multiprocessing.

• With multithreading you can create threads, which naturally share the memory with the main thread, which makes data sharing really easy. The problem is that Python uses GIL(Global interpreter lock) which takes care for the memory management and more specifically it allows only one Python interpreter to in state of execution. In this way the count for the references to a given memory is kept correct and the Garbage Collector can work (we will talk about Garbace Collector (gc) later). Because GIL allows only one interpreter to be in state of execute at a time, all the threads work with one GIL and so with only one core of your CPU.

The multithreading is very memory efficient as one and the same memory is used by all the threads and it is much easier to manage this memory. As always it is trade CPU vs Memory. Multithreading is very memory efficient and CPU inefficient

The real problem is when you need multiple parts of the code to work in parallel in the same time. If you have multiple threads where most of them are waiting most of the time this can be useful, but in cases when you need to „split“ the work between different cores in order speed up things you need to use multiprocessing.

• With multiprocessing you actually create whole new process with it’s own GIL. Having it’s own GIL the new process can use another core. In order to achieve this you should copy one and the same memory to the new process and again it is trade CPU vs Memory. Multiprocessing is very CPU efficient and memory inefficient.

To better illustrate the difference we can use the following example:

You have cpu with 8 cores, list with 1 000 000 000 (one billion) elements. Let’s say this list takes 1GB memory. The calculation of 200 000 000 elements is 5 mins.

• multithreading
  • create 5 threads and the memory consumption will be 5x python thread memory (let’s say 25MB) + 1x memory for the list = 5×0.025 + 1×1 = 1.125GB.
  • Every thread sees the same list and when one thread makes a change to the list, the other threads can see it.
  • No two threads can work at the same time.
  • If you split the list in 5 and give each chunk to a different thread they will wait for each other to do the work: thread1-start, thread1-work, thread1-end, thread2-start, thread2-work, thread2-end, thread3-start, thread3-work, thread3-end, thread4-start, thread4-work, thread4-end, thread5-start, thread5-work, thread5-end.
  • The overall time will be 5×5 mins = 25 mins.
  • The final result for the threads is – 1.125GB memory used, 25mins work.
• multiprocessing
  • create 5 processes and the memory consumption will be 5x python thread memory (let’s say 25MB) and 5x memory for the list = 5x 0.025 + 5x = 5.125GB.
  • Every process will have it’s own copy of the list and when changes are done the other processes are unaware of this.
  • In order to notify the other processes you have to use shared memory/queue/pipe.
  • The processes can work in parallel (simultaneously) and the order of execution will be process1-start, process1-work, process2-start, process2-work, process3-start, process3-work, process4-start, process4-work, process5-start, process5-work, process1-end, process2-end, process3-end, process4-end, process5-end.
  • The overall time will be 5 mins
  • The final result for the processes is – 5.125GB memory used 5mins work.

For the multiprocessing there is one more thing to consider – fork or spawn the process. The difference is the fork(linux only) „copies“ all the memory of the parent process, while spawn creates new process with only the needed memory (link).

To illustrate the difference let’s say we use the example from above but in the main process we have another list with 500 000 000 (0.5GB) elements, which is not related to the main work. With spawn each process will get only the list for the work(1GB) and the overall memory will be like above(5.125GB). With fork each process will get 1GB+0.5GB and the overall memory will be +2.5GB (7.625GB) compared to the above.

These numbers – 5 mins vs 25 mins and 1GB vs 5 GB, 5GB vs 7.5GB may still seem small, but imagine you have 32 cores and the work and the dummy lists are 8 000 000 000 elements. The differences will be 5 vs 160 mins, 8GB vs 256GB, 256GB vs 512GB. The real difference, the real importance to understand this comes with the scale of the data you will process.

Now that we know the differences between multiprocessing and multithreading let’s see how to exchange data between different processed. Sharing the data between threads is easy because it is all done in Python and with Python’s data structures, while data sharing between processes is done at OS level. The real problem with Python data sharing is that the python data should be transformed into C data structures, transferred and then transformed back into Python’s data structure. Simply the OS doesn’t understand Python’s lists or dictionaries for example. To use one and the same object we can either „share“ or „pass“ it. With sharing all the processes can access and modify this memory/data, which with passing one process give the memory/data to another process and the changes from the latter one are not visible from the first process.

We have 3 ways to „share“ data between processes:

• Shared memory – you define the size of the requested memory and you have to take care of freeing it once it is not needed. This can be simplified with SharedMemoryManager. You also need to take care of locking/unlocking in order to avoid to processes to write to the same part simultaneously and lose the results of the first process.
• Shared ctype objects – Array or Value with certain ctype.

We can „pass“ memory in 2 ways:

• Pipe – it will create two connections between two processes which represent the two ends of the pipe. By default you can send and receive messages from each side/connection, but you can make it unidirectional so only one of the sides/connections can be used for sending and the other for receiving
• Queue – Built on top of the pipe, multiple processes put data/messages and multiple processes can receive/get the data.

With passing data you should know that the data is in some ordered way and you can take only what is at the end of the data, but you can’t put it back there. You can put it only at the beginning. Example:

You have a queue:
a -> b -> c -> d

If you make queue.get() you can get only „d“, you can’t ask queue.get(b) or queue.get(1). Once „d“ is taken, noone else can take it, it is gone from the queue/list. Also if you get „d“ you can’t put it back at the end. You can put elements only at the start.

What all this has to do with our task? Well, we want to ensure that one job is done only once. Using queues we will ensure that calculations for one product will be done only once and the results will be read only once.

Queues in Python

Generally in Python you have the queue module. But this is for single core/threaded applications. For multiprocessing there is the multiprocessing.Queue module. Before you rush using it be careful!!! There is an issue with this Queue – if you put an object in the queue and the object gets deleted it is also removed from the queue.

Example code to demonstrate this – generate all couples for all the 10 digits:

"""
Play with:
n
line:26 - print('PUT', chunk)
line:19 - time.sleep(0.000001)
"""
import math
from itertools import combinations
from multiprocessing import Queue, JoinableQueue, SimpleQueue
n = 10
k = 2
chunk_len = 10
q = Queue()  # SimpleQueue # - use SimpleQueue and Queue to see the difference
chunk = []


expected_combinations = 0
if hasattr(math, 'comb'):  # only python 3.8+ has it
    expected_combinations = math.comb(n, k)
else:
    expected_combinations = (n * (n - 1)) / 2  # works only with k=2

print('expected combinations', expected_combinations)
print('expected full chunks', expected_combinations // chunk_len)
for comb in combinations(range(n), k): # generate couples of two digits - 45 combinations
    # generate the chunk
    chunk.append(comb)
    if len(chunk) == chunk_len:
        print('PUT', chunk) # - comment here
        q.put(chunk)
        # time.sleep(0.000001) # - comment here
        chunk.clear()

print('-'*50)
while not q.empty():
    chunk = q.get()
    print('GET', chunk)

In this code we generate chunks of couples, each having 10 couples. With 45 couples you should have 4 full chunks which you put in the queue and you expect to get exactly 4 full chunks. When you use Queue and you don’t have the time.sleep() call you will not get from the queue what you had put in. How and why is this?

It has to do with the internal implementation of the Queue and the Garbage Collector.

In short every object is stored in the memory with 3 parts – name, reference and the object(data) itself. You can have multiple names referencing the same memory, which will increase the reference count. As high level language Python has Garbage Collector(GC) which clears the memory for us. GC frees the memory once the reference count becomes 0 (zero). This is exactly what GIL is used for – to ensure the correct reference count. You can watch the following video to get better idea:

Now to explain our problem – although it may seem that programs are executed line after line, instruction after instruction it is not always the case. With our code about the couples the problem is:

q.put(chunk)
chunk.clear()

It may happen that because „chunk.clear()“ actually decreases the reference count and it may become 0 so the GC is called and the memory is cleared. This will happen for the cases when „q.put(chunk)“ has not yet increased the reference count. More on the subject you can read at link. Actually exactly this bug is the reason for creating and introducing mutiprocessing.SimpleQueue. This is quite simplified version of the original Queue – it has only put(), get() and empty() methods. It cannot be defined with max size and so it doesn’t have qsize() and full() methods. It is not clear from the documentation but actually this queue guarantees that if you put object in the queue it won’t get garbage collected. Also it is worth noting it is not related to the Queue, it is completely different class.

Now back to our problem – consine similarity. We end up having the following processes:

• Cosine similarity (main process) – one process
• Process couples (worker processes) – many processes to calculate the similarity for each couple products. You can define how many are they
• Process top items – one process to filter only the top results for each product

In the main process we generate the list of vectors which is used by the worker processes. Also the main process will start all the other processes and monitor them to know when all the jobs are finished. The worker processes will put the results in a single queue, where they will be picked by the process for top items. At the end this process will write the results where needed.

All this communication is done with queues. The system works as:

1. The main process generates the vectors, vector groups. Generates a job queue with the ids of the products and a results queue for the results, starts the worker processes passing them the vectors, the job and results queues, starts the the process for the top results. At the end of the job queue ‘STOP’ elements will be put – as many times as worker processes we have.
2. The worker processes get product ids one by one and calculate the similarities and put the results in the results/output queue. Once ‘STOP’ element is found the process stops. Be cause the ‘STOP’s are the exact number of worker processes it is guaranteed when they are all done, the queue is also empty.
3. The main process monitors all the worker processes. When they are all done only the process for the top items should be waited. We put ‘STOP’ in the results queue so this process also knows when to finish. This ‘STOP’ will be at the end after all the results, because all the worker process are stopped and put all the results already.
4. The process for the top items generates an array to keep the top matches for each product, gets items from the results queue one by one, checks the array for the product and updates the array if needed. Once ‘STOP’ is found the process writes the results in the database or where we want.

lru_cache

When you have function which has some parameters and returns some result and you know this function will be called many times with the same parameters, it may be better to ‘cache’ the result. Yes, you will use some memory but probably the CPU load will be way lower.

This is exactly the case with getting the relation between two products. Actually it is not relation between the products themselves but the relation between their categories. We have 170k products with 14,5kkk couples, but only 700 categories with 250k couples. It means the relation between two categories will be calculated on average 58 000 times.

We can create our own cache – some dictionary with result, before any calculation we check if we already have result and if not we calculate it and put it in the dictionary. But we can use lru_cache decorator from functools to do this for us. It does exactly what we need – checks if the function is called with these parameters and if yes returns the result directly.

Decorators and context managers

While working with all this we need to measure our time for each piece of code. Usually you will use something like:

import time

def my_func()
    start = time.time()
    a=1
    b=2
    c = a + b
    print('Function execution time:', time.time() - start, 'secs')

my_func()

start = time.time()
x=1
y=2
z = x + y
print('Piece of code execution time:', time.time() - start, 'secs')

If we keep doing this we will add tons of lines to our code, it is not unified and we’ll repeat the same piece of code multiple times. Measuring time is textbook example for decorators and context managers. But what are they? Both allow doing something before and after some code execution. The decorators are designed for functions, while context managers are for any piece of code (you will understand in a bit).

Decorator – it allows you to take the function (1) you want, pass it to another function (2) where you can do something before and/or after (1).

How is that possible? Functions in Python are objects, which means they can be passed to other functions and also returned from other functions. Let’s say we have these functions:

from time import time, sleep

def measure_time(func):  # this accepts function as parameter
    def wrapper():
        start = time()
        func()
        execution_time = time() - start
        print(f'Execution time {execution_time}')
    return wrapper  # this returns function

def real_function(a, b):
    print('We are in the real function')
    sleep(2)

You can pass real_function to measure_time and get resulting function, which you can use:

real_function_before_after = measure_time(real_function)

When you execute this:

>>> real_functio_before_after()
We are in the real function.
Execution time 2.0120627880096436

Good. But if we use this for every function we have to write one line after the function and we will have to use the new name. Here comes the decorator syntax – using ‘@decorator_function_name’ before the function. And our code becomes:

def measure_time(func):  # this accepts function as parameter
    def wrapper():
        start = time()
        func()
        execution_time = time() - start
        print(f'Execution time {execution_time}')
    return wrapper  # this returns function

@measure_time
def real_function():
    print('We are in the real function')
    sleep(2)

Now you can easily run the real_function directly:

>>> real_function()
We are in the real function.
Execution time 2.0021607875823975

Remember:

• @measure_time is the same as real_function_before_after = measure_time(real_function).
• You can nest decorators – you will see this in a bit.
• You can use one and the same function as decorator for multiple other functions.

This doesn’t hurt, does it? But what if real function has parameters? What if real_function returns value? Simply use *args, **kwargs in the wrapper and return value:

def measure_time(func):  # this accepts function as parameter
    def wrapper(*args, **kwargs):
        start = time()
        real_func_value =  func(*args, **kwargs)
        execution_time = time() - start
        print(f'Execution time {execution_time}')
        return real_func_value
    return wrapper  # this returns function

@measure_time
def real_function(a, b):
    print(f'We are in the real function. {a} + {b} = {a+b}')
    sleep(2)
    return a+b

Now you can call real_function with parameters:

>>> outside_value = real_function(2, 3)
We are in the real function. 2 + 3 = 5
Execution time 2.0106985569000244
>>> print(f'outside_value is {outside_value}')
outside_value is 5

Great! But we can do more. Remember that the decorator is function itself and we can pass parameters to it. What could be the use of this?

We want to use one and the same decorator function to measure the time of different functions and to accumulate the execution time of each of them.

To achieve this we will have dictionary where we will keep the values. Exactly this dictionary we want to pass to the decorator plus some id for the specific function. How we are gonna do this? We will wrap our wrapper function once more so we can receive the parameters for the decorator itself. What we get at the end is:

def measure_time(timers: dict, timer_name: str = ''):  # receives the parermeters we passed to the decorator
    """
    Decorator

    :param timers:
    :param timer_name:
    :return:
    """
    def actual_decorator(func): # receives the function as parameter
        def wrapper(*args, **kwargs): # finally receives function parameters
            tic = time()
            value = func(*args, **kwargs)
            if timer_name not in timers:
                timers[timer_name] = 0

            timers[timer_name] += time() - tic

            return value
        return wrapper
    return actual_decorator

Now before we use the decorator we need to have the timers dictionary. We will also add another function to measure:

timers = {}

@measure_time(timers=timers, timer_name='real_function')
def real_function(a, b):
    print(f'We are in the real function. {a} + {b} = {a+b}')
    sleep(2)
    return a+b

@measure_time(timers=timers, timer_name='fast_function')
def fast_function():
    print(f'This is fast function')
    sleep(1)

And we get the following result:

>>> outside_value = real_function(2, 3)
We are in the real function. 2 + 3 = 5
>>> print(f'outside_value is {outside_value}')
outside_value is 5
>>> fast_function()
This is fast function
>>> print(timers)
{'real_function': 2.0069334506988525, 'fast_function': 1.0027618408203125}

If decorators are somewhat complex what about the context managers? They are much simpler, for real 🙂

Context managers are merely a class with __enter__ and __exit__ methods. They are used in Pythons’s with statements.

We want to measure parts of the code, not the whole functions. For the above example (ability to measure time. and keep track of different parts of the code in different timers) if we skip all the steps the context manager will be:

class MeasureTime:
    """
    context manager class
    """
    start = 0

    def __init__(self, timers: dict, timer_name: str = ''):
        self.timers = timers
        self.timer_name = timer_name

    def __enter__(self):
        self.start = time()

    def __exit__(self, exc_type, exc_value, exc_tb):
        if self.timer_name not in self.timers:
            self.timers[self.timer_name] = 0
        self.timers[self.timer_name] += time() - self.start

We can use it in the following way:

timers = {}
print('start of the code')
with MeasureTime(timers=timers, timer_name='first'):
    print('first part of the code')
    sleep(2)
    print('end of first part')

print('some other code')
sleep(1)

with MeasureTime(timers=timers, timer_name='second'):
    print('second part of the code')
    sleep(1)
    print('end of second part')

print(timers)

And we get:

start of the code
first part of the code
end of first part
some other code
second part of the code
end of second part
{'first': 2.0075531005859375, 'second': 1.014967679977417}

For the sake of reference here is the overall code we get:

from time import sleep, time


def measure_time(timers: dict, timer_name: str = ''):  # receives the parermeters we passed to the decorator
    """
    Decorator

    :param timers:
    :param timer_name:
    :return:
    """
    def actual_decorator(func): # receives the function as parameter
        def wrapper(*args, **kwargs): # finally receives function parameters
            tic = time()
            value = func(*args, **kwargs)
            if timer_name not in timers:
                timers[timer_name] = 0

            timers[timer_name] += time() - tic

            return value
        return wrapper
    return actual_decorator

class MeasureTime:
    """
    context manager class
    """
    start = 0

    def __init__(self, timers: dict, timer_name: str = ''):
        self.timers = timers
        self.timer_name = timer_name

    def __enter__(self):
        self.start = time()

    def __exit__(self, exc_type, exc_value, exc_tb):
        if self.timer_name not in self.timers:
            self.timers[self.timer_name] = 0
        self.timers[self.timer_name] += time() - self.start

timers = {}

@measure_time(timers=timers, timer_name='real_function')
def real_function(a, b):
    print(f'We are in the real function. {a} + {b} = {a+b}')
    sleep(2)
    return a+b

@measure_time(timers=timers, timer_name='fast_function')
def fast_function():
    print(f'This is fast function')
    sleep(1)

outside_value = real_function(2, 3)
fast_function()
outside_value = real_function(4, 5)
fast_function()

print('start of the code')
with MeasureTime(timers=timers, timer_name='first'):
    print('first part of the code')
    sleep(2)
    print('end of first part')

print('some other code')
sleep(1)

with MeasureTime(timers=timers, timer_name='second'):
    print('second part of the code')
    sleep(1)
    print('end of second part')

print(timers)

Useful links:

Nina Zakharenko – Memory Management in Python – The Basics – Speaker Deck

https://realpython.com/python-timer/

https://stackoverflow.com/questions/986006/how-do-i-pass-a-variable-by-reference

https://www.tutorialspoint.com/python/python_multithreading.htm