Deep learning approach to images processing

There are many articles where deep learning is depicted, but there is still no common understanding what it means under the hood. On the one hand it is known that this term is related to supervised learning or in other words learning with a teacher. On the other hand it is still in a shadow of what it covers under its secret sauce. But in the meanwhile we do not pledge to provide you with indispensable solution to your production goals. It is just an attempt to look into this pervasive technology with a little more sober look.

Let’s get started with discussing a sufficient input format. In the most common cases images are taken as RGB. This format means converting each pixel of images into red, green, blue number encoding (each number can be any value from 0 to 255). For example, green is (0, 255, 0). Very often case when an input image will be cropped beforehand and saved as e.i. 64x64x3. Then under the hood this cropped picture is converted into some matrix with 3x3x3 dimension size. Each slice of this matrix mean red, blue or green

On this stage convolution can happen, what means that we take 3x3x3 image matrix and multiply it by some so-called template. Templates can be different, some of them can apply for example some filter or gradient. In the result we collect all of the preprocessed pieces into the 64x64x3 picture again.

Then we can apply pooling on some pieces of the picture. It is used for reducing computation. Most common case is max pooling what means getting pieces with maximum number. In the result we get the shorten matrix with e.i 2x2 elements

Finally convolution and pooling can be applied in any turn and multiple times. All of these operations applied by model automatically and usually only creators look into this magic. Model can be consisted of multiple layers but even 2 or 5 layers can be sufficient number for some kind of operations. These layers are preserved and called features. The detailed descriptions of such magic are written in many scientific papers, attended to solve a particular business goal for example emotions detection.

So, feature = layer = convolution/pulling. A model generates a set of features with different predictions for an each image based on predefined weights.

In simple words, a model is a set of operations which should be applied to an image to convert it into clear format fitted for our comparisons and predictions. It receives an input and returns results called predictions. This set of operations (pooling, convolution) are initially pre-determined but their interpretation can be changed by modifying model weights.  

So now let’s talk about model usage, what happens when we call model api and put e.i. image into it.

Mathematically, deep learning predictions are based on polynom. It can be depicted like:

y = c0 + a0X^0 + a1x^1 + a2x^ +... anx^n

What does it actually mean in practice? Our goal will be solving a system of such polynomials

Each polynomial will be assigned for each image. So how many images we have - such amount polynomials we need. What does it mean to solve? Each a is a parameter of business entity - they are determined when model is created and the number of a is called model capacity (a is also called a model weight and additionally means what to do with x to get y). For example it can be mapping between real-world entities and some value. E.i. 1 - is a traffic light, 2 - bike rack and etc. Or it can be 1,8 and if c0 = 32 we get converting from temperature by C into by F: y = 32+1,8*x. X is a prediction. It can be any real number (1, 1.12, 2.35). y is our ground truth (for example, the bounding boxes of annotations).  с0 is a bias. c0 and model weights can be changed in the result of model functioning post factum. The main goal of putting an input into model is getting x (prediction for each model parameter a). it will tell us that the object from model class a1 was detected on this image and a2 was not detected.

The system tries to unearth a certain pattern of real object behaviour. From this perspective there should be another criteria to confidently predict something.

So as we mentioned the first step is to solve the system of polynomials. It will tell us that an object on the images belongs to a particular class.

The second step will be an attempt to find overlapping zones between our results (predictions) and ground truth - y. Results and truth will be also polynomials. In other words we would like to know the percentage of our loss in predictions. 30% of overlapping means that if the predictions are overlapped to the ground truth by 30% of cases we assume that the prediction is succeeded, the object was detected for some classes a. If the result of  loss function is 100% it means that we did not find anything relevant to our model classes. The model overestimated or underestimated results. We need to pick up new parameters a with an optimizer approach again (it will be depicted below)

To do this we need to find differences between x and y. For it we can use a loss function. The most common case is a square loss function: x^2 - y^2 = delta. There can also be a logarithmic loss function, hybrid and others. This equation should be computed for all of our input images. In addition it will be computed in batches on GPU. For that can be used pytorch.

To compute this x^2 - y^2 = delta, we need to use some methods from maths SGD, ADAM  to find our parameters which should be put. This stage is called an optimizer. We can substitute parameters with parameters with significant difference - this approach will be called a large step or we can use parameters will small difference for example with a difference only by 1 (it will be called a small step). Both approaches should be combined because in other cases we can find only local minimum of loss function polynomial but not the real one. So the main idea is to look for the loss function results and seeking to its minimum value.

In the result of this step we get new weights which can be added to model again. Epoch is a model with the best result (minimum loss value). Loss can be calculated across all of loss functions (each function is assigned to an image or other input). Loss can be summed or average can be applied.

Models can be into two phases: training and validation. Training means that we take e.i. 32 images and process them on GPU using model. For the first run model weights can be random). Then we calculate loss. Then we take the next 32 pictures and process them. Calculate loss again and then compare two loss values. If it reduced we need to fix parameters. This method is called fitting. When we use model in validation status, it means that we turned off fitting and just calculate loss to be aware of the situation.

There are also other metrics to validate a model. The first one is recall which means how many research objects we missed in comparison with ground truth (we can believe in annotators results. Precision means how many mistakes we made e.i. a table was found instead of a stool. If recall rises precision reduces and vice versa.

Each model can be tailored for its specific goals: segmentation, detection, classification. Segmentation means image processing by pixel to e.i. colorize some object. Detection is used for finding particular areas in image called bounding boxes with sought objects. Classification means distinction of found objects by country, type or another criteria.

Thus, deep learning consists of data, model, loss function and optimizer and used for prediction based on inputs and for improving their results in the result of supervised learning