What is parallel programming?

What is it? When to use?

Whenever two processes are independent, by definition, you can use parallelization. So far no novelty ... the difficult thing is to guarantee an environment in which parallelization does not cause a problem bigger than the benefit, and it is precisely here that languages, data structures and techniques which make it easier to program in a comparable way.

I would not say that parallel programming is by itself a paradigm, but rather a set of elements that make it easier to have parallelism and gain the benefits brought by it.

Languages

Functional languages

These are examples of languages that are more in the direction of parallelization, especially pure calling functions , that is, do not cause side effects and do not rely on anything other than what is explicitly passed on to them.

Parallelization usually occurs over pure functions that must be applied to a collection of data. Since the function is pure, one can delegate each element of the collection to a parallel processor, either on the same machine or on different machines.

Examples:

There is a endless list of languages that come closest to the parallel approach.

Data Structures

Immutability

There are also data architectures that go well with parallelization, and the most notable of my see are immutable structures . They allow the programmer not to worry about collaterals when manipulating data, because each time you do so, new instances are created instead to modify the original object. The previously presented pure functions benefit greatly immutable structures, since this ensures that there will be no side effects through the arguments passed.

Techniques / Paradigms

Map / Reduce

There are also techniques that use parallelization to do huge data processing (e.g. Big Data processing). One notable one is the Map / Reduce , which follows the paradigm of dividing to conquer. This paradigm (or technique) uses pure functions as well as immutable structures.

For those who know JavaScript, Map and Reduce are here in the same sense as map and reduce available in Arrays:

• Map: An output collection is produced for an input collection.
• Reduce: for an input collection an aggregate result of the elements of the original collection is produced.

I give this JavaScript example because it is now that people have more familiarity (I think), and so I get closer to them what MapReduce is when they do not seem to understand what it's all about.

I would not say that there are big differences between technique and paradigm, besides that of the paradigm to be applied as a foundation, so I'm presenting both under the same title.

Examples:

• Hadoop

No hardware

Image rendering

Parallel programming also exists on the hardware layer. For example, on your video card. Image rendering is clearly one of the main applications of parallelism, since there are millions of pixels to be rendered and all of them can be rendered independently of each other (speaking very plainly just to give an idea).

    

Parallel programming consists of dividing a computational task into two or more instances that are sufficiently independent that they can be executed in parallel - for example on two different processing cores, or even on two different machines. Not every task is parallelizable, it depends heavily on the characteristics of the problem and the dependency relationships between the tasks to be performed.

An example of an easily paralleled problem is "increment in one all values of this huge vector" (divide the vector into N parts and have each process perform the task in one of the parts). An example of a problem that is hardly parallelizable is "make each element of this huge vector have its value added to all elements after it."

The task of paralleling a program and the tools and techniques used are not in themselves a paradigm, and you can adopt this technique in any language that allows you to trigger more than one process that supports multi-threading. (provided threads are deployed so that they can be distributed across more than one process), or that supports network communication (for distribution of processing across more than one machine). However, you can design a whole language or framework with parallelism in mind (see "concurrent programming" ), such as the Erlang . If in the "philosophy" of the language / platform parallelism is the norm, not the exception, then I believe that one can speak of paradigm in these cases ...

About when to use, I would say that wherever possible, and are advantageous. That is, if the problem is not parallelizable (or if the benefit of parallelization is negligible) there is nothing to do; if a sequential alternative has a satisfactory performance, it does not compensate for the extra complexity of parallelizing (and may even worsen performance); and finally if the parallel overhead (critical sections / locks), separation of memory and inter-process communication, network communication, etc.) will nullify most of the benefit , it may not make sense to parallelize for now (on the other hand, what is not advantageous today may be in the future when hardware and communications conditions change).

    

What is parallel programming?

Briefly, it is when you divide a computation between several machines or processors that run at the same time.

Do not confuse parallelism with competition. For example, an operating system will have several processes running concurrently but only one of them uses the CPU at a time.

Is it a programming technique or paradigm?

I think it's more about technique than a paradigm. Generally when thinking about paradigm is something you can use for any program and parallel programming is something that is best used on a case by case basis.

Is there any specific language for parallel programming?

Parallel programming is a very difficult problem, especially when parallel processes need to do some form of communication or sharing of resources with each other. There are several programming languages and frameworks that try to support parallel programming but each will have its strengths and weaknesses. Parallel programming is not like object-oriented programming, functional programming, or logical programming in which you can come and speak "look, the X language is an example of what it means to be a parallel programming language."

That said, an example I can give is CUDA and other languages for writing parallel programs that run on GPUs.

In which cases is it better to use parallel programming?

Parallel programming is something you do when you want to take advantage of the fact that you have more than one processor available so you can try to find the answer to your problem faster. In situations where performance is not important it is not worth doing a complex parallel implementation and depending on the algorithm you have to implement it may be more or less difficult to create a parallel version.

A good example of the kind of thing that works well with parallelism is when you have "obviously parallelizable" computations. For example:

• To render 3d images each processor may be responsible for a different piece of the screen.
• In some numerical algorithms such as multiplication of vectors and matrices, each processor can be responsible for a portion of the separated vector or matrix and can join the partial accounts at the end.
• To compile a multi-module program, it is possible to use parallelism to compile more than one module at a time (the -j option of make is good for this)