Multi-core programming may actually require more than one paradigm. Some current contenders are:

1. MapReduce. This works well where a problem can be easily decomposed into parallel chunks.
2. Nested Data Parallelism. This is similar to MapReduce, but actually supports recursive decomposition of a problem, even when the recursive chunks are of irregular size. Look for NDP to be a big win in purely functional languages running on massively parallel but limited hardware (like GPUs).
3. Software Transactional Memory. If you need traditional threads, STM makes them bearable. You pay a 50% performance hit in critical sections, but you can scale complex locking schemes to 100s of processors without pain. This will not, however, work for distributed systems.
4. Parallel object threads with messaging. This really clever model is used by Erlang. Each "object" becomes a lightweight thread, and objects communicate by asynchronous messages and pattern matching. It's basically true parallel OO. This has succeeded nicely in several real-world applications, and it works great for unreliable distributed systems.

Some of these paradigms give you maximum performance, but only work if the problem decomposes cleanly. Others sacrifice some performance, but allow a wider variety of algorithms. I suspect that some combination of the above will ultimately become a standard toolkit.

Two solutions I really like are join calculus (JoCaml, Polyphonic C#, Cω) and the actor model (Erlang, Scala, E, Io).

I'm not particularly impressed with Software Transactional Memory. It just feels like it's only there to allow threads to cling on to life a little while longer, even though they should have died decades ago. However, it does have three major advantages:

1. People understand transactions in databases
2. There is already talk of transactional RAM hardware
3. As much as we all wish them gone, threads are probably going to be the dominant concurrency model for the next couple of decades, sad as that may be. STM could significantly reduce the pain.

The mapreduce/hadoop paradigm is useful, and relevant. Especially for people who are used to languages like perl, the idea of mapping over an array and doing some action on each element should come pretty fluidly and naturally, and mapreduce/hadoop just takes it to the next stage and says that there's no reason that each element of the array need be processed on the same machine.

In a sense it's more battle tested, because Google is using mapreduce and plenty of people have been using hadoop, and has shown that it works well for scaling applications across multiple machines over the network. And if you can scale over multiple machines across the network, you can scale over multiple cores in a single machine.

For .NET application I choose ".NET Parallel Extensions (PLINQ)" it's extremely easy to use and allows me to parallelize existing code in minutes.

1. It's simple to learn
2. Used to perform complex operations over large arrays of objects, so I can't comment on other applications
3. Supports tasks and piplines
4. Should be supported for a next couple of years, but who knows for sure?
5. CTP version has some performance issues, but already looks very promising.

Mono will likely get support for PLINQ, so it could be a cross-platform solution as well.

For heavy computations and the like, purely functional languages like Haskell are easily parallelizable without any effort on the part of the programmer. Apart from learning Haskell, that is.

However, I do not think that this is the way of the (near) future, simply because too many programmers are too used to the imperative programming paradigm.

kamaelia is a python framework for building applications with lots of communicating processes.

Kamaelia - Concurrency made useful, fun

In Kamaelia you build systems from simple components that talk to each other. This speeds development, massively aids maintenance and also means you build naturally concurrent software. It's intended to be accessible by any developer, including novices. It also makes it fun :)

What sort of systems? Network servers, clients, desktop applications, pygame based games, transcode systems and pipelines, digital TV systems, spam eradicators, teaching tools, and a fair amount more :)

See also the Question Multi-Core and Concurrency - Languages, Libraries and Development Techniques

I'm betting on communicating event loops with promises, as realized in systems like Twisted, E, AmbientTalk, and others. They retain the ability to write code with the same execution-model assumptions as non-concurrent/parallel applications, but scaling to distributed and parallel systems. (That's why I'm working on Ecru.)

Check out Erlang. Google for it, and watch the various presentations and videos. Many of the programmers and architects I respect are quite taken with its scalability. We're using it where I work pretty heavily...

As mentioned, purely functional languages are inherently parallelizable. However, imperative languages are much more intuitive for many people, and we are deeply entrenched in imperative legacy code. The fundamental issue is that pure functional languages express side-effects explicitly, while side effects are expressed implicitly in imperative languages by the order of statements.

I believe that techniques to declaratively express side effects (e.g., in an object oriented framework) will allow compilers to decompose imperative statements into their functional relationships. This should then allow the code to be automatically parallelized in much the same way pure functional code would be.

Of course, just as today it is still desirable to write certain performance-critical code in assembly language, it will still be necessary to write performance-critical explicitly parallel code tomorrow. However, techniques such as I outlined should help automatically take advantage of manycore architectures with minimal effort expended by the developer.

I'm surprised nobody has suggested MPI (Message Passing Interface). While designed for distributed memory, MPI programs with essential and frequent global coupling (solving linear and nonlinear equations with billions of unknowns) have been shown to scale to 200k cores.

This question seems to keep appearing with different wording - maybe there are different constituencies within StackOverflow. Flow-Based Programming (FBP) is a concept/methodology that has been around for over 30 years, and is being used to handle most of the batch processing at a major Canadian bank. It has thread-based implementations in Java and C#, although earlier implementations were fiber-based (C++, and mainframe Assembler - the one used at the bank). Most approaches to the problem of taking advantage of multicore involve trying to take a conventional single-threaded program and figure out which parts can run in parallel. FBP takes a different approach: the application is designed from the start in terms of multiple "black-box" components running asynchronously (think of a manufacturing assembly line). Since the interface between components is data streams, FBP is essentially language-independent, and therefore supports mixed-language applications, and domain-specific languages. For the same reason, side-effects are minimized. It could also be described as a "share nothing" model, and a MOM (message-oriented middleware). MapReduce seems to be a special case of FBP. FBP differs from Erlang mostly in that Erlang operates in terms of many short-lived threads, so green threads are more appropriate there, whereas FBP uses fewer (typically a few 10s to a few hundred) longer-lived threads. For a piece of a batch network that has been in daily use for over 30 years, see part of batch network. For a high-level design of an interactive app, see Brokerage app high-level design. FBP has been found to result in much more maintainable applications, and improved elapsed times - even on single core machines!

Qt concurrent offers an implementation of MapReduce for multicore which is really easy to use. It is multiOS.

If your problem domain permits try to think about a share nothing model. The less you share between processes and threads the less you have to design complex concurrency models.

See also the question Multi-Core and Concurrency - Languages, Libraries and Development Techniques

A job-queue with multiple workers system (not sure of the correct terminology - message queue?)

Why?

Mainly, because it's an absurdly simple concept. You have a list of stuff that needs processing, then a bunch of processes that get jobs and process them.

Also, unlike the reasons, say, Haskell or Erlang are so concurrent/parallelisable(?), it's entirely language-agnostic - you can trivially implement such a system in C, or Python, or any other language (even using shell scripting), whereas I doubt bash will get software transactional memory or join-calculus anytime soon.

I'd use Java - its portable so future processors wont be a problem. I'd also code my application with layers seperating interface / logic and data (more like 3 tier web app) with standard mutex routines as a library (less debugging of parallel code). Remember that web servers scale to many processors really well and are the least painful path to multicore. Either that or look at the old Connection Machine model with a virtual processor tied to data.

Erlang is the more "mature solution" and is portable and open source. I fiddled around with Polyphonic C# , I don't know how it would be to program everyday in it.

There are libraries and extensions for almost every language/OS under the sun, google transactional memory . It's an interesting approach from MS.