Few people will argue that asynchronous computation is cool and useful. In fact, the wholereactive programming idea is based on asynchronous computations being possible. Well, there’s more than that, but the core idea is to allow data and events to flow through your system and do something with the results when they become available.

So let’s look at an example of asynchronous function that everyone has seen and many have written themselves.

$("#book").fadeIn("slow",    function() {    console.log(“hurray”);   }); 

This piece of JavaScript code takes a book element and fades it in. When fading is complete a callback function is called and “hurray” string appears in the console. All is well and good in this trivial case, but once your system grows you can find yourself writing more and more of these nested callbacks.

Callbacks are a common way of dealing with asynchronous or delayed actions. They are not the best option though; the problem with callbacks is that they tend to chain forever, callbacks for callbacks for callbacks, until you find yourself in a complete mess and every change in the code becomes extremely painful and slow.

Maybe there are other ways to organize asynchronous code? In fact, there are: all you need to do is just tweak the perspective a bit. Imagine, if you had a type to represent a result of an async computation. It would be awesome, and your code would pass it around like every other value and be flat, fluid and readable.

Well, why don’t we build it!

When we’re done, we’ll have a monadic type Promise written in Java 8 that will make our asynchronous code wonderful. It’s not like it wasn’t ever done before, but I want to lead you through the process and help you understand what’s happening and why. If you are lazy or just prefer starting from code, check out the github repo.

Getting to love monads in 9.5 minutes

Oh, monads! Every programmer worth their morning coffee has written about them. Monads are what functional programming adepts love, use and praise. And there are thousands of tutorials and posts describing the concept.

So if you know everything there is to know about monads and want to get a closer look onto more interesting things, scroll down to the code below. Otherwise, bear with me just ten minutes, maybe this will become your go-to explanation about what a monad is.

A monad is a type, that represents a context of computation. I bet you’ve heard that before, but have you thought about what it means?

First of all, a monad doesn’t specify what is happening, that’s the responsibility of the computation within the context. A monad says what surrounds the computation that is happening.

Now, if you want an image reference to help you out, you can think of a monad as a bubble. Some people prefer a box, but a box is something concrete so a bubble works better for me.

These monad-bubbles have two properties:

• a bubble can surround something
• a bubble can receive instructions about what should it do with a surrounded thing

The surrounding part is easy to model in a programming language. Just take something and return a bubble! A constructor or a factory method comes to mind immediately here. Let’s look at how it is formalized. I’m assuming that you have some knowledge of Haskell notation (which you probably should have anyway). So the function that takes something and returns a monad is usually called pure or return:

return :: a -> m a 

Or in Java, if we can have some Monad class already.

public class Monad<T> {    public Monad(T t) {     …   } }

See that was easy. In fact, we’re halfway there. Another thing we must add is the ability to receive instructions for working with this value T eaten by our bubble.

What will help us is a bind function, which takes some form of an action and returns a different monad bubble that wraps this action executed on whatever was previously in the bubble.

For the sake of completeness, here is how it looks in Haskell.

(>>=)  :: m a -> (a -> m b) -> m b 

So this bind function takes a monad over a, (m a) and a function from a, and returns a different monad. In Java, we’ll have this definition as follows.

public class Monad<T> {    public abstract <V> Monad<V> bind(Function<T, Monad<V>> f); }

That will complete our generic definition of monads so we can proceed with an implementation.

Wait, what? I can have my monads in Java?

First of all, there are many different types of monads. In that sense, a monad is more like an interface in Java terms. There is a List monad, a Maybe monad, an IO monad (for languages that are very pure and cannot allow themselves to have normal IO), etc.

We will focus on creating a specific monad in Java, more specifically in Java 8. There is a good reason as to why we chose Java 8, since previously we found out that a monad will have to manipulate functions, which is really not that enjoyable in pre-lambda versions of Java.  However, Java 8 introduces lambdas and first-class methods, so it will be much more pleasant to work with them.

Your homemade Promise implementation

Here we go, now we’ve established our goal to have a monadic type to represent async computations. We’ve got our tools, namely Java 8, and we are ready to hack.

What we want to have is a Promise class that represents a result of asynchronous computation, either successful or erroneous.

Let’s pretend that we already have some Promise class that accepts callbacks to execute when the main computation is finished. Luckily, we don’t have to pretend very much, there are many implementations of that available: Akka’s Future, Play’s promise and so forth.

For this post I’m using the one from Play Framework, in which case instances of Promise get redeemed when some thread calls invoke() or invokeWithException() methods. It also accepts callbacks in a form of Play’s Promise specific Action class arguments. Obviously, Promise has constructors already, but not only do I want to create new instances of Promise, I also want to mark them completed with a value immediately. Here is how I can do it.

public static <V> Promise<V> pure(final V v) {     Promise<V> p = new Promise<>();     p.invoke(v);     return p;   }

The returned Promise is already redeemed and is ready to provide us with a result of the computation, which is precisely the given value.

The bind implementation will look like something below. It takes a function and adds that as a callback to this instance. A callback will get a result of this computation and apply given function to it. Whatever that function application returns or throws is used to redeem the resultingPromise.

public <R> Promise<R> bind(final Function<V, Promise<R>> function) {     Promise<R> result = new Promise<>();      this.onRedeem(callback -> {       try {         V v = callback.get();         Promise<R> applicationResult = function.apply(v);         applicationResult.onRedeem(applicationCallback -> {           try {             R r = applicationCallback.get();             result.invoke(r);           }           catch (Throwable e) {             result.invokeWithException(e);           }         });       }       catch (Throwable e) {         result.invokeWithException(e);       }     });     return result;   } 

Both applying the given function and getting a result from this are wrapped into the try-catch blocks, so exceptions are propagated to the resulting instance of Promise, just as one might expect.

With these two constructs, it’s very easy to chain asynchronous computations while avoiding going deeper and deeper into the callback hell. In the following synthetic example, we do exactly that.

public static void example1()                      throws ExecutionException, InterruptedException {     Promise<String> promise = Async.submit(() -> {       String helloWorld = "hello world";       long n = 500;       System.out.println("Sleeping " + n + " ms example1");       Thread.sleep(n);       return helloWorld;     });     Promise<Integer> promise2 = promise.bind(string ->               Promise.pure(Integer.valueOf(string.hashCode())));     System.out.println("Main thread example2");     int hashCode = promise2.get();     System.out.println("HashCode = " + hashCode);   }

That is basically it. We’ve implemented a monadic type Promise to represent a result of an async action.

Production-ready completable future

For those of you who have beared with me this far, I just want to say some final words about the quality of this implementation. Naturally, the above-mentioned GitHub repository has some tests that are proving that in some contexts, this might all work. However, I wouldn’t recommend using those Promises in production.

One reason for that is that Java 8 already contains a class that represents a result of async computation and is monadic…, welcome, CompletableFuture!

It does exactly what we want it to do and features several methods that allow you to bind a function to the result of an existing computation. Moreover, it provides methods to apply a function or a consumer, which is a void function by the way, or a plain old Runnable.

• thenApply(Function<? super T,? extends U> fn)
• thenApplyAsync(Function<? super T,? extends U> fn)
• thenAccept(Consumer<? super T> block)
• thenAcceptAsync(Consumer<? super T> block)
• thenRun(Runnable action)
• thenRunAsync(Runnable action)

On top of that, methods that end on *Async will execute this function asynchronously using a common ForkJoin executor. Otherwise, you can also supply an executor of your own liking.

Conclusion

Hopefully, this post shed some light onto what a monad is and next time you are about write a callback, you might want to consider a different approach.

In the post above we’ve looked at what monads are and how one can implement monadic classes in Java 8. Monads are great help in organizing data flow through your code and we’ve shown it with an example of Promise monad that represents results of an asynchronous computation. All the code from this blogpost is available for pondering in the Github repo.

Stay tuned for my next post, in which I plan to cover how to use the javaflow library to implement asynchronous awaiting for the promise to return a result. So you can get even more reactive :-)

Get the PDF

Threaded vs Evented Servers

July 24 2010

Broadly speaking, there are two ways to handle concurrent requests to a server. Threadedservers use multiple concurrently-executing threads that each handle one client request, while evented servers run a single event loop that handles events for all connected clients.

To chose between the threaded and evented approaches you need to consider the load profile of the server. This post describes a simple mathematical model for reasoning about these load profiles and their implications for server design.

Suppose that requests to a server take c CPU milliseconds and w wall clock milliseconds to execute. The CPU time is spent actively computing on behalf of the request, while the wall clock time is the total time including that time spent waiting for calls to external resources. For example, a web application request might take 5 ms of CPU time c and 95 ms waiting for a database call for a total wall time w of 100 ms. Let’s also say that a threaded version of the server can maintain up to t threads before performance degrades because of scheduling and context-switching overhead. Finally, we’ll assume single-core servers.

If a server is CPU bound then it will be able to respond to at most

(/ 1000 c) 

requests per second. For example, if each requests takes 2 ms of CPU time then the CPU can only handle

(/ 1000 2) => 500 

requests per second.

If the server is thread bound then it can handle at most

(* t (/ 1000 w)) 

requests per second. This expression is similar to the one for CPU time, but here we multiply the result by t to account for the t concurrent threads.

The throughput of a threaded server is the minimum of the CPU and thread bounds since it is subject to both constraints. An evented server is not subject to the thread constraint since it only uses one thread; its throughput is given by the CPU bound. We can express this as follows:

(defn max-request-rate [t c w]   (let [cpu-bound    (/ 1000 c)         thread-bound (* t (/ 1000 w))]     {:threaded (min cpu-bound thread-bound)      :evented  cpu-bound})) 

Now we’ll consider some different types of servers and see how they might perform with threaded and evented implementations.

For the examples below I’ll use a t value of 25. This is a modest number of threads that most threading implementations can handle.

Let’s start with a classic example: an HTTP proxy server. These servers require very little CPU time, so say c is 0.1 ms. Suppose that the downstream servers can receive the relay within milliseconds for a wall time w of, say, 10 ms. Then we have

(max-request-rate 25 0.1 10) => {:threaded 2500, :evented 10000} 

In this case we expect a threaded server to be able to handle 2500 requests per second and an evented server 10000 requests per second. The higher performance of the evented server implies that the thread bound is limiting for the threaded server.

Another familiar example is the web application server. Let’s first consider the case where we have a lightweight app that does not access any external resources. In this case the request parsing and response generation might take a few milliseconds; say c is 2 ms. Since no blocking calls are made this is the value of w as well. Then

(max-request-rate 25 2 2) => {:threaded 500, :evented 500} 

Here the threaded server performs as well as the evented server because the workload is CPU bound.

Suppose we have a more heavyweight app that is making calls to external resources like the filesystem and database. In this case the amount of CPU time will be somewhat larger that the previous case but still modest; say c is 5 ms. But now that we are waiting on external resources we should expect a w value of, say, 100 ms. Then we have

(max-request-rate 25 5 100) => {:threaded 200, :evented 200} 

Even though we are making a lot of blocking calls, the workload is still CPU bound and the threaded and evented servers will therefore perform comparably.

Suppose now that we are implementing a background service such as an RSS feed fetcher that makes high-latency requests to external services and then performs minimal processing of the results. In this case c may be quite low, say 2 ms, but w will be high, say 250 ms. Then

(max-request-rate 25 2 250) => {:threaded 100, :evented 500} 

Here an evented server will perform better. The CPU load is sufficiently low and the external resource latency sufficiently high that the blocking external calls limit the threaded implementation.

Finally, consider the case of long polling clients. Here clients establish a connection to the server and the server responds only when it has a message it wants to send to the client. Suppose that we have a lightweight app such that c is 1 ms, but that response messages are sent to the client after 10 seconds such that the w value is 10000 ms. Then

(max-request-rate 25 1 10000) => {:threaded 2.5, :evented 1000} 

If the server were really limited to 25 threads and each client required its own thread, we could only allow 2.5 new connections per second if we wanted to avoid exceeding the thread allocation. An evented server on the other hand could saturate the CPU by accepting 1000 requests per second.

Even if we increase the maximum number of threads t by an order of magnitude to 250, the evented approach still fares better:

(max-request-rate 250 1 10000) => {:threaded 25, :evented 1000} 

Indeed, a threaded server would need to maintain 10000 threads in order to be able to accept requests at the rate of the evented server.

Now that we have seen some specific examples of the model we should step back and note the patterns. In general, an evented architecture becomes more favorable as the ratio of wall time w to CPU time c increases, i.e. as proportionally more time is spent waiting on external resources. Also, the viability of a threaded architecture depends on the strength of the underlying threading implementation; the higher the thread threshold t, the more wait time can be tolerated before eventing becomes necessary.

In addition to the quantitative performance implications captured by this model, there are several qualitative factors that influence the suitability of threaded and evented architectures for particular servers.

One factor is the fit of the server architecture to the work that the server is doing internally. For example, proxying is well suited to evented architectures because the work being done is fundamentally evented: upon receiving an input chunk from the client the chunk is relayed to a downstream server. In contrast, the business logic implemented by web applications is more naturally described in a synchronous style. The callbacks required by an evented architecture become unwieldy in complex application code.

Another consideration is memory coordination and consistency. Evented servers executing in a single event loop do not need to worry about the correctness and performance implications of maintaining consistent shared memory, but this may be a problem for threaded servers. Threaded servers therefore attempt to minimize memory shared among threads. This approach works well for the servers that we discussed above - proxies, web applications, background workers, and long poll endpoints - as none of them need to share state internally across client sessions. But fundamentally stateful servers like caches and databases cannot avoid this problem.

The threaded approach can be a non-starter if the underlying platform does not support proper threading. In these cases blocking calls to external resources prevent the process from using the CPU in other threads, even if the blocker is not itself using the CPU. C Ruby falls into this category. In these cases t is effectively 1, making evented architectures relatively more appealing.

In the other extreme, the assumption of t being 25 or even 250 may be too modest for some platforms. These low t values are an an artifact of threading implementations and not intrinsic to the threading model itself. More scalable threading implementations make threaded servers viable for higher w to c ratios.

An evented approach can be compromised by a lack of evented libraries for the platform. For evented servers to perform optimally, all external resources must be accessed through nonblocking libraries. Such libraries are not always available, especially on platforms that have typically used threaded/blocking models like the JVM and C Ruby. Fortunately this situation is improving as developers publish more nonblocking libraries in response to the demand from implementors of evented servers. Indeed, the requirement of pervasive evented libraries for optimal performance is one reason that node.js is so compelling for building evented servers.

User’s guide

Notes

目 录 [ - ]

1. 先来个自我介绍吧！
2. Dubbo是什么？能做什么？
3. Dubbo适用于哪些场景？
4. Dubbo的设计思路是什么？
5. Dubbo的需求和依赖情况？
6. Dubbo的性能如何？
7. 和淘宝HSF相比，Dubbo的特点是什么？
8. Dubbo在安全机制方面是如何解决的？
9. Dubbo在阿里巴巴内部以及外部的应用情况？
10. 在分布式事务、多语言支持方面，Dubbo的计划是什么？
11. Dubbo采用的开源协议？商业应用应该注意哪些事项？
12. Dubbo开发团队情况？
13. 其他开发者如何参与？可以做哪些工作？
14. Dubbo未来的发展计划？

• 梁飞 （开发人员/产品管理）
• 刘昊旻 （开发人员/过程管理）
• 刘超 （开发人员/用户支持）
• 李鼎 （开发人员/用户支持）
• 陈雷 （开发人员/质量保障）
• 闾刚 （开发人员/开源运维）

 
从左至右：刘超，梁飞，闾刚，陈雷，刘昊旻，李鼎