Nice Blog, I agree Ldirectord rocks… I too hated paying crazy money for naff hardware load balancers, so built my own using LVS, Ldirectord and Heartbeat. Then I thought he this is pretty good why not sell it… 7 years later Loadbalancer.org owns a small corner of the load balancer appliance market and we’ve added SSL termination, TPROXY, HAProxy, Nginx, feedback agents, SNMP…. all sorts of stuff that initially I couldn’t see the point in (Why doesn’t everyone use LVS in DR mode with Ldirectord?). Still the customer is always right (even when they are wrong). We are a little different ’cause we allow full root access and open source any changes we make. It makes me sad to admit but now that processors are getting so fast… full proxies (SNAT) like HAProxy are so easy to insert into your architecture they will probably make Layer 4 routers aka. LVS almost redundant at some point. Ps. Love the line ‘Never underestimate the value of knowing where the bodies are buried’, exactly why I never worry when a customer says the load balancer is broken….It’s never the load balancer (well mostly never). And here comes the cheeky load balancing link .

If you just want a “regular” IPVS load balancer, that’s all you need. For a high-availability load balancer (if the active node dies, the other node in the pair will take over), then you need to add Pacemaker, and heartbeat or corosync, to the list of things to put together.

Is “all that chain of tools” available within the heartbeat packages?

Is there anything else that I would need to be familiar with to create a complete LVS load balancer aside from these utilities: iptables, ldirectord or keepalived, and ipvsadm?

Thanks again. github rocks and you guys nailed it.

I mostly wanted to ask one question. The article is centered around ldirectord, and as I have been researching, ldirectord doesn’t seem to the the load balancing engine at all. Rather, ipvsadm appears to be the load balancing utility, and ldirectord is the monitoring utility that checks the health of the real servers.

Am I way off here?

first, rest assured that I did not take any of your comments as criticism nor attacks, and I’m not dicussing your choices, just about your perception of what proxies can do, nothing else.

Concerning the performance, it depends what you are doing. Direct-routing load balancers installed in one-leg configuration where they only see upstream traffic will always be faster than a proxy because they see about half the packets. Common figures are about 4 times haproxy’s max connection rate. A load balancer set up in the middle of the stream will still generally be faster than the proxy on connection rate, but can be slower on high bit rates (multi-gigabit), because it cannot make use of TCP speedups the NIC provides, while a proxy can. For instance, haproxy can forward 10 Gbps of HTTP traffic with only 20% CPU on one of my machines, while this machine has high difficulties to simply IP-forward the same stream at this rate. This is because of the TCP off-loading capabilities of the NICs which can only work with the proxy. But granted, this is not everyone’s needs !

Concerning queuing, well, I can tell you that this has saved quite a number of medium to large sites. Yes connections can sometimes pile up. But better accept them, parse them, reject wrong ones and get the good ones ready to be served than simply drop the SYNs and wait for the client to resend them 3 seconds later. Also, it is quite possible that your site has a very predictible load. But gaming sites, sports sites, online newspapers, stock exchange, etc… are extremely unpredictable and need to be prepared for 4-10 times the load without failing, even if some requests get slightly delayed. And quite honnestly, it’s very rare that the queue delay is counted in seconds. This happens when something breaks behind. But even there, having the ability to automatically flush requests out of the queue is nice.

Concerning QoS, well, it also depends on your usages. Gaming sites like to reserve resources for paying customers. Other sites with many external partners will prefer to prioritize some requests vs some others to the same partner so that if they become bandwidth-limited, the most important objects are fetched first. But doing that really does not cost much. It’s just a switching rule between two queues. You may drom from 15000 connections per second to 14980 maybe, this is not particularly noticeable.

Concerning the logs, I don’t agree with you. The web servers will tell what they do, not how it’s perceived. The front LB will see how it’s perceived and will be able to quickly tell you that server XX has higher response times than others or fails to accept a connection 1/1000th of the times, etc… This is very valuable information when people start to complain about performance issues, and it’s even better when you can compare what the LB sees with what the server says. Most often, the difference is in (too short) system queues on the server itself (small TCP backlog, etc…) that can’t even be detected by the server software because the information is dropped before reaching it. That’s also the only place you can detect the famous 3-second stairs indicating some packet loss.

Overall, all a proxy can bring is a tradeoff, adding more intelligence in the decisions taken by the load balancer at the expense of an increased resources usage on one cheap box. If the smart decisions can improve the site’s responsiveness, reliability or scalability, that’s fine. If they don’t bring anything, it’s not a place for a proxy. That’s why you generally see them in front of HTTP servers, sometimes in front of mail servers, database servers and terminal servers, and rarely in front of anything else.

Most likely in your case, as you explain it, it would simply not bring anything for your current usage (and I have no problem with that). And I agree with you that with only 128 MB of RAM it’s hard to run a proxy with large numbers of connections, considering that the system itself will take 16 kB minimum per connection !

I do have something to add, though:

I have experience setting ldirectord up to replace a keepalived setup, because for some reason the keepalived process would go catatonic and drop all states in the kernel. Only a kill -9 and restart would cure it. Very unsatisfying.

So I’m not a particularly big fan of keepalived, but, that was 4 years ago. It could’ve been fixed by now, I don’t know.

Another thing I like about ldirectord is the fact that it does one thing well, then you have something like Heartbeat from Linux-HA to make the highly available part. Heartbeat does that very well.

And if the ldirectord process dies it just leaves the kernel IPVS system in whatever state it was in at the time of death, unlike my experiences with keepalived.

One bad thing about ldirectord: there’s a memory leak if you use negotiate checks on HTTPS services, and you’ll have to restart it every once in a while. This is easily done without interruptions, you just have to do it.

Thanks for taking the time to provide such an in-depth comment.

Sorry if I implied that haproxy couldn’t sustain a certain connection rate. I don’t know what the performance limit is with either haproxy or ldirectord on a given hardware setup — I’ve never benchmarked them to that degree. However, given the amount of work that is required to proxy connections rather than simply load balance them, I’d be very surprised if the load balancer wouldn’t handle a lot more traffic than a proxy.

As to the two “important features for scalability” that proxies provide, well, I don’t really consider them benefits.

Content switching can be achieved more easily with a simple assets domain, which also has the key benefits of avoiding unnecessary cookies, and steps around the per-domain limit on concurrent connections in web browsers.

I don’t think queueing something as time-sensitive as interactive HTTP requests is a good idea — if you’re running close enough to the line, resource wise, to ever get a queue forming, my experience is that it doesn’t stay at the “few milliseconds” of delay level for very long. The overload means that while the first connection might get delayed by, say, 5 milliseconds, the next will be delayed by 10 milliseconds, and so on — it doesn’t take too many requests to hit user-noticable levels, and at several thousand requests per second, it doesn’t take very long either. You’re better off shedding load to a static failover setup and keeping things under control that way. Your users get screwed either way, but at least with load shedding they get told why they’re screwed, instead of getting the spinning throbber of doom.

Implementing QoS in the frontend is an interesting idea, but I subscribe to the view that you only need QoS if you’re oversubscribed. Proper capacity planning saves the need to play such games, which when all is said and done can be no more than a temporary stopgap before rising traffic levels overwhelm the system capacity even with QoS. Doing more complex decision making on the proxy will also put more pressure on it, as it needs to increase the processing it does on requests before passing them through, which has implications for capacity again.

I don’t see logs as being any sort of a benefit of a proxy — the end servers can provide timing and error rate information better than the proxy can, and it’ll scale better as you don’t have to do all the processing and logging at a single point.

There is one point, though, I completely agree with you on: “If the proxy has no added value, let’s not install one”.

I don’t dislike haproxy, it’s used heavily at Github and we couldn’t do a lot of what we do without it. I’ve got another blog post planned on how haproxy is used in Github, and why it works so well. I just don’t think that it adds any value in the locations we’ve used ldirectord, and the rest of my arguments in favour of ldirectord still stand — we’ve got more experience using it, it is a lightweight solution that does exactly what we need to do, and the monitoring and management work was already done.

This post was mentioned on Twitter by FastestFood: Load balancing at Github: Why ldirectord? | Anchor Web Hosting Blog http://tinyurl.com/y9f26w3...