Load balancing apache cluster environment

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Load balancing apache cluster environment

Keeling, Diana

I know this probably isn't the best group, but I was wondering if anyone would be willing to help us out, or could forward this email to their system administrators to answer some questions.

 

We are looking into setting up a load balanced, high-availability apache cluster that runs PHP.  We are just wanting to gather some information from universities to see what others are doing to help gauge what we might decide to do. 

 

1.       What OS is being used?

2.       How does your environment handle PHP Sessions?

3.       Can the environment be scaled?

4.       How do you handle upgrades (different environments?)

5.       We are also going to be using MariaDB in a different clustered environment and we know there some caveats that are tied to that like every table must have a uniqueID.  Do you know if apache has any of those caveats as well?

6.       If anyone has implemented this type of setup and are willing to hop on a phone call, we would definitely appreciate it. 

 

-- 

Thank You,

 

Diana Keeling

Manager, Web Development Group

Enterprise Applications and Programming

Western Kentucky University

 



You are currently subscribed to [hidden email].
To unsubscribe send an email to [hidden email] with the subject line unsubscribe uwebd. Leave the message body blank. If you experience problems, contact list owner Eric Kreider at [hidden email]. More information is available at: http://www.uakron.edu/webteam/university-web-developers.dot




Reply | Threaded
Open this post in threaded view
|

Re: Load balancing apache cluster environment

C. Daniel Chase-2
Diana-

I've done this too... :-)

When we implemented our OU Campus installation at The University of Tennessee at Chattanooga (UTC) I architected a three-VM frontend, load-balanced behind an F5 BigIP. Here is a link to one of the original design docs with the architecture:


To answer your questions:

1) We were using Linux, Red Hat Enterprise Linux v6, which was eventually converted in-place to Oracle Linux due to better educational licensing costs. 
2) Session handling was done via a shared SAN mount point that was read-write and dedicated to sessions. 
3) Yes, this can easily be scaled and this was the concept behind the architecture. (I've lived through a big event on campus taking down our website--I was not going to let that happen again.)
4) Depending on what you mean by "different environments" we upgraded servers by simply taking one out of the load balancer pool, upgrading it, and adding it back, then doing the next...
5) The diagram doesn't show it, but parallel to this I implemented a MySQL cluster. Due to the age at the time of our F5, it was not capable of the complex iRule that was required to separate SQL reads from writes, so instead of distributing reads across the cluster as I originally intended, all traffic went to a Master. The replicas ended up being strictly for redundancy, instead of load. Today, the F5 has been upgraded and could do what I originally planned. However, I would recommend you consider using Percona Server, which is another MySQL compatible database, as they have a multi-master capability. So, you could easily distribute all reads & writes based on load.

Two options you should consider at this point:
1) Moving to the cloud. (Before I was offered early retirement from UTC, this is the plan I had in place. Accounting / political issues were the only hold-up.)
2) Using NGINX Plus (https://www.nginx.com/products/) as both load balancer & web server. We used NGINX for our web servers when we deployed the above architecture. If we did not have an F5 already, I would have deployed NGINX as the load balancer, potentially with a Varnish cache server in the middle.

I'd be happy to discuss more as needed. 


-Dan

C. Daniel Chase
Lead Web Developer
SUNY Empire State College
(518) 581-2238


On Mon, Dec 19, 2016 at 8:55 PM, Keeling, Diana <[hidden email]> wrote:

I know this probably isn't the best group, but I was wondering if anyone would be willing to help us out, or could forward this email to their system administrators to answer some questions.

 

We are looking into setting up a load balanced, high-availability apache cluster that runs PHP.  We are just wanting to gather some information from universities to see what others are doing to help gauge what we might decide to do. 

 

1.       What OS is being used?

2.       How does your environment handle PHP Sessions?

3.       Can the environment be scaled?

4.       How do you handle upgrades (different environments?)

5.       We are also going to be using MariaDB in a different clustered environment and we know there some caveats that are tied to that like every table must have a uniqueID.  Do you know if apache has any of those caveats as well?

6.       If anyone has implemented this type of setup and are willing to hop on a phone call, we would definitely appreciate it. 




You are currently subscribed to [hidden email].
To unsubscribe send an email to [hidden email] with the subject line unsubscribe uwebd. Leave the message body blank. If you experience problems, contact list owner Eric Kreider at [hidden email]. More information is available at: http://www.uakron.edu/webteam/university-web-developers.dot




Reply | Threaded
Open this post in threaded view
|

Re: Load balancing apache cluster environment

Charles Fulton
Diana,

We’ve implemented something very similar to what Dan has done. The main difference is that we’re using Memcache for session handling, Red Hat Storage for shared uploads, and MariaDB’s Galera database cluster (very similar to Percona). Galera’s been stable and fault-tolerant; the only application that we couldn’t host on it was Moodle and that’s a known issue with Moodle’s database architecture. We’re also about to stand up Nginx Plus as a replacement for generic Nginx.

I’d also be happy to discuss our implementation.

Charles

--
Charles Fulton
Senior Web Applications Developer
Lafayette College
(610) 330-3046

On Dec 20, 2016, at 9:06 AM, C. Daniel Chase <[hidden email]> wrote:

Diana-

I've done this too... :-)

When we implemented our OU Campus installation at The University of Tennessee at Chattanooga (UTC) I architected a three-VM frontend, load-balanced behind an F5 BigIP. Here is a link to one of the original design docs with the architecture:


To answer your questions:

1) We were using Linux, Red Hat Enterprise Linux v6, which was eventually converted in-place to Oracle Linux due to better educational licensing costs. 
2) Session handling was done via a shared SAN mount point that was read-write and dedicated to sessions. 
3) Yes, this can easily be scaled and this was the concept behind the architecture. (I've lived through a big event on campus taking down our website--I was not going to let that happen again.)
4) Depending on what you mean by "different environments" we upgraded servers by simply taking one out of the load balancer pool, upgrading it, and adding it back, then doing the next...
5) The diagram doesn't show it, but parallel to this I implemented a MySQL cluster. Due to the age at the time of our F5, it was not capable of the complex iRule that was required to separate SQL reads from writes, so instead of distributing reads across the cluster as I originally intended, all traffic went to a Master. The replicas ended up being strictly for redundancy, instead of load. Today, the F5 has been upgraded and could do what I originally planned. However, I would recommend you consider using Percona Server, which is another MySQL compatible database, as they have a multi-master capability. So, you could easily distribute all reads & writes based on load.

Two options you should consider at this point:
1) Moving to the cloud. (Before I was offered early retirement from UTC, this is the plan I had in place. Accounting / political issues were the only hold-up.)
2) Using NGINX Plus (https://www.nginx.com/products/) as both load balancer & web server. We used NGINX for our web servers when we deployed the above architecture. If we did not have an F5 already, I would have deployed NGINX as the load balancer, potentially with a Varnish cache server in the middle.

I'd be happy to discuss more as needed. 


-Dan

C. Daniel Chase
Lead Web Developer
SUNY Empire State College
(518) 581-2238


On Mon, Dec 19, 2016 at 8:55 PM, Keeling, Diana <[hidden email]> wrote:

I know this probably isn't the best group, but I was wondering if anyone would be willing to help us out, or could forward this email to their system administrators to answer some questions.

 

We are looking into setting up a load balanced, high-availability apache cluster that runs PHP.  We are just wanting to gather some information from universities to see what others are doing to help gauge what we might decide to do.  

 

1.       What OS is being used?

2.       How does your environment handle PHP Sessions?

3.       Can the environment be scaled?

4.       How do you handle upgrades (different environments?)

5.       We are also going to be using MariaDB in a different clustered environment and we know there some caveats that are tied to that like every table must have a uniqueID.  Do you know if apache has any of those caveats as well?

6.       If anyone has implemented this type of setup and are willing to hop on a phone call, we would definitely appreciate it. 



You are currently subscribed to [hidden email].
To unsubscribe send an email to [hidden email] with the subject line unsubscribe uwebd. Leave the message body blank. If you experience problems, contact list owner Eric Kreider at [hidden email]. More information is available at: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.uakron.edu_webteam_university-2Dweb-2Ddevelopers.dot&d=CwIDaQ&c=R_4lvufQDu25rflw2-oxk1ta8gMshLx4NEAyywNmS2w&r=Ytr4KhsDS3k7STCIzOnd3ZTaVYk6usDlnyh2YuwKckk&m=2imTmGQcNo2eA0WySaK4aEn-qkXw_eCPWjCtmU56-Jc&s=_d8QYNvLgt_ARyjbnbyry6f0UJD0_KdhCuvisMGaqcI&e=



You are currently subscribed to [hidden email].
To unsubscribe send an email to [hidden email] with the subject line unsubscribe uwebd. Leave the message body blank. If you experience problems, contact list owner Eric Kreider at [hidden email]. More information is available at: http://www.uakron.edu/webteam/university-web-developers.dot




Reply | Threaded
Open this post in threaded view
|

RE: Load balancing apache cluster environment

Jason R Pitoniak

Same thing at RIT as well.  We actually have several load-balanced systems, but I’ll highlight our web hosting environment, since that is the closest match to what you seem to want.

 

We have two data centers on campus.  Everything is mirrored in both, so that if one entire DC goes down, the web environment should stay up.  All file-based data is mirrored between two NFS servers (I don’t know what technology is being used to do the mirroring), including the PHP session data.  Only one NFS server can be active at a time, so the passive one constantly monitors the active one and, if it sees the active node malfunctioning, it takes over automatically.  The front-end nodes run Apache and mount the NFS server to get their content.  In addition to storing PHP session data in NFS, the load balancer uses sticky sessions based on IP (once an IP hits a server, all future requests are directed to the same server if it is available).  These sticky sessions are configured to time out after 20 minutes, I belive.

 

We also use a MariaDB Galera cluster in this environment.  The biggest gotchas with this setup are that only InnoDB tables can be replicated (or at least this was the case when we implemented ours) and that auto-incremented unique IDs are not contiguous. When we first launched our cluster, we had a few crashes because of existing MyISAM tables that were imported from our old stand-alone MySQL server.  One of our engineers created a patch to block the creation of MyISAM tables altogether.  Most people don’t even notice it’s there, but every once in a while some no-name CMS installer will crash because it tries to create MyISAM tables, though it’s usually an easy problem to fix.  The non-contiguous numbers are something that people (usually students) like to complain about, usually in the form of a ticket reading “please fix the MySQL server so that it counts correctly.” J

 

As Dan noted, upgrades are a breeze…just take one node out of the balancer, patch it, return it to the balancer, and then move on to the next.

 

I’m not a sys admin (though I play one on TV), but I’d be happy to talk a bit more if you have any burning questions.

 

--Jason

 

______________________________________

Jason R. Pitoniak '00

Technical Team Lead

Information and Technology Services

Rochester Institute of Technology

 

Email: [hidden email]

Phone: 585.475.4186

 

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information.

 

From: Charles Fulton [mailto:[hidden email]]
Sent: Tuesday, December 20, 2016 9:20 AM
To: [hidden email]
Subject: Re: [uwebd] Load balancing apache cluster environment

 

Diana,

 

We’ve implemented something very similar to what Dan has done. The main difference is that we’re using Memcache for session handling, Red Hat Storage for shared uploads, and MariaDB’s Galera database cluster (very similar to Percona). Galera’s been stable and fault-tolerant; the only application that we couldn’t host on it was Moodle and that’s a known issue with Moodle’s database architecture. We’re also about to stand up Nginx Plus as a replacement for generic Nginx.

 

I’d also be happy to discuss our implementation.

 

Charles

 

--
Charles Fulton
Senior Web Applications Developer
Lafayette College

(610) 330-3046

 

On Dec 20, 2016, at 9:06 AM, C. Daniel Chase <[hidden email]> wrote:

 

Diana-

 

I've done this too... :-)

 

When we implemented our OU Campus installation at The University of Tennessee at Chattanooga (UTC) I architected a three-VM frontend, load-balanced behind an F5 BigIP. Here is a link to one of the original design docs with the architecture:

 

 

To answer your questions:

 

1) We were using Linux, Red Hat Enterprise Linux v6, which was eventually converted in-place to Oracle Linux due to better educational licensing costs. 

2) Session handling was done via a shared SAN mount point that was read-write and dedicated to sessions. 

3) Yes, this can easily be scaled and this was the concept behind the architecture. (I've lived through a big event on campus taking down our website--I was not going to let that happen again.)

4) Depending on what you mean by "different environments" we upgraded servers by simply taking one out of the load balancer pool, upgrading it, and adding it back, then doing the next...

5) The diagram doesn't show it, but parallel to this I implemented a MySQL cluster. Due to the age at the time of our F5, it was not capable of the complex iRule that was required to separate SQL reads from writes, so instead of distributing reads across the cluster as I originally intended, all traffic went to a Master. The replicas ended up being strictly for redundancy, instead of load. Today, the F5 has been upgraded and could do what I originally planned. However, I would recommend you consider using Percona Server, which is another MySQL compatible database, as they have a multi-master capability. So, you could easily distribute all reads & writes based on load.

 

Two options you should consider at this point:

1) Moving to the cloud. (Before I was offered early retirement from UTC, this is the plan I had in place. Accounting / political issues were the only hold-up.)

2) Using NGINX Plus (https://www.nginx.com/products/) as both load balancer & web server. We used NGINX for our web servers when we deployed the above architecture. If we did not have an F5 already, I would have deployed NGINX as the load balancer, potentially with a Varnish cache server in the middle.

 

I'd be happy to discuss more as needed. 

 

 

-Dan

 

C. Daniel Chase

Lead Web Developer

SUNY Empire State College

(518) 581-2238

 

 

On Mon, Dec 19, 2016 at 8:55 PM, Keeling, Diana <[hidden email]> wrote:

I know this probably isn't the best group, but I was wondering if anyone would be willing to help us out, or could forward this email to their system administrators to answer some questions.

 

We are looking into setting up a load balanced, high-availability apache cluster that runs PHP.  We are just wanting to gather some information from universities to see what others are doing to help gauge what we might decide to do.  

 

1.       What OS is being used?

2.       How does your environment handle PHP Sessions?

3.       Can the environment be scaled?

4.       How do you handle upgrades (different environments?)

5.       We are also going to be using MariaDB in a different clustered environment and we know there some caveats that are tied to that like every table must have a uniqueID.  Do you know if apache has any of those caveats as well?

6.       If anyone has implemented this type of setup and are willing to hop on a phone call, we would definitely appreciate it. 

 


You are currently subscribed to 
[hidden email].
To unsubscribe send an email to 
[hidden email] with the subject line unsubscribe uwebd. Leave the message body blank. If you experience problems, contact list owner Eric Kreider at [hidden email]. More information is available at: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.uakron.edu_webteam_university-2Dweb-2Ddevelopers.dot&d=CwIDaQ&c=R_4lvufQDu25rflw2-oxk1ta8gMshLx4NEAyywNmS2w&r=Ytr4KhsDS3k7STCIzOnd3ZTaVYk6usDlnyh2YuwKckk&m=2imTmGQcNo2eA0WySaK4aEn-qkXw_eCPWjCtmU56-Jc&s=_d8QYNvLgt_ARyjbnbyry6f0UJD0_KdhCuvisMGaqcI&e=

 



You are currently subscribed to [hidden email].
To unsubscribe send an email to [hidden email] with the subject line unsubscribe uwebd. Leave the message body blank. If you experience problems, contact list owner Eric Kreider at [hidden email]. More information is available at: http://www.uakron.edu/webteam/university-web-developers.dot




Reply | Threaded
Open this post in threaded view
|

Re: Load balancing apache cluster environment

Keeling, Diana
Thank you everyone for the replies!! We are currently on break till the new year but I'll be sure to get back with my group and discuss everything. I'm sure we will have more questions.

Thank You,
Diana Keeling

Sent from my iPhone. Random auto-corrects and typos are my special gift to you.

On Dec 20, 2016, at 5:30 PM, Jason R Pitoniak <[hidden email]> wrote:

Same thing at RIT as well.  We actually have several load-balanced systems, but I’ll highlight our web hosting environment, since that is the closest match to what you seem to want.

 

We have two data centers on campus.  Everything is mirrored in both, so that if one entire DC goes down, the web environment should stay up.  All file-based data is mirrored between two NFS servers (I don’t know what technology is being used to do the mirroring), including the PHP session data.  Only one NFS server can be active at a time, so the passive one constantly monitors the active one and, if it sees the active node malfunctioning, it takes over automatically.  The front-end nodes run Apache and mount the NFS server to get their content.  In addition to storing PHP session data in NFS, the load balancer uses sticky sessions based on IP (once an IP hits a server, all future requests are directed to the same server if it is available).  These sticky sessions are configured to time out after 20 minutes, I belive.

 

We also use a MariaDB Galera cluster in this environment.  The biggest gotchas with this setup are that only InnoDB tables can be replicated (or at least this was the case when we implemented ours) and that auto-incremented unique IDs are not contiguous. When we first launched our cluster, we had a few crashes because of existing MyISAM tables that were imported from our old stand-alone MySQL server.  One of our engineers created a patch to block the creation of MyISAM tables altogether.  Most people don’t even notice it’s there, but every once in a while some no-name CMS installer will crash because it tries to create MyISAM tables, though it’s usually an easy problem to fix.  The non-contiguous numbers are something that people (usually students) like to complain about, usually in the form of a ticket reading “please fix the MySQL server so that it counts correctly.” J

 

As Dan noted, upgrades are a breeze…just take one node out of the balancer, patch it, return it to the balancer, and then move on to the next.

 

I’m not a sys admin (though I play one on TV), but I’d be happy to talk a bit more if you have any burning questions.

 

--Jason

 

______________________________________

Jason R. Pitoniak '00

Technical Team Lead

Information and Technology Services

Rochester Institute of Technology

 

Email: [hidden email]

Phone: 585.475.4186

 

CONFIDENTIALITY NOTE: The information transmitted, including attachments, is intended only for the person(s) or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and destroy any copies of this information.

 

From: Charles Fulton [[hidden email]]
Sent: Tuesday, December 20, 2016 9:20 AM
To: [hidden email]
Subject: Re: [uwebd] Load balancing apache cluster environment

 

Diana,

 

We’ve implemented something very similar to what Dan has done. The main difference is that we’re using Memcache for session handling, Red Hat Storage for shared uploads, and MariaDB’s Galera database cluster (very similar to Percona). Galera’s been stable and fault-tolerant; the only application that we couldn’t host on it was Moodle and that’s a known issue with Moodle’s database architecture. We’re also about to stand up Nginx Plus as a replacement for generic Nginx.

 

I’d also be happy to discuss our implementation.

 

Charles

 

--
Charles Fulton
Senior Web Applications Developer
Lafayette College

(610) 330-3046

 

On Dec 20, 2016, at 9:06 AM, C. Daniel Chase <[hidden email]> wrote:

 

Diana-

 

I've done this too... :-)

 

When we implemented our OU Campus installation at The University of Tennessee at Chattanooga (UTC) I architected a three-VM frontend, load-balanced behind an F5 BigIP. Here is a link to one of the original design docs with the architecture:

 

 

To answer your questions:

 

1) We were using Linux, Red Hat Enterprise Linux v6, which was eventually converted in-place to Oracle Linux due to better educational licensing costs. 

2) Session handling was done via a shared SAN mount point that was read-write and dedicated to sessions. 

3) Yes, this can easily be scaled and this was the concept behind the architecture. (I've lived through a big event on campus taking down our website--I was not going to let that happen again.)

4) Depending on what you mean by "different environments" we upgraded servers by simply taking one out of the load balancer pool, upgrading it, and adding it back, then doing the next...

5) The diagram doesn't show it, but parallel to this I implemented a MySQL cluster. Due to the age at the time of our F5, it was not capable of the complex iRule that was required to separate SQL reads from writes, so instead of distributing reads across the cluster as I originally intended, all traffic went to a Master. The replicas ended up being strictly for redundancy, instead of load. Today, the F5 has been upgraded and could do what I originally planned. However, I would recommend you consider using Percona Server, which is another MySQL compatible database, as they have a multi-master capability. So, you could easily distribute all reads & writes based on load.

 

Two options you should consider at this point:

1) Moving to the cloud. (Before I was offered early retirement from UTC, this is the plan I had in place. Accounting / political issues were the only hold-up.)

2) Using NGINX Plus (https://www.nginx.com/products/) as both load balancer & web server. We used NGINX for our web servers when we deployed the above architecture. If we did not have an F5 already, I would have deployed NGINX as the load balancer, potentially with a Varnish cache server in the middle.

 

I'd be happy to discuss more as needed. 

 

 

-Dan

 

C. Daniel Chase

Lead Web Developer

SUNY Empire State College

(518) 581-2238

 

 

On Mon, Dec 19, 2016 at 8:55 PM, Keeling, Diana <[hidden email]> wrote:

I know this probably isn't the best group, but I was wondering if anyone would be willing to help us out, or could forward this email to their system administrators to answer some questions.

 

We are looking into setting up a load balanced, high-availability apache cluster that runs PHP.  We are just wanting to gather some information from universities to see what others are doing to help gauge what we might decide to do.  

 

1.       What OS is being used?

2.       How does your environment handle PHP Sessions?

3.       Can the environment be scaled?

4.       How do you handle upgrades (different environments?)

5.       We are also going to be using MariaDB in a different clustered environment and we know there some caveats that are tied to that like every table must have a uniqueID.  Do you know if apache has any of those caveats as well?

6.       If anyone has implemented this type of setup and are willing to hop on a phone call, we would definitely appreciate it. 

 


You are currently subscribed to 
[hidden email].
To unsubscribe send an email to 
[hidden email] with the subject line unsubscribe uwebd. Leave the message body blank. If you experience problems, contact list owner Eric Kreider at [hidden email]. More information is available at: https://urldefense.proofpoint.com/v2/url?u=http-3A__www.uakron.edu_webteam_university-2Dweb-2Ddevelopers.dot&d=CwIDaQ&c=R_4lvufQDu25rflw2-oxk1ta8gMshLx4NEAyywNmS2w&r=Ytr4KhsDS3k7STCIzOnd3ZTaVYk6usDlnyh2YuwKckk&m=2imTmGQcNo2eA0WySaK4aEn-qkXw_eCPWjCtmU56-Jc&s=_d8QYNvLgt_ARyjbnbyry6f0UJD0_KdhCuvisMGaqcI&e=

 


You are currently subscribed to [hidden email].
To unsubscribe send an email to [hidden email] with the subject line unsubscribe uwebd. Leave the message body blank. If you experience problems, contact list owner Eric Kreider at [hidden email]. More information is available at: http://www.uakron.edu/webteam/university-web-developers.dot






You are currently subscribed to [hidden email].
To unsubscribe send an email to [hidden email] with the subject line unsubscribe uwebd. Leave the message body blank. If you experience problems, contact list owner Eric Kreider at [hidden email]. More information is available at: http://www.uakron.edu/webteam/university-web-developers.dot