What I actually wanted to show in this post was a major version upgrade (from CDH4 to CDH5, for example). Unfortunately, CDH4 is no longer available for download (at least not directly from Cloudera). So instead I will do a minor upgrade, from my CDH 5.6.0 that is running on my lab cluster to the newly released 5.7.0
The upgrade is done in two steps, first we have to upgrade Cloudera manager itself along with its agents and then the whole CDH platform.
First, let’s take a look at the current version. In Cloudera manager go to support menu and click “about”. This will bring up the about screen that also shows the current version:
For this upgrade I followed the instructions from here: http://www.cloudera.com/documentation/enterprise/latest/topics/cm_ag_upgrade_cm5.html
First, we should stop some services like Hive and Oozie and remove Kafka if we are using it.
In my cluster I do not have Kafka and since this is for educational purpose only, I will stop the entire cluster.
Now, on the node which runs cloudera manager server, stop the server, agent and database:
[[email protected] ~]# service cloudera-scm-agent stop Stopping cloudera-scm-agent: [ OK ] [[email protected] ~]# service cloudera-scm-server stop Stopping cloudera-scm-server: [ OK ] [[email protected] ~]# service cloudera-scm-server-db stop waiting for server to shut down............................................................... failed pg_ctl: server does not shut down
Hmmm… something is wrong, embedded postgresql database will not stop.
We will have to force stop it. First of all, locate user cloudera-scm in /etc/passwd since this is the user who runs postgresql.
cloudera-scm:x:496:493:Cloudera Manager:/var/lib/cloudera-scm-server: /sbin/nologin
Change the last section to /bin/bash or any other shell of your choice to enable this user to login.
First we find the postgresql data directory:
cat /etc/init.d/cloudera-scm-server-db | grep DATA_DIR= DATA_DIR=/var/lib/cloudera-scm-server-db/data Then change to cloudera-scm user and do this: pg_ctl -D /var/lib/cloudera-scm-server-db/data -m immediate stop waiting for server to shut down.... done server stopped
The database stopped. Probably there were some clients connected. To make sure the db is ok, we will start it and shut it down again using the service (exit back to root user):
[[email protected] ~]# service cloudera-scm-server-db start DB initialization done. waiting for server to start.... done server started [[email protected] ~]# service cloudera-scm-server-db stop waiting for server to shut down.... done server stopped
it worked. Lets get back to our upgrade.
Note: This procedure only works if you’re using the embedded postgresql database. If you are using external database you should manually stop it now and start it later.
Make sure you have the cloudera yum repository setup in /etc/yum.repos.d:
[cloudera-manager] # Packages for Cloudera Manager, Version 5, on RedHat or CentOS 6 x86_64 name=Cloudera Manager baseurl=https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/5/ gpgkey =https://archive.cloudera.com/cm5/redhat/6/x86_64/cm/RPM-GPG-KEY-cloudera gpgcheck = 1"
Now upgrade the cloudera manager server:
Yum clean all yum update cloudera-manager-server cloudera-manager-daemons cloudera-manager-server-db-2 cloudera-manager-agent
Let yum download and install everything.
Now start the database and the cloudera manager server:
[[email protected] yum.repos.d]# service cloudera-scm-server-db start DB initialization done. waiting for server to start.... done server started [[email protected] yum.repos.d]# service cloudera-scm-server start Starting cloudera-scm-server: [ OK ]
The Cloudera manager service was upgraded succesfuly. Now we will upgrade the cloudera manager agents on all nodes.
Login to cloudera manager and you will immediately see the cloudera manager upgrade wizard (it may take a while until the scm-server is up and receiving connections, since it has to process some extra data from the upgrade):
Just follow the wizard which is basically the same one you used when you first installed CDH (if you used cloudera manager to install CHD).
At the end of the upgrade the nodes are inspected and the inspection raised a warning that not all component versions are identical across all hosts:
Underneath was a very long and complicated list of components. So I decided to ignore it for now, because my cluster was functioning just fine until now.
Then it says that some configuration parameters need to be changed, that’s OK, just let it change them:
And finally, you will discover that client component is stale for some components. Just click on the icon and then on “deploy client configuration”, let it change the configuration and restart.
Now we can check gain and see that the Cloudera manager version has changed.
In this post we finished upgrading Cloudera manager along with its agents. In the next post we will go on and upgrade CDH itself.