Wednesday, June 15, 2011

CIS(theta) 2010-2011 - Scaling the cluster! - Meeting XV - XVIII


Aim: 
Scaling the cluster! (New Step 5)

Attending: 
CIS(theta) 2010-2011: DavidG, HerbertK, JoshG, RyanH

Reading:
NA

I'm adding a new Step 5: Scaling the Cluster. This step, once we figured it out, was actually very simple but very time consuming!. It boils down to replicating Step 4 on every single node. We have 25 nodes with 50 cores, so this step took several meetings to debug everything (dang stupid mistakes)! As far as compilers are concerned, only the Master Node needs to be configured for mpif77, mpif90, mpicc and mpicc++ to work properly. So, we can compile once and copy many. So, Step 5, listed below, is mainly concerned with getting mpirun to work correctly on every node. 

This was our last meeting of the year, so this is our last update. Step 6 will have to wait for next year's CIS(theta) 2011-2012 team (GeorgeA, GrahamS, KennyK, LukeE) to actually produce something with this cluster. Congrats, CIS(theta) 2010-2011 (DavidG, HerbertK, JoshG, RyanH), you are the first team to actually get openMPI to work natively on our Ubuntu partitions! Thanx to CIS(theta) 2009-2010 (DevinB, JeremyA, SteveB) for getting us a head start on this project last year. We stand upon the shoulders of giants! Good job guys!


InstantCluster Step 1: Infrastructure
Make sure your cores have enough ventilation. The room has to have powerful air conditioning too. These two factors may seem trivial but will become crucial when running the entire cluster for extended periods of time! Also, you need to have enough electrical power, preferably with the cabling out of the way, to run all cores simultaneously. Don't forget to do the same with all your Ethernet cabling. We have CAT6E cables to support our gigE Ethernet cards and switches. We are lucky that this step was taken care of for us already!

InstantCluster Step 2: Hardware
You need up to date Ethernet switches plus Ethernet cards and cores as well as plenty of RAM in each Linux box. As stated above, our gigE LAN and switches were already setup for us. Also, we have 64bit dual-core AMD Athlons and our HP boxes have 750 MB of RAM. I'd rather 1 or 2 GB of RAM, but that will have to wait for an upgrade!

InstantCluster Step 3: Firmware
We wasted way too much time last year trying out all kinds of Linux distros looking for a good 64bit base for our cluster. This year we spent way too much time testing out different liveCD distros. Recently, we downgraded from 64bit Ubuntu 10.04 Desktop edition to the 32bit version on our Linux partitions. 64bit gives us access to more RAM and a larger maxint, but was proving to be a pain to maintain. Just to name one problem, jre and flash were hard to install and update on FireFox. Last year we tried Fedora, Rocks, Oscar, CentOS, Scientific Linux and, finally, Ubuntu. 32bit Ubuntu has proven very easy to use and maintain, so I think we'll stick with it for the cluster! We've done this several times over the years using everything from Slakware and KNOPPIX to Fedora and Ubuntu!

Current Ubuntu version:

http://shadowfaxrant.blogspot.com/2011/05/2-so-many-hard-drives-so-little-time.html


InstantCluster Step 4: Software Stack
On top of Ubuntu we need to add openSSH, public-key authentication and openMPI. In step 6 we can discuss an application to scatter/gather over the cluster whether it be graphical (fractals, povray, blender, openGL, animations) or number crunching (C++ or python app for Mersenne Primes or Beal's Conjecture). So, what follows is a summary of what we did to get up to plublic-key authentication. This summary is based on the http://cs.calvin.edu/curriculum/cs/374/MPI/ link listed below. First, we installed openSSH-server from http://packages.ubuntu.com using the proxy server, then:
  1. If you have no .ssh directory in your home directory, ssh to some other machine in the lab; then Ctrl-d to close the connection, creating .ssh and some related files. 
  2. From your home directory, make .ssh secure by entering:
    chmod 700 .ssh
  3. Next, make .ssh your working directory by entering:
    cd .ssh
  4. To list/view the contents of the directory, enter:
    ls -a [we used ls -l]
  5. To generate your public and private keys, enter:
    ssh-keygen -t rsa
    The first prompt is for the name of the file in which your private key will be stored; press Enter to accept the default name (id_rsa).The next two prompts are for the password you want, and since we are trying to avoid entering passwords, just press Enter at both prompts, returning you to the system prompt.
  6. To compare the previous output of ls and see what new files have been created, enter:
    ls -a [we used ls -l]
    You should see id_rsa containing your private key, and id_rsa.pub containing your public key.
  7. To make your public key the only thing needed for you to ssh to a different machine, enter:
    cat id_rsa.pub >> authorized_keys
    [The Linux boxes on our LAN, soon to be cluster, have IPs ranging from 10.5.129.1 to 10.5.129.24 So, we copied each id_rsa.pub file to temp01-temp24 and uploaded these files to our ftp server. On the ftp server we just ran cat tempnn >> authorized_keys for each temp file to generate one master authorized_keys file for all nodes that we could just download to each node's .ssh dir.]
  8. [optional] To make it so that only you can read or write the file containing your private key, enter:
    chmod 600 id_rsa
  9. [optional] To make it so that only you can read or write the file containing your authorized keys, enter:
    chmod 600 authorized_keys
We then installed openMPI (see dependencies below) and tested multi-core with flops. Testing the cluster as a whole will have to wait until the next meeting! We followed openMPI install instructions for Ubuntu from http://www.cs.ucsb.edu/~hnielsen/cs140/openmpi-install.html


These instructions say to use sudo and run run apt-get install openmpi-bin openmpi-doc libopenmpi-dev However, the way our firewall is setup at school, I can never update my apt-get sources files properly. So, I used http://packages.ubunutu.com to look up these files and got the following dependencies!


If you think that's bad, look at all the dependencies I ran into to install VLC on Ubuntu Maverick Meerkat 10.04 which was required to view *.mp4 files and to instrall handbrake on the Teacher Station!


We finally got all this working (sans vlc which is only on my Teacher station). Then we used the following FORTRAN code to test multi-core. FORTRAN, really? I haven't used FORTRAN77 since 1979! ...believe it or don't!


we compiled on the Master Node flops.f after installing gfortran too:
mpif77 -o flops flops.f

and tested openmpi and got around 900 MFLOPS using 2 cores:
mpirun -np 2 flops

Next, we generated a "machines" file to tell mpirun where all the nodes (Master and Workers) are (2 nodes with 2 cores each for example):
mpirun -np 4 --hostfile machines flops

InstantCluster Step 5: Scale the Cluster!
The focus here is to get mpirun working on every node. I don't know if you have to set up every node exactly the same way, but everything finally worked when we did! IE: I didn't think we needed the machines file on every box. Here's what we did on each node:


as root on every node:
install openmpi
install openssh
adduser shadowfax

as shadowfax on Master node:
create machines file and copy /home/shadowfax/machines to ftp site
compile flops.f and copy /home/shadowfax/flops to ftp site

as shadowfax on every node:
create /home/shadowfax/.ssh
run ssh-keygen -t rsa
cat id_rsa.pub tempXX (XX = 1-25 = node number)
copy all tempXX files to ftp site and cat tempXX >> authorized_keys 
copy from ftp site to /home/shadowfax/flops 
chmod 755 flops
copy from ftp site to /home/shadowfax/machines
copy from ftp site to /home/shadowfax/.ssh/authorized_keys

Note: uniformity is the key! This appoarch is very convenient as, with just a little bit of tweaking of the "machines" and "authorized_keys" files as well as the command line, any box can be the Master node and you can use as many or as few of the nodes as you want! 


Every node has the same "machines" text file in /home/shadowfax listing all the IPs, one per line. Every node has the same "flops" executable file (or whatever your executable will be) in /home/shadowfax. Every node has the same "authorized_keys" text file with all 25 keys in /home/shadowfax/.ssh


InstantCluster Step 6: Running Apps!
This step will have to wait for next year. Thanx to the hard work of the last 2 years' CIS(theta) teams, we finally know how to setup an openMPI cluster on our Linux partitions: no more 64bit, no more liveCDs, no more hunting for a new distro! Next year's team can redo steps 1-5 to learn how all this works with a new version of Ubuntu (bye bye Lucid Lemur and Maverick Meerkat, hello Natty Narwhal or Oneiric Ocelot). Than we can finally play with fractals, povray, tachyon, blender, animations and number crunching!


Research: 
openSSH and openMPI
Parallel Python
IPython

Well, that's all for now, enjoy!

Posted by at No comments:
Labels: , , ,

Saturday, June 11, 2011

Is there life after Calculus?

Screening all 9 hours of LOTR!!!

Yes, there is life after my AP Calculus BC class! This is by far my favorite time of year. This "season," for want of a better term, starts with LIMACON around the ides of March and ICON around the nones of April. Of course, we also start AP Review around this time followed by AP Exam Week during the calens of May! The AP Calculus exams are usually given early during the 2 AP Weeks, so we have a lot of time to kill after Calculus! 

View of my classroom from left rear to left front side.

As a result, Life After Calculus, aka LAC, has evolved into an art form. Right after the AP Exam, we have a Movie Marathon. Traditionally, our Movie Marathon, starts with "Stand and Deliver," but I couldn't bear to watch that movie this year as Jaime Escalante passed away not so long ago. We usually show one or two other movies to do with math, such as "A Beautiful Mind" or "The Proof." In recent years, we've been doing Science Fiction trilogies like "Star Wars" or "Planet of the Apes," etc. I justify this odd behavior by naming each and every TI Graphing Calculator we've ever used after a scifi character.

View of my classroom from right rear to right front side.

TI81 = Obi Wan Kenobi (Star Wars IV)
TI82 = Klaatu (The Day The Earth Stood Still)
TI83 = Ziggie (Quantum Leap TV series)
TI84 = Joshua (Wargames)
TI85 = Johnny5 (Short Circuit)
TI86 = Spock (Star Trek TV series)
TI89 = Hal 9000 (2001: A Space Odysee)
TI92 = Colossus (The Forbin Project)

I use names of characters from these movies and TV series. So, I claim that I'm still looking for a good name for the latest GC, the TI Inspire CAS. For example, last year we saw all 3 versions of "I Am Legend" (Vincent Price, Charleton Heston and Will Smith) and decided to name the Inspire after Dr. Neville! The year before that, I was still looking for a good name for the Inspire, so we viewed "Back to the Future" and chose the name Doc Brown, but it didn't stick. This year, we went crazy and decided to watch all 9 hours of "The Lord of the Rings!" We viewed 2 DVDs worth of LOTR during AP Week and saved the last DVD for these last few days of school. Maybe we'll rename the Inspire as "Nazgul?"

In between these 2 screenings of LOTR, we actually managed to do a lot of math! What follows is a record of some of the topics we covered. We spent some time, LAC Days 1-8, on loose ends in BC Calculus previewing some topics in Calc III and IV. Then I showed my students how to use a bit of SAGE, http://www.sagemath.org 

Life aftercalcday1
View more presentations from A Jorge Garcia
"10 Trig Identities you must know cold, from Pythagoras to Euler!" 


Life aftercalcday2
View more presentations from A Jorge Garcia
"Everything you ever wanted to know about Complex Exponents using Euler's Identity!" 


Life aftercalcday3
View more presentations from A Jorge Garcia
"Second Order DiffEqus modeling springs without damping!" 


Life aftercalcday4
View more presentations from A Jorge Garcia
"Second Order DiffEqus modeling springs with damping!"  


Life aftercalcday5
View more presentations from A Jorge Garcia
"Second Order DiffEqus modeling springs, the Complex case!"  


Life aftercalcday6
View more presentations from A Jorge Garcia
"Solving DiffEqus without Anti-Differentiating?: 


Life aftercalcday7
View more presentations from A Jorge Garcia
"Solving Simultaneous DiffEqus modeling competing populations using Slope Fields!" 


Life aftercalcday8

Here's some screen-casts of these SAGE tutorials!







What do ya think? 
Teaching with Technology, 
Posted by at No comments:
Labels: , , , , , , , , ,

Wednesday, June 8, 2011

Chief Reader Rosenstein vs. Betty Sue, who wins?


Thanx, Andrew, for a great rendition of "Our Chief Reader, Rosenstein" by Dan Kennedy. I mentioned this Calculus Song, Parody, Filk, or whatever you want to call it, on the AP Calculus ListServ a few months ago as I had lost the words. The song was about the solution to AP Calculus AB Free Response Question 88AB6:


Let f be a differentiable function, defined for all real numbers x, with the following properties
(i)   f '(x) = ax^2 + bx
(ii)  f '(1) = 6 and f "(1) = 18
(iii) int(f(x),x,1,2) = 18
Find f(x). Show your work.



I finally found the lyrics sheet I was looking for in my desk, so my students have had the dubious honor of singing this song in class! Since I mentioned it on the ListServ, I have been asked several times for the words, but I haven't had a chance to share them until now. What follows are the long, sought-after lyrics:

"Our Chief Reader, Rosenstein" by Dan Kennedy
Sung to the tune of "My Darling, Clementine"

Peerless leader, dirty deeder, our Chief Reader, Rosenstein,
You're our hero, you gave zero to a kid we thought was nine!

In a classroom, in Topeka, on the 13th day of May,
sat the class of Mrs. Bentley, for their AP Judgment Day!

After months of AP HomeWork, Betty Sue was well prepared,
she moved quickly through the problems, never nervous, never scared!

AB6 was tough for some, but Betty Sue was feeling fine,
as she smugly wrote her answer, she was sure she had a nine!

Then the tests were wrapped and bundled and were sent to ETS,
where a group of expert graders would do justice, more or less!

In the world of Mathematics, sometimes things aren't meant to be,
so, alas, at George's briefing, Betty Sue was Sample G!

All the readers at the briefing gave her 9s with pleasure when,
they could see she had four x cubed minus 3 x squared plus ten!

George was wincing with displeasure, and his head just gently shook.
Then he sighed, "That's not what I had, so let's take a closer look!"

"In the third line, she had f prime where she wants to integrate.
Then she never quite recoups it, so she can't get more than 8!

"In the 4th line, she miscopies, and a 12 becomes 11.
I'll admit that it's a pity, but her 8 just fell to 7!

"Now she sees that a's 11, but her a and b don't mix.
So, she changes b to match it, that's too bad, she's down to 6!

"Then she puts those terms together, for a function to derive.
But a plus becomes a minus, so she drops from 6 to 5!

"Then she integrates quite nicely, but we dock the kid once more.
For her C is clearly missing, so her 5 is down to 4!

"In the next line, by some magic, she is solving for a C.
But she never justifies it, so we drop her score to 3!

"If you closely check her limits, where the integral is through.
You will see she had them backwards, so at most she gets a 2!

"Now her error, if consistent, could not yield the constant 10.
So her right C is the wrong C, and we burn her once again!

"If she puts it all together, this poor girl could still get one.
But recall, she's lost a plus sign, now it's back, so she gets none!"

HTH, I hope you enjoyed our little filking interlude!

Learning with Technology, 
Posted by at No comments:
Labels: , , , , ,
Subscribe to: Posts (Atom)