Showing posts with label openmosix. Show all posts
Showing posts with label openmosix. Show all posts

Thursday, November 28, 2013

CIS(theta) 2013-2014 November Meeting II Firing up the cluster for the first time!

CIS(theta) 2013-2014 
November Meeting II
Firing up the cluster for the first time!
We are trying to figure out if we can use a bootable Linux CD to boot up the cluster or if we want to add an MPI stack with public key authenticated sshd to our Ubuntu Desktop. So, we downloaded the pelicanHPC ISO, burned a DVD and got 17 compute nodes or 68 cores running at about 35 GFLOP/s! 

Hey, that's not too shabby, just 13 years ago, top500.org reports that all you needed were 50 cores running at 55 MFLOP/s to make the list of one of the 500 fastest clusters in the world! We may even get up to 50 MFLOP/s if we get all 100 cores up and running. 

So, we're looking at pelicanHPC, BCCD and ClusterByNight for MPI. We used to run clusterKNOPPIX, BCCD and Quantian with openMOSIX. Too bad openMOSIX isn't supported anymore. It was so easy to code for MOSIX. In C or C++ all you had to do was use the fork() function to start a new process. openMOSIX ran in the Linux Kernel. Whenever your processes came close to using 100% of you CPU, MOSIX would automagically send one of your forked jobs to another processor. When we started using MOSIX, we only had one core per PC, so that meant send the job to another PC on the LAN. In those days (late 1990s) we managed to run 100 cores anyway by taking over 4 PC Labs on my floor at school....



November Meeting I
Last Linux Install Fest!
We finally got the new 64bit Ubuntu 13.10 Saucy Salamader installed correctly on all student PCs. I'll upgrade my station at a later date. We got this down to a science: 
(1a) download latest ISO and burn 8 DVDs
(1b) install DVD with plain vanilla defaults
(2) edit eth0 connection
(3a) edit system settings/network/proxy
(3b) edit system settings/appearance
(3c) edit system settings/brightness
(3d) edit system/settings/printers
(4) edit sources to main in USC
(5a) sudo apt-get update
(5b) sudo apt-get upgade
(6a) USC install WINE and TIEMU
(6b) add TI89 and TI92 ROMs to TIEMU
(7a) download, extract and install TilEm
(7b) add TI83 and TI84 ROMs to TilEm
(8a) downlaod and extract SAGE.lzma
(8b) echo "alias sage=$HOME/Desktop/SAGE/sage" >> ~./bashrc
(8c) source ~/.bashrc
(9) all we left on the desktop was the SAGE directory, a short cut to TilEm and our handy dandy Trig Table (see below)
(10a) to use TilEm, the students just click on the short cut
(10b) to use TIEMU and SAGE open a terminal and type "tiemu" or "sage" on a command line.
(11) I'll add TilEm and TIEMU to the start-up menu. I was going to add "sage -n) too, but I don't want FireFox hogging all my Desktops at bootup!

October Meeting II
Linux Install Fest Again!
I downloaded 8 copies the latest version of Ubuntu Desktop 32bit ISO: 13.10 Saucy Salamander. Then I burned 8 DVDs and reinstalled the Linux partition on the second row of 8 PCs in our classroom. The geek squad did all the tweaks this time! 

We are having a massive problem with VTI running under WINE. It's so slow as to be unusable. We have to figure out a solution. Maybe 13.04 is better? Or maybe I can get WINE's direct ppa and use their latest version?

Another problem is that we have a SAGE directory on the Desktop extracted from the latest compiled version of SAGE. In there we have a SAGE bash script. I usually make it executable and students can just click it to run SAGE. In 13.10, the script will not execute, the script just opens in gedit even though I made the file executable. I even ran "which bash" to see if bash moved (it did) so I could update the #! line but still no joy!

How about an alias like:
echo "alias sage=$HOME/Desktop/SAGE/sage" >> ~/.bashrc

then:
source ~/.bashrc

October Meeting I
Linux Install Fest!
We downloaded 8 copies the latest version of Ubuntu Desktop 64bit ISO: 13.10 Saucy Salamander. Then we burned 8 DVDs and reinstalled the Linux partition on the first row of 8 PCs in our classroom. I did all the tweaks the following day.

Next week I'll reinstall the 2nd row and the Geek Squad will do all the tweaks. I like to keep the Student Stations very simple. So, we will only tweak as follows:
(1) configure Network (eth0, proxy)
(2) configure Appearence and Brightness 
(3) configure Printers
(4) sudo apt-get update (after setting sources to main).
(5) sudo apt-get upgrade
(6) install WINE
(7) copy VTI to the desktop
(8) extract SAGE to the desktop

September Meeting I
Administrativa!
(1) Wreath of the Unknown Server: We visited our first ssh server, Colossus, which is still in the switch room though dormant. I set it up for the first time in 1995 running Slackware Linux. Colossus ran for 12 years straight, 24x7 never having to shut down, reboot or even have anything re-installed!

(2) Display Case Unveiled: We took down a ton of fractal prints and ray tracings from Room 429 to the 2 cases on the 1st floor near the art wing. We decorated both cases as best we could and left before anyone saw us. Must have been gremlins.

(3) Recruiting: We decided that we have more than enough qualified CIS(theta) members for this year's Geek Squad, so we tabled that topic.

(4) Planning: We have to wait another 2 weeks 10/4 at which point Ubuntu 13.10 Desktop Edition should be available for a mini install fest. After that, we may use bootable cluster Linux CD distros such as BCCD and pelicanHPC to learn MPI using C++ or Python. We also talked about installing an MPI stack on each hdd along with public key authenticated ssh. We would like to make a fractal zoom animation.

(5) Summary: This year's CIS(theta) team is off to a good start. Shadowfax, our 100 core cluster, is in good hands!
==================================
What we are researching IV (Jan):MPI
==================================
What we are researching IV (Dec):MOSIX
==================================
What we are researching III (Nov)
==================================
What we are researching II (Oct)
==================================
What we are researching I (Sept)
Thomas Jefferson High courses
Thomas Jefferson High paper
Thomas Jefferson High ftp
Thomas Jefferson High teacher
==================================
Daily Attendance:
CIS(theta) 2013-2014: Taty absent

BiWeekly Topic:
CIS(theta) 2013-2014 - Firing up the cluster! 

Monthly Reading:
Chapter 5: Building Parallel Programs (JAN)
Chapter 4: Building Parallel Programs (DEC)
Chapter 3: Building Parallel Programs (NOV)
Chapter 2: Building Parallel Programs (OCT)
Chapter 1: Building Parallel Programs (SEP)
==================================
Membership (alphabetic by first name):

CIS(theta) 2013-2014: BryanS, CheyenneC, DanielG, HarineeN, RichardH, RyanW, TatianaR, TylerK

CIS(theta) 2012-2013: Kyle Seipp

CIS(theta) 2011-2012: Graham Smith, George Abreu, Kenny Krug, LucasEager-Leavitt

CIS(theta) 2010-2011: David Gonzalez, Herbert Kwok, Jay Wong, Josh Granoff, Ryan Hothan

CIS(theta) 2009-2010: Arthur Dysart, Devin Bramble, Jeremy Agostino, Steve Beller

CIS(theta) 2008-2009: Marc Aldorasi, Mitchel Wong

CIS(theta) 2007-2008: Chris Rai, Frank Kotarski, Nathaniel Roman

CIS(theta) 1988-2007: A. Jorge Garcia, Gabriel Garcia, James McLurkin, Joe Bernstein, ... too many to mention here!
==================================
CIS(theta) DropBox!
==================================
Well, that's all folks!

Friday, October 25, 2013

CIS(theta) 2013-2014 October Meeting I Linux Install Fest!




CIS(theta) 2013-2014 
October 
Meeting I
Linux Install Fest!
 
We downloaded 8 copies the latest version of Ubuntu Desktop 64bit ISO: 13.10 Saucy Salamander. Then we burned 8 DVDs and reinstalled the Linux partition on the first row of 8 PCs in our classroom. I did all the tweaks the following day.

Next week I'll reinstall the 2nd row and the Geek Squad will do all the tweaks. I like to keep the Student Stations very simple. So, we will only tweak as follows:

(1) configure Network (eth0, proxy)
(2) configure Appearence and Brightness 
(3) configure Printers
(4) sudo apt-get update (after setting sources to main).
(5) sudo apt-get upgrade
(6) install WINE
(7) copy VTI to the desktop
(8) extract SAGE to the desktop

September: Administrativa!
Meeting I

(1) Wreath of the Unknown Server: We visited our first ssh server, Colossus, which is still in the switch room though dormant. I set it up for the first time in 1995 running Slackware Linux. Colossus ran for 12 years straight, 24x7 never having to shut down, reboot or even have anything re-installed!

(2) Display Case Unveiled: We took down a ton of fractal prints and ray tracings from Room 429 to the 2 cases on the 1st floor near the art wing. We decorated both cases as best we could and left before anyone saw us. Must have been gremlins.

(3) Recruiting: We decided that we have more than enough qualified CIS(theta) members for this year's Geek Squad, so we tabled that topic.

(4) Planning: We have to wait another 2 weeks 10/4 at which point Ubuntu 13.10 Desktop Edition should be available for a mini install fest. After that, we may use bootable cluster Linux CD distros such as BCCD and pelicanHPC to learn MPI using C++ or Python. We also talked about installing an MPI stack on each hdd along with public key authenticated ssh. We would like to make a fractal zoom animation.

(5) Summary: This year's CIS(theta) team is off to a good start. Shadowfax, our 100 core cluster, is in good hands!
==================================
What we are researching II (Oct+Nov)
MPI, PVM, MOSIX
Fractals, PovRay, Blender
==================================
What we are researching I (Sept)
Look what this school did in the 80s: 
Thomas Jefferson High courses
http://academics.tjhsst.edu/compsci/parallel
Thomas Jefferson High paper
http://www.tjhsst.edu/~rlatimer/techlab07/BWardPaperQ3-07.pdf
Thomas Jefferson High ftp
http://www.tjhsst.edu/~rlatimer/techlab07
Thomas Jefferson High teacher
http://www.tjhsst.edu/~rlatimer
==================================
Today's Topic:
CIS(theta) 2013-2014 - Linux Install Fest! - Meeting 1
Today's Attendance:
CIS(theta) 2013-2014: 100%
Today's Reading:
Chapter 2: Building Parallel Programs (BPP)
==================================
Membership (alphabetic by first name):
CIS(theta) 2013-2014: BryanS, CheyenneC, DanielG, HarineeN, RichardH, RyanW, TatianaR, TylerK

CIS(theta) 2012-2013: Kyle Seipp
CIS(theta) 2011-2012: Graham Smith, George Abreu, Kenny Krug, LucasEager-Leavitt
CIS(theta) 2010-2011: David Gonzalez, Herbert Kwok, Jay Wong, Josh Granoff, Ryan Hothan
CIS(theta) 2009-2010: Arthur Dysart, Devin Bramble, Jeremy Agostino, Steve Beller
CIS(theta) 2008-2009: Marc Aldorasi, Mitchel Wong
CIS(theta) 2007-2008: Chris Rai, Frank Kotarski, Nathaniel Roman
CIS(theta) 1988-2007: A. Jorge Garcia, Gabriel Garcia, James McLurkin, Joe Bernstein, ... too many to mention here!
==================================




Well, that's all folks!








Tuesday, September 6, 2011

CIS(theta) 2011-2012 - Introductory Meeting - Meeting I

What was discussed:
We've been building clusters for some time now! We used PVM in the late 1990s. Until recently we were using openMOSIX, but that project closed. Lately we've been trying to get openMPI to work with out new multi-core 64bit hardware.

Once we got a cluster running, we have done some Large Integer number crunching (Beal, Mersenne) as well as graphics applications such as Fractals vs Ray Tracing (povray). This year, we may try our hand at animation (blender). We decorated the room and a display case with some framed zazzle prints of fractals and ray tracings.

What was accomplished:
All my computers were replaced last week. School started this week. We didn't have a lot of time to install anything. So, here's a minimal install for my students to get some work done! We tried the KISS (Keep It Simple Stupid) every Linux box. So, we didn't install SAGE/JRE or WINE/VTI on the Student Stations. I may add VTI if they need it. We are running SAGE online.
(1a) Installed 32bit Ubuntu Desktop 10.10 on all 24 Student Stations (KISS plain vanilla install)
(1b) Installed 32bit Ubuntu Desktop 10.10 on both Servers (password login enabled)
(1c) Installed 32bit Ubuntu Desktop 10.10 on the Teacher Station (password login enabled)
(2) Dissabled the ScreenSaver and enabled Ethernet (10.5.129.1-10.5.129.25, 255.255.0.0, 10.5.0.254, 10.1.1.19) + Printing on all stations (10.10.254.212)

(3) Servers also got:
(3a) vsFTP 
(3b) openSSH

(4) Teacher Station also got (in addition to configuring a slow mouse and bluetooth for my tablet):
(4a) Flash to watch YouTubes
(4b) Chrome for eSchool (only need VLC for HandBrake and DVDlib, Chrome plays *.mp4s )
(4c) Xournal pen app for tablet (will add SmartNotebook later)
(4d) WINE to run Virtual TI (VTI)
(4e) didn't have time for JRE (3D graphs in SAGE + screencast-o-matic) but we will...

UPDATES:
(1) I tried Xournal on the Teacher Station for a while as a pen app but using the tablet it was really hard to see the pointer! My SmartBoard is still down, I couldn't use Xournal on there. So, I went to http://www.smarttech.com and tried http://express.smarttech.com which is an online SmartNotebook. This solution has the opposite problem. You get a pen and a mouse pointer at the same time! That's too cumbersome to use. So, I broke down and installed the Linux version of SmartNotebook 10.2 which comes as a *.gz file. I extracted that file and found a pile of *.debs to install in a directory called dists/stable/multiverse. All is well with the world of SmartNotebook!
(2) I installed JRE 1.6.0_27 in Firefox and it automagically works in Chrome as well. This way I can do animated 3D graphs in SAGE with viewer='jmol' which is the default in show(). The Student Stations can do 3D graphs in SAGE but they need to use .show(viewer='tachyon') and they don't get to animate the graphs. So, I have JRE joy!
(3) I tried also installed JRE so as to get http://screencast-o-matic.com (SOM) working. SOM captures the video of my screencast and renders *.mp4s for YouTube.
(4) for more detailed installation instructions please see, http://shadowfaxrant.blogspot.com/2011/05/2-so-many-hard-drives-so-little-time.html

What we are researching (look what this school did in the 80s and 90s): 

Today's Topic: 
CIS(theta) 2011-2012 - Introductory Meeting - Meeting I

Today's Attendance: 
CIS(theta) 2011-2012: GeorgeA, GrahamS, KennyK, LucasE

Today's Reading:
Chapter 1 Building Parallel Programs (BPP) using clusters and parallelJava

Well, that's enough food for thought, enjoy!

Sunday, January 16, 2011

CIS(theta) Meeting IX (2010-2011) - try, try, try, try, try, try, ...., again!!!


Aim: 
pelican HPC 2.2 64bit: If at first you don't succeed, try, try, try, try, try, try, ...., again!!!

Attending: 
CIS(theta) 2010-2011: DavidG, HerbertK, JoshG, RyanH

Reading: 
pelican HPC CD 64bit version 2.2: /home/usr/*
pelican HPC CD 64bit version 2.2: /usr/bin/pelican*

http://pelicanhpc.788819.n4.nabble.com/pelicanHPC-ReMaster-td3163841.html

It's been a while since I blogged, but that's just because we've been busy trying out several Linux liveCD distributions for setting up High Performance Clusters!  As you will see in the nabble link above, we have a thread on the http://www.pelicanhpc.org forum regarding our efforts in getting pelicanHPC working on our LAN even though we have tons of DHCP conflicts.  We also tried out Cluster By Night which does not suffer from the DHCP problem as it sets up public key authenticated ssh to connect nodes which works right out of the box!  I'm also revisiting BCCD as detailed below.

BCCD Update
I just recently revisited the Bootable Cluster CD project sponsored by the University of Northern Iowa.  Years ago, when Paul Gray started the project, he was very helpful to my students in setting up an openMOSIX cluster using BCCD.  This was back in the day when clusterKNOPPIX and QUANTIAN were the only other competitors.  Now, Paul is not so intimately involved with the project.  Take a look at the BCCD links listed above.  You will see the main site, the site for version 3.0 and the developers' forum to which I have contributed a little in December and January.  So, now it seems there's a team of developers involved from the littleFe project to Earlham and Contra Costa Colleges.

Last week I downloaded their latest ISO (3.0 for 64bit).  The problem with this distro is that its slightly too big for a CD, so I had to burn a DVD.  Another issue is that when BCCD boots, it copies itself to RAM.  An ISO file has some compression associated with it, so the 800MB ISO expands to over 1GB in RAM.  I cannot use this ISO as my worker nodes, 64bit Athlon dual-cores, only have 750MB RAM.  Next step will be to try the 32bit ISO which is smaller.  In fact, I did try the 32bit version on my Xeon dual-cores and the system seems to boot all the way but then fails just when the X windowed desktop should appear.  I'll try to boot with the linux 4 cheat code next time.



Cluster by Night Screencast from Dept. Dirigible Flightcraft on Vimeo.
Cluster By Night Update
Cluster By Night is proving to be very interesting.  Kevin Lynagh created 2 ISOs for 32bit arch.  One boots the Master and then each Worker node uses a separate ISO.  So, we burned 1 master and 7 workers to test out CbN.  Each ISO is only about 15MB so its a fast download, burn and boot!  You get a  full openMPI installation and you can boot up the cluster using public key authenticated ssh in a mater of minutes.  There are sample apps on CbN but not much source code or dox.  So, we booted up a 32bit pelicanHPC 2.2 CD and compiled flops.f using mpif77 -o flops flops.f since doing so in CbN causes errors.  It would seem that the openMPI installation on CbN is incomplete.  At least gfortran is missing.  Then I copied the executable program flops to the CbN master node from sftp user@10.10.129.21 as pelicanHPC which was running on worker node #21.  Then we ran cbn_run -np 16 flops from the master and got 8 nodes, 16 cores running at nearly 7GFLOPS!  Again, openMPI is incomplete as using mpi_run causes a ton of errors too!  All in all, CbN is very promising. 


Pelican Update
We are focusing this week's readings on the files on a booted pelicanHPC 2.2 64bit CD.  We printed out all the files in /home/user (including pelican_config) and all the files starting /usr/bin/pelican* (system scripts for starting the cluster).  Michael Creel says we should be able to modify how pelicanHPC boots by editing pelican_config and putting it in a persistent home volume called PELHOME.  In this manner, we hope to disable pelican's DHCP server so as to use the existing DHCP service on eth0 already.  Once we eliminate the DHCP conflicts, we should be able to boot the worker nodes from the one master CD via PXE after all.  Wish us luck!

Well, that's all for this week!  Please stay tuned.  We hope this record will help others trying to make their own clusters.  Also, if you have any hints or pointers, please leave a comment to let us know.  

Happy Clustering,

CIS(theta) Meeting IX (2010-2011) - try, try, try, try, try, try, ...., again!!!


Aim: 
pelican HPC 2.2 64bit: If at first you don't succeed, try, try, try, try, try, try, ...., again!!!

Attending: 
CIS(theta) 2010-2011: DavidG, HerbertK, JoshG, RyanH

Reading: 
pelican HPC CD 64bit version 2.2: /home/usr/*
pelican HPC CD 64bit version 2.2: /usr/bin/pelican*

http://pelicanhpc.788819.n4.nabble.com/pelicanHPC-ReMaster-td3163841.html

It's been a while since I blogged, but that's just because we've been busy trying out several Linux liveCD distributions for setting up High Performance Clusters!  As you will see in the nabble link above, we have a thread on the http://www.pelicanhpc.org forum regarding our efforts in getting pelicanHPC working on our LAN even though we have tons of DHCP conflicts.  We also tried out Cluster By Night which does not suffer from the DHCP problem as it sets up public key authenticated ssh to connect nodes which works right out of the box!  I'm also revisiting BCCD as detailed below.

BCCD Update
I just recently revisited the Bootable Cluster CD project sponsored by the University of Northern Iowa.  Years ago, when Paul Gray started the project, he was very helpful to my students in setting up an openMOSIX cluster using BCCD.  This was back in the day when clusterKNOPPIX and QUANTIAN were the only other competitors.  Now, Paul is not so intimately involved with the project.  Take a look at the BCCD links listed above.  You will see the main site, the site for version 3.0 and the developers' forum to which I have contributed a little in December and January.  So, now it seems there's a team of developers involved from the littleFe project to Earlham and Contra Costa Colleges.

Last week I downloaded their latest ISO (3.0 for 64bit).  The problem with this distro is that its slightly too big for a CD, so I had to burn a DVD.  Another issue is that when BCCD boots, it copies itself to RAM.  An ISO file has some compression associated with it, so the 800MB ISO expands to over 1GB in RAM.  I cannot use this ISO as my worker nodes, 64bit Athlon dual-cores, only have 750MB RAM.  Next step will be to try the 32bit ISO which is smaller.  In fact, I did try the 32bit version on my Xeon dual-cores and the system seems to boot all the way but then fails just when the X windowed desktop should appear.  I'll try to boot with the linux 4 cheat code next time.



Cluster by Night Screencast from Dept. Dirigible Flightcraft on Vimeo.
Cluster By Night Update
Cluster By Night is proving to be very interesting.  Kevin Lynagh created 2 ISOs for 32bit arch.  One boots the Master and then each Worker node uses a separate ISO.  So, we burned 1 master and 7 workers to test out CbN.  Each ISO is only about 15MB so its a fast download, burn and boot!  You get a  full openMPI installation and you can boot up the cluster using public key authenticated ssh in a mater of minutes.  There are sample apps on CbN but not much source code or dox.  So, we booted up a 32bit pelicanHPC 2.2 CD and compiled flops.f using mpif77 -o flops flops.f since doing so in CbN causes errors.  It would seem that the openMPI installation on CbN is incomplete.  At least gfortran is missing.  Then I copied the executable program flops to the CbN master node from sftp user@10.10.129.21 as pelicanHPC which was running on worker node #21.  Then we ran cbn_run -np 16 flops from the master and got 8 nodes, 16 cores running at nearly 7GFLOPS!  Again, openMPI is incomplete as using mpi_run causes a ton of errors too!  All in all, CbN is very promising. 


Pelican Update
We are focusing this week's readings on the files on a booted pelicanHPC 2.2 64bit CD.  We printed out all the files in /home/user (including pelican_config) and all the files starting /usr/bin/pelican* (system scripts for starting the cluster).  Michael Creel says we should be able to modify how pelicanHPC boots by editing pelican_config and putting it in a persistent home volume called PELHOME.  In this manner, we hope to disable pelican's DHCP server so as to use the existing DHCP service on eth0 already.  Once we eliminate the DHCP conflicts, we should be able to boot the worker nodes from the one master CD via PXE after all.  Wish us luck!

Well, that's all for this week!  Please stay tuned.  We hope this record will help others trying to make their own clusters.  Also, if you have any hints or pointers, please leave a comment to let us know.  

Happy Clustering,