Showing posts with label mpitb. Show all posts
Showing posts with label mpitb. Show all posts

Saturday, November 13, 2010

CIS(theta) Meeting V (2010-2011) - pelican HPC CD on Overdrive!

Aim: 
pelican HPC CD on Overdrive!

Attending: 
DavidG, HerbertK, JayW, JoshG, RyanH

Reading: 
Building Parallel Programs, Chapter5

Research5: 
pelican HPC website
http://idea.uab.es/mcreel/ParallelKnoppix
pelican HPC tutorial
http://pareto.uab.es/mcreel/PelicanHPC/Tutorial/PelicanTutorial.html
pelican HPC forum
http://pelicanhpc.788819.n4.nabble.com
pelican HPC article
http://www.linux-magazine.com/w3/issue/103/030-035_pelicanHPC.pdf
pelican HPC based
http://birg1.fbb.utm.my/birghpc
pelican HPC based
http://kestrelhpc.sourceforge.net
pelican HPC like
http://www.ehu.es/AC/ABC.htm
pelican HPC like
http://www.dirigibleflightcraft.com/index.html




This week we went into pelicanHPC overdrive!  pelicanHPC was developed by Michael Creel at the University of Barcelona for his research in Econometrics using Octave and MPI.  We downloaded the ISO and burned one CD.  We then booted one of our 64bit dualcore AMD Athlons to try out pelicanHPC.  We booted the remaining 24 PCs (total of 50 2GHz cores) from that CD via PXE or netboot.  The video above follows the tutorial on the pelicanHPC website.  Take a look at the BIRG website with a version for Bioinformatics and protein folding.  It also has a tutorial.  Kestrel looks interesting too but it has to be installed on a hard drive and I don't think I want to give up the Ubuntu desktop we use in class during the day.  Our clustering experiments will be conducted after hours.  Take a look at CLuster By Night which uses ssh instead of PXE and Ruby instead of C++ for MPI jobs!



We ran into a bit of a SNAFU, however, as the client PCs got stuck somewhere in the middle of the boot process via PXE (so PXE boot works at least).  So, I dragged out a CrossOver Ethernet cable (http://www.littlewhitedog.com/content-8.html) I had spliced together ahead of time for just such an eventuality.  



This way we can fool 2 PCs (4 cores on the Athlons or 8 cores on the Xeons) into "thinking" they were still on an isolated LAN (PXE boot only works over Ethernet on a network with no other DHCP servers). 



We attached the Ethernet cards on eth1 (we also have eth0 for a public Windows LAN) that we usually use for the private Linux LAN, but ran into the same problem!




We also discussed using pelicanHPC or parallel Java "out of the box."  In other words these clustering solutions work fine on multicore PCs, the problem is networking more than one node.  We should have no problem making pelican or PJ use 1 or 2 cores on one of our AMD Athlons and 1, 2 or 4 cores on one of our Intel Xeons!


Well, "That's All Folks," for this meeting. We'll tackle the new server again next meeting in 2 weeks. Check our new Facebook Group, CIS(theta), for our next event!  BTW, this 1 hour meeting every 2 weeks is mandatory.  






Happy Clustering,
Posted by at No comments:
Labels: , , , , , , ,

CIS(theta) Meeting V (2010-2011) - pelican HPC CD on Overdrive!

Aim: 
pelican HPC CD on Overdrive!

Attending: 
DavidG, HerbertK, JayW, JoshG, RyanH

Reading: 
Building Parallel Programs, Chapter5

Research5: 
pelican HPC website
http://idea.uab.es/mcreel/ParallelKnoppix
pelican HPC tutorial
http://pareto.uab.es/mcreel/PelicanHPC/Tutorial/PelicanTutorial.html
pelican HPC forum
http://pelicanhpc.788819.n4.nabble.com
pelican HPC article
http://www.linux-magazine.com/w3/issue/103/030-035_pelicanHPC.pdf
pelican HPC based
http://birg1.fbb.utm.my/birghpc
pelican HPC based
http://kestrelhpc.sourceforge.net
pelican HPC like
http://www.ehu.es/AC/ABC.htm
pelican HPC like
http://www.dirigibleflightcraft.com/index.html




This week we went into pelicanHPC overdrive!  pelicanHPC was developed by Michael Creel at the University of Barcelona for his research in Econometrics using Octave and MPI.  We downloaded the ISO and burned one CD.  We then booted one of our 64bit dualcore AMD Athlons to try out pelicanHPC.  We booted the remaining 24 PCs (total of 50 2GHz cores) from that CD via PXE or netboot.  The video above follows the tutorial on the pelicanHPC website.  Take a look at the BIRG website with a version for Bioinformatics and protein folding.  It also has a tutorial.  Kestrel looks interesting too but it has to be installed on a hard drive and I don't think I want to give up the Ubuntu desktop we use in class during the day.  Our clustering experiments will be conducted after hours.  Take a look at CLuster By Night which uses ssh instead of PXE and Ruby instead of C++ for MPI jobs!



We ran into a bit of a SNAFU, however, as the client PCs got stuck somewhere in the middle of the boot process via PXE (so PXE boot works at least).  So, I dragged out a CrossOver Ethernet cable (http://www.littlewhitedog.com/content-8.html) I had spliced together ahead of time for just such an eventuality.  



This way we can fool 2 PCs (4 cores on the Athlons or 8 cores on the Xeons) into "thinking" they were still on an isolated LAN (PXE boot only works over Ethernet on a network with no other DHCP servers). 



We attached the Ethernet cards on eth1 (we also have eth0 for a public Windows LAN) that we usually use for the private Linux LAN, but ran into the same problem!




We also discussed using pelicanHPC or parallel Java "out of the box."  In other words these clustering solutions work fine on multicore PCs, the problem is networking more than one node.  We should have no problem making pelican or PJ use 1 or 2 cores on one of our AMD Athlons and 1, 2 or 4 cores on one of our Intel Xeons!


Well, "That's All Folks," for this meeting. We'll tackle the new server again next meeting in 2 weeks. Check our new Facebook Group, CIS(theta), for our next event!  BTW, this 1 hour meeting every 2 weeks is mandatory.  






Happy Clustering,
Posted by at No comments:
Labels: , , , , , , ,

Wednesday, September 8, 2010

CIS(theta) Meeting I (2010-2011) - Welcome Back!

Aim: 
Welcome Back!


Attending: 
DavidG, HerbertK, JayW, JoshG, RyanH

Reading: Building Parallel Programs, Chapter1

Research1: 
pelicanHPC
http://idea.uab.es/mcreel/ParallelKnoppix/
mpitb
http://atc.ugr.es/javier-bin/mpitb_eng


I welcomed our new CIS(theta) team today!  We will meet every other Wednesday.  Our task is to learn about cluster computing hardware in general.  Investigate the current state of cluster computing software and find a problem worthy of running on our cluster.  


First, let's get some vocabulary straight:
CIS(theta) = Computing Independent Study class
CIS(team) = DavidG, HerbertK, JoshG, JayW, RyanH (aka Geek Squad)
Shadowfax = 25 node, 64bit dual-core AMD Athlon 2GHz, gigE cluster


How about some history:
CIS(team) 0: We've done a bit of cluster programming over the years as a final project in AP Computer Science after AP Week all the way back to clusterKNOPPIX, i386/486s and slow ethernet!  Actually, about 25 years ago I helped JamesM with a robotics project for the Westinghouse Talent Search.  So, that's when the bug bit me to get into all this research with my students!  I think the computing independent study started about 20 years ago when JoeB wanted to learn about Mathematica and Vector Calculus.


CIS(team) I: ChrisR, FrankK, NathanielR successfully installed openMOSIX on 25 Pentium IVs over fastE and created several Mandelbrot plots using C++ and fork().  We followed the advice from this blog, http://nullprogram.com (search there for mosix, octave, c++ or fractal).  Take a look at the fractal prints on my deviantart site, http://cistheta2007.deviantart.com/gallery
CIS(team) II: MarcA, MitchellW were the first to use the current cluster hardware.  They installed the Quantian Linux DVD and used bash scripts to scatter/gather povray jobs over publicly authenticated ssh.  Take a look at the ray tracings on my zazzle site: http://www.zazzle.com/cistheta2008/gifts
CIS(team) III: ArthurD, DevinB, JeremyA, StevenB were the first to attempt using the current cluster hardware in 64bit mode.  We tried Fedora, CentOS and even Rocks Clusters and OSCAR to no avail.  I think we barely got HelloCluster.cc running on openMPI.
CIS(team) IV: DavidG, HerbertK, JayW, JoshG, RyanH will try 64bit mode again on Ubuntu Linux 10.04 that I have already installed on all nodes (see my post on this, http://shadowfaxrant.blogspot.com/2010/06/so-many-linux-distros-so-little-time.html).  You may want to try pelicanHPC too. This is a 32bit live Linux CD.  With this CD you are supposed to be able to reboot the whole room as an MPI cluster in minutes without installing a thing!  What's nice about this solution is that it includes openMPI and has a lot of demos.  We had this working on the new hardware last summer but sometimes its hard to use as its based on PXE and there's some DHCP server on our 10.5.*.* subnet that interferes with booting the nodes.  However, another benefit of this CD is that it includes Octave (like MATLAB/SAGE) and mpiTB (an interface for Octave to work with MPI)!  Hence your online research assignment listed above.  Another CD solution with MPI is BCCD.  An older DVD with openMOSIX is QUANTIAN.





So, our first meeting consisted of discussing all the above.  Also, I mentioned that the CIS(theta) team will act as my personal Geek Squad and help me install hardware, firmware and software in our PC Classroom (since I am still recuperating from surprise hernia surgery).  So, we had 2 "field trips."
Firld Trip 1: We went to the display case with some fractal and ray tracing prints we wanted to swap out.  We also went to Mrs. Murthy's new office to help decorate it with some prints.  We saved some nice big prints for the PC Classroom.  We are getting 6 new marker boards in there which will be installed all around the room.  I will hang a big print above each one.  So, my decorative work here is done!
Firld Trip 2: I also showed the current CIS(theta) team around to all the servers
centauri: ftp server in the book room
colossus: ssh server in the business office
guardian and caprica: new servers in the PC Classroom
So, the Geek Squad will also be helping me to install these new servers.  The new servers will eventually replace the old ones.  These new servers each have 4x72GB RAID drives and have 64bit Intel Xeon quadcores!  Maybe we should make a cluster out of these?  Google Cloud Computing using Ubuntu and Cloud Computing for HPC.
Wow, we have our work cut out for us.  But, I think we are going to have a lot of fun this year!





Happy Clustering,
Posted by at 2 comments:
Labels: , , , , , , , , , , , ,

CIS(theta) Meeting I (2010-2011) - Welcome Back!

Aim: 
Welcome Back!


Attending: 
DavidG, HerbertK, JayW, JoshG, RyanH

Reading: Building Parallel Programs, Chapter1

Research1: 
pelicanHPC
http://idea.uab.es/mcreel/ParallelKnoppix/
mpitb
http://atc.ugr.es/javier-bin/mpitb_eng


I welcomed our new CIS(theta) team today!  We will meet every other Wednesday.  Our task is to learn about cluster computing hardware in general.  Investigate the current state of cluster computing software and find a problem worthy of running on our cluster.  


First, let's get some vocabulary straight:
CIS(theta) = Computing Independent Study class
CIS(team) = DavidG, HerbertK, JoshG, JayW, RyanH (aka Geek Squad)
Shadowfax = 25 node, 64bit dual-core AMD Athlon 2GHz, gigE cluster


How about some history:
CIS(team) 0: We've done a bit of cluster programming over the years as a final project in AP Computer Science after AP Week all the way back to clusterKNOPPIX, i386/486s and slow ethernet!  Actually, about 25 years ago I helped JamesM with a robotics project for the Westinghouse Talent Search.  So, that's when the bug bit me to get into all this research with my students!  I think the computing independent study started about 20 years ago when JoeB wanted to learn about Mathematica and Vector Calculus.


CIS(team) I: ChrisR, FrankK, NathanielR successfully installed openMOSIX on 25 Pentium IVs over fastE and created several Mandelbrot plots using C++ and fork().  We followed the advice from this blog, http://nullprogram.com (search there for mosix, octave, c++ or fractal).  Take a look at the fractal prints on my deviantart site, http://cistheta2007.deviantart.com/gallery
CIS(team) II: MarcA, MitchellW were the first to use the current cluster hardware.  They installed the Quantian Linux DVD and used bash scripts to scatter/gather povray jobs over publicly authenticated ssh.  Take a look at the ray tracings on my zazzle site: http://www.zazzle.com/cistheta2008/gifts
CIS(team) III: ArthurD, DevinB, JeremyA, StevenB were the first to attempt using the current cluster hardware in 64bit mode.  We tried Fedora, CentOS and even Rocks Clusters and OSCAR to no avail.  I think we barely got HelloCluster.cc running on openMPI.
CIS(team) IV: DavidG, HerbertK, JayW, JoshG, RyanH will try 64bit mode again on Ubuntu Linux 10.04 that I have already installed on all nodes (see my post on this, http://shadowfaxrant.blogspot.com/2010/06/so-many-linux-distros-so-little-time.html).  You may want to try pelicanHPC too. This is a 32bit live Linux CD.  With this CD you are supposed to be able to reboot the whole room as an MPI cluster in minutes without installing a thing!  What's nice about this solution is that it includes openMPI and has a lot of demos.  We had this working on the new hardware last summer but sometimes its hard to use as its based on PXE and there's some DHCP server on our 10.5.*.* subnet that interferes with booting the nodes.  However, another benefit of this CD is that it includes Octave (like MATLAB/SAGE) and mpiTB (an interface for Octave to work with MPI)!  Hence your online research assignment listed above.  Another CD solution with MPI is BCCD.  An older DVD with openMOSIX is QUANTIAN.





So, our first meeting consisted of discussing all the above.  Also, I mentioned that the CIS(theta) team will act as my personal Geek Squad and help me install hardware, firmware and software in our PC Classroom (since I am still recuperating from surprise hernia surgery).  So, we had 2 "field trips."
Firld Trip 1: We went to the display case with some fractal and ray tracing prints we wanted to swap out.  We also went to Mrs. Murthy's new office to help decorate it with some prints.  We saved some nice big prints for the PC Classroom.  We are getting 6 new marker boards in there which will be installed all around the room.  I will hang a big print above each one.  So, my decorative work here is done!
Firld Trip 2: I also showed the current CIS(theta) team around to all the servers
centauri: ftp server in the book room
colossus: ssh server in the business office
guardian and caprica: new servers in the PC Classroom
So, the Geek Squad will also be helping me to install these new servers.  The new servers will eventually replace the old ones.  These new servers each have 4x72GB RAID drives and have 64bit Intel Xeon quadcores!  Maybe we should make a cluster out of these?  Google Cloud Computing using Ubuntu and Cloud Computing for HPC.
Wow, we have our work cut out for us.  But, I think we are going to have a lot of fun this year!





Happy Clustering,
Posted by at 2 comments:
Labels: , , , , , , , , , , , ,

Tuesday, February 16, 2010

Eine Kliene Nacht Lesung...

A little nightime reading re openMPI, enjoy!

Happy Clustering,
Posted by at No comments:
Labels: , , ,

Eine Kliene Nacht Lesung...

A little nightime reading re openMPI, enjoy!

Happy Clustering,
Posted by at No comments:
Labels: , , ,

Tuesday, September 22, 2009

Meeting II

Today's Aim: Boot up an MPI cluster using the Pelican HPC 64-bit bootable CD from the University of Barcelona on the Master Node and PXE booting the Worker Nodes
Tonight's Reading: Building Parallel Programs, Chapter 2
This Week's Research: look up info on GIMPS, PovRay and Fractals
Attending: JeremyA, SteveB, DevinB, ArthurD 

We had a problem with conflicting DHCP servers on our LAN when setting up Pelican HPC, even though the Linux LAN is supposed to be isolated.  I was surprised as I had this working last June!  We'll try again next meeting.  We are also looking into downloading and buring the Red Hat Fedora 11 64-bit installation DVD to reinstall the whole room.  Eventually, we'll try to emulate the MPI+MPITB+Octave cluster a la Pelican HPC.  Maybe we can install Sage from the University of Washington and use dSage?  What about the proprietary version of MOSIX, now that openMOSIX is dead, from the University of Israel?


Happy Clustering,
Posted by at No comments:
Labels: , , , , , , , , , , , , , ,

Meeting II

Today's Aim: Boot up an MPI cluster using the Pelican HPC 64-bit bootable CD from the University of Barcelona on the Master Node and PXE booting the Worker Nodes
Tonight's Reading: Building Parallel Programs, Chapter 2
This Week's Research: look up info on GIMPS, PovRay and Fractals
Attending: JeremyA, SteveB, DevinB, ArthurD 

We had a problem with conflicting DHCP servers on our LAN when setting up Pelican HPC, even though the Linux LAN is supposed to be isolated.  I was surprised as I had this working last June!  We'll try again next meeting.  We are also looking into downloading and buring the Red Hat Fedora 11 64-bit installation DVD to reinstall the whole room.  Eventually, we'll try to emulate the MPI+MPITB+Octave cluster a la Pelican HPC.  Maybe we can install Sage from the University of Washington and use dSage?  What about the proprietary version of MOSIX, now that openMOSIX is dead, from the University of Israel?


Happy Clustering,
Posted by at No comments:
Labels: , , , , , , , , , , , , , ,
Subscribe to: Posts (Atom)