Showing posts with label octave. Show all posts
Showing posts with label octave. Show all posts

Saturday, May 14, 2011

CIS(theta) 2010-2011 - Eureka, we did it! - Meeting XIV

Aim: 
Completing Step 4!

Attending: 
CIS(theta) 2010-2011: DavidG, RyanH

Absent: 
CIS(theta) 2010-2011: HerbertK, JoshG

Reading:
NA

Parallel Python
IPython
Large Integer number crunching Mersenne Primes
http://www.hoise.com/primeur/03/articles/weekly/AE-PR-01-04-37.html
Large Integer number crunching Beal Conjecture
http://www.bealconjecture.com/



InstantCluster Step 4: Software Stack (Completed)
By Jove, I think we've done it! OK, we had a really short meeting today as everyone had to come late due to xtra help and AP exams.... Anyway, we actually got openMPI working last time and didn't even know it! All we had to do was add the IPs to a file called "machines" to get nearly 2700 MFLOPs using 2 cores per node on 3 nodes!

Now, we can scale the cluster up from 3 nodes to all 24 students boxes and get over 20 GFLOPs. The trick was to use the 10.10.*.* IPs and not the 10.5.*.* ones. Maybe we could get the 3 servers in on this too for an additional 3 or more GFLOPs? 

BTW, we figured out how to setup the "machines" file by running ClusterByNight first, thanx Kevin Lynagh! That's also where we got the idea to run openMPI over openSSH. Sorry, but PXE boot just didn't work for us! We should also be able to add the octave/mpitb functionality from pelicanHPC as well, thanx Michael Creel!




We only installed gfortran on one box so we could compile flops.f on that node (lets call it the master node):
mpif77 -o flops flops.f

and copied "flops" to all the worker boxes (and used chmod 755 to make it executable) getting about 2650 MFLOPs on 6 cores over 3 nodes:
mpirun -np 6 --hostfile machines flops

What we messed up last time was making the "machines" file based on the 10.5.*.* static IPs we set up on eth1. When we listed all the 10.10.*.* DHCP IPs used on eth0, all worked fine! So, openMPI just defaults to eth0?

Well, it was almost fine. When the remote processes terminated, the master node hung. I think that's because the 10.5.*.* IPs are listed in the authorized_keys file. Maybe we can just edit that and make them 10.10.*.* too?

Well, that's all for now, enjoy!

Posted by at No comments:
Labels: , , , , , ,

CIS(theta) 2010-2011 - Eureka, we did it! - Meeting XIV

Aim: 
Step 4 is completed!

Attending: 
CIS(theta) 2010-2011: DavidG, RyanH

Absent: 
CIS(theta) 2010-2011: HerbertK, JoshG

Reading:
NA

Parallel Python
IPython
Large Integer number crunching Mersenne Primes
http://www.hoise.com/primeur/03/articles/weekly/AE-PR-01-04-37.html
Large Integer number crunching Beal Conjecture
http://www.bealconjecture.com/



InstantCluster Step 4: Software Stack (Completed)
By Jove, I think we've done it! OK, we had a really short meeting today as everyone had to come late due to xtra help and AP exams.... Anyway, we actually got openMPI working last time and didn't even know it! All we had to do was add the IPs to a file called "machines" to get nearly 2700 MFLOPs using 2 cores per node on 3 nodes!

Now, we can scale the cluster up from 3 nodes to all 24 to get over 20 GFLOPs. The trick was to use the 10.10.*.* IPs and not the 10.5.*.* ones. Maybe we could get the 3 servers in on this too for an additional 3 or more GFLOPs? 

BTW, we figured out how to setup the "machines" file by running ClusterByNight first, thanx Kevin Lynagh! That's also where we got the idea to run openMPI over openSSH. Sorry, but PXE boot just didn't work for us! We should also be able to add the octave/mpitb functionality from pelicanHPC as well, thanx Michael Creel!



We only installed gfortran on one box so we could compile flops.f on that node (lets call it the master node):
mpif77 -o flops flops.f

and copied "flops" to all the worker boxes (and used chmod 755 to make it executable) getting about 2650 MFLOPs on 6 cores over 3 nodes:
mpirun -np 6 --hostfile machines flops

What we messed up last time was making the "machines" file based on the 10.5.*.* static IPs we set up on eth1. When we listed all the 10.10.*.* DHCP IPs used on eth0, all worked fine! So, openMPI just defaults to eth0?

Well, it was almost fine. When the remote processes terminated, the master node hung. I think that's because the 10.5.*.* IPs are listed in the authorized_keys file. Maybe we can just edit that and make them 10.10.*.* too?

Well, that's all for now, enjoy!

Posted by at No comments:
Labels: , , , , , ,

Saturday, November 27, 2010

CIS(theta) Meeting VI (2010-2011) - pelican HPC the movie!

Aim: 
pelican HPC the movie!

Attending: 
DavidG, HerbertK, JoshG, RyanH

Reading: 
Building Parallel Programs, Chapter6

Research6: 

We went a little cuckoo this week and watched all the Pelican HPC related videos we could find online!  We wanted to see that other peeps were getting Pelican to work and how they did it.  Maybe we missed something?  We started with a series of four:









Then we viewed the tutorial video from the Pelican website:



We found two more on YouTube:





OK, here's the CbN (Cluster by Night) video too.  This one is nice as it uses a CD for the Master Node and multiple CDs for the worker nodes.  So, this solution does not use PXE+openMPI+Octave, it uses ssh+openMPI+Ruby!



Well, that's all for this week.  I hope you enjoyed our movie night!  Next week, we're going to bring in our own cables, switches, routers and laptops to see if we can get Pelican to work in our lab....

Happy Clustering,
Posted by at No comments:
Labels: , , , , , ,

CIS(theta) Meeting VI (2010-2011) - pelican HPC the movie!

Aim: 
pelican HPC the movie!

Attending: 
DavidG, HerbertK, JoshG, RyanH

Reading: 
Building Parallel Programs, Chapter6

Research6: 

We went a little cuckoo this week and watched all the Pelican HPC related videos we could find online!  We wanted to see that other peeps were getting Pelican to work and how they did it.  Maybe we missed something?  We started with a series of four:









Then we viewed the tutorial video from the Pelican website:



We found two more on YouTube:





OK, here's the CbN (Cluster by Night) video too.  This one is nice as it uses a CD for the Master Node and multiple CDs for the worker nodes.  So, this solution does not use PXE+openMPI+Octave, it uses ssh+openMPI+Ruby!



Well, that's all for this week.  I hope you enjoyed our movie night!  Next week, we're going to bring in our own cables, switches, routers and laptops to see if we can get Pelican to work in our lab....

Happy Clustering,
Posted by at No comments:
Labels: , , , , , ,

Saturday, November 13, 2010

CIS(theta) Meeting V (2010-2011) - pelican HPC CD on Overdrive!

Aim: 
pelican HPC CD on Overdrive!

Attending: 
DavidG, HerbertK, JayW, JoshG, RyanH

Reading: 
Building Parallel Programs, Chapter5

Research5: 
pelican HPC website
http://idea.uab.es/mcreel/ParallelKnoppix
pelican HPC tutorial
http://pareto.uab.es/mcreel/PelicanHPC/Tutorial/PelicanTutorial.html
pelican HPC forum
http://pelicanhpc.788819.n4.nabble.com
pelican HPC article
http://www.linux-magazine.com/w3/issue/103/030-035_pelicanHPC.pdf
pelican HPC based
http://birg1.fbb.utm.my/birghpc
pelican HPC based
http://kestrelhpc.sourceforge.net
pelican HPC like
http://www.ehu.es/AC/ABC.htm
pelican HPC like
http://www.dirigibleflightcraft.com/index.html




This week we went into pelicanHPC overdrive!  pelicanHPC was developed by Michael Creel at the University of Barcelona for his research in Econometrics using Octave and MPI.  We downloaded the ISO and burned one CD.  We then booted one of our 64bit dualcore AMD Athlons to try out pelicanHPC.  We booted the remaining 24 PCs (total of 50 2GHz cores) from that CD via PXE or netboot.  The video above follows the tutorial on the pelicanHPC website.  Take a look at the BIRG website with a version for Bioinformatics and protein folding.  It also has a tutorial.  Kestrel looks interesting too but it has to be installed on a hard drive and I don't think I want to give up the Ubuntu desktop we use in class during the day.  Our clustering experiments will be conducted after hours.  Take a look at CLuster By Night which uses ssh instead of PXE and Ruby instead of C++ for MPI jobs!



We ran into a bit of a SNAFU, however, as the client PCs got stuck somewhere in the middle of the boot process via PXE (so PXE boot works at least).  So, I dragged out a CrossOver Ethernet cable (http://www.littlewhitedog.com/content-8.html) I had spliced together ahead of time for just such an eventuality.  



This way we can fool 2 PCs (4 cores on the Athlons or 8 cores on the Xeons) into "thinking" they were still on an isolated LAN (PXE boot only works over Ethernet on a network with no other DHCP servers). 



We attached the Ethernet cards on eth1 (we also have eth0 for a public Windows LAN) that we usually use for the private Linux LAN, but ran into the same problem!




We also discussed using pelicanHPC or parallel Java "out of the box."  In other words these clustering solutions work fine on multicore PCs, the problem is networking more than one node.  We should have no problem making pelican or PJ use 1 or 2 cores on one of our AMD Athlons and 1, 2 or 4 cores on one of our Intel Xeons!


Well, "That's All Folks," for this meeting. We'll tackle the new server again next meeting in 2 weeks. Check our new Facebook Group, CIS(theta), for our next event!  BTW, this 1 hour meeting every 2 weeks is mandatory.  






Happy Clustering,
Posted by at No comments:
Labels: , , , , , , ,

CIS(theta) Meeting V (2010-2011) - pelican HPC CD on Overdrive!

Aim: 
pelican HPC CD on Overdrive!

Attending: 
DavidG, HerbertK, JayW, JoshG, RyanH

Reading: 
Building Parallel Programs, Chapter5

Research5: 
pelican HPC website
http://idea.uab.es/mcreel/ParallelKnoppix
pelican HPC tutorial
http://pareto.uab.es/mcreel/PelicanHPC/Tutorial/PelicanTutorial.html
pelican HPC forum
http://pelicanhpc.788819.n4.nabble.com
pelican HPC article
http://www.linux-magazine.com/w3/issue/103/030-035_pelicanHPC.pdf
pelican HPC based
http://birg1.fbb.utm.my/birghpc
pelican HPC based
http://kestrelhpc.sourceforge.net
pelican HPC like
http://www.ehu.es/AC/ABC.htm
pelican HPC like
http://www.dirigibleflightcraft.com/index.html




This week we went into pelicanHPC overdrive!  pelicanHPC was developed by Michael Creel at the University of Barcelona for his research in Econometrics using Octave and MPI.  We downloaded the ISO and burned one CD.  We then booted one of our 64bit dualcore AMD Athlons to try out pelicanHPC.  We booted the remaining 24 PCs (total of 50 2GHz cores) from that CD via PXE or netboot.  The video above follows the tutorial on the pelicanHPC website.  Take a look at the BIRG website with a version for Bioinformatics and protein folding.  It also has a tutorial.  Kestrel looks interesting too but it has to be installed on a hard drive and I don't think I want to give up the Ubuntu desktop we use in class during the day.  Our clustering experiments will be conducted after hours.  Take a look at CLuster By Night which uses ssh instead of PXE and Ruby instead of C++ for MPI jobs!



We ran into a bit of a SNAFU, however, as the client PCs got stuck somewhere in the middle of the boot process via PXE (so PXE boot works at least).  So, I dragged out a CrossOver Ethernet cable (http://www.littlewhitedog.com/content-8.html) I had spliced together ahead of time for just such an eventuality.  



This way we can fool 2 PCs (4 cores on the Athlons or 8 cores on the Xeons) into "thinking" they were still on an isolated LAN (PXE boot only works over Ethernet on a network with no other DHCP servers). 



We attached the Ethernet cards on eth1 (we also have eth0 for a public Windows LAN) that we usually use for the private Linux LAN, but ran into the same problem!




We also discussed using pelicanHPC or parallel Java "out of the box."  In other words these clustering solutions work fine on multicore PCs, the problem is networking more than one node.  We should have no problem making pelican or PJ use 1 or 2 cores on one of our AMD Athlons and 1, 2 or 4 cores on one of our Intel Xeons!


Well, "That's All Folks," for this meeting. We'll tackle the new server again next meeting in 2 weeks. Check our new Facebook Group, CIS(theta), for our next event!  BTW, this 1 hour meeting every 2 weeks is mandatory.  






Happy Clustering,
Posted by at No comments:
Labels: , , , , , , ,

Saturday, October 2, 2010

CIS(theta) Meeting II (2010-2011) - Burn Fest!!


Aim: 
Burn Fest!!


Attending: 
DavidG, HerbertK, JayW, JoshG, RyanH


Reading: Building Parallel Programs, Chapter2


tjhsst
http://www.tjhsst.edu/~dhyatt/supercomp/index.html
tjhsst
http://www.tjhsst.edu/~rlatimer/compsys/compsys2001.html




This week we reviewed the reading from Chapter 1 and the links from Reasearch 1.  We talked about setting up pelicanHPC in the near future via PXEboot to run some jobs using Octave, MPI and MPITB.  This week's reading is Chapter 2 and Research 2.  I put up Research 3 as it is related to TJHSST, but you can skip those links for now.





The next version of Ubuntu is coming soon
Then we talked about our 64bit Ubuntu install as discribed here: http://shadowfaxrant.blogspot.com/2010/06/so-many-linux-distros-so-little-time.html whereby all the PCs in our lab are installed identically (24 student clients, 1 teacher client and 2 servers) via the Ubuntu 10.04 live install CD (dualboot with WinXP making linux the primary boot partition on grub2).  I added JRE to the teacher and student stations so we could do 3D graphs on SAGE.  I also added VLC to the teacher station so I could display my crazy videos.  



We discussed adding flash (for youtube viewing), WINE (for VTI in math class), handbrake (to edit DVDs for youtube upload) and xournal (for writing notes in class with my wireless tablet) to the teacher station.  We will also be installing vsftp and openssh-server on the servers. 




Note: student and teacher PCs are 64bit dualcore AMD Athlons and the new servers are 64bit quadcore Intel Xeons.  So, the "geek squad" has its hands full with helping me do all this in the near future!







Meanwhile, the "geek squad" burned, labeled and cased 50 Ubuntu 32bit liveCDs that I can now give out in my classes for my new students to work from home in the same environment as they do in class.  BTW, we made a mini field trip to the Math Office to get some CDs where we found 10x100 CD and 15x50 DVD spindles!  Thanx guys!


Happy Clustering,
Posted by at No comments:
Labels: , , , , , , , , , , , , , , ,
Subscribe to: Posts (Atom)