server 08 enterprise vs HPC

[Khasiar]

Hey guys,

I am wanting to setup a cluster and am wondering if enterprise version of server 08 offers the same processing power as HPC when setting up a cluster... my main intention is to get my programs to compile faster using Visual Studios 2010...

any help would be greats.

Thanks,
Khas

[michaewlewis]

You can only get HPC if it is preinstalled on a computer. (Are you thinking of datacenter edition?)
Either one would do the trick. They're the same operating system, they just have different features. Also, enterprise edition can work with 16 nodes, while HPC can work with thousands. How many computers are you trying to cluster?

[BC_Programmer]

my main intention is to get my programs to compile faster using Visual Studios 2010...

Unless your programs are several millions lines long, that won't take long enough to justify such a upgrade.

clustering is used for database servers, not for development or build machines.

[michaewlewis]

Also, take a look at the Nvidia tesla technology. It's basically a small cluster stuffed into a pcie card.
http://www.nvidia.com/object/tesla_computing_solutions.html

[Khasiar]

Unless your programs are several millions lines long, that won't take long enough to justify such a upgrade.

clustering is used for database servers, not for development or build machines.

program is only about 250 lines, but has to process over a million calculations. with each calculation taking about 5 seconds because it needs to compare a result with an array with size about 270 000 using binary search. so you can imagine how fustrating it is doing this on a core 2 duo 2.2ghz machine when VS2010 limits the max process it can use to 50% of CPU max, which is good so my computer doesnt freeze but i need more grunt.

maybe i should look into using a different search method but regardless it will still take a long time without a cluster

[Khasiar]

Also, take a look at the Nvidia tesla technology. It's basically a small cluster stuffed into a pcie card.
http://www.nvidia.com/object/tesla_computing_solutions.html

Thanks mike, ive seen this one already, isnt it meant for graphics clustering? i need this for mass calculations

You can only get HPC if it is preinstalled on a computer. (Are you thinking of datacenter edition?)
Either one would do the trick. They're the same operating system, they just have different features. Also, enterprise edition can work with 16 nodes, while HPC can work with thousands. How many computers are you trying to cluster?

to begin with just 10, but if this still doesnt improve my speed much i may have to set one up at work use all our computers there haha

[BC_Programmer]

program is only about 250 lines, but has to process over a million calculations.

That has nothing to do with compilation speed, then.

with each calculation taking about 5 seconds because it needs to compare a result with an array with size about 270 000 using binary search.

A binary search of 270,000 elements would only take 17 iterations. an Imperceptible time- certainly less them 5 seconds. Clearly you are doing something wrong, or not explaining something; the binary search is clearly the less time consuming part of your "calculations".

VS2010 limits the max process it can use to 50% of CPU max, which is good so my computer doesnt freeze but i need more grunt.

VS2010 doesn't impose any limitation on the CPU usage of your process. But with a Dual core, you are going to have to have at least two threads (one for each core). This same restriction is imposed in a cluster- you cannot have a single process suddenly parallelized simply by running it on a cluster, it needs to have discrete chunks so that it's work can be divvy'd up.

maybe i should look into using a different search method but regardless it will still take a long time without a cluster

Binary search is one of the most efficient methods. From the sounds of things, you aren't using it. If it takes that long on that configuration it sounds more like you are doing a sequential search, not a binary search.

[michaewlewis]

Thanks mike, ive seen this one already, isnt it meant for graphics clustering? i need this for mass calculations

Take a further look at the page I linked to. There are several references to C, C++, and floating point kinds of stuff. I'm not a programmer but I know programming lingo when I see it. It does more than just graphics.

[Khasiar]

That has nothing to do with compilation speed, then.
A binary search of 270,000 elements would only take 17 iterations. an Imperceptible time- certainly less them 5 seconds. Clearly you are doing something wrong, or not explaining something; the binary search is clearly the less time consuming part of your "calculations".
VS2010 doesn't impose any limitation on the CPU usage of your process. But with a Dual core, you are going to have to have at least two threads (one for each core). This same restriction is imposed in a cluster- you cannot have a single process suddenly parallelized simply by running it on a cluster, it needs to have discrete chunks so that it's work can be divvy'd up.
Binary search is one of the most efficient methods. From the sounds of things, you aren't using it. If it takes that long on that configuration it sounds more like you are doing a sequential search, not a binary search.

definetly a binary search...

while (first <= last){
      mid = (first + last) / 2;

      if (word == dictFile.at(mid))
         return true;//if
      else{
         if (word < dictFile.at(mid))
            last = mid - 1;//if
         if (word > dictFile.at(mid))
            first = mid + 1;//if
      }//else
   }//while

im creating possibilties to be checked against my DB (dictFile) which takes less than a quarter of the time it does to check against the DB, again my DB is 270,000 elements and my possibilities are into the millions, ive managed to reduced the checking time by splitting my DB into smaller files and checking when necessary so its about a second per calc but with millions of calcs i need this to be faster and i think it comes down to faster processing speed... if i had lots of cash to spare id just go buy a 50,000$ supercomputer but i think itll be cheaper to setup a cluster

[BC_Programmer]

Clearly you aren't using an Actual database, otherwise you could just run a query on it. Sounds like when you say "database" you really mean "text file with data inside".

It doesn't come down to faster processing speed. It never comes down to faster processing speed.

[Khasiar]

Clearly you aren't using an Actual database, otherwise you could just run a query on it. Sounds like when you say "database" you really mean "text file with data inside".

It doesn't come down to faster processing speed. It never comes down to faster processing speed.

Do you have something against me BC? yes it is a text file with data inside but a database doesnt need to be in SQL format:

http://searchsqlserver.techtarget.com/definition/databaseDefinition
database

database is a collection of information that is organized so that it can easily be accessed, managed, and updated. In one view, databases can be classified according to types of content: bibliographic, full-text, numeric, and images.



if it never came down to processing speed then why do universities invest in such machines? im not saying my program matches the complexity of a program that can graph movements of atoms but it does require processing power.

imagine processing the following:

if x = 4
x! (x * x-1 * x-2 .... *x-(x-1)
= 4*3*2*1 = 24

if x=11
x=39,916,800

now try and make each of these 39,916,800 elements different to all the rest, and check them against a DB file, yes a DB file.

tell me, even if your computer was processing 100 calcs per second against 270,000 elements using a binary search it would still take

399,168 seconds
6652.8mins
110.88 hours
4.62 days

who has that much time to wait for one instance of a program, and this is only with x as 11. my intention of x is about 200

consider that

x = 25 = 15,511,210,043,330,985,984,000,000

currently im doing about 2 calcs per second which means for a size of 11 i would take 50* 4.62 days = 231 days

[BC_Programmer]

Do you have something against me BC? yes it is a text file with data inside but a database doesnt need to be in SQL format:

No.

database is a collection of information that is organized so that it can easily be accessed, managed, and updated.

A text file is none of those things.

if it never came down to processing speed then why do universities invest in such machines?

You are taking what I said out of context. The solution to a slow program shouldn't be "get a faster machine". That's just lazy. Universities and large corporations get powerful computers because their calculations are entirely intractable otherwise.

imagine processing the following:

if x = 4
x! (x * x-1 * x-2 .... *x-(x-1)
= 4*3*2*1 = 24

if x=11
x=39,916,800

It's a factorial. So?

now try and make each of these 39,916,800 elements different to all the rest

What "elements"? Why are you generating 39 million elements? What are they elements of?

and check them against a DB file, yes a DB file.

If by "DB" file you are referring to a text file, then, I repeat- that is your bottleneck.

tell me, even if your computer was processing 100 calcs per second against 270,000 elements using a binary search it would still take

100 calculations per second is 100Hz. Don't know how you managed to get a machine like that.

who has that much time to wait for one instance of a program, and this is only with x as 11. my intention of x is about 200

consider that

x = 25 = 15,511,210,043,330,985,984,000,000

I STILL don't see how factorials fit in or why you are generating a number of items equal to the factorial of some other arbitrary value.



Either way, even if you wanted to take the brute force approach and just throw more Ghz and the problem, you are going to have to somehow parallelize the process of... well, whatever it is you are doing. And, since File IO cannot be parallelized, I am going to guess that you will see very little improvement.

Instead of throwing more Ghz at the problem, why not perform some actual profiling of your application and see where the most time is spent, and write that area better? I managed to speed up my game's framerate by a factor of 3 by changing a single line of code I would have never even suspected without the use of dotTrace.