| Author |
Message |
|
|
|
Post subject: QNX for scientific computing
 Posted: Apr 02, 2004 - 10:50 PM
|
|
New Member
Joined: Apr 02, 2004
Posts: 3
|
|
I have sent this query as an email to QNX and have posted in on ww.osnews.org but have had no replies. Perhaps someone in this community has some information. Thanks!
------------
I am interested in the possibility of using QNX as a platform for heavy technical computing.
The oil industry uses many compute and graphics intensive applications in analyzing its large (multi-gigabyte) data volumes. The processing and visualization applications that are used have been, and are still written for high-end UNIX systems.
Recently these applications have begun to be ported to Linux machines using high-end PC hardware.
Examples of these applications are processing of volumetric seismic wavefield data for the purpose of imaging in three dimensions the earth's subsurface, interactive 3D interpretation of this image data, construction of numerical models of the subsurface, including geologic features and assignment of fluid related properties, and simulation of oil field production based on these models. Besides our internally developed codes, a large suite of software vendors address these computing needs. Some of these are Schlumberger, Landmark Graphics, Paradigm Geophysical, Roxar, ESRI. A perusal of these companies web pages may serve to give some idea of the nature of the applications that are in use in the oil industry.
I am interested to find out if QNX is a good fit to this sort of environment. Has it been so used, if not in the oil industry, then perhaps in other scientific large scale computing/visualization environments? Of particular interest is whether QNX is used in cluster computing or grid computing environments. The oil field simulation calculations and sometimes visualization are of such magnitude and complexity that clusters computing is very important. Specifically, our Linux environment consists of high end PCs connected by Infiniband high-speed switches.
A move to 64-bit LINUX applications (UNIX apps have been 64 bit for several years) is underway now. Does QNX support 64-bit computing?
Here are some typical issues that are faced in developing Linux port of our UNIX applications:
1. high speed switched cluster computing application development:
We have to make sure all drivers for hardware operate on the operating system.
We have to recompile all of our Fortran90 and C code on the operating system using an available fortran and c compiler that optimizes well for the hardware and operating system being used (e.g., dual opteron compute server from IBM, HP or DELL).
We have to be able to debug our code in case it does not run the same as on other systems; we do this using the Etnus Totalview debugger. Totalview must be able to work on the new operating system with the compiler being used.
MPI libraries have to be available for our parallel processing capabilities.
We have to have access to scripting languages available within things like Korn and C shells.
We need to be able to run awk scripts.
We don’t have the luxury of putting many people on the porting effort. It needs to go smoothly from beginning to end.
2. desktop technical Pcworkstation running 3D interactive visualization:
Similar issues as above, but also have the issue of getting the vendor to port their application to the operating system and continue supporting it.
Graphics cards will need to work on the operating system (e.g., Nvidia Quadro FX 3000)
VMWARE OS emulation software will need to be ported to allow us to run simultaneous the Windows operating system with the other operating system.
Part of the motivation for moving to Linux in the cluster area was that none of the traditional UNIX vendors had a complete solution that would run on the chosen hardware (machines and switches network).
Would the QNX environment be able to address these issues effectively, and in fact more effectively than Linux ? A survey of the capabilities outlined on your web site and in the white papers linked there seems to indicate that it should be at least as easy to develop in QNX as in Linux. One would expect that a QNX based cluster might be more reliable and run faster than a Linux based cluster.
We are attracted to and admire the QNX design because of its pursuit of the principles of layering and modularization that characterize UNIX to their logical conclusion in kernel and total o/s design. In addition, the explicit focus on communication between executing program elements seems to make computing, and especially distributed computing more understandable, and acts as a source of ideas.
In the long term, it seems inevitable the QNX design principles should prevail. Would one expect a radical enough improvement in the whole development cycle that a port of our applications to QNX would be a desirable goal to pursue?
I would be interested to receive any comments and suggestions you may have in regard to this query. Many thanks for your attention to my queries! |
|
|
| |
|
|
|
 |
|
|
|
Post subject:
Posted: Apr 02, 2004 - 11:52 PM
|
|
QNX Master
Joined: Sep 01, 2002
Posts: 2667
|
|
QNX is going more and more after the embedded market and leaving the industrial market behind (that's my opinion). Although it's technicaly superior to Linux and would make a better Linux in many cases, availability of software (vmware being one example) is a problem and as well as support for hardware (forget 3D on off the shelf card or high-end card)
Fortran isn't officialy supported
64 bit (Intel/Amd) is apparrently not on their radar
QNX is NOT binary compatible with Linux so unless you have access to sources to port them over, you are out of luck( I guess that rules usage of TotalView out)
You are right that the design quality should prevail, technicaly speaking, but as you know there are other factors. QSS seems to have given up on that market (intentionaly or not) hence Linux is gaining in that area. |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Apr 03, 2004 - 12:09 AM
|
|
New Member
Joined: Apr 02, 2004
Posts: 3
|
|
Thanks for the reply.
If there were some expectation that QNX could offer superior performance or at least a better "computing experience" with essentially the same performance, source would be available. Our recent move from UNIX to Linux involved this.
The main concern would be to deploy techical apps on QNX on high end PC hardware. 3D graphics would be a must. Distributed computing would be a must. Could not at least the graphics drivers used by Linux be ported to QNX ? They make big claim for ease of migration after all!
Getting a project like this started is tough because there seem to be no case studies, only theoretically based hopes.
I am surprised the grid computing community has not tried QNX out. They certainly have source and lots of smart people to do the porting.
Sigh!
PS:
I also wondered if anyone knows how to get in touch with the scientists at QNX ?
Thanks again!
PPS:
Another experiment that would be fascinating would be to get QNX running on multi-cpu Macs ideally the dual G5. Then see what would happen with a cluster of these machines. The Mac is making a lot of inroads in scientific computing circles now.
Thanks yet again! |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Apr 03, 2004 - 04:24 AM
|
|
QNX Master
Joined: Jul 18, 2002
Posts: 288
|
|
| I thought QNX fit into distributed computing very well. I does get contact by someone from "oil industry" a couple of years ago, talking about distributed computing with QNX. If you have serious consideration, try to bring it up to your local sales rep. |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Apr 03, 2004 - 02:43 PM
|
|
QNX Master
Joined: Feb 01, 2003
Posts: 737
|
|
If you're not interrested in games or realtime and you aren't a cheapskate then the Mac wins everytime. The scientific workstation fits that like a glove.
As for QNX, it's a designers OS. QNX's ace is it performs realtime natively.
If I read http://www.qnx.com/news/pr_621_2.html correctly then accelerated 3D support is available from 6.3.0 onwards but, given that not many platforms are supported yet, you will be paying custom rates to support your chosen platform. On the good side you get to choose your prefered hardware. 
As for development tools, if they can interface with Eclipse then you are set.
And this may be of relevance, http://www.qnx.com/news/pr_812_2.html |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Apr 05, 2004 - 04:33 PM
|
|
QNX Master
Joined: Sep 01, 2002
Posts: 2667
|
|
|
hdmi wrote:
If there were some expectation that QNX could offer superior performance or at least a better "computing experience" with essentially the same performance, source would be available.
CPU wise you'd probably get better performance (and with Intel compiler possibly comming out that should be interesting). But that's about disk, HD performance is lame, files can't be bigger then 2G, no support for RAID, tape
hdmi wrote:
The main concern would be to deploy techical apps on QNX on high end PC hardware. 3D graphics would be a must.
That's a big problem 3D is out of the question (for now at least). Support for HT is not as good as it could be and support for high end SMP machine (read NEMA) is also lacking
hdmi wrote:
Distributed computing would be a must.
The OS does provide some feature to build distributed computing but it's not enough. Some extra layer needs to be build, these layers already exists (PVM) and can be port to QNX but they use TCP/IP and TCP/IP performance is no better then other OSes
hdmi wrote:
Could not at least the graphics drivers used by Linux be ported to QNX ? They make big claim for ease of migration after all!
No cause companies building 3D card are not making the source public thus porting is out of the question. Furthermore QNX claims are relative to user application, most drivers required a rewrite.
hdmi wrote:
Getting a project like this started is tough because there seem to be no case studies, only theoretically based hopes.
QNX may be technicaly attractive, but QSS has not provided any incentive.
Matter of priority I guess.
hdmi wrote:
I am surprised the grid computing community has not tried QNX out. They certainly have source and lots of smart people to do the porting.
What would be the point, there are some many problem to overcome and I doubt there would be any major benefit.
hdmi wrote:
I also wondered if anyone knows how to get in touch with the scientists at QNX ?
How do you define scientists? The QNX community is very small, as most QNX user are professional that are working in a private (read BIG OEM) environment.
hdmi wrote:
Another experiment that would be fascinating would be to get QNX running on multi-cpu Macs ideally the dual G5. Then see what would happen with a cluster of these machines. The Mac is making a lot of inroads in scientific computing circles now.
Too many legal problem. |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Apr 06, 2004 - 02:01 AM
|
|
Active Member
Joined: Mar 09, 2004
Posts: 88
|
|
| And the Mac OS can do clustering very well since it can fully harness the unerlying hardware. |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Apr 14, 2004 - 11:22 PM
|
|
New Member
Joined: Apr 02, 2004
Posts: 3
|
|
Thanks, everyone, for your replies.
It seems like the consensus is that, although QNX is probably technically superior to other o/s's, the "user" infrastructure for technical computing just doesn't exist. A massive coding effort would be required to enable QNX to function in a general technical computing environment. QNX is in much the same position Linux would have been in if the GNU project had not been in place.
On the other hand, QNX seems such a much more rational design that the effort might be worth undertaking.
In particular, QNX seems, potentially at least, to more naturally accomodate distributed computing. It treats remote and local thread processing in a uniform way, and natively allows any thread to be executed on any processor. This seems like it would make the development of distributed computing easier on QNX than on other systems, and was the reason I felt the Grid computing world might want to take a look at it.
QNX also seems to offer a new way to look at computing. It suggests a view of the network as a pool of interacting "capabilities". Each component is present because it provides a capability, and components have maximal flexibility to interact without constraints. The o/s acts less like a local governor and more like an enabler, whose main role is to provide orderly and efficient communications between components present potentially anywhere in the system.
I would still like to get in touch with the designers ("scientists") at QNX. Is this possible?
Many thanks again! |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Apr 15, 2004 - 01:19 AM
|
|
QNX Master
Joined: Sep 01, 2002
Posts: 2667
|
|
|
Quote:
Thanks, everyone, for your replies.
In particular, QNX seems, potentially at least, to more naturally accomodate distributed computing. It treats remote and local thread processing in a uniform way, and natively allows any thread to be executed on any processor
.
Threads within a process have to run on the same machine (but can run on different CPU in the case of SMP)
Quote:
This seems like it would make the development of distributed computing easier on QNX than on other systems, and was the reason I felt the Grid computing world might want to take a look at it.
Maybe, but at a cost of portability
Quote:
QNX also seems to offer a new way to look at computing.
New way? Well they have based their OS around the same philosophy for over 20 years ;-)
Quote:
I would still like to get in touch with the designers ("scientists") at QNX. Is this possible?
Unless you have a business case, and money to wave at them, these people are real busy taking care of ongoing project. That being said the senior staff is sometimes made accessible through conferences/presentation. Some of them even raise their head and pop up in here or news:inn.qnx.com. |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Nov 04, 2007 - 11:22 PM
|
|
New Member
Joined: Nov 04, 2007
Posts: 2
|
|
|
Quote:
Fortran isn't officialy supported
Does it mean that there is no possibility to use fortran sources in QNX? Is there some workarounds i.e. use some GNU/Linux compiler? I'm just beginning with QNX in hope that it can handle my computations on weak hardware. (not looking for desktop os but for some software to change desktop pc into reliable machine for mathematic calculations)
How does look support for basic - that's the second language i need.
What would You adwice: translating all my sources to C and then compile it and test on QNX or some cross-platform compiling directly from fortran to binary? What is easier? |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Nov 05, 2007 - 05:08 AM
|
|
QNX Master
Joined: Sep 01, 2002
Posts: 2667
|
|
|
|
|
|
|
|
|
Post subject:
Posted: Nov 05, 2007 - 07:51 PM
|
|
QNX Master
Joined: Jun 25, 2003
Posts: 974
|
|
Some interesting questions and answers here. I can't add much but here goes.
1) It's not necessary to add a clustering layer to QNX, a big advantage.
2) There might be an application out there for which the QNX architecture and short thread switching times might make a good compute engine.
3) QNX is usually a little behind on the GNU compiler version. I'm not sure how that affects optimizations.
4) As Mario pointed out, the disk I/O is pretty lame. This is true for general file I/O, although for a specific application you might be able to get around this by doing direct to disk I/O.
5) If the existing Open GL support is not adequete, don't count on much improvement.
6) Forget about native Nvidia support unless you think you can bribe them yourself.
7) Getting the Fortran compiler going should be a reasonable task. For years there has been a Fortran to 'C' translator, although I don't know how good the code produced is.
 A better fit for QNX would be if you need to compute something in realtime.
9) I recall running a ported version of gbasic on QNX 4 (or was it 2) at one time. It was cute, but I would not want to have to count on it for serious computing. |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Nov 07, 2007 - 11:53 PM
|
|
New Member
Joined: Jul 22, 2002
Posts: 7
|
|
The fortran compiler - gfortran is accessible in QNX - http://www.ajam.org.pl/
There are also accessible libraries:
MPFR library => a C library for multiple-precision floating-point computations with correct rounding. - http://www.mpfr.org/
GMP => a free library for arbitrary precision arithmetic, operating on signed integers, rational numbers, and floating point numbers. - http://gmplib.org/ |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Nov 08, 2007 - 10:28 PM
|
|
New Member
Joined: Nov 04, 2007
Posts: 2
|
|
Thank you for your help.
I'll try both translators to C and gfortran.
Why not use (only) GNU/Linux:
1) I'm looking for possibility to do my simulations faster, and to get better control on time that each process communicate with another (it's few processes running simultaneously)
2) While porting application to another operating system I can find mistakes I haven't noticed earlier, also I think it might be a good way to get know QNX by working on the same task on known and unknown system.
3) I'm not sure if I could rely on any system without checking the result on another. Mathematic is one thing but programs made to check this ideas are another thing. I cannot misuse any possibility of falsification if I want to check the theory. |
|
|
| |
|
|
|
|
|
|
|
Post subject:
Posted: Nov 09, 2007 - 12:18 PM
|
|
QNX Master
Joined: Sep 01, 2002
Posts: 2667
|
|
1 ) I see, however real time does not always equal performance. For example gcc 4.2.1 which is officially available for LINUX produces faster code then what is available on QNX6 ( although now 4.2.1 is unofficially available ). Even better you can use Intel compilers and again, latest Intel Compiler is not available on QNX6 ( I think they are 2 version behind ). I'm talking C/C++ here. There might be better fortran compiler for Linux or even Windows. Intel compiler is very good at using SSEx.
My own test between Watcom 10.6 which is still pretty good for an old compiler, against Intel show a 25% increase in speed. That's for a real application not a benchmark.
If you do any disk intensive operation QNX6 might hurt you there, performance wise. |
|
|
| |
|
|
|
|
|
|
FAQ
Search
Register
Log in to check your private messages
Log in
