Remote File Access With Emacs

bridged with qdn.public.qnxrtp.applications
Bill Caroselli

Re: Remote File Access With Emacs

Post by Bill Caroselli » Fri Aug 01, 2003 2:14 pm

Maybe QSSL could appoligize to Ted Green and we can get a copy of vedit for
QNX6.

"Rick Duff" <rick@astranetwork.com> wrote in message
news:bg5uvr$201$1@inn.qnx.com...
Andrew Thomas wrote:
A few more voices might make QNX notice.


For what's worth, add my voice. ;-) I would hate to be forced to use
vim or that stupid editor in the ide to do anything. I would be forced
to use Charles approach and run the editor on a linux machine.

Hey QSS, what's the issue from your point of view with making these
changes?

Rick

--
Rick Duff Internet: rick@astranetwork.com
Astra Network URL: http://www.astranetwork.com
QNX Consulting and Custom Programming Phone: +1 (204) 997-NETW (6389)

Guest

Re: Remote File Access With Emacs

Post by Guest » Fri Aug 01, 2003 6:09 pm

On Fri, 1 Aug 2003 09:14:10 -0500, "Bill Caroselli"
<QTPS@Earthlink.net> wrote:
Maybe QSSL could appoligize to Ted Green and we can get a copy of vedit for
QNX6.
I couldn't stand vedit myself but there were a couple of developers
around here that used it. Everyone has a favorite I guess. I was a
big fan of JED on QNX4 and it seems to run equally well on QNX6 (it's
color syntax highlighting works properly and it's even fast under
Phindows). I particularly like JED's macro language, SLang. It's
simple and powerful. Emacs' Lisp causes my brain to meltdown turning
me into a quivering blob of Jell-o. I actually think JED is a better
editor than Emacs.

That being said, I'm used to Emacs and use Emacs on every other
platform I work on. The JED Emacs emulation is not bad but it's not
Emacs. I've had trouble getting JED on different platforms to behave
the same way with my QNX JED configuration file.


C.

Andrew Thomas

Re: Remote File Access With Emacs

Post by Andrew Thomas » Tue Aug 05, 2003 6:09 pm

On Thu, 31 Jul 2003 13:45:21 -0400, Adam Mallory <amallory@qnx.com> wrote:
Andrew Thomas <andrew@cogent.ca> wrote in message
news:oprs6jdtplw9rgrq@nntp.qnx.com...
Thanks for responding, Adam. In reality this is not a matter of policy.
QNX has already established the policy of presenting certain parts of
the source tree to public CVS. The problem is one of implementation of
that policy.

True, but policies change (not saying they will/are, just saying...), and
the 'powers that be' are still the decision makers in this case, not the
technical people. Anyways, I digress...
Public policies are supposed to change publically. Has QNX decided not
to maintain the public CVS archive?
Well as a start, you can setup a SIGSEGV handler and setjmp/longjmp (ie.
try/catch) on pages in the heap to figure out what is where. Each new
area you discover can be a new data section in the ELF excutable which on
load, would be collected into the data loadable segment at execution time
(IIRC).
Ouch. That sounds like a last resort.
Well the UNIX standard says that the use of brk()/sbrk() is unspecified
when an application uses any other memory functions such as (not limited
to)
malloc/free/mmap. I'm not trying to cop out, but the use of brk/sbrk is
very fragile in the Emacs case IMHO.
[...]

Well the reliance on sbrk/brk is a historical thing, and it's meaning is
not very clear in a virtual memory system (since it was put in long
before
virtual memory). I can see a world of hurt coming on for Emacs and
anything that relies on brk/sbrk.
Emacs actually only uses sbrk(0) to determine the end of virtual memory
for the process. If you can tell me another method, I'll remove sbrk
in a heartbeat.
I really think this is harmless, and I also suspect that it is a
necessary precondition for a functioning sbrk() implementation.

The change is already in, as an option to mallopt() (MALLOC_MEMORY_HOLD
is the flag if memory serves) as part of a larger scheme to control the
return of memory to the system at different rates (rather than just
always or never).
Excellent. Which version of the OS has this? I'm running
QNX 6.2.1 2003/01/08-02:15:12EST
I don't know what patch level that would be. This is not documented
and the constant does not exist in this version.

Thanks,
Andrew

Adam Mallory

Re: Remote File Access With Emacs

Post by Adam Mallory » Tue Aug 05, 2003 7:35 pm

Andrew Thomas <andrew@cogent.ca> wrote in message
news:oprtf11r0kw9rgrq@nntp.qnx.com...
Public policies are supposed to change publically. Has QNX decided not
to maintain the public CVS archive?
Like I said, I have no clue to policy direction, and I'm not suggesting that
the policy will/won't change. Anyways this is OT and I have nothing of
value to add except 'I don't know'.
Ouch. That sounds like a last resort.
Much like using a relic like sbrk(), and since this is 'setup' time the
extra work really isn't that much in the grand scheme of Emacs land.
Emacs actually only uses sbrk(0) to determine the end of virtual memory
for the process. If you can tell me another method, I'll remove sbrk
in a heartbeat.
If Emacs uses any memory function together with sbrk() the results are not
defined regardless of what the rational for using sbrk() is. In addition,
the assumption that the heap is virtually contiguous is, IMHO, frail to say
the least - lots of allocator schemes out there break that assumption (not
just ours).

The best method I think would be for Emacs to provide its own hunk
allocator, which can setup the heap anyway it sees fit. This would solve
the issue of heap information (start/stop/contig?) and layout. Also it
would give you a much more robust way of doing what you're trying to
acomplish.
Excellent. Which version of the OS has this? I'm running
QNX 6.2.1 2003/01/08-02:15:12EST
I don't know what patch level that would be. This is not documented
and the constant does not exist in this version.
It's not out, it will be in a future release - hence the fact you don't have
it, nor is it doc'd.

Cheers,
-Adam

Andrew Thomas

Re: Remote File Access With Emacs

Post by Andrew Thomas » Wed Aug 06, 2003 8:55 pm

On Tue, 5 Aug 2003 15:35:24 -0400, Adam Mallory <amallory@qnx.com> wrote:
Andrew Thomas <andrew@cogent.ca> wrote in message
news:oprtf11r0kw9rgrq@nntp.qnx.com...

If Emacs uses any memory function together with sbrk() the results are
not defined regardless of what the rational for using sbrk() is. In
addition, the assumption that the heap is virtually contiguous is, IMHO,
frail to say the least - lots of allocator schemes out there break that
assumption (not just ours).
Interestingly, the *very* many operating systems supported by emacs either
provide a system allocator that does maintain contiguous virtual memory
or provide a functioning sbrk() that allows gmalloc to work. While I agree
with you in principle, I don't see much evidence for it in practise.

Emacs can be configured to use gmalloc, which uses sbrk() exclusively.
The problem is that sbrk() does not work as documented, even when it is
the only memory allocator in use. Part of this is likely due to other
library functions using other memory allocation techniques (shared memory
or segment allocation?), which makes it impossible not to "use any memory
function together with sbrk()".
The best method I think would be for Emacs to provide its own hunk
allocator, which can setup the heap anyway it sees fit. This would solve
the issue of heap information (start/stop/contig?) and layout. Also it
would give you a much more robust way of doing what you're trying to
acomplish.
This is the gmalloc route. It fails. I cannot remember how, frankly,
but as I said, it has to do with sbrk() not really obeying its own
documentation.

Generally speaking, I am willing to put in a few days of work to port
emacs to QNX6, but I am not willing to totally rewrite its memory
allocators or ELF dumping code. I was rather hoping that QNX would
try to support emacs, at least passively - it's one of the most popular GNU
tools. It is unfortunately also a pretty severe test case for
"compatibility", where compatibility is defined as
"conformance to a legacy assortment of random side-effects". We don't have
to like it, but it's the world we live in.

Cheers,
Andrew

Adam Mallory

Re: Remote File Access With Emacs

Post by Adam Mallory » Wed Aug 06, 2003 10:01 pm

Andrew Thomas <andrew@cogent.ca> wrote in message
news:oprth4dznow9rgrq@nntp.qnx.com...
Interestingly, the *very* many operating systems supported by emacs either
provide a system allocator that does maintain contiguous virtual memory
or provide a functioning sbrk() that allows gmalloc to work. While I
agree
with you in principle, I don't see much evidence for it in practise.
Partly because they have the luxury of doing so - if they actually need the
physical memory backing the virtual address space, they can swap it out and
the next reference will fault it back in on demand - we not going to do that

<snip>
This is the gmalloc route. It fails. I cannot remember how, frankly,
but as I said, it has to do with sbrk() not really obeying its own
documentation.
Well, actually you can do a hunk allocator without sbrk - just mmap() in
hunks of memory and provide a slab like object allocator from that. The
fact you do your own allocator already means you have the memory map, and
can do away with sbrk all together.
Generally speaking, I am willing to put in a few days of work to port
emacs to QNX6, but I am not willing to totally rewrite its memory
allocators or ELF dumping code. I was rather hoping that QNX would
try to support emacs, at least passively - it's one of the most popular
GNU
tools.
Well, putting features into the allocator to get some desktop domain program
ported isn't what we want to target IMHO. In this narrow case, we can put a
feature in to control the rate of release back to the system (the
pathalogical case being never) which can help the port. But if the choice
has to come between a 'traditionally conforming' sbrk() and a more efficient
allocator - the choice isn't hard.
It is unfortunately also a pretty severe test case for
"compatibility", where compatibility is defined as
"conformance to a legacy assortment of random side-effects". We don't
have
to like it, but it's the world we live in.
I don't debate that - but then one needs to understand that one day those
'side-effects' won't be around.

FYI - thanks for all you input Andrew, much appreciated!

-Adam

Andrew Thomas

Re: Remote File Access With Emacs

Post by Andrew Thomas » Fri Aug 08, 2003 5:11 pm

On Wed, 6 Aug 2003 18:01:46 -0400, Adam Mallory <amallory@qnx.com> wrote:
Andrew Thomas <andrew@cogent.ca> wrote in message
news:oprth4dznow9rgrq@nntp.qnx.com...

Interestingly, the *very* many operating systems supported by emacs
either provide a system allocator that does maintain contiguous virtual
memory
or provide a functioning sbrk() that allows gmalloc to work. While I
agree with you in principle, I don't see much evidence for it in
practise.

Partly because they have the luxury of doing so - if they actually need
the physical memory backing the virtual address space, they can swap it
out and the next reference will fault it back in on demand - we not going
to do that.
I don't understand the correlation between the last two paragraphs.
I didn't think that contiguous virtual memory in any way implied a
swap implementation or a relationship with physical memory.
Well, actually you can do a hunk allocator without sbrk - just mmap() in
hunks of memory and provide a slab like object allocator from that. The
fact you do your own allocator already means you have the memory map, and
can do away with sbrk all together.
*I* don't do *my* own allocator. I'm configuring emacs to use one
of the many memory allocation schemes that it has available to it.
It requires one of two possibilities:
a) a functioning sbrk() that can be used in a malloc() replacment
that all library functions use to the exclusion of other memory
allocation techniques. QNX seems to fail here by not respecting
malloc() replacements. Many people have asked for this.
b) a memory allocator that can be optionally made to maintain
contiguous virtual memory, and a way to determine the beginning
and end of that virtual memory.
These are not ridiculous requirements.
Well, putting features into the allocator to get some desktop domain
program ported isn't what we want to target IMHO. In this narrow case,
we can put a feature in to control the rate of release back to the system
(the
pathalogical case being never) which can help the port. But if the
choice has to come between a 'traditionally conforming' sbrk() and a more
efficient allocator - the choice isn't hard.
This is more than just emacs. Like I said, Common LISP environments,
some scheme and smalltalk environments, and many C/C++ garbage collectors
require similar capabilities.

In any case, this is not a choice between 'traditionally conforming' and
'efficient'. Both are possible simultaneously. If QNX actually allowed
drop-in replacments for malloc() then all that would be required is to
fix sbrk(), and the job is done. If you use sbrk() to grab system memory,
you can have a 'traditionally conforming' allocator. If you do not, then
you can have an arbitrarily efficient one.

QNX publishes a library with sbrk() in it. This function does *something*,
just not a correct thing. You could end this line of questioning by
removing sbrk() from the library, but as long as you continue to ship it,
I am just hoping you will also make it work.
It is unfortunately also a pretty severe test case for
"compatibility", where compatibility is defined as
"conformance to a legacy assortment of random side-effects". We don't
have to like it, but it's the world we live in.

I don't debate that - but then one needs to understand that one day those
'side-effects' won't be around.
Not in the lifetime of Linux, is my guess. As long as you want to claim
that QNX is compatible with existing UNIX software, supporting some of
those side-effects is a necessity. This particular side-effect is just
one of the harder ones. And you nearly have it. Why not take the
extra baby step here?

Cheers,
Andrew

Adam Mallory

Re: Remote File Access With Emacs

Post by Adam Mallory » Fri Aug 08, 2003 8:59 pm

Andrew Thomas <andrew@cogent.ca> wrote in message
news:oprtljcvj2w9rgrq@nntp.qnx.com...
I don't understand the correlation between the last two paragraphs.
I didn't think that contiguous virtual memory in any way implied a
swap implementation or a relationship with physical memory.
The fact that if you have a virtually contiguous heap, in QNX also means you
have physical backing store for it (ie. real memory). So if we avoid using
'real' memory by releasing it back to the system, then we do so. Other OS's
(FreeBSD, Linux etc) have the luxury of not doing that (ie releasing
directly back via munmap()) because if the system needs more 'real' memory,
it's possible to just take away the backingstore, while still leaving the
heap virtually contiguous (although nothing is behind it).
*I* don't do *my* own allocator. I'm configuring emacs to use one
of the many memory allocation schemes that it has available to it.
It requires one of two possibilities:
a) a functioning sbrk() that can be used in a malloc() replacment
that all library functions use to the exclusion of other memory
allocation techniques. QNX seems to fail here by not respecting
malloc() replacements. Many people have asked for this.
b) a memory allocator that can be optionally made to maintain
contiguous virtual memory, and a way to determine the beginning
and end of that virtual memory.
These are not ridiculous requirements.
No one said they were ridiculous. When you have fixed resources you must
prioritize where you spend them - as such getting an emacs port isn't high
on the list. I'm not suggesting that you do the implementation of the
allocator, I just think emacs could stand to be a tad more portable.
This is more than just emacs. Like I said, Common LISP environments,
some scheme and smalltalk environments, and many C/C++ garbage collectors
require similar capabilities.
The key word is 'some' - 'some' software is written in less portable
manners, that's life too. And I agree that we should do what we can to ease
the pain, no agruments there.
In any case, this is not a choice between 'traditionally conforming' and
'efficient'. Both are possible simultaneously. If QNX actually allowed
drop-in replacments for malloc() then all that would be required is to
fix sbrk(), and the job is done. If you use sbrk() to grab system memory,
you can have a 'traditionally conforming' allocator. If you do not, then
you can have an arbitrarily efficient one.
I'm not sure why you say we don't support 'drop in replacement', you can use
any allocator you like. Just ensure that symbols for malloc/free etc
resolve to your allocator first, then resolve against libc.
QNX publishes a library with sbrk() in it. This function does
*something*,
just not a correct thing. You could end this line of questioning by
removing sbrk() from the library, but as long as you continue to ship it,
I am just hoping you will also make it work.
Well, I'll impose on you once more to tell me what exactly you find broken
and I'll do my best to see what can be done about it.
Not in the lifetime of Linux, is my guess. As long as you want to claim
that QNX is compatible with existing UNIX software, supporting some of
those side-effects is a necessity. This particular side-effect is just
one of the harder ones. And you nearly have it. Why not take the
extra baby step here?
I agree that we'll do what we can, and I don't recall anyone saying 'no'.

Cheers,
-Adam

Post Reply

Return to “qdn.public.qnxrtp.applications”