Skip navigation.
Home
The QNX Community Portal

View topic - unexpected interrupt delay

unexpected interrupt delay

Read-only archive of qnx.rtos (Writing resources managers, and general discussion around the QNX Neutrino RTOS) at inn.qnx.com

unexpected interrupt delay

Postby dadji » Wed Dec 19, 2007 11:55 am

Hello,
i'm writing an application that handles incomming interrupts from an
external device and with a constant rate of 8kHz.
The application has the highest piority on the system (qnx6.3.2, x86). The
interrupt handling needs a few microsec. It notice that some interrupts come
in the system with delay of 40 - 200 and more microsec. To find it out, it
just measure the time delay of two consecutive interrupts in the handling
function. My question is, what can be the cause of this unexpected delay? Is
there any system operation that causes this behaviour and how can i suppress
it. As i started the tracelogger, i also notice that it causes an interrupt
delay.

Thanks in advance for your help.
Yannick
dadji
 

Re: unexpected interrupt delay

Postby kwschumm » Wed Dec 19, 2007 1:42 pm

"dadji" <ydadji@hotmail.com> wrote in message
news:fkb0qe$l1v$1@inn.qnx.com...
Hello,
i'm writing an application that handles incomming interrupts from an
external device and with a constant rate of 8kHz.
The application has the highest piority on the system (qnx6.3.2, x86). The
interrupt handling needs a few microsec. It notice that some interrupts
come in the system with delay of 40 - 200 and more microsec. To find it
out, it just measure the time delay of two consecutive interrupts in the
handling function. My question is, what can be the cause of this
unexpected delay? Is there any system operation that causes this behaviour
and how can i suppress it. As i started the tracelogger, i also notice
that it causes an interrupt delay.

The interrupt you are using could be preempted by a higher priority
interrupt, or an application somewhere could be disabling interrupts for a
period of time, or (depending on hardware) the system mgmt. interrupt could
be preempting it, or your video (depending on hardware) could be preempting
it. The fact that loading tracelogger causes a delay may indicate that the
media that tracelogger is loaded from (disk drive, flash, ?) uses an
interrupt that preempts your applications (or the driver disables interrupts
for a lengthy time period).

On x86 you can move interrupt priorities to some extent to make yours higher
than others.

What is your hardware platform? What interrupt does your app use?
kwschumm
Senior Member
 
Posts: 306
Joined: Fri Sep 17, 2004 12:46 pm

Re: unexpected interrupt delay

Postby dadji » Thu Dec 20, 2007 10:05 am

Hello Ken,

"Ken Schumm" <kwschumm@qsolv.com> schrieb im Newsbeitrag
news:fkb74u$pbf$1@inn.qnx.com...
"dadji" <ydadji@hotmail.com> wrote in message
news:fkb0qe$l1v$1@inn.qnx.com...
Hello,
i'm writing an application that handles incomming interrupts from an
external device and with a constant rate of 8kHz.
The application has the highest piority on the system (qnx6.3.2, x86).
The interrupt handling needs a few microsec. It notice that some
interrupts come in the system with delay of 40 - 200 and more microsec.
To find it out, it just measure the time delay of two consecutive
interrupts in the handling function. My question is, what can be the
cause of this unexpected delay? Is there any system operation that causes
this behaviour and how can i suppress it. As i started the tracelogger, i
also notice that it causes an interrupt delay.

The interrupt you are using could be preempted by a higher priority
interrupt, or an application somewhere could be disabling interrupts for a
period of time, or (depending on hardware) the system mgmt.

The thread handling the interrupt has the higher priority of thte system. Is
it common that a system application disable the interrupts for more the 200
µs on a real time operating system? i didn't consider this issue. Yesterday,
i did an additional measurement. i measure the delay between the event
signaling (return event) in the Interrupt Service routine and the event
handling (unblock of InterruptWait() ) in the handling thread. i notice that
this delay is in best case 2,5 and worst case 28 µs. I don t have any
explanation for this variation, do you?


interrupt could
be preempting it, or your video (depending on hardware) could be
preempting it. The fact that loading tracelogger causes a delay may
indicate that the media that tracelogger is loaded from (disk drive,
flash, ?) uses an interrupt that preempts your applications (or the driver
disables interrupts for a lengthy time period).

On x86 you can move interrupt priorities to some extent to make yours
higher than others.

What is your hardware platform? What interrupt does your app use?

I work on a x86 platform under qnx 6.3.2. my app handle hardware interrupt
from a IEEE1394 (FireWire) PCI card



thank you and best regards.
dadji
 

Re: unexpected interrupt delay

Postby Mario Charest » Thu Dec 20, 2007 1:46 pm

The thread handling the interrupt has the higher priority of thte system.
Is
it common that a system application disable the interrupts for more the
200 µs on a real time operating system?

Although it's never guruantied 100% system application don't disable
interrupt during normal operation. However 3rd party application or your
own may do so.

i didn't consider this issue. Yesterday, i did an additional measurement. i
measure the delay between the event signaling (return event) in the
Interrupt Service routine and the event handling (unblock of
InterruptWait() ) in the handling thread. i notice that this delay is in
best case 2,5 and worst case 28 µs. I don t have any explanation for this
variation, do you?

Any other interrupt will have priority over the thread handling the
interrupt. Try the system profiler in the IDE to figure it out.
Mario Charest
 

Re: unexpected interrupt delay

Postby kwschumm » Thu Dec 20, 2007 2:28 pm

"dadji" <ydadji@hotmail.com> wrote in message
news:fkdepj$dos$1@inn.qnx.com...
Hello Ken,

"Ken Schumm" <kwschumm@qsolv.com> schrieb im Newsbeitrag
news:fkb74u$pbf$1@inn.qnx.com...
"dadji" <ydadji@hotmail.com> wrote in message
news:fkb0qe$l1v$1@inn.qnx.com...
[...]

The thread handling the interrupt has the higher priority of thte system.

The thread priority only matters once it receives the interrupt. If the
interrupt delivery is delayed for any reason (interrupts being disabled,
preemption by a higher priority interrupt, something hogging the bus), then
your thread won't even be scheduled until the interrupt is delivered.


Is it common that a system application disable the interrupts for more the
200 µs on a real time operating system? i didn't consider this issue.
Yesterday, i did an additional measurement. i measure the delay between
the event signaling (return event) in the Interrupt Service routine and
the event handling (unblock of InterruptWait() ) in the handling thread. i
notice that this delay is in best case 2,5 and worst case 28 µs. I don t
have any explanation for this variation, do you?

QNX code probably does not disable interrupts for more than a microsecond or
two, they're generally very careful, but any third party driver or
application could do it. When programmers aren't geared towards real-time
they sometimes find disabling interrupts to be a quick and easy way to solve
a problem.
kwschumm
Senior Member
 
Posts: 306
Joined: Fri Sep 17, 2004 12:46 pm

Re: unexpected interrupt delay

Postby Robert Craig » Thu Dec 20, 2007 5:35 pm

Mario also mentioned the System Management Interrupt and this one is a
particularly nasty one. I remember the SMI interrupt causing delays of
200usec (or more) on one platform that I was working on. This 200usec
is effectively a HW delay (i.e. the OS isn't involved at all. I believe
that it jumps into the BIOS and starts executing code (?)). I'm not
sure what the interrupt does, but the only way to remove the delay was
to disable the SMI. How you do this is CPU dependent.

On another platform, we found that the way that the USB was set up in
the BIOS would affect the interrupt latency. Disabling USB Support and
USB Legacy support fixed things for us.

I'd still agree with Mario that you should use the system profiler to
check and see if your interrupt is getting pre-empted by a higher
priority interrupt. You might also want to look at adjusting the
interrupt priorities themselves to make yours the highest. 200 usec does
seem like quite a long time for a pre-emption though.

Robert.


Ken Schumm wrote:
"dadji" <ydadji@hotmail.com> wrote in message
news:fkdepj$dos$1@inn.qnx.com...
Hello Ken,

"Ken Schumm" <kwschumm@qsolv.com> schrieb im Newsbeitrag
news:fkb74u$pbf$1@inn.qnx.com...
"dadji" <ydadji@hotmail.com> wrote in message
news:fkb0qe$l1v$1@inn.qnx.com...
[...]
The thread handling the interrupt has the higher priority of thte system.

The thread priority only matters once it receives the interrupt. If the
interrupt delivery is delayed for any reason (interrupts being disabled,
preemption by a higher priority interrupt, something hogging the bus), then
your thread won't even be scheduled until the interrupt is delivered.


Is it common that a system application disable the interrupts for more the
200 µs on a real time operating system? i didn't consider this issue.
Yesterday, i did an additional measurement. i measure the delay between
the event signaling (return event) in the Interrupt Service routine and
the event handling (unblock of InterruptWait() ) in the handling thread. i
notice that this delay is in best case 2,5 and worst case 28 µs. I don t
have any explanation for this variation, do you?

QNX code probably does not disable interrupts for more than a microsecond or
two, they're generally very careful, but any third party driver or
application could do it. When programmers aren't geared towards real-time
they sometimes find disabling interrupts to be a quick and easy way to solve
a problem.

Robert Craig
 

RE: Re: unexpected interrupt delay

Postby maschoen » Thu Dec 20, 2007 7:15 pm

SMI is usually used to emulate in software some hardware feature that is missing and can be a cost saver. I've seen it used to emulate missing video modes by doing pixel stretching and also emulation of a Sound Blaster card. It probably could be used to make a USB keyboard look like one attached to the standard keyboard interface.
maschoen
QNX Master
 
Posts: 2644
Joined: Wed Jun 25, 2003 5:18 pm

Re: unexpected interrupt delay

Postby Robert Craig » Thu Dec 20, 2007 9:12 pm

Interesting.... My technical assessment? Yuck! :->>

Robert.


maschoen wrote:
SMI is usually used to emulate in software some hardware feature that
is missing and can be a cost saver. I've seen it used to emulate
missing video modes by doing pixel stretching and also emulation of a
Sound Blaster card. It probably could be used to make a USB keyboard
look like one attached to the standard keyboard interface.
Robert Craig
 

RE: Re: unexpected interrupt delay

Postby maschoen » Fri Dec 21, 2007 6:36 am

Yuck! is exactly right. You might as well be running on VM under XP. SMI turns a cpu into a NON-realtime piece of hardware. All that careful work to minimize interrupt latency in the OS all gone to waste.
maschoen
QNX Master
 
Posts: 2644
Joined: Wed Jun 25, 2003 5:18 pm

Re: unexpected interrupt delay

Postby Mario Charest » Fri Dec 21, 2007 6:49 pm

"Robert Craig" <rcraig_at_qnx@nowhere.com> wrote in message
news:fkelqf$9k1$1@inn.qnx.com...
Interesting.... My technical assessment? Yuck! :-

Here is another one for you:


On a system with about 10 PC, one in a while after a reboot, a few of the
computer were running real slow. It was the strangest thing ever, any
timing based on rdtsc was ok but every else was out to lunch. #sleep 1
would last 2-3 seconds...

The culprit was a smart switch box we use to share one USB keyboard/Monitor
with the 10 PC. Turns out the switch box was buggy and was sending data on
the USB port at a very high rate. This data cause constant invokation of
the BIOS/SMI, which in turn slowed down the system.... On these perticular
PC we can't turn USB emulation because they won't boot from a USB key which
is something we need. Upgrading the firmware in the keyboard switch box
solved the problem... That was a nasty one.


Robert.


maschoen wrote:
SMI is usually used to emulate in software some hardware feature that
is missing and can be a cost saver. I've seen it used to emulate
missing video modes by doing pixel stretching and also emulation of a
Sound Blaster card. It probably could be used to make a USB keyboard
look like one attached to the standard keyboard interface.
Mario Charest
 

Re: unexpected interrupt delay

Postby John Nagle » Tue Dec 25, 2007 5:27 am

Yes. This is a known problem with some CPU boards.
Even some embedded boards have this problem. Look for
emulated old-style serial ports, IDE disks emulated with
flash memory, USB keyboard emulation, and similar
stuff being done in system management mode.
When you have unexpected interrupt delays, give the full
hardware configuration.

There should be no long interrupt lockouts in any
standard QNX software.

How are you measuring the time delay between interrupts?

John Nagle

Robert Craig wrote:
Interesting.... My technical assessment? Yuck! :-

Robert.


maschoen wrote:
SMI is usually used to emulate in software some hardware feature that
is missing and can be a cost saver. I've seen it used to emulate
missing video modes by doing pixel stretching and also emulation of a
Sound Blaster card. It probably could be used to make a USB keyboard
look like one attached to the standard keyboard interface.
John Nagle
 

RE: Re: unexpected interrupt delay

Postby maschoen » Tue Dec 25, 2007 9:14 am

One irony is that SMI is more likely to appear in a SBC than a desktop PC. The board manufactures do this to keep costs down, and either aren't considering, or aware of the real-time issues involved. Some all-in-one cpu chips will come with SMI features to reduce board space and provide these features at a relatively reduced cost compared with having the actual hardware. This is great for the embedded market, and hell for the real-time market.
maschoen
QNX Master
 
Posts: 2644
Joined: Wed Jun 25, 2003 5:18 pm

Re: unexpected interrupt delay

Postby dadji » Wed Jan 02, 2008 11:09 am

"John Nagle" <nagle@downside.com> schrieb im Newsbeitrag
news:fkq45f$mt$1@inn.qnx.com...
Yes. This is a known problem with some CPU boards.
Even some embedded boards have this problem. Look for
emulated old-style serial ports, IDE disks emulated with
flash memory, USB keyboard emulation, and similar
stuff being done in system management mode.
When you have unexpected interrupt delays, give the full
hardware configuration.

There should be no long interrupt lockouts in any
standard QNX software.

How are you measuring the time delay between interrupts?


to measure the interrupt delay i use the Clockcycle() call, and get the
system time by each incoming interrupt. Then i build the time difference
between two consecutive interrupts. Since the interrupts are expected at
least every 125µs, i can set an upper time range to detect the missing
interrupts. For examples i measure the time difference of two consecutive
interrupt of more than 300 µs.
I can also see the effect of the interrupt delay on the scope using the
parallel port pins.

best regards.


John Nagle

Robert Craig wrote:
Interesting.... My technical assessment? Yuck! :-

Robert.


maschoen wrote:
SMI is usually used to emulate in software some hardware feature that
is missing and can be a cost saver. I've seen it used to emulate
missing video modes by doing pixel stretching and also emulation of a
Sound Blaster card. It probably could be used to make a USB keyboard
look like one attached to the standard keyboard interface.



Yannick
dadji
 

Re: unexpected interrupt delay

Postby Armin » Thu Jan 03, 2008 10:28 am

dadji wrote:
"John Nagle" <nagle@downside.com> schrieb im Newsbeitrag
news:fkq45f$mt$1@inn.qnx.com...
Yes. This is a known problem with some CPU boards.
Even some embedded boards have this problem. Look for
emulated old-style serial ports, IDE disks emulated with
flash memory, USB keyboard emulation, and similar
stuff being done in system management mode.
When you have unexpected interrupt delays, give the full
hardware configuration.

There should be no long interrupt lockouts in any
standard QNX software.

How are you measuring the time delay between interrupts?


to measure the interrupt delay i use the Clockcycle() call, and get the
system time by each incoming interrupt.

Are you using an ISR + interrupt handler thread or just an interrupt thread?

If you are using an ISR ... what is the priority of the returned event?
Is the SIGEV_PULSE_PRIO_INHERIT flag set?

If yes, the interrupt handler thread inherits the low(?) priority of the
pulse event. If no, what is the priority of the interrupt handler ?

If your device is a PCI device ... is its interrupt shared by an other
device?

If yes, is the other driver disabling the interrupt for a long time?

Could it be that the interrupt logic of the hardware of you device has a
bug? Is the interrupt logic implemented in a FPGA??

--Armin



Then i build the time difference
between two consecutive interrupts. Since the interrupts are expected at
least every 125µs, i can set an upper time range to detect the missing
interrupts. For examples i measure the time difference of two consecutive
interrupt of more than 300 µs.
I can also see the effect of the interrupt delay on the scope using the
parallel port pins.

best regards.


John Nagle

Robert Craig wrote:
Interesting.... My technical assessment? Yuck! :-

Robert.


maschoen wrote:
SMI is usually used to emulate in software some hardware feature that
is missing and can be a cost saver. I've seen it used to emulate
missing video modes by doing pixel stretching and also emulation of a
Sound Blaster card. It probably could be used to make a USB keyboard
look like one attached to the standard keyboard interface.



Yannick

Armin
 

Re: unexpected interrupt delay

Postby dadji » Fri Jan 04, 2008 2:42 pm

"Armin" <a@nospam.org> schrieb im Newsbeitrag
news:fligg4$dm5$1@inn.qnx.com...
dadji wrote:
"John Nagle" <nagle@downside.com> schrieb im Newsbeitrag
news:fkq45f$mt$1@inn.qnx.com...
Yes. This is a known problem with some CPU boards.
Even some embedded boards have this problem. Look for
emulated old-style serial ports, IDE disks emulated with
flash memory, USB keyboard emulation, and similar
stuff being done in system management mode.
When you have unexpected interrupt delays, give the full
hardware configuration.

There should be no long interrupt lockouts in any
standard QNX software.

How are you measuring the time delay between interrupts?


to measure the interrupt delay i use the Clockcycle() call, and get the
system time by each incoming interrupt.

Are you using an ISR + interrupt handler thread or just an interrupt
thread?

If you are using an ISR ... what is the priority of the returned event?
Is the SIGEV_PULSE_PRIO_INHERIT flag set?

If yes, the interrupt handler thread inherits the low(?) priority of the
pulse event. If no, what is the priority of the interrupt handler ?

If your device is a PCI device ... is its interrupt shared by an other
device?

If yes, is the other driver disabling the interrupt for a long time?

Could it be that the interrupt logic of the hardware of you device has a
bug? Is the interrupt logic implemented in a FPGA??

--Armin

i use ISR + interrupt handler thread. The signaling event is of type
SIGEV_INTR, so no priority allocated.
The interrupt line was shared with other devices. Now i have set my device
to a single (not shared) interrupt line manually, but i have the same error.
I don't think about hardware bug (standard IEEE 1394 device), may be i
didn't consider something in the software.
thanks.
Yannick




Then i build the time difference between two consecutive interrupts.
Since the interrupts are expected at least every 125µs, i can set an
upper time range to detect the missing interrupts. For examples i measure
the time difference of two consecutive interrupt of more than 300 µs.
I can also see the effect of the interrupt delay on the scope using the
parallel port pins.

best regards.


John Nagle

Robert Craig wrote:
Interesting.... My technical assessment? Yuck! :-

Robert.


maschoen wrote:
SMI is usually used to emulate in software some hardware feature that
is missing and can be a cost saver. I've seen it used to emulate
missing video modes by doing pixel stretching and also emulation of a
Sound Blaster card. It probably could be used to make a USB keyboard
look like one attached to the standard keyboard interface.



Yannick
dadji
 

Next

Return to qnx.rtos

Who is online

Users browsing this forum: Google [Bot] and 1 guest

cron