Skip navigation.
Home
The QNX Community Portal

View topic - Application hang without debug information

Application hang without debug information

anything that doesn't fit to other groups.

Re: Application hang without debug information

Postby Tim » Thu Nov 05, 2015 4:24 pm

I've never used Mudflap myself. Maybe Nico knows if he checks this thread again.

What I would suggest you do is to create a small test program that compiles in the same manner as your application. A 'Hello World' kind of simple program that does say 1 memory allocation and then exits. Then try launching that with Mudflap enabled to see if it crashes. If it does you know it's nothing to do with your application and something to do with how you've configured the IDE. At that point you can play with the IDE options until you get it to work.

I'll assume you have configured your IDE as per this link
http://www.qnx.com/developers/docs/6.5. ... nIDE_.html

Tim
Tim
Senior Member
 
Posts: 1388
Joined: Wed Mar 10, 2004 12:28 am

Re: Application hang without debug information

Postby Ericxx » Thu Nov 19, 2015 4:12 am

Thanks, Tim. I compiled a simple application with mudflap and can work.

Then my application accidentally generated a core dump file during execution. However my application was compiled in release mode. When I launched the postmortem analysis, I cannot get the whole picture of the code, but I can see the address where each thread got segmented fault. Then can I locate the culprit code with this dump file? Thanks.

Regards,
Eric
Ericxx
Senior Member
 
Posts: 158
Joined: Mon Jun 09, 2008 1:38 pm

Re: Application hang without debug information

Postby Ericxx » Thu Nov 19, 2015 10:02 am

Hello, I just used the ntoarm-gdb.exe with my stripped executable program and the core dump file. As shown in the attached screen shot, I can use the x/20i 0x103a145 to check the 20 instructions from address 0x103a145. However, from the assembly instructions, it is a bit hard to identify the crashed line. Is there a way out there based on the stripped program and core dump file? Thanks.

Regards,
Eric
Attachments
Program_crash_coreDump_screenshot.png
(33.18 KiB) Not downloaded yet
Ericxx
Senior Member
 
Posts: 158
Joined: Mon Jun 09, 2008 1:38 pm

Re: Application hang without debug information

Postby Tim » Thu Nov 19, 2015 3:36 pm

Eric,

Many of your threads are on the exact same point. Those are probably 'waiting' for something (receiving a message, waiting on a mutex etc) and are unlikely to be the culprit. It's most likely thread 1 that crashed.

There are 2 things you can do to help figure out what went wrong.

1) Add the creation of a map file as part of your build process. Then without making any changes to code, just re-link (I think map files are creating during linking) to get the map file. Look in the map file and you'll find the addresses of every function. You can cross reference those with the ones reported in the core file to see what function each thread was in.

It's possible (and likely) that all the crashes will end up being in functions in the standard library (strcpy for example if you walked off the end of a buffer) in which case you'll learn nothing from figuring out the actual line/function. If that happens proceed to step 2.

2) For each thread, do a stack trace. This shows all the functions that were called. That will let you work backwards up the function calls until you get to one that's in your code. That will at least pin point where in your code you were.

It's also possible one of the stack traces itself is corrupted. If that occurs you won't be able to go any further up the chain but you will know it means you have a stack corruption problem (local variable in some function) as opposed to a heap one (malloc based).

Tim

P.S. Note: Pin pointing the exact line of code is almost impossible unless you want to start counting assembly instructions from the start of your function (once you identify from step 1&2 above). Usually just getting the right function is enough because unless you have really complex functions there is normally only one or two variables that could be causing the crash.
Tim
Senior Member
 
Posts: 1388
Joined: Wed Mar 10, 2004 12:28 am

Re: Application hang without debug information

Postby Ericxx » Fri Nov 20, 2015 4:09 am

Dear Tim, thanks for your kind explanation.

I have generated the map file and found the memory location starts around from 0x00102850 (.init section) to 0x0013b494 (.text section). However, the crash location from the core dump 0x103a145 is out of the range above. During the core dump analysis, the gdb said the registers were corrupted. So the PC register points to the wrong location which is 0x103a145? The back trace seems not working in the release mode of core dump as well.

Regards,
Eric
Ericxx
Senior Member
 
Posts: 158
Joined: Mon Jun 09, 2008 1:38 pm

Re: Application hang without debug information

Postby Tim » Fri Nov 20, 2015 3:06 pm

Eric,

What do you mean by backtrace isn't working?

Is it showing something like:

#0 0x(some address) in ?? ()

The ?? indicates a stack corruption. A successful trace should always show several layers of function addresses (but not names since you are in release mode) that go all the way back up to the start point of the program/thread.

You can see this if you create a simple test program like:

Code: Select all
void foo()
{
    char buf[4];   // local buffer

    memset(buf, 0, 20);   // buffer over run corrupts stack

    int a = 10/0;   // divide by 0 does not corrupt stack

    printf("A is %d\n", a);    // Needed or compiler optimizes out the a=10/0 line
}

main {
    foo();
}


If you compile with the memset line active it will mangle the backtrace in the core file. If you comment that memset out and instead do the divide by 0 you will get a valid backtrace.

So it appears you mangled the stack in a local variable via a buffer overrun of some kind. I would think your other threads would still have valid backtraces...

This will be hard to solve from this core dump because you have no real info on where it happened. The only clue is its a stack issue and you have to look for local buffers that could be over run.

Tim
Tim
Senior Member
 
Posts: 1388
Joined: Wed Mar 10, 2004 12:28 am

Re: Application hang without debug information

Postby Ericxx » Wed Nov 25, 2015 1:08 pm

Thanks, Tim.
Yes, it shows like "#0 0x00000000 in ?? ()" in the debugger.

Then I put the debug version of binary and finally got a core dump after a few times of execution. Then I tried the postmortem debugger, it gave me the same result: no back trace information available... still have the "#0 0x00000000 in ?? ()" in the console debugger, though the symbols of the binary is available. Then is there any possible way to find the crash line information from this core dump file? I have tried to search for the possible local buffer overrun issues. Thanks.

Regards,
Eric
Ericxx
Senior Member
 
Posts: 158
Joined: Mon Jun 09, 2008 1:38 pm

Re: Application hang without debug information

Postby Tim » Sat Nov 28, 2015 3:22 am

Eric,

Unfortunately I don't know any way to find stack corruptions from dump files whether they are from release or debug executables.

You will literally have to hand trace the problem by examining the local variables in every function. This shouldn't be too bad because normally there aren't a lot of stack variables that can over flow. It's pretty much limited to local arrays like the one in my sample program OR wrong arguments to printf %'s.

If you are really struggling to find it you should be able to add some printf debug statements in those functions that have local arrays and print out something like 'enter XXX' when entering and 'exit XXX' when exiting function XXX.

Tim
Tim
Senior Member
 
Posts: 1388
Joined: Wed Mar 10, 2004 12:28 am

Previous

Return to General Programming

Who is online

Users browsing this forum: No registered users and 3 guests