[Omake] Omake hangs when building a large product

[Michael Steinhaus]

> Does the hang happen in the -j1 build, or is it only happening when

You see, with -j4 the problem usually appears every second run or every
third run, and I would guess that with -j1 it's much less likely to
reproduce. And with -j1 building the entire product takes about one hour...
I tried to reproduce with -j1 today, but to no avail. Will continue
tomorrow.

> One obvious thing would be to try running OMake with the --verbose option
and trying to figure
> out what command(s) it is executing at the time of the hang

I tried to analyze the output, but noticed no pattern. It hangs at randomly.
Usually it's somewhere in the middle of building 7000+ targets, but today a
more interesting event happened: it hung after completing the entire build
(there even was the "omake: done" line in the output)! Every time it hangs
there are _no_ unterminated external processes (like compiler/linker).

I tried making snapshots with the "handle" utility from
www.sysinternals.comin parallel with building and noticed that at the
same moment omake hangs, a
handle to a non-existant process appears in its handle table -- a build rule
process that's just finished. Don't know whether it's cause of the hang or
just a consequence. According to debugger, no one of the multiple omake
threads hanging in the WaitForMultipleObjects/WaitForSingleObject is waiting
on those handles to non-existant processes.

> Did you try the latest 0.9.8.x sources from SVN - is that what you meant
by "the latest 0.9.8.5 sources"?

Yes, I did built the latest 0.9.8.x from SVN (I erroneously mentioned it as
0.9.8.5).

2008/3/19, Aleksey Nogin <nogin at metaprl.org>:
>
> Michael,
>
> Does the hang happen in the -j1 build, or is it only happening when
> building in parallel? One obvious thing would be to try running OMake
> with the --verbose option and trying to figure out what command(s) it is
> executing at the time of the hang (keeping in mind that the in the
> parallel version the command that is still executing might have been
> printed a while ago).
>
> It's quite possible that there is some internal deadlock in OMake, but
> it's also possible that something in your OMakefile fails to terminate
> (and if it's an internal command, then it will show up as an OMake
> thread).
>
> P.S. Did you try the latest 0.9.8.x sources from SVN - is that what you
> meant by "the latest 0.9.8.5 sources"?
>
> Aleksey
>
>
> On 18.03.2008 10:25, Michael Steinhaus wrote:
>
> > I'm designing an omake-based build system for a large product (Windows
> > platform, Microsoft toolchain, ~2.5MLOC of C++/C# code in more than a
> > hundred dll/lib projects). Everything works fine when building only
> > parts of the product. A problem appears when building the entire product
> > -- omake just hangs at some point.
> >
> > The box is Intel Core2Quad, 3GB RAM.
> > The command line is "omake -j4".
> > When the build process hangs, in the Task Manager I see omake consuming
> > 0% CPU resources and the are no other build-related processes running
> > (like compiler/linker).
> >
> > When I inject into the omake process with debugger, I see 11 threads.
> > Typically one of them is in the _Sleep at 4, another one in _ReadFile at 20, a
> > third one is in _GetNamedPipeInfo at 20, and the rest are in
> > WaitForMultipleObjects/WaitForSingleObject calls. On top of the call
> > stack, all the threads are hanging in the _KiFastSystemCallRet at 0 system
> > call.
> >
> > This looks like deadlock, but I have no idea how to troubleshoot it.
> > Unfortunately, the problem is elusive and doesn't always reproduce.
> >
>
> > I built the latest 0.9.8.5 <http://0.9.8.5> sources, but the problem is
>
> > still there. Tried to build 0.9.9 from sources, but failed.
> >
> > The product in question is a production system at Deutsche Bank, with
> > lots of legacy code and convoluted dependencies between individual
> > projects. I'm fascinated with the ease omake could cover the product's
> > complexity in a concise and generic way. I would like to convince the
> > management to choose omake as the official build tool. This problem with
> > the hang, however, is a critical block. I'd be greatful for any help or
> > advise.
> >
> > Regards,
> > Michael.
> >
> >
> >
> >
> >
> >
>
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > Omake mailing list
> > Omake at metaprl.org
> > https://lists.metaprl.org/mailman/listinfo/omake
>
> _______________________________________________
> Omake mailing list
> Omake at metaprl.org
> https://lists.metaprl.org/mailman/listinfo/omake
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.metaprl.org/pipermail/omake/attachments/20080320/25645c16/attachment.html