[m-dev.] Declarative debugging paper accepted.

[Ian MacLarty]

The declarative debugging paper has been accepted!

Below are the reviewer's comments.  Some of the issues raised (such as  
trusting and "don't know" answers) are implemented for the Mercury  
declarative debugger, but were omitted from the paper because of space  
limitations and the fact that they are in themselves not hugely  
original contributions.

The main critism seems to be the lack of references to related work.

Cheers,

Ian.

Begin forwarded message:

>
> Dear Mr. Ian MacLarty:
>
> On behalf of the AADEBUG 2005 Program Committee, we are
> delighted to inform you that the following paper has
> been accepted to appear at the conference:
>
>      Divide-and-query and subterm dependency tracking in
>            the Mercury declarative debugger
>
> The program committee decided to assign a member
> to help "shepherd" your paper.  Please contact this
> person, and double-check on the specific revisions
> we're requesting.
>
> The PC member to contact is:
>
>     Henrik Nilsson at  nhn at cs.nott.ac.uk
>
> This person is the editor responsible for the end-result
> of the paper.  Before you send in your final copy, he
> will need to notify us that the revision addresses
> some issues raised in the reviews.
>
> Aside from this provision, consider your paper accepted
> to AADEBUG 2005.
>
> The Program Committee worked very hard to thoroughly
> review all the submitted papers.  Please repay their
> efforts, by following their suggestions closely in the
> revision - even those not explicitly requested by your
> paper's assigned editor.
>
> When you are finished, you can upload your final manuscript
> at the following site:
>
>      http://www.softconf.com/start/AADEBUG05/final.html
>
> At the top of the page, enter the passcode associated with
> your paper. Your passcode is as follows:
>
>          13X-F1G9A0E9H8
>
> Alternatively, you can click on the following URL, which
> will take you directly to a form to submit your final paper:
>
>   
> http://www.softconf.com/start/AADEBUG05/cgi-bin/finalCopy.cgi? 
> paperCode=13X-F1G9A0E9H8
>
>
> The reviews and comments are attached below.  Again, do
> your best to follow the advice in revising your paper.
>
> If you have any additional questions, please feel
> free to get in touch with us.
>
> Best Regards,
> Jong-Deok Choi and Raimondas Lencevicius
> AADEBUG 2005 program co-chairs
>
> ======================================================================= 
> =====
> AADEBUG 2005 Reviews for Submission #13
> ======================================================================= 
> =====
>
> Title: Divide-and-query and subterm dependency tracking in the Mercury  
> declarative debugger
>
> Authors: Ian MacLarty, Zoltan Somogyi and Mark Brown
> ======================================================================= 
> =====
>                             REVIEWER #1
> ======================================================================= 
> =====
>
>
>                            Reviewer's Scores
>                   -----------------------------------
>
>                  Originality of the Work: 3
>                        Technical Quality: 6
>                             Presentation: 8
>                                Relevance: 10
>                          Overall Quality: 7
>                               Paper Type: full paper
>                    Reviewer's Confidence: 3
>
>
> ----------------------------------------------------------------------- 
> ----
>                           Detailed Comments
> ----------------------------------------------------------------------- 
> ----
>
> The submission is dedicated to two improvements of the Mercury  
> declarative
>    debugger. The first one is inspired by old work of Shapiro (1983),  
> the
>    second one by similarly old work of Pereira (1986). Authors claim  
> that
>    these early works cannot be applied to realistic programs and  
> therefore
>    should be revised.
>
>    The main contribution of the paper is therefore in providing a  
> practical
>    weighting metrics for the divide-and-query approach and a subterm  
> tracking
>    algorithm that does not require modifying the usual Prolog  
> unification.
>    Moreover, the second technique had to be adapted for Mercury.
>
>    While the ideas present seem to be interesting and promissing I  
> miss a
> report
>    on experimental evaluation. For instance, the authors claim that the
> "pseudo-
>    randomality" of the questions asked by the d&q makes answering them  
> hard.
>    Can this statement be qualified by some empirical data? Can you  
> show that
>    the average time spent on answering the d&q questions is longer  
> than the
>    average time spent on answering the subterm questions? The same for  
> the
>    number of questions asked?  How can the approximation metric and  
> the biased
>    metric be compared?
>
>    A debugger for Mercury should be able to locate the logical errors.
>    How does the current approaches tackle these?  For example, lists  
> that are
>    assumed to be ordered, but in practice they are not. What kind of  
> debugging
>    support would be needed?
>    About example 5.3, the error here is in the base case. Therefor, it  
> seems
>    normal that subterm tracking is a good approach to find the error.   
> Another
>    interesting case is when the error would be in the recursive  
> clause. What
>    would be the behaviour like? Suppose the computation of the sum is  
> wrong,
>    how many queries would there be?
>
>    The paper is well-written and presents the approaches nicely.
>
>    Detailed comments:
>
>    p.3 Is it possible to extend the framework with the possibility of  
> the
> oracle
>    answering "do not know"? For large computations it might be  
> difficult for
> the
>    user to foresee the exact result. Clearly, one can run the debugger  
> twice,
> once
>    with the oracle giving the "correct" answer, another time with her  
> giving
> the
>    "erroneous" answer, but is there no better solution possible?
>
>    p.3 Would it be possible to include some dicsussion of the case  
> with nested
>    negation?
>
>    p.7 Can you provide some additional details on why does not it  
> matter
> whether
>    2, 3 or [3,2] are marked as a wrong subterm?
>
>    p.7 rational_add What does M stand for?
>
>    p.8 It seems that finding the origin of a term is easier for  
> Mercury than
> for
>    Prolog since in Prolog "multiple" origins may be possible in the  
> case that
>    a variable is instantiated more than once. Would it be possible  
> that the
> fact that
>    Pereira needs to change the unification and you do not is related  
> to Prolog
>    vs. Mercury?
>
>    Some papers the authors might like to consult:
>
>    1. Stephen Lee Johnson, Teabag: a debugger for Curry, M.Sc. thesis,  
> Portland
>    State University, 2004.
>
>    2. Deursen, A. v., Klint, P., and Tip, F. Origin tracking. Journal  
> of
> Symbolic
>    Computation 15 (1993), 523--545.
>
> ======================================================================= 
> =====
>                             REVIEWER #2
> ======================================================================= 
> =====
>
>
>                            Reviewer's Scores
>                   -----------------------------------
>
>                  Originality of the Work: 3
>                        Technical Quality: 7
>                             Presentation: 8
>                                Relevance: 10
>                          Overall Quality: 7
>                               Paper Type: full paper
>                    Reviewer's Confidence: 3
>
>
> ----------------------------------------------------------------------- 
> ----
>                           Detailed Comments
> ----------------------------------------------------------------------- 
> ----
>
> The article presents variants of algorithmic debugging for Mercury. In
> particular about some strategies to construct and traverse the data
> structure to represent executions. This data structure, the Evaluation
> Dependency Tree (EDT), is not a contribution as it comes from articles
> of Naish.
>
> The related work is very poor, only related to logic programming. Six
> citations out of the 10 are refences to the autjors'work or they lab
> mates. The remaining refs are old (1980, 1883, 1986 !), except for
> one...
>
> The authors do not seem to realize that they are doing some sort of
> slicing, and that slicing has alerady been studied for logic
> programming (see for example the work of Ducasse et al. and Szilagyi
> et al.).  Algorithmic debugging together with slicing has been
> extensively studied at least by the debugging team of Linkoeping
> University (see the work of Fritzson, Kamkar, Shahmehri et al.).
>
> There is, however, a contribution, about the approximative estimation
> of the weight of not-yet-calculated nodes of the EDT.
>
> The article is well written.
>
> I recommend acception of the article but the bibliography and the
> related work MUST be made decent.
>
> ======================================================================= 
> =====
>                             REVIEWER #3
> ======================================================================= 
> =====
>
>
>                            Reviewer's Scores
>                   -----------------------------------
>
>                  Originality of the Work: 4
>                        Technical Quality: 6
>                             Presentation: 7
>                                Relevance: 9
>                          Overall Quality: 6
>                               Paper Type: full paper
>                    Reviewer's Confidence: 3
>
>
> ----------------------------------------------------------------------- 
> ----
>                           Detailed Comments
> ----------------------------------------------------------------------- 
> ----
>
> This paper describes two techniques for speeding up the process of
> finding bugs when using the Mercury declarative (algorithmic)
> debugger. The first is divide-and-query. This is about dividing the
> search space in a balanced way, so as to maximize the guaranteed
> information content of each answer provided by the user. The result is
> that the number of questions asked becomes logarithmic in the size of
> search space. The key contribution of the paper is a method of
> achieving this that works even the search space is too large to keep
> in primary memory. The second technique is tracking dependences among
> subterms. If it is possible to say that a particular part of something
> large is wrong, and it is known where that part came from, then it is
> possible to take a "short cut" to the relevant part of the
> computation. The end result is quite impressive: a declarative
> debugger that is sufficiently good to aid in debugging the Mercury
> compiler itself.
>
> The paper is overall well written. However, I would like to see it
> become a bit more accessible to non logic programmers. It is not that
> the paper has been written with only logic programmers in mind: on the
> contrary, an effort is made to explain what is going on to a general
> audience, but I don't think the paper quite succeeds. If at least the
> first part of section 3 could be made a bit clearer, e.g. by an
> example illustrating paragraphs 3 to 5, I think quite a bit would be  
> won.
> I also have to admit that I got lost in the discussion on positive and
> negative contexts, despite the example. Some more effort there would
> no doubt be quite useful.
>
> However, I have one major problem with this paper, and that is that it
> is very weak on related work. For example, over the past decade,
> substantial work on declarative debugging was carried out in the
> functional programming community. In particular, the Haskell Abstract
> Tracer (HAT) by Runciman, Wallace, and Chitil spring to mind. HAT is
> also based on tracing, and it supports a very sophisticated form of
> "subterm tracking" as well as algorithmic debugging. The techniques
> used may be quite different, but that is EXACTLY what a paper like
> this should clarify, because the basic ideas are certainly very
> similar indeed.
>
> Yet, none of this work is even mentioned. I find that very surprising,
> to say the least. (Somewhat ironically, the term EDT that was coined  
> in the
> functional declarative debugging community is, however, used quite a  
> bit.)
>
> For another example, there are certainly also connections between
> subterm tracking and work on program slicing,
>
> Finally, I wonder if other ways of shrinking the EDT were considered.
> A fairly obvious technique is to allow the user to express trust
> in certain modules (like standard libraries). This technique,
> extended to work also for higher order functions, has been used
> in at least two declarative debuggers in the functional community
> and found to be quite effective.
>
> To sum up, I like this paper. It reports on very solid work, and
> the end result appears to be a very usable declarative debugger.
> It is also well written, although I think some more effort to make
> it more accessible to non logic programmers would be warranted.
> However, the paper really needs to do a better job on related
> work. My technical and overall ratings of the paper reflects this.
> They would have been significantly higher had this been done.
>

--------------------------------------------------------------------------
mercury-developers mailing list
Post messages to:       mercury-developers at cs.mu.oz.au
Administrative Queries: owner-mercury-developers at cs.mu.oz.au
Subscriptions:          mercury-developers-request at cs.mu.oz.au
--------------------------------------------------------------------------