LPs on Halide

[dawidborycki]

Before submitting a pull request for a new Learning Path, please review Create a Learning Path

• I have reviewed Create a Learning Path

Please do not include any confidential information in your contribution. This includes confidential microarchitecture details and unannounced product information.

• I have checked my contribution for confidential information

By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the Creative Commons Attribution 4.0 International License.

[stevesuzuki-arm]

I put my 1st round of feedback, only for intro, atm.

[stevesuzuki-arm]

Can we merge two section "Overview of Mobile Integration with Halide" and "Benefits of Using Halide on Mobile" into one, focusing on how Halide helps some of challenges in image processing on mobile device?

[dawidborycki]

I am leaving it up to Copy Editors

[balintelias]

please make the punctuation consistent between list items.

[dawidborycki]

?

[stevesuzuki-arm]

Please note some of the additional comments are in reply of Resolved comment.

[stevesuzuki-arm]

Please note some comments are in resolved comment

[stevesuzuki-arm]

Note: I will raise a separate review comment for fusion.md

[stevesuzuki-arm]

I suspect this indentation is not intentional?

[stevesuzuki-arm]

Precisely speaking, this filter is called Binomial filtering, which offer a computationally efficient approximation of Gaussian filtering

[stevesuzuki-arm]

Again, this should be done as part of Halide pipeline because OpenCV generates intermediates which causes R/W memory access, while in Halide this would be a good example to show the benefit of tiling.

[stevesuzuki-arm]

int16_t should be the better type for accumulate reduction in this example as it won't overflow and should be memory efficient.

[stevesuzuki-arm]

int16_t would be better because of the reason written in below comment.

[stevesuzuki-arm]

I don't understand why this code block is here as the alternative way is explained later.

[stevesuzuki-arm]

I need to repeat my previous comment..
This is not the good example to show tiling because the consumer of the tile is simple pixel-wise process which fusing makes sense more.
Tiling would be beneficial if we store intermediates of RGB-to-Gray stage in tile and then apply blur for each tile.
This is because, if we fuse RGB-to-Gray with blur, all 9 neighbors used in blur computes RGB-to-Gray, which ends up with much redundant computation as a entire frame.

[stevesuzuki-arm]

I think mentioning the example of NN and IP here sounds redundant.

[stevesuzuki-arm]

Please write consistent code for the same thing throughout this LP as long as it makes sense.
I mean to say here specifically is that the halide code below is slightly different from what we presented in the previous chapter.

[stevesuzuki-arm]

I think we need some comments to explain what these constants are.

[stevesuzuki-arm]

I don't think we can conclude that we made performance improvement without comparing any performance numbers

[stevesuzuki-arm]

Unfortunately for this LP, (but fortunately for users), this pipeline is too simple to showcase that. Halide compiler simplifies this into something like:

final_stage(x, y) = expensive_intermediate(x, y) * 3 + 1

You may check the output after Halide IR transformation by :
final_stage.compile_to_lowered_stmt(...)

Therefore, we need to show different example. I think if the consumer of expensive_intermediates is a spatial filter (e.g. blur), it will work.

[stevesuzuki-arm]

I would suggest this chapter need overhaul as follows.

1. Remove anything about Loop-fusion
   Reason:

• It is not in initial ideas about the agenda which this LP is going to cover.
• I suppose Loop-fusion is less commonly used in Halide scheduling. As the target reader of this LP is beginner, it would be better not to touch.
• Some of the argument in this draft does not look right. Loop-fusion is beneficial where the number of dimension is limited such as GPU target where we are restricted by GPU programming model, or parallelizing across CPU cores. In other cases, I suppose the benefit is tiny or limited and prone to the loop optimization by backend compiler (e.g. LLVM).

2. Focus on operator fusion

• First, understanding the concept of operator fusion is essential when scheduling Halide.
  This can be explained by comparing to scheduling with compute_root(). Pseudo code of loop structure would help readers to understand (please refer to offical tutorial lesson_08_scheduling_2). Note that operator fusion is the default Halide scheduling. (you could explicitly set compute_inline(), although we don't do that usually)
• Second, we can explain the benefit of operator fusion by mentioning R/W memory access of intermediates which otherwise happens.
• Third, we can explain the case where compute_root() makes sense. For example, producer is expensive and the consumer has a large spatial filter with many access to producer to compute single output.
• Finally, we can provide some guide about the practice of scheduling a long pipeline with many operators. I recommend to start with a naive scheduling where all Func is scheduled with compute_root, and then fusing the intermediate one-by-one where fusion simplifies (e.g. pixel-wise process), or apply tiling for small filter.
  It is worth mentioning again that the default scheduling of Halide is fusion/inlining. i.e. if you don't set any scheduling to Func, it is fused/inlined.
  The pitfall of scheduling a long pipeline is that if everything is inlined, it could end up with very complex generated code with huge compute redundancy, which both compilation time and runtime are extremely long.