detect repeated STL container lookups

[firewave]

@kazutakahirata has been fixing a lot of duplicated lookups with STL containers in recent months - like
c531255
94e9813

I do not know how these were found but it would be great if clang-tidy could detect these.

See the history for way more examples: https://github.com/llvm/llvm-project/commits?author=kazutakahirata.

[llvmbot]

@llvm/issue-subscribers-clang-tidy

Author: Oliver Stöneberg (firewave)

@kazutakahirata has been fixing a lot of duplicated lookups with STL containers in recent months - like c531255 94e9813

I do not know how these were found but it would be great if clang-tidy could detect these.

See the history for way more examples: https://github.com/llvm/llvm-project/commits?author=kazutakahirata.

[nicovank]

I think this is a job for the static analyzer. It can keep track of the last lookup for each map, and on lookup check if the saved value equals the new value. I wrote something similar in Infer for a different purpose (removing some false positives).

I actually had started writing a CSA MapDoubleLookup version of such a check a few months ago but I'm not familiar with it so it took time. Is this something that you can see being added to the analyzer as a new check @steakhal @NagyDonat @Xazax-hun @haoNoQ?

[kazutakahirata]

Yeah, automated discovery would be nice. I'm using clang-query to find Map[Item] and then look for find, count, and contains a few lines prior to Map[Item], so the process is half automated.

There are quite a few variations, but here are some common ones:

if (Map.contains(Item))  // or count
  // Read from Map[Item];

"then" and "else" may be flipped:

if (!Map.contains(Item))
  break; // or continue or return.
// Read from Map[Item].

Checking for membership in advance:

if (!Map.contains(Item))
  Map[Item] = 123;

One challenge is to examine the side effects between the membership check and insertion:

if (!Map.contains(Item)) {
  auto Val = foo();
  Map[Item] = Val;
}

Here, foo() might rely on the fact that Map does not contain Item yet. Inserting an element ahead of time in the "if" condition may change the semantics of the original program.

[nicovank]

Here, foo() might rely on the fact that Map does not contain Item yet.

This is why I think inter-procedural analysis is necessary. Plus all the logic regarding methods invalidating references would probably be easier to implement.

[llvmbot]

@llvm/issue-subscribers-clang-static-analyzer

Author: Oliver Stöneberg (firewave)

@kazutakahirata has been fixing a lot of duplicated lookups with STL containers in recent months - like c531255 94e9813

I do not know how these were found but it would be great if clang-tidy could detect these.

See the history for way more examples: https://github.com/llvm/llvm-project/commits?author=kazutakahirata.

[steakhal]

Here, foo() might rely on the fact that Map does not contain Item yet.

This is why I think inter-procedural analysis is necessary. Plus all the logic regarding methods invalidating references would probably be easier to implement.

Uh, that should be possible to do, but in general really hard to achieve.
Tracking how and where objects escape is a pain in the neck as soon as you start reasoning about globals and pointers.
Like with everything, this should be implemented in baby-steps, and refine the implementation and take shortcuts. Otherwise, if we play by the book, it's not going to be a useful checker.

That said, I hacked together a checker that should work in general - although I don't reason about the number of contained elements in a container.

My idea was to look for set/map "lookups", while looking at who could access and mutate the container between the "lookups". If it was mutated or escaped, then conservatively assume that the second "lookup" makes sense and not warn.

You can find my alpha checker at my branch.
Remember, this is not production ready, likely has a bunch of bugs and limitations. Maybe crashes as well.
Anyone can use it, just let me know you use it and give attribution accordingly.

I don't think I'll have the time to ever come back to this checker, so anyone willing to push it forward and publish it as a PR is more than welcomed.

---

Have you used weggli for coming up with a query to find refactoring opportunities?
It appears to my a bit more lightweight than writing AST matchers and clang-query. WDYT @kazutakahirata

[Xazax-hun]

My idea was to look for set/map "lookups", while looking at who could access and mutate the container between the "lookups". If it was mutated or escaped, then conservatively assume that the second "lookup" makes sense and not warn.

Absolutely, this is the biggest challenge. It is very easy to run into iterator invalidation or similar errors when the map ends up being mutated between the lookups.

On the other hand, if Clang ever adopts something akin to a borrow checker and that information is available to tidy or the static analyzer, for lifetime annotated code this question will be trivial to answer. This direction is not really relevant now but wanted to mention it just in case it happens in the next couple of years and someone takes a stab at this problem after that.

[steakhal]

Now that I think about it, in my prototype, I should have restricted the checker to only consider double lookups if they are present in the same function body (aka. LocarionContext).

[nicovank]

This is what I was playing around with a couple months ago: MapDoubleLookup.cpp. It was specialized to maps and handling the different methods one-by-one. I might get back to it but not in the immediate future and take things from / build upon your draft.

I should have restricted the checker to only consider double lookups if they are present in the same function body (aka. LocarionContext).

That's one thing I wanted to do but couldn't figure out how (first time touching CSA).

[firewave]

@kazutakahirata I was wondering if the query logic could possibly "just" be encapsulated in code to get this somehow started.

And then suddenly this appeared: #123734 (I am always amazed by such coincidences).

[kazutakahirata]

Have you used weggli for coming up with a query to find refactoring opportunities? It appears to my a bit more lightweight than writing AST matchers and clang-query. WDYT @kazutakahirata

No, I've never heard of it, but I just tried it. It's super fast! I'll keep it in my tool box. Thanks!

To find repeated map lookups, however, I do care about the type of Map in Map[Item]. I'm not sure weggli can accommodate that.