After digging the docs, I found no option to filter objects in the scene to do raycasting. When my scene have some big mesh, the performance is not good. If I do the raycast manually, instead of
raycaster.intersectObjects(this.scene.children, true)
I will do
raycaster.intersectObjects(objectsShouldListenToEvent, true)
So I think an option such as func filterSceneChildren or array objectsShouldListenToEvent is good for cases like mine