epoll support

[libumem]

Checklist

Which kind of PR do you create?

• This PR only contains minor fixes.
• This PR contains major feature update.
• This PR introduces a new function/api for Qiling Framework.

Coding convention?

• The new code conforms to Qiling Framework naming convention.
• The imports are arranged properly.
• Essential comments are added.
• The reference of the new code is pointed out.

Extra tests?

• No extra tests are needed for this PR.
• I have added enough tests for this PR.
• Tests will be added after some discussion and review.

Changelog?

• This PR doesn't need to update Changelog.
• Changelog will be updated after some proper review.
• Changelog has been updated in my PR.

Target branch?

• The target branch is dev branch.

One last thing

• I have read the contribution guide

---

Comments

This PR is intended to provide initial support for the epoll_create, epoll_create, epoll_ctl, and epoll_wait syscalls using the select library; similar to how poll is done already. The binaries for testing have also been PR'd here; but I have since changed the names slightly after checking out the examples/src folder.

[xwings]

Hi,

Thanks for the PR and welcome to Qiling Framework.

Please take a look at the error. it seems lik here is some error

[libumem]

@xwings

Ok, I disabled the unit tests I had provided, as they won't yet work until my examples are accepted upstream in rootfs; let alone the issues I outlined in my PR description.

[xwings]

let me accept that pr

[xwings]

you can enable the test. i updated the CI test and the merged rootfs

[xwings]

We still have tons of issue witl multithread.

not sure how to fix it and it seems like we dont have a betteer way too.

@wtdcode @elicn you guys might want to take a look at this PR

[libumem]

We still have tons of issue witl multithread.

FWIW, one problem I've pointed out is with a single test case -- and only under unit testing, not the underlying Qiling emulation of multithreaded code. The example binaries are single-threaded, as is the epoll implementation I have so far.

[elicn]

Hi there,
Thanks for your contribution. See my review comments inline.
Also please add comments to explain the tests, what is happening on each stage and what is the expected outcome. Comments are key to make the tests usable; otherwise nobody will know what is actually tested and how to react when they fail.

[libumem]

I appreciate the feedback @elicn

I'm running into a couple of issues with the tests I have written, both in test_elf.py

1: When trying to feed my epoll_0 example input via stdin, similar to the examples in the qiling docs; the input doesn't seem to make it through. If I am not doing anything to modify stdin; I can run the binary under Qiling normally, as it will echo my input or exit on receiving "stop", which is what it's supposed to do. I've disabled the test case for now.

2: When running the test ELFTest.test_elf_linux_x8664_epoll_server by itself, it works fine. But when I run the whole test suite, with PYTHONPATH=</path/to/this/pr/> python3 test_elf.py; I get a "PermissionError: [Errno 1] Operation not permitted" error. I haven't been able to isolate the cause of this, as my server doesn't do anything that ought to require any sort special privileges outside of opening port 8000. If the port was an issue, I would expect the test to also fail when run standalone.

Would you please take a look at these problems?

[elicn]

I've patched some of the code earlier today and posted a PR to your repo. Please review the changes I made to see if they make sense. There were a few snippets in which I could not figure out what should be the expected behavior; look for my inline comments starting with "elicn".

[libumem]

Looking into the permission denied errors I'm getting when running the whole test suite here

[elicn]

The commits you pushed after my PR are introducing wrong 'fixes'.
You modified the pointer handling with a wrong implementation, and replaced a useful comment with a useless one. There is no need to fix what not needs fixing.

[libumem]

Rolled back to b724f4cf63de7a2fd88239b12c3eaca9a8122c80
which fires a UC_ERR_READ_UNMAPPED error in the test_elf_linux_x8664_epoll_server test case.

Ran in PDB, which shows what I think is the issue -- this is what I was trying to deal with in the next commit

-> event_ptr = ql.mem.read_ptr(event)
(Pdb) list
151                 return -EEXIST
152  
153             if not event:
154                 return -EINVAL
155  
156 B->         event_ptr = ql.mem.read_ptr(event)
157             events = ql.mem.read_ptr(event_ptr, 4)
158  
159             # EPOLLEXCLUSIVE was specified in event and fd refers to an epoll instance
160             if isinstance(fd_obj, QlEpollObj) and (op & EPOLLEXCLUSIVE):
161                 return -EINVAL

(Pdb) hex(event)
'0x7ffffffffab8'
(Pdb) n
> <removed>/epoll.py(157)ql_syscall_epoll_ctl()
-> events = ql.mem.read_ptr(event_ptr, 4)
(Pdb) hex(event_ptr)
'0x380000005' -> # invoking events = ql.mem.read_ptr(event_ptr, 4) will cause an access violation

[elicn]

Got it.
Undo these two commits so I can rebase my new work on top of yours:
e4242ca741116c66fdde120e003b608559ef0b92
f9497bc7d6998c02bef5dd4cddf18e9132b24c7a

[libumem]

@elicn Done

[elicn]

I've created a new PR.
Please let me know if you have any questions.

[elicn]

The test errors suggest that you relocate the x8664_onestraw_server to the bin directory under the rootfs you are using.

[libumem]

Commented out the test case, will dig deeper with what's going on in unittest.

[libumem]

@elicn Up to you if you want to merge as is, or I can convert to draft PR.

The blocker was/is that the code works outside of unittest, but gets "Operation not permitted" errors when running the ELFTest group.

[libumem]

@elicn Thoughts?

[elicn]

I prefer avoiding incorporating changes that cannot be tested.
Is there anything we can do to have the test work?

[libumem]

@elicn Haven't looked at this in a while, but I finally figured out what was the issue.
Now have one unit test.

[elicn]

Good to hear. Providing a brief explanation could help us look into issues around this in the future.
Please refer to my comments in this review.

[elicn]

This is very unusual.
Care to explain what this is about?

[libumem]

When running this test case, there was a dangling FD pointing to ql.log on the disk after os.remove got invoked. This FD ended up being added to the epoll args, and not the legitimate server socket as I observed outside the test suite.

So here, I just force a closure to keep the FD list in a consistent state.

[libumem]

@elicn Thoughts?