Change scenario

[waghanza]

Hello all,

In order to make this project more reliable, we need to implement more tests.

The idea is to create a v2 of this project, so we can seamlessly switch to this version when implementations. So v1 and v2 will exists in parallel.

For v2, we will use a SQLite database (in-memory) to make sure we use a lot of data so as we can serialize/ deserialize (JSON) data, but without having latency and db performance issues.

Purpose of this is to have all implementations setup for mid 2025.

Here is a list of endpoints listed for this purpose :

PS : The purpose of this project is to have some raw indicators about frameworks letting any one to choose between stack in context of product creation, or tech migration.

PSS : I'll create a comment per endpoint, please argue for each one in threads, and fill free to suggest in comments

[waghanza]

First endpoint is /, in order to check response time

[waghanza]

Second endpoint is /user/:id, to check for routing parameters

[waghanza]

Third endpoint will be /hello?user=John to test also query parad in url

[waghanza]

Fourth endpoint will be /all, to have a big serialization json content, from database

[waghanza]

Fifth endpoint will be /filter. The idea is to deserialize filter (given as JSON), pick some data in database and serialize it as json

[pi0]

Suggestions from #6726:

• Nested calls (/x calls /y when generating response)
• Response streaming

[waghanza]

Thanks for your insights.

Response streaming could be hard to maintain, especially with the (huge) number of frameworks we have here. Will file upload endpoint be ok ? I mean will test the file part of a framework features set ?

Not sure to understand nested calls, I mean in a real application @pi0

[trikko]

I think you should set a endpoint /dump that must reply with a dump of the request in a simple format (to avoid relying on 3rd party libs: are we testing them?)

For example:
path: /dump
query params: hello=world
cookie: ....
post body: ....

Etc.

Please randomize params to avoid hard coding of requests. For the same reason I would add some cross check. Eg: /sum?a=35 must return 35+the number on post body. Then try some extra fuzzy test (what if a=pizza or a=-5 or a is missing or...)

Every other URL should be a 404 (again: random testing to avoid a strict rule)

[waghanza]

This is more about test, right ?

Then it's a good idea to make sure no explictly define endpoint returns any 4xx.

However, we are more talking about how we can be consistent, I mean to compare frameworks with something more realistic that a simple hello word

Having requirements for a framework to be included in this list is a good idea. Feel free to open a discussion about that. Keep in mind that some used frameworks (I mean in production), fastify for example could be inconsistent with web standard => #8116 (comment)

[waghanza]

Hi @the-benchmarker/web-frameworks ,

I think about using a real database (postgresql) and a real cache (redis) to compare each framework here. After some reflexion using a mock (harcoded list of data) to mimic database behavior could avoid latency problem (messing results because of this latency), but since the same architecture will be used to tests all frameworks, I guess the latency issue will not be so huge comparing what database testing could bring.

Endpoint	Expected response	Feature tested
GET /health	{"status":"ok"}	Routing
GET /hello/:user	{"message":"Hello $user"}	URL parsing
GET /query?page=2&size=50&sort=name	Parsed query parameters	Query parsing
POST /upload	{"filename":"test.bin","size":4096}	Multipart parsing
POST /echo	Echoes the JSON body	JSON Deserialization
GET /serialization	JSON array with at least 100 objects containing id and name	JSON Serialization
GET /db/1	{"id":1,...}	Database reading
GET /cached/1	First call: {"cached":false,...}, second call: {"cached":true,...}	Cache usage

The final purpose is to test raw performance of each framework.

For me the more important is not to say that framework A have absolute this figures, but having a relative comparison between framework A and framework B.

What do you think ?

[trikko]

To ensure the routing engine and query parser actually work (and aren't just matching hardcoded strings), we should test with dynamically generated, nested paths and query parameters.

• Test Case: Request a randomized URL structure:
  GET /<random_1>/<random_2>/<random_3>?<key_1>=<val_1>&<key_2>=<val_2>
• Expected Response (JSON):

{
  "path": "/<random_1>/<random_2>/<random_3>",
  "params": {
    "<key_1>": "<val_1>",
    "<key_2>": "<val_2>"
  }
}

Many minimal/benchmark-oriented servers cheat by returning static, pre-serialized buffer responses. We need a test that forces CPU work, memory allocation, and actual serialization.

• Test Case:
  Send a POST request to /serialization?start=<dynamic_int> with a raw payload of comma-separated random strings (e.g., asdldfssdf,werwerw,"hello" world,...)
• Expected Behavior:
  The server must parse, split, sort the strings, and serialize them into a JSON array of objects where each item gets an incremental ID starting from the query parameter:

[
  { "id": 343, "value": "asdldfssdf" },
  { "id": 344, "value": "sdfsdfewr" }
]

To filter out "fake" webservers optimized exclusively for happy-path benchmarks, the test suite must include basic RFC compliance and error handling tests.

• Invalid HTTP Methods: Send BET / HTTP/1.1 instead of GET. The server must return a 405 Method Not Allowed or 400 Bad Request, not a 200 OK.
• Invalid Protocol Version: Send GET / HTTP/1.99 or GET / ASDS/1.3. The server must handle this gracefully.
• Header Smuggling & Edge Cases: Send incomplete/empty/invalid headers or requests missing the trailing \r\n\r\n to ensure the server doesn't hang indefinitely or crash.

I have many tests I run against my webserver if needed.

Properly implementing protocol compliance and robust request handling inevitably increases code complexity, which naturally impacts raw performance. Writing ad-hoc, hyper-optimized code just to win synthetic benchmarks is significantly easier than building a real, compliant, and secure web server.

[ujibang]

I agree, IMO the current test just measures the framework that is fastest at doing nothing!

Time ago I also proposed this #3803 (comment)

I won't be bound to an actual database but rather have both cpu-bound and io-bound tasks.