Performance problems

[Brimstedt]

I have some quite simple code:

        final ShardedCounterService shardedCounterService = new ShardedCounterServiceImpl();
        shardedCounterService.increment(counterName, 1);

        final Optional<Counter> optCounter = shardedCounterService.getCounter(counterName);
        BigInteger newQueueItemCount = optCounter.get().getCount();

If I do a load test on this, I get quite bad performance: spikes several seconds long, for a load of less than 20r/s.
This when I already have a bunch of instances warmed up and 10 shards for my counter.

I've tried higher shard counts, but it does not give much better result.

If I remove the "increment" line I get much better result (300ms max responsetimes for the same amount of instances, 20r/s, etc) (But of course; I dont get a working counter :)

Am I misunderstanding something gravely here? :-)
Should this not be possible?

[sappenin]

Hey, thanks for your question.

Can you post the code you use to configure your ShardedCounterServiceImpl in your other tests (i.e., where you construct a ShardedCounterServiceConfiguration)? From what you posted, the default number-of-shards will be 3, and this is not enough to support 20 r/s (though you indicated trying higher shard-counts, so just want to be sure that happening correctly).

Also, the datastore is highly variable, by day and even per-project. I've seen places online (over the years) that claim the datastore can support various numbers of transactional writes per second (sometime 5, sometime 1, etc). That's what's occurring in the increment operation - it has to do a transactional write on the shard to make sure the "increment" is atomic. However, I would expect a 20shard counter to be able to support 20 r/s.

It would be interesting to tweak your load test to see what load you can support, as a baseline for our investigation. For example, can your code handle 1 r/s? What about 3r/s or 10r/s?

Also, what do you see in the logs? I would expect a bunch of ConcurrentModificationException being thrown from the Datastore. Is that happening?

Last but not least, you're load testing in the actual GAE Datastore, correct (not in a unit test, locally)?

[Brimstedt]

Hi, thanks for the response :-)

I just increased the shard count in data store browser in the app engine console. Then I flushed memcache and ran tests again. I did see new shards being created. I tried 3, 30 and 90 shards.

I did not go only be "first" test neither, but ran several to ensure instance count was up and memcache was warm. Also, my tests had gradual load increase.

20r/s should be easily handled by 20 instances and 30 shards I belive :-) But even more instances than that were running. (And as i mentioned, if i remove the increment line, it works much much better).

If I test on local dev server I get response times on a few ms or so, the load tests Im writing about are against actual GAE on a paid account.

So, if I understand you correctly - I should be seeing much better response times and performance?

Because of time restraints I moved on to a memchache counter backed to storage in cron job, but if I get the time Ill create a stand alone sample and load test against it and update this issue with the results.

br