Optimal YaCy Senior Peer performance

[kami4ka]

I tried to find an answer, but didn't get much clear information about the matter.

I want to run a performant Senior peer for YaCy using Hetzner server. Server specs are:

AMD Ryzen™ 9 7950X3D
16 cores / 32 threads @ 4.2 GHz
128 GB ECC DDR5 RAM
2 x 1.92 TB NVMe SSD Datacenter Edition

I assume that this server alone should be able to perform thousands of crawls simultaneously, but the initial setup with just an increased amount of heap to 100Gb only provided 1.5K PPM for a minute or two, and then I got stuck at 100-200 PPM. I feel like I'm missing something, but can't get where the bottleneck is.

I would appreciate community suggestions, as it looks really odd.

I built it from sources and running it using ./startYACY.sh

[okybaca]

Hi, thanks, that's a nice evaluation on really powerful machine! There could be something happening in that moment of change from 1.5K PPM to 100-200 PPM. Did you observe the memory / word cache graphs during that change? Something significant? Without having such a powerful server, I got just some guesses: wild guess #1: word cache saturation? did you try to rise word cache limit? https://eldar.cz/yacydoc/operation/performance.html#increase-indexing-cache wild guess #2: problem with GC? Recetly, I had been experimenting with GC parameters of Java, having OK results with: `-XX:+UseG1GC -XX:MaxGCPauseMillis=250 -XX:+UseLargePages` (in JAVA_ARGS= line of startYACY.sh) YaCy was using old Garbage Collector, but I'm not sure, whether newer Java uses newer GC automatically or not. But you can try to change this parameter to force. But if you see a lot of GC messages in the log, you got a trace to follow. wild guess #3: some limit in internal Solr server? You can also try to use external one: https://eldar.cz/yacydoc/dev/solr.html or a Solr Cloud: https://eldar.cz/yacydoc/dev/solrcloud.html For debugging the moment of change, it could be useful to live-monitor yacy00.log: https://eldar.cz/yacydoc/operation/logging.html and switch of all CRAWLER and NETWORK entries to see only the important details in real-time. Just guesses of non-programming long-time user, hope it helps...

[Orbiter]

YaCy does not load more than four pages from the same host in one second - on purpose. If you make a wide crawl with no limits to the host, then the crawl speed is fast because YaCy balances its request over all known hosts. As soon as there is a dry-out situation of the host set and only a few are left in the crawl, then those hosts get only those four requests per second and that would explain the effect you see.

[kami4ka]

Thank you for your answers. I tried to overcome the YaCy's 4 requests limitation by adding around 800k domains to crawl list. It appeared to worsen the situation and the crawl speed dropped to 0-10 PPM.

I'm also going to apply GC tricks to check the performance (and check the logs).

Could you, please, help to specify the most performant setup in terms of crawl? Like amount of distinct domains, how deep should it go, etc. Are there any P2P bottlenecks that could drop PPM? I'd really like to contribute to the network and not close my index.

[kami4ka]

Ok, looks like I found a root cause. I believe that the parsing queue is limited to the number of CPU threads. But 50s per parse is something odd, isn't it?

Restart fixed it and I'm up at 1k PPM at the moment.

After the restart, the numbers look better

[okybaca]

Great! that 'documentParse'? it's probably defined in [source/net/yacy/search/Switchboard.java](https://github.com/yacy/yacy_search_server/tree/master/source/net/yacy/search/Switchboard.java), line 1088. You can try to change the hardwired value, recompile, and test, whether that helps. If so, I believe it could be added as a configuration parameter.