Server Deployments
Week 47 marks Thanksgiving in the USA so as reported last time, there have been no server-side deployments for the Release Candidate or main channels, and no rolling restarts. This is liable to continue into week 48 (week commencing Monday 26th November), as there is unlikely to be any deployment to the main channel. There will, however, be deployments to the RC channels, details TBA.
HTTP Updates – Texture Fetching
After indicating that there would be no viewer releases during week 47 in the run-up to Thanksgiving, the Lab rolled out the first of the 3.4.3 beta releases – 3.4.3267135 – on November 20th. The major change to this is the inclusion of the first phase of Monty Linden’s new HTTP-based texture fetch capability, designed to significantly improve texture rezzing within the viewer. As the release notes state:
A new scheme for performing HTTP operations is introduced with this release. It is intended to reduce crashes and stalls while performing HTTP operations and generally enable performance and reliability improvements in the future. In this release, it is being used by the viewer’s texture retrieval code. Our expectation is that it will provide consistent and predictable downloading of textures. As well as the usual problem reporting, we’re also interested in confirmation of improvements where this release improves your experience.
The code for these improvements has already started appearing in some TPVs, and will doubtless be available across all flavours of the viewer in the near future.
Observable improvements in rezzing times have been reported by those who have used the project viewer releases of this code, so it should yield benefits for those using the beta. Monty Linden, who is handling this project is apparently now working on further improvements to the server-side of the equation, which should see additional improvements in the future.
Also pushed out during the week was a new version of the development viewer – 3.4.3.267201.
It is currently not clear when the renewed roll-out od beta and development viewers will result in updates appearing with the production version of the viewer, I believe that this may be additionally delayed while other requirements are put in place related to the Steam link-up (the code for the Steam link-up already having been merged into the beta viewer).
Volumetric Pathfinding
Also during the Tuesday Simulator meeting on November 20th, the question of volumetric pathfinding came up, and how pathfinding might be extended into the air, to allow birds, etc., and under Linden water. There are a range of issues with doing this – perhaps the biggest being the actual demand. There is also the matter of keeping birds and the like from crashing into buildings and skyborne objects, or in keeping fish in the water.
During the meeting, Baker Linden passed a question on the subject to Falcon Linden and indicated that Falcon felt, “It’d be about 3 months of work to get volumetric pathfinding — and that still wouldn’t handle dynamic avoidance (which is the hard part). Theoretically, it’s not that hard — it’s having to rework some Havok systems to work with intermediate data.”
This doesn’t mean that the work is about to be undertaken in any way whatsoever – just that were LL to consider it, getting the basics going for volumetric pathfinding going would take around three months. However, even then, unless the issue of dynamic hazard avoidance, it is unlikely this is something we’ll be seeing in SL for a while yet.
Server-side Object Rezzing Performance
Baker Linden indicated that he has started looking at server-side object rezzing. This work isn’t connected to Andrew Linden’s Interest list work, which is related to which assets the simulator should be loading ready for rezzing, but is rather focused on reducing the server-side lag which is induced when an object physically rezzes in a region. As Baker explained during the meeting, “If you get a really complex object, with many large meshes, or large LLSD files, it takes a while to rez into the world. I’m trying to reduce that.”
There are no timescales associated with the work, although it is expected that it will include avatar attachments as well as in-world objects have less of a performance / fps hit on the region when rezzing complex items, particularly in Baker can get the parsing of large object files to work asynchronously, which currently does not occur. Whether this will translate to visible viewer-side improvements is debatable.
SL Issues
Homestead Performance / Memory Issues
There have been growing reports of region performance issues occurring across the grid. These primarily appear to be impacting Homestead regions – although it can be encountered on full regions as well.
Essentially the problem manifests itself (for most users) when they find they are unable to rez objects in-world and / or as attachments, while raw prims created in-world may rez, but are reported as turning phantom on creation. The issue appears to be large and abnormal memory usage by the region’s physics system, although the precise causes as to why it is occurring are currently unknown.
Regions are allocated a fixed amount of memory that can use. In the case of a full simulator, this is about 1GB, while Homesteads are allocated around 250MB. Generally, physics memory usage for a region – even a busy one – is around 40-80MB. However, on affected Homestead regions, the physics memory use is reaching or exceeding 200MB.
When the physics memory for a simulator gets abnormally high (close to or on 90% of the allowed maximum) internal region safeguards kick-in and prevent object rezzing in an attempt to limit further calls of the region’s memory and keep it alive. This is the behaviour people are witnessing in their regions. The safeguards themselves are designed to help prevent regions from becoming unstable during griefing attacks. However, the problems people are experiencing appear to be entirely unrelated to any form of griefing, and are thus causing a certain amount of head-scratching at Linden Lab.
Reed Linden, in responding to a support request from Motor Loon, provides clear guidelines on what to do if you have a Homestead and experience these issues. It is thought that the most likely culprit for the problem is an unidentified memory leak, but this has yet to be confirmed. Reeds comments regarding particle systems are fascinating. Particles tend to be more viewer-intensive than server, and as many commented at the Simulator User Group meeting on Tuesday 20th November, it would take something bizarre to be going on for particles to be impacting region performance; however, at least one region affected by the issue appears to have a large number of particle emitters in operation.
A further interesting twist came at the meeting itself, when a pathfinding snake and a number of pathfinding characters were rezzed, and the region suffered a severe performance hit (sharp FPS drop experienced by all attendees, sharp increase in both physics memory use and time taken to ping the region) which appeared to be linked with the snake (which was set to follow its creator around) re-calculating its path to both follow its creator and avoid other avatars / objects. However, when the snake was re-rezzed a short time later, no similar issue was noticed, with the region using around 116MB physics memory, with no other outward performance issues.
I the meantime, and as the Linden dev team continue to investigate the issue, if you experience this kind of problem, please ensure you raise a support ticket, supplying as much information as possible, including region name / simulator version (from HELP > ABOUT (either Second Life or the name of your viewer), the time the problem occurred, how it manifested and, if possible, information from the Statistics bar: memory used, FPS and physics performance details, etc.
Inara Pey: Living in a Modemworld 
I certainly see the Particle problem. As you say, you would expect this to be mostly in the Viewer. There are places I can double the visual FPS when I turn off particles and it doesn’t seem to be any particular Viewer. There might be something lurking in the Linden code, but why should it have become apparent now?
It’s a very different problem to what’s afflicting the servers, I suppose, but turning off particles is worth trying when things get sluggish. There might be something else happening, maybe in server-viewer communication, which can affect all viewers.
LikeLike
It’s a weird situation. Several at the meeting expressed some surprise at Reed Linden’s comments in his response to Motor’s issue. It’s very possible / probably that the fact someone at the meeting had encountered similar issues while using a large number of particle generators (ironically trying to chase down a viewer-side issue) on their homestead may simply have been coincidental.
The underpinning problem does seem to be more server-side than viewer overall, as it is affecting all users within a region simultaneously. That fact that it appears (primarily) impacting Homesteads is interesting as well; given they have much smaller memory allocations than Full regions, one might expect memory issues to make themselves felt a lot sooner on Homesteads than Full regions. The question then becomes, why some and not others? It’ll also be interesting to see if – dare I say it – the pathfinding characters placed out during the meeting were the cause of the very dramatic performance collapse during the meeting; the thing here again is that the symptoms didn’t seem to repeat during a further test, and some of the characters (not the snake mentioned in the piece) have been in use elsewhere on the grid without incident…
Obviously, with Thanksgiving in the USA, we’re unlikely to get a update this coming week directly from the Lindens, but it’ll be interesting to see if any users have been poking and prodding any more.
LikeLike
Oddly enough, I think volumetric pathfinding is the sort of thing that pathfinding would actually be good for. birds and fish are about the only thing I’d find interesting with those tools anyway 😉
LikeLike
It would be interesting to see how it could be done without pigeons or whatever splatting into skyhomes or fish taking a walk over the garden :). The latter could be handled by a different navmesh surface category.
If my part, however, I think I’d much rather see effort bent towards volumetric lighting :).
LikeLike
yup, i am gonna concur with the faster texture loads on the beta viewer. the difference between wednesday night and thursday (i think) morning was STUNNING. taking a not very scientific guess, I would say for me maybe 500% faster. log in, and see no grey (!). it was unbelievable at first 🙂
LikeLike
I was surprised with the project viewer after an initial false start. And things are going to get even better :).
LikeLike
I am going to steer clear of the Particles issue. It is likely to involve trying to talk to Lindens.
It’s maybe more useful to look at the Stats for “ktris drawn per second”, which is a bit more stable than frames per second. It’s a measure of the graphics work being done and scene complexity seems to have little effect on the figure. I’ve just started using the Firestorm Public Beta and the effects persist.
LikeLike
Given the long weekend, etc. Thursday’s beta meeting is liable to be the earliest we’ll hear anything in-depth from LL. Tuesday if we’re very lucky and someone had time last week to do more digging with the dev teams. I’ll be at both meetings, so will have updates here in the week :).
LikeLike