2024.03.1b

Lots of changes in this update! With the introduction of working microphones and even more reduced bandwidth! However, there is a cost to all these new features that developers NEED to be aware of. Lets go into some changes.

Changes

• Fixed Microphones!
  • Microphones are now encoded with Opus properly and don't sound like a jittery mess!
  • HUGE Thank you to Virtual for fixing this!
• Fixed Mirrors!
  • Now you can Mirror dwell all you want...
• Improved bandwidth 10 fold!
  • There was a complete rewrite of the underlying networking of Hypernex.Unity. Please see the API changes for BOTH Hypernex.Unity and Hypernex.Networking!
  • Switched from MessagePack's built-in compression (LZ4) to Meta's ZStandard (zstd)
    • This has a much higher compression rate with low compression/decompression times (~8-10ms)
  • Reduced syncing timeframes for PlayerUpdates and WeightedObjectUpdates
  • Bulked WeightedObjectUpdates to reduce bandwidth
    • This is done by taking a list of all WeightedObjectUpdates, and compressing the whole thing, which yields better compression results.
  • Added MessagePackListener.cs for developers to debug how much bandwidth they're using.
    • Shows all messages sent outbound and how much data they are per-second.
  • Bumped Nexport to 1.5.0
• SmoothTransforms!
  • This allows all NetworkSync components to smoothly transform objects
  • Instead of objects being all jittery moving, they now move nice and smooth!
  • This is also applied to HandleCameras
  • For developers, the smoothing can be controlled from Init.Instance.SmoothingFrames
• Made AvatarCreator abstract
  • This was for the network rewrite
  • This change allows users to implement their own AvatarCreator abstraction for any purposes
  • For example, there is an AvatarCreator for both LocalPlayers and NetPlayers
• Added MathF
  • Check the Engine API docs for more information!
• Fixed Cobalt support
  • This fixed the issue where videos would never load
  • This also fixed an issue where having a malformed/unsupported url would never invoke the result of GetOptions()
  • Bumped CobaltSharp to 1.1.0
• Added LocalAvatar.IsLocalClient()
  • Returns true if the LocalAvatarScript is being run on the client with the avatar
  • Returns false if the LocalAvatarScript is running on a NetPlayer
• Improved performance for Handling Blendshapes on NetAvatars
• Improved Full-Body Tracking with OpenXR
• Grabbable outlines are now the Primary Vector color for themes
• Added object to handle LocalPlayerSyncing (LocalPlayerSyncController.cs)
• Cleaned up LocalPlayer.cs and moved functions to their proper places
• Refresh list of input devices when opening the General Settings tab
• Added close button for Warn/Kick Message Window
• Fixed bug when a Player left an instance, the client would add a NetworkSync to every transform in the loaded scene
• Fixed bug where disabling a GameObject containing a AvatarNearClip component would break
• Fixed outline shader only rendering in one eye in VR
• Fixed issue where bones with different tracking spaces wouldn't have correct head rotations on desktop
• Fixed issue where Weights would be excluded due to the Reset message arriving after the weight reset
• Fixed issue where Refreshing the Friends list would clear Instances
• Fixed issue with Avatars sharing the same AnimatorController
• Fixed logging issue in FaceTrackingServices.cs
• Fixed issue where the Client execute button would always be visible
• Fixed naming for LocalAvatar.IsLocalPlayerId
• Added Extensions.cs to introduce easy methods to get or set information
• Bumped SteamAudio to 4.5.0
• Bumped MagicaCloth2 to 2.5.2
• Bumped DynamicBone to 1.3.3
• Removed ability to use RAW microphone output
• Removed ability to Create more than one HandleCamera
  • This is due to a bug. This feature will be re-opened once working and stable

API Changes

Transform Networking

Avatar transforms are no longer synced automatically; only required transforms are synced. Here is an example of how transforms are synced and handled.

LocalPLayerSyncController.cs

// Get the Current Instance
GameInstance gameInstance = GameInstance.FocusedInstance;
// Don't do anything if it doesn't exist, or isn't open
if(gameInstance == null || !gameInstance.IsOpen) return;
// Create a PlayerObjectUpdate
PlayerObjectUpdate playerObjectUpdate = new PlayerObjectUpdate
{
    Auth = new JoinAuth
    {
        UserId = APIPlayer.APIUser.Id,
        TempToken = gameInstance.userIdToken
    },
    Objects = GetCoreTransforms()
};
// Send the message over an unreliable pipeline
gameInstance.SendMessage(typeof(PlayerObjectUpdate).FullName, Msg.Serialize(playerObjectUpdate),MessageChannel.Unreliable);

private CoreBone TrackerRoleToCoreBone(XRTrackerRole xrTrackerRole)
{
    // Simply converts a tracker-role to a CoreBone
    switch (xrTrackerRole)
    {
        case XRTrackerRole.Hip:
            return CoreBone.Hip;
        case XRTrackerRole.LeftFoot:
            return CoreBone.LeftFoot;
        case XRTrackerRole.RightFoot:
            return CoreBone.RightFoot;
    }
    // If none, assume Camera
    return CoreBone.Camera;
}

// Note that all Positions and Rotations are global
private Dictionary<int, NetworkedObject> GetCoreTransforms()
{
    Dictionary<int, NetworkedObject> coreTransforms = new Dictionary<int, NetworkedObject>();
    // If there's no LocalPlayer, return an empty Dictionary
    LocalPlayer localPlayer = LocalPlayer.Instance;
    if(localPlayer == null) return;
    // Root is where the Player's CharacterController is
    coreTransforms.Add((int) CoreBone.Root, localPlayer.transform.GetNetworkTransform());
    // Head is usually the Camera, but it could also be the Avatar's head (if it has one)
    coreTransforms.Add((int) CoreBone.Head,
        localPlayer.Camera.transform.GetNetworkTransform(localPlayer.transform));
    if (LocalPlayer.IsVR)
    {
        // Almost all VR headsets have a LeftHand and RightHand
        // The VRIKTarget bones are used only for their VRIK specific calibration
        // This means that you could transform these bones for specific avatars (if needed)
        coreTransforms.Add((int) CoreBone.LeftHand,
            localPlayer.LeftHandVRIKTarget.GetNetworkTransform(localPlayer.transform));
        coreTransforms.Add((int) CoreBone.RightHand,
            localPlayer.RightHandVRIKTarget.GetNetworkTransform(localPlayer.transform));
        if (XRTracker.CanFBT)
        {
            // Get all supported trackers (currently Hip, LeftFoot, and RightFoot) and add them
            foreach (string name in Enum.GetNames(typeof(XRTrackerRole)))
            {
                XRTrackerRole xrTrackerRole = (XRTrackerRole) Enum.Parse(typeof(XRTrackerRole), name);
                if (XRTracker.Trackers.TryFind(
                        x => x.TrackerRole == xrTrackerRole,
                        out XRTracker xrTracker))
                    coreTransforms.Add((int) TrackerRoleToCoreBone(xrTrackerRole),
                        xrTracker.transform.GetNetworkTransform(localPlayer.transform));
            }
        }
    }
    // Finally, add any Cameras
    foreach (HandleCamera handleCamera in HandleCamera.allCameras)
    {
        NetworkedObject networkedObject = handleCamera.transform.GetNetworkTransform();
        networkedObject.IgnoreObjectLocation = true;
        networkedObject.ObjectLocation = "*" + handleCamera.gameObject.name;
        coreTransforms.Add((int) CoreBone.Camera, networkedObject);
    }
    return coreTransforms;
}

NetPlayer.cs

// This is invoked first whenever the client receives a PlayerObjectUpdate
public void NetworkObjectUpdate(PlayerObjectUpdate playerObjectUpdate)
{
    foreach (KeyValuePair<int, NetworkedObject> keyValuePair in playerObjectUpdate.Objects)
    {
        /*
         * PLUGIN DEVELOPERS!
         * If handling custom keys, HarmonyPrefix this function, and don't run the original IF it contains unknown keys.
         * Simply handle and remove your known keys, then pass the object without keys to the original function
         * OR
         * You can rewrite the function to gracefully handle unknown keys with something like an Event
         * This may be the preferred method for libraries
         */
        // Convert the key to a CoreBone
        CoreBone coreBone = (CoreBone) keyValuePair.Key;
        // Get the NetworkedObject
        NetworkedObject networkedObject = keyValuePair.Value;
        try
        {
            if (coreBone == CoreBone.Camera)
            {
                // FInd and handle the camera
                NetHandleCameraLife n = GetHandleCamera(networkedObject.ObjectLocation);
                n.Ping();
                SmoothTransform c = n.SmoothTransform;
                c.Position = NetworkConversionTools.float3ToVector3(networkedObject.Position);
                c.Rotation = Quaternion.Euler(new Vector3(networkedObject.Rotation.x,
                    networkedObject.Rotation.y, networkedObject.Rotation.z));
                c.Scale = new Vector3(0.01f, 0.01f, 0.01f);
                return;
            }
        }
        // Camera probably wasn't found.
        catch (Exception) {}
        // Make sure something required isn't null
        if (string.IsNullOrEmpty(networkedObject.ObjectLocation))
            networkedObject.ObjectLocation = "";
        // Update the transform!
        UpdatePlayerUpdate(coreBone, networkedObject);
    }
}

private void UpdatePlayerUpdate(CoreBone coreBone, NetworkedObject networkedObject)
{
    // If a SmoothTransform doesn't exist, create and cache it!
    if (!smoothTransforms.TryGetValue(coreBone, out SmoothTransform smoothTransform))
    {
        smoothTransform = new SmoothTransform(GetReferenceFromCoreBone(coreBone), false);
        if (smoothTransforms.ContainsKey(coreBone)) smoothTransforms.Remove(coreBone);
        smoothTransforms.Add(coreBone, smoothTransform);
    }
    // Update the SmoothTransform
    networkedObject.Apply(smoothTransform);
}

SmoothTransforms

SmoothTransform is a new tool that allows transforms to be smoothly moved from point A to point B. The example below demonstrates how to create a GameObject that uses a SmoothTransform to set it's position, when the GameObject is not being driven by an outside factor.

danger

SmoothTransforms are NOT thread-safe. Do not touch them outside of the main thread.

public class MySmoothTransformDriver : MonoBehaviour
{
    // This would be set to true when an outside factor needs to control the transform
    public bool isBeingDriven = false;

    private GameObject gameObject;
    private SmoothTransform smoothTransform;

    public MySmoothTransformDriver()
    {
        GameObject gameObject = new GameObject("ExampleObject");
        SmoothTransform smoothTransform = new SmoothTransform(gameObject.transform, false);
    }

    public void MoveTransformGlobal(Vector3 position, Quaternion rotation, Vector3 scale)
    {
        smoothTransform.SetLocalSpace(false);
        MoveTransform(position, rotation, scale);
    }

    public void MoveTransformLocal(Vector3 position, Quaternion rotation, Vector3 scale)
    {
        smoothTransform.SetLocalSpace(true);
        MoveTransform(position, rotation, scale);
    }

    private void MoveTransform(Vector3 position, Quaternion rotation, Vector3 scale)
    {
        // Doesn't make a difference, but it may for your use-case
        if(isBeingDriven) return;
        // Update transforms
        smoothTransform.Position = position;
        smoothTransform.Rotation = rotation;
        smoothTransform.Scale = scale;
    }

    void Update()
    {
        // Tell the SmoothTransform to stop smoothening the transform and just pull data from it instead when isBeingDriven is true
        smoothTransform.PullFromTransform = isBeingDriven;
        // Update the SmoothTransform (will also update the position/rotation/scale is smoothTransform.PullFromTransform is false)
        smoothTansform.Update();
    }
}

Parameter Ids

There are new Parameter Ids for WeightedObjectUpdates when being handled by a NetPlayer!

• *main
  • Applies the parameter to the AvatarCreator's MainAnimator
• *all
  • Applies the parameter to all AvatarCreator Animators (including playables)