Improved Mouse Support

[SurfingNerd]

I am working on a cross platform game, and Cocos2D-XNA currently lacks on Mouse Support. So i started to write my own implemention.

I see that the other platform specific input devices are hold together via interfaces and implemented on CCNode.
These are: ICCTargetedTouchDelegate, ICCStandardTouchDelegate, ICCTouchDelegate, ICCKeypadDelegate, ICCKeyboardDelegate.

According to this, the next logical step is to create new interfaces, and implement them on CCNode ?
Thats what i have got so far:

The Tooltip thing may be Bloadware, and i am unsure if its clever to implement it on CCNode.

Major problem currently, i sheduled the Node for Updates, and on Update i ask MonoGames MousePosition.
1.) that is not wise to do for ALL Nodes
2.) its unaccurate, i dont get all clicks atm. (Timing Problems)

Am i doing right, or am i on a missleading way ?

[totallyeviljake]

As for getting the mouse input and handling it, there is a class called CCInputState that is designed to handle general input controls. You should be using CCInputState to get the current input device state, which will give you a wide variety of inputs that can be processed with every update() call.

You should only poll the input state in the update() call, and only in the CCDirector's update method. Then when you get an input state update, you propagate that object state to all of the input listeners. The CCDirector should have a method to manage the receivers of CCInputState. You could add some event handlers that take the CCInputState as their signature. Then in your CCNode, you could have an option to register with the input state handler in CCDirector.

[SurfingNerd]

thx for help,
i just wanted to inform, that i still work on it, but has no priority for me.
I have to fix bugs, refactor to your suggested design, and improve it.
i also wanna to separate MouseHandling and the Tooltip feature in different classes,
and also to make a concept for Tooltips on touch screens.

Update 2014 05 06

  public class CCMouseClickedEventArgs
{
    public CCMouseHandlingNode Node { get; private set; }
    public MouseState MouseState { get; private set; }

    public CCMouseClickedEventArgs(CCMouseHandlingNode node, MouseState mouseState)
    {
        Node = node;
        MouseState = mouseState;
    }
}

public class CCMouseHandlingNode : CCNode
{
    public event EventHandler<CCMouseClickedEventArgs> MouseClicked;

    public CCMouseHandlingNode()
    {
        this.ScheduleUpdate();
    }

    enum MouseStatus
    {
        None,
        Clicked
    };

    public CCNode Tooltip { get; set; }
    public CCPoint TooltipOffset { get; set; }

    private bool m_MouseIsOver;
    public bool MouseIsOver
    {
        get
        {
            return m_MouseIsOver;
        }
        private set
        {
            m_MouseIsOver = value;
        }
    }

    private static MouseStatus m_currentMouseStatus = MouseStatus.None;
    //Dirty hack to prevent that 1 mouseclick is handlet multiple times in the program.
    DateTime m_lastMouseClickHandled = DateTime.MinValue;

    public override void Update(float dt)
    {
        base.Update(dt);
        MouseState state = Mouse.GetState();
        CCPoint point = this.ConvertToWorldSpace(this.Position);

        var rect = this.WorldBoundingBox;
        CCPoint realMousePos = new CCPoint(state.X, CCDrawManager.VisibleSize.Height - state.Y);
        //rect = new CCRect(rect.Origin.X - rect.Size.Width, rect.Origin.Y, rect.Size.Width, rect.Size.Height);
        rect = new CCRect(rect.Origin.X - rect.Size.Width / 2, rect.Origin.Y, rect.Size.Width, rect.Size.Height);

        if (rect.ContainsPoint(realMousePos.X, realMousePos.Y))
        {
            if (!MouseIsOver)
            {
                MouseIsOver = true;
                OnMouseEntered(state);
            }
            if (state.LeftButton == ButtonState.Pressed)
            {
                //dont recognize a mouse click more than once.
                if (m_currentMouseStatus != MouseStatus.Clicked)
                {
                    DateTime now = DateTime.Now;
                    m_currentMouseStatus = MouseStatus.Clicked;
                    if (now.Subtract(m_lastMouseClickHandled).TotalMilliseconds > 50)
                    {
                        OnMouseClick(state);
                    }
                    m_lastMouseClickHandled = now;
                }
            }
            else if (state.LeftButton == ButtonState.Released)
            {
                m_currentMouseStatus = MouseStatus.None;
            }
        }
        else
        {
            if (MouseIsOver)
            {
                MouseIsOver = false;
                OnMouseLeaved(state);
            }
            m_currentMouseStatus = MouseStatus.None;
        }
    }

    protected virtual void OnMouseClick(MouseState state)
    {
        if (MouseClicked != null)
        {
            MouseClicked(this, new CCMouseClickedEventArgs(this, state));
        }
    }

    protected virtual void OnMouseEntered(MouseState state)
    {
        if (Tooltip != null)
        {
            Tooltip.AnchorPoint = new CCPoint();
            Tooltip.Position = TooltipOffset;
            AddChild(Tooltip);
        }

    }

    protected virtual void OnMouseLeaved(MouseState state)
    {
        if (Tooltip != null && Children != null)
        {
            if (Children.Contains(Tooltip))
            {
                RemoveChild(Tooltip);
            }
        }
    }
}

[totallyeviljake]

I see what you are after now. Supporting proper mouse activity like hover actions will require the changes you are proposing. Thanks!

[SurfingNerd]

I updated my code, its still buggy since i still not understand the coordinate systems well enought, but thats not the problem, i will come behind it somesays :D, and i have to get rid of the dirty hacks in it (now.Subtract(m_lastMouseClickHandled).TotalMilliseconds > 50)...

another problem i encountered is the clicking on overlapping areas.
i think i have to make the checks in a central function like myLayer.Update() and handle there the Node accordingto their Z-Order.

Since you know much better than me, have you any tipps ?

[totallyeviljake]

I have an issue for the z-ordering of clicks on overlapping nodes. Right now the touch passes through all of the touch handlers until it finds one that processes it. That usually does not work because the touch handlers are not re-ordered in LIFO order. To fix this in cocos2d-x they used the global Z ordering, which we do not have in cocos2d-xna.

[SurfingNerd]

yeah, i just saw your entry for this Issue. #396
so maybe we can work together for solving this. Shouldnt it be possible to travel the Node Tree and calculate the global Z-Order ??

[totallyeviljake]

I am not sure. Consider the case of highly nested nodes with deep ancestry. You need to know the rank of the tree so you can compute a sufficient number space to contain each of the branches in the tree. The worst part is that this tree will change constantly, so it can become computationally hard with respect to fast performance during the update() loop.

I suppose each time a child is added to a parent you could increase the branch count back to the root (when a new child is added). Hmm. That could work to give us the tree rank in real time. Then it's just a matter of applying a magnitude factor, e.g. 10x, on each branch rank and filling in the global z-order for the nodes using their local z-order as the minor.

A (3) <- B (2) <- C(1)

Z(A) = 103, Z(B) = 102, Z(C) = 10**1

Z(C[n]) = Z(C) + C[n].zOrder * normalization_factor
Z(B[n]) = Z(B) + B[n].zOrder * normalization_factor

[SurfingNerd]

ok, thank you for your help,
I initially thought about keeping a double endet Queue or something in sync, so i have fast access to the most top notes, these have the highest chance that they will be the processing one.
of course, it uses more ram, but the update() is so performance critical.
i will mediate about it :-), just keep in contact.

[SurfingNerd]

i prototyped my idea - seems to work - if i understood the ZOrder / Node.Children correctly.
will continiue tomorrow, stay tuned :)

public class ZOrderManager
{
    public CCNode RootNode { get; private set; }

    /// <summary>
    /// Primary storage in the Visual Order,
    /// where the first position is the one most in the background.
    /// </summary>
    public System.Collections.Generic.LinkedList<CCNode> VisualOrder { get; private set; }

    /// <summary>
    /// Fast retrival of the LinkeNode in the LinkedList.
    /// </summary>
    private Dictionary<CCNode, LinkedListNode<CCNode>> m_lookup = new Dictionary<CCNode,LinkedListNode<CCNode>>();

    /// <summary>
    /// points to the last node known with that Z-Order value.
    /// </summary>
    private SortedDictionary<int, LinkedListNode<CCNode>> m_zOrderIndex = new SortedDictionary<int, LinkedListNode<CCNode>>();
    /// <summary>
    /// points to the first node known whith that Z-Order value.
    /// </summary>
    private SortedDictionary<int, LinkedListNode<CCNode>> m_zOrderIndexHead = new SortedDictionary<int, LinkedListNode<CCNode>>();

    public ZOrderManager(CCNode rootNode)
    {
        RootNode = rootNode;
        //VisualOrder = new List<CCNode>();
        VisualOrder = new LinkedList<CCNode>();
    }


    public void UpdateTree()
    {
        VisualOrder.Clear();
        m_lookup.Clear();
        m_zOrderIndex.Clear();
        m_zOrderIndexHead.Clear();
        //assumption: ZOrder is global value ?!
        AddNode(RootNode);
        UpdateTree(RootNode);
    }

    private void UpdateTree(CCNode node)
    {
        if (node.Children != null)
        {
            foreach (CCNode child in node.Children)
            {
                AddNode(child);
            }

            foreach (CCNode child in node.Children)
            {
                UpdateTree(child);
            }
        }
    }

    private void AddNode(CCNode node)
    {
        LinkedListNode<CCNode> listNode = null; 
        LinkedListNode<CCNode> currentZTail;
        int zorder = node.ZOrder;
        if (m_zOrderIndex.TryGetValue(zorder, out currentZTail))
        {
            listNode = VisualOrder.AddAfter(currentZTail, node);
            m_zOrderIndex[zorder] = listNode;
        }
        else
        {
            if (VisualOrder.First == null)
            {
                listNode = VisualOrder.AddFirst(node);
                m_zOrderIndexHead.Add(zorder, listNode);
                m_zOrderIndex.Add(zorder, listNode);
            }
            else
            {
                int[] keys = m_zOrderIndex.Keys.ToArray();

                if (zorder < keys[0])
                {
                    LinkedListNode<CCNode> lowestHead = m_zOrderIndexHead[keys[0]];
                    listNode = VisualOrder.AddBefore(lowestHead, node);
                    m_zOrderIndexHead.Add(zorder, listNode);
                    m_zOrderIndex.Add(zorder, listNode);
                }
                else if (zorder > keys[keys.Length - 1])
                {
                    LinkedListNode<CCNode> highestTail = m_zOrderIndex[keys[keys.Length - 1]];
                    listNode = VisualOrder.AddAfter(highestTail, node);
                    m_zOrderIndexHead.Add(zorder, listNode);
                    m_zOrderIndex.Add(zorder, listNode);
                }
                else
                {
                    for (int i = 0; i < keys.Length; i++)
                    {
                        if (zorder > keys[i] && zorder < keys[i + 1])
                        {
                            LinkedListNode<CCNode> highestTail = m_zOrderIndex[keys[i]];
                            listNode = VisualOrder.AddAfter(highestTail, node);
                            m_zOrderIndexHead.Add(zorder, listNode);
                            m_zOrderIndex.Add(zorder, listNode);
                            break;
                        }
                    }
                }
                //find the highest Z-Order Element that is lower then the current one

            }
        }
        m_lookup.Add(node, listNode);   
    }
}

[totallyeviljake]

This looks like it would work if the zOrder is maintained as a global z-order by the programmer.

[totallyeviljake]

what I am considering is a global z-order that is computed as nodes are added, removed, and updated in the draw tree. Then use that global z-order as a priority value in the touch handler interface to sort the touch dispatcher targets in a proper priority order.

[SurfingNerd]

what do you mean "if the zOrder is maintained as a global z-order by the programmer" ?
it also handles parent/child relationship in the correct order, or not ?

yeah the prototype has no life update yes, i wanted to do this as well, thats the reason, why i have the m_lookup dictionary, what is completly useless in the current implementation. it shall be used if something is changing in the tree the keep the tree sync very fast. as far i tested, my implementation is satisfying for me right now, since i only need to update the tree, if really a click is happening:

       MouseState state = Mouse.GetState();
        if (state.LeftButton != m_lastLeftButtonState)
        {
            if (state.LeftButton == ButtonState.Pressed)
            {
                m_zOrderManager.UpdateTree();

[totallyeviljake]

The current problem with zOrder is that many nodes can have the same zOrder value. That makes the zOrder local scoped instead of global.

I started the refactoring of the touch handler to take the TouchPriority from CCNode when the priority is not specified. Currently, the priority is always set to zero, so my refactoring code is not being exercised yet.

fa52305

[SurfingNerd]

ok thanks for the detailled explaination.
i activated now the cocos2D Touch Handling instead of my own Mouse interpretation. however i am more confused than before, because for me it looks like the Touch*() functions get called without any checking of the bounding box. i receive the touch event on Node B, if Node A gets clicked. i saw posts in the web for the original cocos2d that suggest to do the bounding box check within the node, wich makes no sense for me... dit i understand everything ???

[totallyeviljake]

You are understanding the problem more clearly now. Often times, even in cocos2d-x, the node's bounding box is the entire window, so the bounds check is useless. The only reliable solution is to set the touch priority on the node so that it can handle the touches properly. Once the touch is in the node, it should know whether or not it is in relevant content. There may be times when you want to handle a touch that exists outside of your bounding box ... so I don't think that check is useful.

[totallyeviljake]

Send us your mouse support changes and we'll merge them in.

Then we can tackle the touch/mouse priority handling...

[SurfingNerd]

thanks for the offer, its really an honor. Actually its still a prototyp and currently i am switching from "per node handling" of the click to a cenral handling,Ffurthermore i want to build the solution on the touch implementation instead of retrieving the mouse position from MonoGame. but i need still to dig deeper in the possibilities and impossibilities of Cocos2D. So for a merge its far to early, unless you need it right now and are willing to complete it by yourself.

[SurfingNerd]

hmm actually i think about supporting the right and middle mouse key as well,
what seems impossible if i rely on CCTouch and co.
Do you know if MonoGame is simulating a Mouse on Touch devices ?
What do you think about supporting actions that are not available for all platforms ?

[totallyeviljake]

MonoGame does not simulate the mouse on touch devices.

I have no qualms about supporting platform-specific input. Our framework is the layer that is supposed to handle that type of support.

[SurfingNerd]

thx, good to know, i am still working on it, made some improvements yesterday, so stay tuned.
After all i am handling now the Mouse Clicks in the Update of the most top layer, but currently only taking care of the Parent/Child Order. To be continued with Z-Order, and i am thinking about interprating TouchPriority as well, hoping that will not be too confusion for the people. MouseEnter, MouseLeave and so on is handlet within the Update() Routine of the MouseHandlingNode itself, because i dont have concurrency between the nodes there. Multiple nodes can have a MouseOver = true, but only one Node can handle the Click Event. Another Topic on the "nice to have List" is Bubbeling and Tunneling of the events, i keep this in mint. WPF uses this concept, and its quite good: http://www.codeproject.com/Articles/464926/To-bubble-or-tunnel-basic-WPF-events
but i dont focus on it right now.

[totallyeviljake]

Ah, so tunneling is how the framework currently propagates touches, but does so in a mostly unexpected manner. Once the nodes are properly sorted by draw priority, then the WPF tunneling protocol will also map to the touch/mouse propagation in cocos2d-xna.

Bubbling is curious I suppose. I suppose a game could respond to deep click/touch events and bubble the event up the draw chain.

on z-order, or draw order, you COULD just keep a priority queue that is rebuilt any time the sortChildren method is invoked without short circuiting. That would mean only building it only now and then, and you could even get clever and reconnect the list at a specific node that is causing the order break. The priority value would still be the draw order of the node, implemented as a simple list of nodes and using reverse iteration to process touches and mouse.

the draw loop always computes the proper order of the nodes for focus. ah, focus, something I started to implement for xbox support to handle menu item button actions properly.

[SurfingNerd]

hi jake, i dit not change my implementation the last days. Its still a prototyp, but actually it works for me. i only consider the parent/child relation of my nodes. so i wont change it for a while, sorry about that.
but finally its a good time to share it, if you are interessted.
I added a lot of warnings, so you can directly jump to the points where i think those are problematic.
https://drive.google.com/file/d/0ByFHwohpMiznSlVIOTJwMXpBZXM/edit?usp=sharing

[kjpou1]

Hey guys.

Just some input here. The Cocos2D-X has a new EventDispatcher implementation that really does everything you are talking about. It would be a little work but probably less time than what you are spending on getting that implemented in cocos2d-xna right now.

Take a look here: http://www.cocos2d-x.org/wiki/EventDispatcher_Mechanism

There are a few classes that need implementing for sure but it really is flexible and takes care of everything you have outlined and have been discussing. It will also work for any of the events that you want to create not just mouse events, for example keyboard events.

Just a thought.

[totallyeviljake]

Thanks Kenneth! The v3 Event Dispatcher is pretty much how events are dispatched in -xna. We just encapsulate the delegate into the body of the CCNode instead of leaving it dangling as an anonymous implementation (per the v3 example).. We definitely need to improve how the node priority conveys to input delegation. The real problem, beyond scene graph priority, is focus. When I did the xbox implementation the first problem I had was menu selection. There is no cursor and no concept of touch on a console, so you have to focus on nodes and delegate input to the focus node. That's where the focus manager was born. I stopped work on the focus code when I realized that it did not convey well to the CCNode's encapsulation of the input delegates. That part of the framework needs the most attention nowadays.