⌘ Development Progress Day 3

Nothing technical, so let's see some screenshots:

⌘ Development Progress Day 2

Now that UILabel is all set, I continue to hook other texts that are long enough to be "commanded", e.g. the UITableHeaderFooterView. There are also some reluctant UILabel that refuses to be hooked. For these I hooked their superviews (UINavigationBar & UIPickerTable) for extracting the text info.

And here comes the real challenge: hooking the UIWebDocumentView. Yes I knew the Clippy 0.95-5 experiment was failed. But this does not prevent anyone else from finding another way to achieve the same goal right?

My first naïve approach is to get the DOM node at the touch point, like this:

[[self webView] elementAtPoint:(touchPoint)];

and like Clippy 0.95-5, it crashes immediately at the 2nd touch. If I trace the class of the returned object*, I can see the class changes from e.g. DOMHTMLAnchorElement to NSCFType. In fact I encountered this before once when trying to implement setSelection() to <textarea/> and <input/> in ℏClipboard using the DOMRange object. It fails on the 2nd call. I call this the "observer effect". Right after you "observed" any DOM objects, the object will "collapse" and you can't reference it reliably anymore.

A workaround is to -retain it right after the DOM object is accessed. But this introduces memory leak. This is really a last-resort option.

And actually it turns out to be work done too complicated. There is a -[UIWebDocumentView approximateNodeAtViewportLocation:] method in the Interaction category which can easily be overlooked. It returns a DOMNode without any observer effect AFAIK. Except it crashes on 2.2 complaining ASSERTION FAILED: !WebThreadIsEnabled() || WebThreadIsLocked(). Well that's easy enough to solve. Just lock the WebThread() everytime we need to access this method.

So I used:

WebThreadLock();
DOMNode* activeNode = [self approximateNodeAtViewportLocation:&(touchPoint)];
WebThreadUnlock();

and done! The DOMNode is got and we can do anything we want. Too good to be true. Turns out the DOMNode will only be returned on any objects that have actions, i.e., <a>, <input/>, <textarea/>, <img/>, any thing that has onXXX events, but not static texts. If touchPoint is on a static text, nil will be returned.

But why this method prevents the observer effect? After lots of experiments, finally it turns out that it's the WebThreadLock() and WebThreadUnlock() doing the tricks. In fact, if I do

WebThreadLock();
[[self webView] elementAtPoint:(touchPoint)];
WebThreadUnlock();

then everything works perfectly! At least in the simulator. (Hey Ryan, if you're reading this, as now I've solved this problem, can you add Safari copying back in Clippy 0.95-6? :D)

So, at the end of Day 2, I can hook to any DOMNodes, besides UILabel, UITableHeaderFooterView, UINavigationBar and UIPickerTable.



Note: -[WebView elementAtPoint:] returns a subclass of NSDictionary. By the returned object I mean the value corresponding to the WebElementDOMNode key.

⌘ Development Progress Day 1

OK, now iKeyEx 0.1-6 has been released, let's turn into the other project left over: ⌘.

⌘ was proposed on Jan 26th as a solution to issue 10. You can read the design concept here. As I develop the app, I'll keep updating the progress on this blog.

The first thing I've done is to intercept touches coming to a UILabel using -[UILabel touchesBegan:withEvent:]. It turns out not so easy. By default UILabel ignores all touches. But setting userInteractionEnabled = YES does not solve the problem at all (at least in 2.0 firmware). As usual, when something does not work, I look into the UIKit's assembly code. And it turns out that there is a -[UILabel ignoresMouseEvents] method. So I quickly hooked it to always return NO. Then the test application crashed, complaining -[UIScrollView old_touchesBegan:withEvent:] not found. But hey, I did not do anything to the UIScrollView! What's wrong? Turns out the implementation of -[UIResponder touchesBegan:withEvent:] looks like this:

-(void)touchesBegan:(NSSet*)touches withEvent:(UIEvent*)event {
forwardMethod2(self, _cmd, touches, event);
}

where forwardMethod2() is equivalently the statement [[self nextResponder] performSelector:_cmd withObject:touches withObject:event]. This pattern is used for all touchesXXX:withEvent: messages. Now since I have renamed the old touchesBegan:withEvent: for UILabel only, and my code forwards the message to the next responder (the UIScrollView), and it does not recognize old_touchesBegan:withEvent: -- blam, crash.

As a result, I have to store the old IMP before MobileSubstrate-ing it, and invoke the old message like this:

-(void)touchesBegan:(NSSet*)touches withEvent:(UIEvent*)event {
NSLog(@">>> %@", self.text);
old_touchesBegan_withEvent(self, _cmd, touches, event);
}

With this, I can hook all UILabels in the simulator.