Saturday, September 26, 2009

Compiling iPhoneOS (3.1) apps with Xcode 3.2 without Provisioning Profile

Note: I don't know if this method still works, and I don't care. (And do not use the dd method mentioned in the article. It is extremely fragile to patch a binary file.)

Note 2: Please also check http://iphonedevwiki.howett.net/index.php?title=Xcode#Developing_without_Provisioning_Profile.

I want to compile.


This is easiest to solve. To compile you still need a certificate that can code-sign, but think this as a lip-service. Here is the procedure to create a self-signed code-signing certificate using Keychain Access. Make sure you create the certificate in the "login" (default) keychain, not the "System" keychain. After the certificate is created, perform these steps:
  1. Open /Developer/Platforms/iPhoneOS.platform/Info.plist. (Backup if you want to be safe.)
  2. Go to line 46. Replace the XCiPhoneOSCodeSignContext with XCCodeSignContext
  3. Go to line 79. Replace the XCiPhoneOSCodeSignContext with XCCodeSignContext
  4. Save the file.
  5. Restart Xcode.
  6. Compile!

After doing this, you can't use entitlements with Xcode anymore. But you have ldid -Sxyz.xml that does the same job.



The theory behind this is simple. When Xcode wants to code-sign, it has to consult DevToolsCore.framework on the procedures to code sign. The procedures are coded in the XCCodeSignContext class. The iPhoneOS placed more restrictions on the code-signing requirements, e.g. you must have a provision file. These extra procedures are provided in the class XCiPhoneOSCodeSignContext, as a plug-in at /Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Plug-ins/iPhoneOS Build System Support.xcplugin. Xcode isn't omniscience, so something must tell it that XCiPhoneOSCodeSignContext should be used, instead of the default XCCodeSignContext. This thing is the /Developer/Platforms/iPhoneOS.platform/Info.plist. In the past (iPhoneOS 2.x day), you add the PROVISIONING_PROFILE_[ALLOWED|REQUIRED] keys to give extra instructions to the plug-in, which it can promptly ignore. That's why eventually you need to take the dangerous way of binary-patching the plug-in. But why not just hide XCiPhoneOSCodeSignContext entirely? This is my hack's principle - tell Xcode to meet the new boss, same as the old boss.

I want to install and debug too.



(Note: I'm not fond of patching MobileInstallation.framework. In fact, on 3.1 there's no framework you can statically patch with.)

Unfortunately, installing is more complex than compiling because there are much more checking in the device side. Therefore the procedures will be so complex that ordinary developers should not follow ;).
  1. Make sure you have ldid on Mac. (e.g. this). Place it in /usr/local/bin.
  2. Create the file /usr/local/bin/ldid2. Make it executable. Fill it with:
    #!/bin/sh

    hasGTA=`expr "$*" : '.* -gta .*'`;
    objpath=${!#}/`expr ${!#} : '.*/\([^/]\{1,\}\)\.app$'`;

    if [[ $hasGTA == 0 ]]; then
    /usr/local/bin/ldid -S $objpath;
    else
    TF=`mktemp -t x`;
    echo "<!DOCTYPE plist PUBLIC \"-//Apple//DTD PLIST 1.0//EN\" \"http://www.apple.com/DTDs/PropertyList-1.0.dtd\"><plist version=\"1.0\"><dict><key>get-task-allow</key><true/></dict></plist>" > $TF;
    /usr/local/bin/ldid -S$TF $objpath;
    rm $TF;
    fi;
  3. Open /Developer/Platforms/iPhoneOS.platform/Developer/Library/Xcode/Plug-ins/iPhoneOS Build System Support.xcplugin/Contents/Resources/iPhoneCodeSign.xcspec
  4. Replace the line saying CommandLine = "/usr/bin/codesign" with CommandLine = "/usr/local/bin/ldid2"
  5. Now, copy /usr/libexec/installd from your device to your Mac.
  6. Run this:
    install_name_tool -change /usr/lib/libmis.dylib /usr/lib/libmiss.dylib installd
    ldid -S installd
  7. Now, create a file named libmiss.c, and enter these into the file:
    extern int MISValidateSignature() { return 0; }
  8. Compile libmiss.c to libmiss.dylib with gcc targeting iPhone. Make sure t you have all these flags:
    -dynamiclib
    -install_name /usr/lib/libmiss.dylib
    -current_version 1
    -compatibility_version 1
    -Wl,-reexport-lmis
    -flat_namespace
  9. ldid -S libmiss.dylib, of course.
  10. Copy the new installd to the device's /usr/libexec, and the libmiss.dylib to the device's /usr/lib.
  11. Restart Xcode.
  12. Install!

If you want to debug, add the -gta flag in "Other Code Signing Flags" in your Xcode Project.

OK, so WTF is going on? The first 4 steps allows you to debug the app on the device. The essence of this is replace codesign with ldid. Why this is necessary? Because of the change in Part 1, entitlements are no longer respected. But to debug you must have the get-task-allow entitlement. To fix it, we have to entirely replace codesign with our executable that adds the entitlement ourselves. ldid2 is created with this purpose.

The rest of the steps allows mobile installation to succeed without validation. When the app is installed from Xcode, "mobile_installation_proxy" is invoked, which spawned "installd". The validation part of installd is this:
 +00cd4 0000af2c 0010A0E3 loc_000cd4: mov               r1,#0x0
+00cd8 0000af30 D19701EB bl MISValidateSignature (stub)
+00cdc 0000af34 005050E2 subs r5,r0,#0x0
+00ce0 0000af38 0600000A beq loc_000d00
+00ce4 0000af3c 54049FE5 ldr r0,[pc,#0x454] ; -> 0xb398 "verify_executable"
+00ce8 0000af40 5C149FE5 ldr r1,[pc,#0x45c] ; -> 0xb3a4 "Could not validate signature: %x"
+00cec 0000af44 0520A0E1 mov r2,r5 ; "_CodeSignature/CodeResources"
+00cf0 0000af48 C9E3FFEB bl proc_00003E74
+00cf4 0000af4c 0400A0E1 mov r0,r4
+00cf8 0000af50 B49701EB bl CFRelease (stub)
+00cfc 0000af54 060000EA b loc_000d1c
If we can force MISValidateSignature() to always return 0, any binaries will pass the test. This function is part of libmis.dylib, which is now part of the shared cache, so you can't binary patch this file. Replacing the implementation of a function is a perfect job with MobileSubstrate, unfortunately, no matter how I tried MS can't be injected. Therefore I use a trick: create a "proxy dynamic library" that changes only the MISValidateSignature function, and let the rest pass through. This is the purpose of libmiss.dylib. First, install_name_tool is used to repoint the libmis.dylib in installd to libmiss.dylib. Then, in libmiss.dylib we define a single function, MISValidateSignature, which returns 0 as we wanted. It linked with libmis.dylib, and reexport all the symbols so that the original symbols can still be found. But we have to use flat namespace to make sure our new MISValidateSignature can shadow the old one. Finally, there is a quirk in installd that required the dylib to be at version 1.0.0, so the -compatibility_version and -current_version flags are added.

Ouch these are all so difficult. Any easier methods?


Yes. Pay $99 to Apple. :p

43 comments:

  1. Thank you so much! It worked!

    ReplyDelete
  2. Sorry to ask, but i don´t need to have my iphone jailbroken to do this, right?
    Thanks, bye!

    ReplyDelete
  3. @Maxi:

    For compiling, no, but what's the point of compiling with an invalid code signature when you're not targeting non-JB'ed devices.

    ReplyDelete
  4. Hi. Nice guide. I would ask you if you know the difference from your patched installd against the AppSync 3.1 from Cydia (Hackulo.us repository). Because I have successfully compiled the app using your first step (and so with your without patching the build system), but (with AppSync) I can install it without provisioning but I can't reach to debug it because the app installed on iPhone terminates before gdb reach to start. Running it for the second time on the iPhone works but I have lost the gdb linking. With your patching, gdb works, right? Thanks!

    ReplyDelete
  5. @ Dokugogagoji:

    I don't know what the Hackulous guy did to installd (I can't find their source _code_), so I can't comment on it. But (some general steps):
    0. Did you add the -gta flag in "Other Code Signing Flags" (you should)?
    1. Did you use your device as development in Xcode (you should)?
    2. And also have ldid on Mac (you should)?

    ReplyDelete
  6. 0. Yes (without the -gta flags ldid returns error code 133 - assert(21:false))
    1.Yes...so I can install application..only gdb is broken
    2. same as 0 :p

    I'm looking into the issue...
    To be sure, one question about compiling libmiss.c. Do you have used the XCode bundled compiler or iphone-dev toolchain compiler?

    Thanks!

    ReplyDelete
  7. @ Dokugogagoji:

    0. Oh. That's strange. ldid itself should work with or without -gta.

    I use the compiler from Xcode to compile libmiss.c. Specifically, /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/gcc-4.2 .

    ReplyDelete
  8. Ehm..solved my issue. I have done a typo error on objpath=${!#}/`expr ${!#} : '.*/\([^/]\{1,\}\)\.app$'`; line. Now it works even with AppSync...thanks and sorry for my oversight.

    ReplyDelete
  9. Hi, thanks for the post. I have two questions bothering me.
    In Step 6:
    install_name_tool -change /usr/lib/libmis.dylib /usr/lib/libmiss.dylib installd

    Do I copy libmis.dylib and installd from iPhone device, placeit somewhere and run this command ?
    install_name_tool --help (tells me -change can only have 2 parameters, why are 3 here ?)

    In Step 8:

    I tried

    # /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/gcc-4.2 -dynamiclib -install_name /path/to/libmiss.dylib -current_version 1 -compatibility_version 1 -Wl,-reexport-lmis -flat_namespace /path/to/libmiss.c

    It would complain saying could not locate -lmis
    It would be really nice if you could help me get through' this.

    I am with XCode 3.2 and OS X 10.6

    Thanks.

    ReplyDelete
  10. Great job, man! I can debug my apps now, this is priceless. Although I have amended the procedure slightly and used Installd Patch from iphone.org.hk. I am a lazy bastard and couldn't be asked to follow the steps 5-10 if I didn't have to. Thanks once more!

    ReplyDelete
  11. @Amit:

    6. You don't need to copy libmis.dylib. In fact, there's no libmis.dylib on the device any more.
    8. But you can find libmis.dylib from the SDK (/Developer/Platforms/iPhoneOS.**/blah/blah/blah/iPhoneOS3.1.sdk/usr/lib/libmis.dylib).

    ReplyDelete
  12. I had to restart my iPhone (3GS, running OS 3.0) to get it to work.
    @Amit: You need to set it use the iPhone SDK. Add "-isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.0.sdk" as a flag when calling gcc. If you're using OS 3.1, you'll need to change it accordingly.

    Thanks for the hack!

    ReplyDelete
  13. Hi all,

    Thanks for helping out. But I am still in trouble,
    When I use gcc to make the dylib file I get following output :

    $ gcc4.2 -c libmiss.c

    $ gcc4.2 -isysroot /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.0.sdk/ -dynamiclib -install_name /usr/lib/libmiss.dylib -current_version 1 -compatibility_version 1 -Wl,-reexport-lmis -flat_namespace -o ./libmiss.dylib libmiss.o

    ld: warning: in /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.0.sdk//usr/lib/libmis.dylib, missing required architecture x86_64 in file
    ld: warning: in /Developer/Platforms/iPhoneOS.platform/Developer/SDKs/iPhoneOS3.0.sdk//usr/lib/libSystem.dylib, missing required architecture x86_64 in file


    Then, when I do

    $ ldid -S libmiss.dylib (i get)

    util/ldid.cpp(249): _assert(0:Swap(mach_header_->magic) == MH_MAGIC)
    Trace/BPT trap

    Hope someone could help on what has gone wrong.
    My iPhone device froze last time I tried to install application with the incorrect binaries that I compiled. This time I have to sure, to avoid re-installing firmware and jailbreaking again :(

    Thanks in advance.

    ReplyDelete
  14. @Amit:

    You should use the gcc in /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/gcc-4.2, and you must provide the flag -arch armv6 and also the -isysroot stuff to compile for the iPhoneOS.

    ReplyDelete
  15. Got it working with steps 1-5 on 3.1.4 - other methods weren't working for me.

    Can't get steps 6 onwards install_name_tool isn't playing ball with my appsync installd - could anyone email me the binaries of installd and libmiss.dylib? (BIG thank you in advance)

    quixers@hotmail.com

    Cheers!

    ReplyDelete
  16. I tried first step from your manual, but unfortunately still nothing work. I replaced 46 and 79 lines and restarted Xcode then make new project for 2.2.1, 3.0 and 3.1 iPhone versions and when i tried to compile i had:

    Line Location Tool:0: Code Signing Identity 'iPhone Developer' does not match any valid, non-expired, code-signing certificate in your keychain.

    My Xcode version 3.1.4, Xcode IDE: 1203.0, Xcode Core: 1204.0, ToolSupport 1186.0, I have Mac OS X 10.5.8

    ReplyDelete
  17. @Anonymous and Joe:

    This method is specific for Xcode 3.2. There are already other working methods for Xcode 3.1.* on the web.

    ReplyDelete
  18. Anyone get this working on XCode 3.1.4? I can get it to build but cannot install it on the device.

    ReplyDelete
  19. My XCode 3.2 compiled the app successfully, and install it without provisioning but when gdb start app on iPhone it terminate.

    Messages on XCode Console:
    Running…
    Error launching remote program: failed to get the task for process 230.
    Error launching remote program: failed to get the task for process 230.
    The program being debugged is not being run.
    The program being debugged is not being run.

    Messages on iPhone Console:
    http://nopaste.com/p/azhZg22xbb

    But after that if I start my app, it works.

    ReplyDelete
  20. Any updates about XCode 3.2.1?

    ReplyDelete
  21. @macnow:

    Please add the -gta flag.

    ReplyDelete
  22. @ark:

    The same steps work for 3.2.1. (And don't ask me about 3.1.4.)

    ReplyDelete
  23. Kenny, man, please tell us something about GriP! Why do you avoid the question??

    ReplyDelete
  24. Jeez just install patched installd from hackulous. Done.

    ReplyDelete
  25. Hi! I hope that someone can help me!

    I followed all steps of your article, but when I build my app, an error occurred. This is the result:


    CodeSign "build/Release-iphoneos/Unica Manager.app"
    cd "/Users/tommy/Documents/Xcode Apps/Unica Manager"
    setenv CODESIGN_ALLOCATE /Developer/Platforms/iPhoneOS.platform/Developer/usr/bin/codesign_allocate
    setenv PATH "/Developer/Platforms/iPhoneOS.platform/Developer/usr/bin:/Developer/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin"
    /usr/local/bin/ldid2 -f -s "iPhone" "--resource-rules=/Users/tommy/Documents/Xcode Apps/Unica Manager/build/Release-iphoneos/Unica Manager.app/ResourceRules.plist" "/Users/tommy/Documents/Xcode Apps/Unica Manager/build/Release-iphoneos/Unica Manager.app"

    expr: syntax error
    ./minimal/mapping.h(54): _assert(2:false)
    /usr/local/bin/ldid2: line 14: 3761 Trace/BPT trap /usr/local/bin/ldid -S $objpath
    Command /usr/local/bin/ldid2 failed with exit code 133


    Can someone help me to find the error? The script is copied and pasted as is in the article.

    Sorry for my english if is not perfect!

    Tommy

    ReplyDelete
  26. The last week i found my error. All works now, but the iPhone application crash on startup. Any idea?

    ReplyDelete
  27. hmmm...still the "failed to get the task for process X" happening..

    > firmware 3.1.2
    > "installd patch" for it (not step 5 to ..)
    > -gta option

    I'd like to try your way for "installd" but don't have the original file..would someone have it ?

    Well...any idea guys ?

    Thanks

    ReplyDelete
  28. I can't even get the "I want to compile" working! I followed the instructions, and the erroe message is:

    Command /usr/bin/codesign failed with exit code 1

    Any idea? I'm on OSX 10.6.1 and Xcode 3.2.1.

    ReplyDelete
  29. same error...
    Command /bin/sh failed with exit code 1
    Command /usr/bin/codesign failed with exit code 1

    anny idea??

    Mac OS 10.6.1
    Xcode 3.2.1
    Iphone 3.2.1

    ReplyDelete
  30. No provisioned iPhone OS device connected
    how do i fix this, by the way the iphone is connected

    ReplyDelete
  31. @Tommy: How did you fix your error? I have the same one now...

    ReplyDelete
  32. My error was wrong copy of ldid2 script. I copy it well and it works. Initially my app crashed on startup, but simply changing permission it works fine. i change the permissions of my application directory in the iPhone to 777 and after that, it works.

    Sorry for my english... :)

    In my website i make a script to do all that this article say automatically.

    http://www.ttech.it/en/article/2009/10/iphone-come-compilare-applicazioni-con-xcode-3-2-senza-il-provisioning-profile/

    ReplyDelete
  33. Hi, stil having issue with the application crashing on startup...

    - firmware 3.1.2
    - XCode 3.1.4
    - unable to do "installd step", because :
    install_name_tool: object: installd malformed object (unknown load command 4). > So i use the patch from Cydia.
    - flag "-gta"
    - chmod 777 on the application directory...

    ReplyDelete
  34. @Quentin: This article is about XCode 3.2.x
    Update your XCode...

    ReplyDelete
  35. Hi. Its a great tutorial. Thanks!
    But i have some problems with -gta.
    if i write -gta, i get error: codesign: invalid option -- g
    -Xcode 3.2
    -iPhone 3.0 (mb its a problem? )

    -Alex

    ReplyDelete
  36. :(
    Im still stuck with -gta flag.
    Can anyone help me?

    -Alex

    ReplyDelete
  37. I FOUND BUG IN LDID2 SCRIPT!
    -------------------------
    When u compile app the path of which contains spaces it will end up with expr: syntax error. Solution is simple... add quotes:
    -------------------------
    #!/bin/sh

    hasGTA=`expr "$*" : '.* -gta .*'`;
    objpath=${!#}/`expr "${!#}" : '.*/\([^/]\{1,\}\)\.app'`;

    if [[ $hasGTA == 0 ]]; then
    /usr/local/bin/ldid -S "$objpath";
    else
    TF=`mktemp -t x`;
    echo "<plist version=$
    /usr/local/bin/ldid -S$TF "$objpath";
    rm $TF;
    fi;

    ReplyDelete
  38. I follow this link:

    http://iphonedevwiki.howett.net/index.php?title=Xcode#Developing_without_Provisioning_Profile

    which is based on this page. It just works. Thanks.

    ReplyDelete
  39. Hi Tommy,
    You managed to attach debugger to iPhone app unto the device? Please help.

    Tried:
    1. chmod 777 applications folder on iPhone
    2. i guarantee ldid2 is working.

    Anyone solved the -gta flag to work on debugger? I got error xcode error:

    Error launching remote program: failed to get the task for process 471

    But the app is deployed to device. Tapping the app works fine. It's just the debugger won't attach to the app using xcode.

    Some logs on organizer:

    unknown mobile_installationd[463] : 00808a00 verify_signer_identity: Could not copy validate signature: -402620402
    unknown mobile_installationd[463] : 00808a00 load_application_info: Could not load signer identity from /private/var/mobile/Applications/96C94DD5-B324-41F6-AB13-70D400ED91AC/iPractice2.app/iPractice2
    unknown com.apple.debugserver-43[470] : error: MachTask::StartExceptionThread (): task invalid, exception thread start failed.

    ReplyDelete
  40. Great job, man! It works perfectly even in 3.2.1 xcode

    ReplyDelete
  41. Possible to patch debugserver ondevice using the entitlements patch?????

    im searching for the elusive sandbox destruction that comes with pwning debugserver and i hate modifying my mac because the new Xcodes come out so quickly and from the mac side of things they are really hairy.

    haha it would be good if you could do that app sync patch, then a mobile_image_mounter patch then upload a different developerdiskimage and all you have to change is that first thing at the top.... (thats what im working towards)

    any words of advice? two heads are better than one! :)

    ReplyDelete
  42. Hi,

    I just wanted to say thank you. It worked like a charm. I just didn't follow the last few steps, that makes unsigned applications to run. Beacuse I had already installed the MobileInstallion patch.

    ReplyDelete