Posted by planetbeing on under large data structures, iphe, ftl, comex, iboot, redundancies, dev team, underbrush, pumpkin, pointers, insight, job | 
I don't know how I was talked into reversing a FTL, but we're actually on our way. I've managed to enlist the aid of CPICH (who has been helping with the lower layers as well, he's our human HexRays) and just recently, pumpkin, who you will know from the Dev Team. pumpkin will be the heavy support that's necessary to take down _FTLRestore, which is the most complex function I've seen in 1.1.4 iBoot (and I've pretty much have seen all of it). pumpkin is very good, so this task should be now be cut down to "fairly difficult" from "completely impossible".
The strategy so far has been me methodically hacking through the functions in the order that they are called, completely decompiling them, understanding them, and assimilating them into openiboot. Toward this end, I've been working on FTL_Open, which is a fairly large (but as it turns out, boring) function, but has been useful in enlightening us on several of the large data structures FTL uses.
Meanwhile, CPICH works on functions ahead of me, so that when I reach them, a lot of the thorny underbrush has been cleared out and my job becomes much easier and faster. Toward this end, he has been working on FTL_Read, which uses the data structures that the now-completed FTL_Open should populate.
_FTLRestore is sort of a "bonus", since it's not normally called if the iPhone was shut down normally and everything is cleaned up. However, since recovering faulty data structures require all redundancies to be exploited, reversing this would let us gain a lot of insight into how the FTL works. It's also, naturally, an enormously complex function, and hence I wisely delegated it to pumpkin. =P (We will probably end up working on it together)
The one thing that troubled me was that the code we were reversing is for 1.1.4 whereas we primarily need it to work on 2.0. However, due to the fact that I had it better mapped out than the 2.0 iBoot, and the fact that the equivalent 2.0 code was much more complex (lots of function pointers flying around, and a weird switch idiom I haven't quite figured out yet), We decided to stick to the 1.1.4 iBoot.
After completing FTL_Open, I had a bit of a panic when I discovered it did not work at all on my 2.1 phone, and I could not find any obvious bugs with it. This might've meant that all our work on 1.1.4's FTL was for naught. Forgoing sleep, I tore through the 2.1 iBoot, locating the analogues to my already reversed 1.1.4 functions (I had given up trying to trace through the function pointers the first time around), and called them directly with my special version of iBoot (patched so that one of the commands was able to call arbitrary iBoot functions with arbitrary arguments). I managed to find a couple of bugs with my VFL code, and after having fixed them, FTL_Open appears to have worked. I think. It just finds and reads several data structures from NAND. It remains to be seen if I'm even reading the right thing.
Now for some sleep.
Tagi: large data structures, iphe, ftl, comex, iboot, redundancies, dev team, underbrush, pumpkin, pointers, insight, job
Posted by planetbeing on under linux kernel, iphe, bootmenu, muscle work, different numbers, source tree, deci, ftl, ipod touch, filesystem, screenshot, ports, lt, linux |
Amazingly enough, the FTL_Read stuff from last night was pretty much correct! After that, it was relatively trivial to port over the HFS+ code I've already written (which was in pure C... finally that [fail] design decision has been vindicated =P).
As you can see in the screenshot below, with the latest Git revision, you can browse the filesystem from openiboot!
Next on the list is to port openiboot over to the iPod touch and iPhone 3G. It's probably just a matter of putting in different numbers for the GPIO ports, but we'll see.
After that, I will implement poorlad's bootmenu (which everyone seems to like).
After that, well... We have pretty much all the devices now, so we'll start looking at the Linux kernel. If you're a Linux kernel guy who would be willing to help (preferrably you have experience porting Linux to new ARM platforms), please leave a comment here. I can do most of the muscle work, but it'd be nice if someone can show me how to set up the source tree properly for the new port.
Tagi: linux kernel, iphe, bootmenu, muscle work, different numbers, source tree, deci, ftl, ipod touch, filesystem, screenshot, ports, lt, linux
Posted by planetbeing on under linux kernel, simultaneous projects, boot menu, iphone, nand, ftl, 2g, iphe, ipod touch, hey guys, 3g, few days, resolve, peoe, linux |
Hey guys,
The lack of updates for the past few days is because many of you decided to visit us in IRC, thus enabling work to be done on porting openiboot to the iPod touch and the iPhone 3G (in particular because I don't have an iPod touch at the moment).
I'm pleased to report that everything now seems to be working on the iPhone 2G and the iPhone 3G (albeit NOR read/write on the iPhone 3G is unoptimized and is unacceptably slow). There is apparently an outstanding issue with the NAND ECC on some (?) iPod touchs, and also some people can't seem to actually install openiboot to NOR on both iPhone 2G and iPod touch. Unfortunately, the problem is that these things happen on devices that I don't have physical access to, and IRC is often a frustrating medium for communicating with testers. I'm confident these issues will be resolved soon, though.
So, current simultaneous projects:
1. Resolve openiboot porting issues
2. Implement poorlad's boot menu
3. Work on write support for FTL
After at least one of those things are done, we'll be working on the Linux kernel.
Tagi: linux kernel, simultaneous projects, boot menu, iphone, nand, ftl, 2g, iphe, ipod touch, hey guys, 3g, few days, resolve, peoe, linux
Posted by planetbeing on under raw hardware, e care, bad blocks, locati, ftl, data structures, slew, gadget, amount of time, algorithms, kernel, sectors, ace, yaffs, proposal, benefit, linux |
We've made some progress on the USB gadget driver for Linux, and we're now running a generic serial gadget for communication. This implementation is important because USB is now a lot less laggy and things like ethernet over USB, etc., can eventually be supported, easing access.
We've also got pretty far with porting the NAND driver to Linux. Most of the read support is now there, and we've isolated the routines in the iPhone kernel where the raw hardware write occurs. CPICH and c1de0x are working on reversing it. Hopefully, it will be analogous enough to reads that it won't take a huge amount of time to work out.
This is different from reversing their FTL, however, which is a complicated slew of data structures, merge buffers and other exotic algorithms that take care of evenly distributing writes throughout the device and also making writes take less time.
I think reversing all of that would take too much time and effort. Instead, my proposal is to just reverse the hardware NAND writes. Instead of using a partition, we would have a loop-mounted root filesystem (similar to how Wubi is setup), with the root filesystem being a file on the Media partition. Since there's a non-empty file at that location, the FTL system, whatever it is, must create a one-to-one mapping from logical sectors to physical NAND pages. We can already read the mapping it creates (we have already reversed the read-side FTL code), and so all we have to do to alter the data is to write to the same pages we would've read from. Of course, this means that wear-leveling and bad block handling is not performed. However, if we use a filesystem that's aware of bad blocks and can wear-level (YAFFS or JFFS2), then it amounts to the same thing. The wear-leveling would then take place over the particular physical pages belonging to the rootfs image, rather than the entirety of the NAND. This would make the physical pages belonging to the rootfs image wear out a little faster than the rest of the NAND, but the actual effect of this should be inconsequential.
The additional benefit of this setup is that there's no repartitioning required, so setup is cinch. See
this wiki document for specific proposed implementation details.
Tagi: raw hardware, e care, bad blocks, locati, ftl, data structures, slew, gadget, amount of time, algorithms, kernel, sectors, ace, yaffs, proposal, benefit, linux
Posted by planetbeing on under kernel patches, volatile storage, package managers, serial terminal, csts, myriads, iphone, ftl, device driver, iphe, userland, gadget, shells, operating system, linux, pool |
NAND writing is now semi-reliable (although one has to be VERY careful not to interrupt the device in the middle of a write operation), but it is enough to have something akin to a full-functional OS, backed by non-volatile storage.
People interested in the project should be familiar with the myriads of Linux "distributions" floating around. An operating system consists of two major domains: one is the kernel, which is what manages the hardware, and one is the userland which contains things like shells and other UIs, package managers, etc. Software that help users install and run useful programs. Ubuntu is a popular distribution that I run on my personal machine. Android could also be considered a distribution (though I believe it has some
apparently messy kernel patches).
I decided that Debian would be an interesting thing to try, since we would then instantly have a userland and a pool of ready-compiled applications. Using a slightly dated root filesystem here:
http://lists.debian.org/debian-arm/2007/01/msg00034.html, a initrd and further kernel configurations were sufficient to get it to run. Thus, we can now
compile programs for iPhone Linux
on iPhone Linux. The process is rather slow due to the processor and inefficient NAND device driver (pending a real FTL), but at least theoretically, iPhone Linux is now
self-hosting.
This should be pretty much enough for those who are more into the userland development side of things to come in, possibly using Debian as a base to build anything else (as I believe it is standard enough).
I will be offering instructions on how to get this all to work soon. The (modified for gadget serial terminal) rootfs is fairly hefty (around 130 MB), so I'm not sure how we'll handle distribution of that.
Tagi: kernel patches, volatile storage, package managers, serial terminal, csts, myriads, iphone, ftl, device driver, iphe, userland, gadget, shells, operating system, linux, pool
