As far as capabilities go, both chips are remarkably similar, with the Gameboy supporting an extra background. They both do a handful of sprites and tiled backgrounds with attributes for things like palette selection. However, the NES has one difference that throws a wrench in things. Unlike the Gameboy, which has VRAM in the console, the NES reads all of its graphics data from the cartridge directly.
The NES's PPU is simple in operation, but many cartridge types (MMC3 and MMC5 in particular) relied on the precise behavior of the PPU's reads and writes to time some of their extra features, often something like a scanline counter. This makes the complete system much more difficult to emulate, even though the base features of the console are straightforward to grasp.
What makes the NES such an attractive target for a first emulator is that, due to this quirk and the system's popularity, its hardware is just about perfectly understood. Take a look at the Nesdev Wiki's article on rendering, which has an extremely thorough breakdown of the timing, including the bugs in the sprite rendering subsystem and how to correctly emulate them:
I would think the thing that would make the NES unsuitable would actually be the very large number of mapper chips for the platform. The NES seems less like one platform and more like tens of them at that point. AFAIK the Gameboy didn't really have this proliferation of chips that manipulated the system so directly.
At the very least, you might be able to badly emulate most gameboy games, but a mapperless NES emulator is only going to even be able to try to run the first generation of games from the console.
So you can cover about 95% of the library (and 99% of the games people want to play) with about 20 or so mappers. From there it can be a slog to support them all, but conceptually mappers are really very simple. Almost all of them are just registers that let you page in chunks of memory.
The most advanced mapper is the MMC5, that one can take some doing. And the VRC7's OPL audio is also a nightmare (yet used only for one game.) But beyond that, it's all really simple. Usually around 10-30 lines of code per mapper. Mostly very well documented, with the exception of how mappers detect scanlines (important for accurate emulation, not so much for full compatibility. You can just cheat and give them the PPU scanline counter.)
But, maybe I'm biased. I've emulated thirteen systems so far, and the NES was one of my favorites to work on due to the documentation and general simplicity. I know the rabbit hole goes on forever even with the NES, though.
This one walks you through writing an emulator for the classic Space Invaders game: http://emulator101.com/
But really, if you've had a computer architectures class (or do equivalent research on your own), it's doable to just start looking up info about a specific system, and digging down as necessary.
This is definitely true. The gameboy does have memory mapped cartridges to deal with, but they don't hook quite so deeply into the system; they're much more straightforward to implement, and for simpler games you can ignore the mapper entirely.
The NES is a different beast; nametables _alone_ will give an early NES author headaches, and they're incredibly important to get right, otherwise games won't be able to scroll properly. Even simpler games like Mario and Metroid rely on basic nametable support being there. In this case I think it's the limitations of the NES itself that led to the mappers doing so much more externally; game authors had no other choice if they wanted to pull off advanced tricks. This trend continued with the Super Nintendo; remember the SuperFX chip? It almost turns the Super Nintendo into a dumb terminal at that point, generating entire screens worth of graphics in the gamepak directly.
> In this case I think it's the limitations of the NES itself that led to the mappers doing so much more externally
Agreed. The Game Boy's ability to generate interrupts at various times in the scanline, and access to the current scanline number, make timing things with vanilla hardware much easier on the Game Boy than the NES.
The NES's PPU is simple in operation, but many cartridge types (MMC3 and MMC5 in particular) relied on the precise behavior of the PPU's reads and writes to time some of their extra features, often something like a scanline counter. This makes the complete system much more difficult to emulate, even though the base features of the console are straightforward to grasp.
What makes the NES such an attractive target for a first emulator is that, due to this quirk and the system's popularity, its hardware is just about perfectly understood. Take a look at the Nesdev Wiki's article on rendering, which has an extremely thorough breakdown of the timing, including the bugs in the sprite rendering subsystem and how to correctly emulate them:
https://wiki.nesdev.com/w/index.php/PPU_rendering