Back to topic, I don't see a need for film or video because they replace imagination with concrete images. For this reason, I'm going to have to say the popularity of the novel over the theatrical form -- replacing it in Shakespeare's time -- is not an accident.
It's certainly no accident, but it has nothing to do with the artistic viability of the forms, and everything to do with the structure subsequent history. The rise of the novel (in English literature) was a response not only to the historical conditions of the 17th century, but also to the changing structure of British society.
Whatever else is true of Tudor and early Stuart England, it remained, in essence, an aristocratic, feudal society structured around essentially medieval social and governmental mechanisms. Drama - as well as other public performance traditions: bards, minstrelsy, oral epic poetry etc. - is a natural artistic fit for such societies. While it can be enjoyed by men of all ranks, such performing arts were traditionally supported by aristocratic patronage. As it is a performance-based artforms experienced publicly and collectively (before a time when performance could be recorded for posterity), theater did not lend itself to mass production and commodification. The public nature of drama made it vulnerable to disruption by public disorder.
And here we come to the crux of it. In the 17th century, public order was routinely disrupted through much of Europe, England included. The English experience was, while perhaps less devastating than what occurred in the Germanies, probably the most severe in terms of long-term consequences. The 17th century saw the power of the feudal aristocracy in England decisively broken, first by defeat in the Civil War, and then by the implementation of an extremely limited monarchy following the Glorious Revolution.
The rest of Europe emerged from the era with their previous social systems more or less intact, however. In this context, it is worth noting that drama retained its primacy in much of the rest of Europe long after it was a fossil relic of a bygone era in merry old England. It is, for instance, no accident that Moliere was the central literary figure of the Sun King's reign, nor that Chekov similarly dominated the literary scene of the last truly aristocratic society in Europe.
In England though, the aristocratic, essentially medieval society that had nurtured drama in Shakespeare's day was gone. The society that emerged in its wake was a mercantile one, built on money and dominated by the commercial interests of the middle class. Drama receded in importance precisely because it was a remnant of the aristocratic tradition. What Napoleon's 'nation of shopkeepers' needed was art that:
1.) could be commodified and sold as a mass product
2.) affirmed the importance of the individual
3.) could be enjoyed individually and in private
Enter the novel.