https://old.reddit.com/r/DataHoarder/comments/h02jl4/lets_say_you_wanted_to_back_up_the_internet/
So, you think you want to back up the Internet Archive.
This is a gargantuan project and not something to be taken lightly. Definitely consider why you think you need to do this, and what exactly you hope to have at the end. There’s thousands of subcollections at the Archive and maybe you actually want a smaller set of it. These instructions work for those smaller sets and you’ll get it much faster.
Or you’re just curious as to what it would take to get everything .
Well, first, bear in mind there’s different classes of material in the Archive’s 50+ petabytes of data storage. There’s material that can be downloaded, material that can only be viewed/streamed, and material that is used internally like the wayback machine or database storage. We’ll set aside the 20+ petabytes of material under the wayback for the purpose of this discussion other than you can get websites by directly downloading and mirroring as you would any web page.
That leaves the many collections and items you can reach directly. They tend to be in the form of https://archive.org/details/identifier where identifier is the “item identifier”, more like a directory scattered among dozens and dozens of racks that hold the items. By default, these are completely open to downloads, unless they’re set to be a variety of “stream/sample” settings, at which point, for the sake of this tutorial, can’t be downloaded at all - just viewed.
To see the directory version of an item, switch details to download , like archive.org/download/identifier - this will show you all the files residing for an item, both Original, System, and Derived. Let’s talk about those three.
Original files are what were uploaded into the identifier by the user or script. They are never modifier or touched by the system. Unless something goes wrong, what you download of an original file is exactly what was uploaded.
Derived files are then created by the scripts and handlers within the archive to make them easier to interact with. For example, PDF files are “derived” into EPUBs, jpeg-sets, OCR’d textfiles, and so on.
System files are created by the processes of the Archive’s scripts to either keep track of metadata, of information about the item, and so on. They are generally *.xml files, or thumbnails, or so on.
In general, you only want the Original files as well as the metadata (from the *.xml files) to have the “core” of an item. This will save you a lot of disk space - the derived files can always be recreated later.
So Anyway
The best of the ways to download from Internet Archive is using the official client. I wrote an introduction to the IA client here:
The IA Client – The Swiss Army Knife of Internet Archive | Internet Archive Blogs
The direct link to the IA client is here: GitHub - jjjake/internetarchive: A Python and Command-Line Interface to Archive.org
So, an initial experiment would be to download the entirety of a specific collection.
To get a collection’s items, do ia search collection:collection-name --itemlist Then, use ia download to download each individual item. You can do this with a script, and even do it in parallel. There’s also the --retries command, in case systems hit load or other issues arise. (I advise checking the documentation and reading thoroughly - perhaps people can reply with recipes of what they have found.
There are over 63,000,000 individual items at the Archive. Choose wisely. And good luck.
Edit, Next Day:
As is often the case when the Internet Archive’s collections are discussed in this way, people are proposing the usual solutions, which I call the Big Three:
- Organize an ad-hoc/professional/simple/complicated shared storage scheme
- Go to a [corporate entity] and get some sort of discount/free service/hardware
- Send Over a Bunch of Hard Drives and Make a Copy
I appreciate people giving thought to these solutions and will respond to them (or make new stand-along messages) in the thread. In the meantime, I will say that the Archive has endorsed and worked with a concept called The Distributed Web which has both included discussions and meetings as well as proposed technologies - at the very least, it’s interesting and along the lines that people think of when they think of “sharing” the load. A FAQ: Decentralized Web FAQ | Internet Archive Blogs