I’m sure that you’ve heard the adage that the “Internet is forever” or “the Internet never forgets.” But as it turns out, that’s not quite true—unless, of course, you’re a celebrity who said the wrong thing on MySpace 15 years ago. Then, it sticks around forever.
Perhaps, based on licensing and publishing rights, you could gather up your old articles and essays and put them together as a book. It won’t solve the problem overall but you at least would have a personal record.
I would not blame companies like Google, etc. here. Search engine algorithms prioritize the order of results but should not remove the results (in general). What I could experience is that real sources of information are disappearing too - see how many discussion boards are locking threads just for being too old, despite containing useful and valid information. Some sites are disappearing just because the maintainer has no longer resources to keep it running. Recently there was a merger of two retailers that eradicated two decades of quality source of information on computer products just because the new owner did not want to run it anymore and was not interested in preserving/archiving it. All that eventually leads to that the info would not be retrievable in longer or shorter term.
My other concern is a call to forcibly demonopolize Google as a search engine just because it is too big. See what happened and still happens in area of TV streaming: if you like certain genre and would like to see “all” the shows, there is no single provider that would provide it but have to subscribe to several of them. And - because of mergers and licensing - the content is disappearing there as well. I can see an analogy here. Especially when useful content is or will turn to a paid one.
Honestly, it feels good but also weird to read posts like this, by people who have been writing online for years, only in 2024. But OK, better late than never.
Because this has been a well known problem for years. For many more details about both the causes and the fixes, check out these posts from me, and the links they contain:
There are archive sites besides Wayback Machine these days. They’re really popular for making copies of paywalled articles (you have to be able to access the article to archive it).
Big sites like the New York Times not only keep all their articles but many from pre Internet days are available: some are text and some are a digitized display of the actual newspaper. I’m impressed.
Other stuff that seems lost may just be on a new page. You might find it with Google. Of course some stuff is truly lost but I’ve not found it a problem personally.
I'd hazard a guess that it's not strictly google who have triggered the un-indexing but rather the sites hosting the content have chosen to update their robots.txt in response to AI learning and data scraping activities.
Many prominent sites like reddit have recently done this, much to the dismay of users who have relied on doing google searches for sites that have poor search functionality themselves and i suspect this is only going to get worse.. is going to be interesting to see how google try to maintain revenue from there bread n butter, AdSense, in the coming years as web searches become a thing of the past like altavista, lycos, yahoo etc did :)
And yeah yeah yahoo is still around but its not what it used to be just like myspace :P~
PS congrats on the 5000, i'd say i'm shocked it's not higher but in this clown world it's a testament you haven't been cancelled for simply talking with Musk yet! 🥰
I routinely suggest to people that they investigate Google alternatives. I pay a few bucks a month for https://Kagi.com, which has no ads, provides excellent search results, and has more useful features than Google, notably the ability to lower, raise, or outright block sites in future search results (so they can better reflect your own judgement of accuracy and relevance.) I suggest giving it a try for a month, including searching for your own past work.
It’s an interesting thought to be had.
Perhaps, based on licensing and publishing rights, you could gather up your old articles and essays and put them together as a book. It won’t solve the problem overall but you at least would have a personal record.
Thanks 😊
I would not blame companies like Google, etc. here. Search engine algorithms prioritize the order of results but should not remove the results (in general). What I could experience is that real sources of information are disappearing too - see how many discussion boards are locking threads just for being too old, despite containing useful and valid information. Some sites are disappearing just because the maintainer has no longer resources to keep it running. Recently there was a merger of two retailers that eradicated two decades of quality source of information on computer products just because the new owner did not want to run it anymore and was not interested in preserving/archiving it. All that eventually leads to that the info would not be retrievable in longer or shorter term.
My other concern is a call to forcibly demonopolize Google as a search engine just because it is too big. See what happened and still happens in area of TV streaming: if you like certain genre and would like to see “all” the shows, there is no single provider that would provide it but have to subscribe to several of them. And - because of mergers and licensing - the content is disappearing there as well. I can see an analogy here. Especially when useful content is or will turn to a paid one.
Honestly, it feels good but also weird to read posts like this, by people who have been writing online for years, only in 2024. But OK, better late than never.
Because this has been a well known problem for years. For many more details about both the causes and the fixes, check out these posts from me, and the links they contain:
https://mfioretti.substack.com/p/ever-wondered-why-we-are-in-a-digital
https://mfioretti.substack.com/p/google-sucks-snow-white-is-woke-in
And, specifically about professionals who ignore how fragile their online presence is:
https://mfioretti.substack.com/p/what-we-should-all-learn-by-a-famous
There are archive sites besides Wayback Machine these days. They’re really popular for making copies of paywalled articles (you have to be able to access the article to archive it).
Big sites like the New York Times not only keep all their articles but many from pre Internet days are available: some are text and some are a digitized display of the actual newspaper. I’m impressed.
Other stuff that seems lost may just be on a new page. You might find it with Google. Of course some stuff is truly lost but I’ve not found it a problem personally.
I'd hazard a guess that it's not strictly google who have triggered the un-indexing but rather the sites hosting the content have chosen to update their robots.txt in response to AI learning and data scraping activities.
Many prominent sites like reddit have recently done this, much to the dismay of users who have relied on doing google searches for sites that have poor search functionality themselves and i suspect this is only going to get worse.. is going to be interesting to see how google try to maintain revenue from there bread n butter, AdSense, in the coming years as web searches become a thing of the past like altavista, lycos, yahoo etc did :)
And yeah yeah yahoo is still around but its not what it used to be just like myspace :P~
PS congrats on the 5000, i'd say i'm shocked it's not higher but in this clown world it's a testament you haven't been cancelled for simply talking with Musk yet! 🥰
I routinely suggest to people that they investigate Google alternatives. I pay a few bucks a month for https://Kagi.com, which has no ads, provides excellent search results, and has more useful features than Google, notably the ability to lower, raise, or outright block sites in future search results (so they can better reflect your own judgement of accuracy and relevance.) I suggest giving it a try for a month, including searching for your own past work.