Lessons Learned: Why We Lost Our Website (And How We’re Fixing It)
December 21, 2025 | By Prof. Galen Fontaise
The Story No Research Institute Wants to Tell
In September 2025, our website went down. Not temporarily—permanently. A routine server migration at our hosting provider went catastrophically wrong, corrupting our database beyond recovery. When we tried to restore from backups, we discovered a painful truth: our backup system had been silently failing for months.
Six years of blog posts, research updates, conference announcements, and community discussions—gone.
This is the story of that failure, what we learned, and how we’re rebuilding FICSS’s digital presence on a foundation that won’t crumble again.
What We Had (And What We Lost)
From FICSS’s founding in 2019 until September 2025, we ran a basic WordPress installation on shared hosting. No dedicated domain—just a subdomain provided by our hosting service. It worked well enough for a small research institute: we could post updates, share working papers, and maintain a minimal web presence.
Our content included:
- 90+ blog posts spanning 2019-2025
- Research updates and project announcements
- Working paper drafts and preprints
- Conference presentations and talks
- Collaboration calls and opportunities
- Commentary on developments in computational social science
We assumed our hosting provider’s “automated backup” feature had us covered. We were wrong.
The Failure Chain: What Went Wrong
1. No Redundancy
We relied entirely on our hosting provider’s backup system. We didn’t maintain our own independent backups—not to a separate server, not to cloud storage, not even to local drives.
The lesson: Never rely on a single backup source. If your hosting provider is both your primary and backup solution, you don’t actually have a backup.
2. No Testing
We never tested our backup restoration process. The hosting provider’s dashboard showed “Backup: Success” every night. We took that at face value.
When we actually tried to restore, we discovered the backups were incomplete—database dumps without the corresponding file structure, making them useless.
The lesson: A backup you haven’t tested is a backup that doesn’t exist. Quarterly restoration tests should be mandatory.
3. No Monitoring
We didn’t have alerts configured for backup failures. The backups had been silently failing since early 2025, and we had no idea.
The lesson: Implement monitoring and alerting. If a backup fails, you need to know immediately—not six months later when disaster strikes.
4. No Documentation
We had no written disaster recovery plan. When the crisis hit, we were scrambling to remember configuration details, plugin versions, and content structure.
The lesson: Document everything. Your disaster recovery plan should assume the person implementing it has never seen your site before.
What We Managed to Salvage
Despite the catastrophic failure, we weren’t starting from absolute zero:
Partial Recovery Sources
Internet Archive (Wayback Machine)
We recovered approximately 30-40% of our content from archive.org snapshots. The coverage was spotty—some months were well-archived, others completely missing.
Personal Email Archives
Every blog post notification we’d sent via email became a recovery source. We were able to reconstruct about 20% of posts from our sent mail folder.
Google Cache
A handful of recent posts (5-10) were still in Google’s cache when we started recovery efforts.
Collaborator Copies
Some colleagues had saved PDFs or screenshots of specific posts they’d found useful. These helped fill gaps, though formatting was lost.
Our Own Notes
Draft versions of some posts existed in our local notes. Not final versions, but better than nothing.
Total Recovery Rate: Approximately 60-65% of original content, with varying quality and completeness.
The Silver Lining: A Chance to Rebuild Properly
As devastating as the loss was, it forced us to confront years of technical debt and infrastructure inadequacy. We decided to treat this not as a disaster recovery, but as an opportunity to establish FICSS’s digital presence on professional foundations.
Our New Infrastructure
1. Dedicated Domain
We secured ficss.institute—a proper institutional domain that conveys legitimacy and permanence. No more subdomain hosting.
2. Professional Hosting
We migrated to managed WordPress hosting with:
- Guaranteed 99.9% uptime SLA
- Automatic scaling for traffic spikes
- Built-in CDN for global performance
- SSL certificates included
- Daily security scans
3. Redundant Backup Strategy
We now implement a 3-2-1 backup approach:
- 3 copies of all data
- 2 different storage media types
- 1 copy stored off-site
Specifically:
- Daily automated backups to hosting provider’s storage
- Weekly backups to Amazon S3 (different provider)
- Monthly backups to local encrypted drives (off-site)
All backups are automatically tested via scripted restoration to a staging environment.
4. Monitoring & Alerting
We configured:
- UptimeRobot for site availability monitoring (5-minute checks)
- Email alerts for backup failures
- Slack notifications for any critical issues
- Weekly backup verification reports
5. Version Control for Content
We are planning to give “Critical content” (particularly research updates and working papers) a secondary home:
- GitHub repository for markdown versions of all posts
- arXiv for all formal working papers
- Open Science Framework for datasets and documentation
This will create a distributed backup that exists independently of our website.
6. Documentation
We created comprehensive documentation:
- Complete infrastructure diagram
- Step-by-step disaster recovery procedures
- Plugin inventory with version history
- Contact information for all services
- Quarterly disaster recovery drills scheduled
The Recovery Process: Where We Stand
As of December 2025, we’re systematically recovering and republishing content:
Current Progress
- Website launched: December 15, 2025
- Posts recovered: ~40 of ~70 (ongoing)
- Date range: 2019-2025
- Recovery rate: 2-3 posts per week
Our Approach
We’re republishing posts with their original dates to preserve the historical record. Each recovered post includes a restoration note explaining the migration.
Posts appear chronologically as we recover them, which means you might see a 2020 post appear after a 2023 post. This is normal—we’re prioritizing content importance over strict chronological recovery.
What We’re Still Missing
Approximately 35-40% of original content remains unrecovered:
- Some 2019-2020 posts (poor archive coverage)
- Comment threads (not archived separately)
- Some images and media files
- Exact formatting and styling (being recreated)
We’re continuing recovery efforts and welcome help—if you saved any FICSS blog posts or have screenshots, please contact us at info@ficss.institute.
Lessons for Other Small Research Institutes
If you’re running a research group, lab, or small institute with a web presence, learn from our mistakes:
Essential Practices
1. Own Your Domain
Use a proper institutional domain (yourname.edu, .org, or .institute). Avoid subdomain hosting that can disappear.
2. Implement 3-2-1 Backups
Three copies, two media types, one off-site. No exceptions.
3. Test Your Backups
Quarterly restoration tests to a staging environment. If you can’t restore it, it’s not a backup.
4. Monitor Everything
Set up alerts for site downtime, backup failures, security issues, SSL expiration—anything that could cause problems.
5. Document Everything
Write disaster recovery procedures assuming total team turnover. Future you will be grateful.
6. Version Control Critical Content
Use GitHub, GitLab, or similar for markdown versions of important content. Distributed backups are resilient backups.
7. Budget for Infrastructure
Professional hosting, backup storage, monitoring services—these aren’t luxuries. They’re operational necessities. Budget $500-2000/year minimum.
8. Separate Research from Web Infrastructure
Your research publications, datasets, and code should live on platforms designed for permanence (arXiv, GitHub, OSF, Zenodo). Your website can complement these, but shouldn’t be the sole repository.
The Human Cost
Beyond the technical lessons, there’s an emotional dimension worth acknowledging.
Losing six years of work—even “just” blog posts—was demoralizing. Those posts represented conversations, ideas in development, responses to current events in our field. They were part of FICSS’s institutional memory.
Recovery is tedious. Manually recreating posts from fragmented sources, reformatting text, tracking down missing images—it’s draining work that takes time away from actual research.
But it’s also been humbling. The experience has made us better stewards of our digital infrastructure and more appreciative of the invisible labor that keeps research institutions running.
Moving Forward
We’re treating this migration not as a return to the status quo, but as an opportunity to build something better:
- Improved accessibility: Better mobile experience, faster loading
- Enhanced discoverability: Proper SEO, better categorization
- Stronger security: HTTPS everywhere, security hardening
- Long-term sustainability: Infrastructure that scales with FICSS
We’re also committing to greater transparency about our operations—this post being Exhibit A.
Thank You
To everyone who has reached out with sympathy, support, or salvaged content: thank you.
To our collaborators who patiently waited while we rebuilt: thank you.
To the Internet Archive and other digital preservation efforts: your work matters more than you know.
Comments and Discussion
We welcome your thoughts, similar experiences, or additional advice in the comments below. This is a learning opportunity for the entire research community.
Originally published: December 21, 2025
Category: Institute News, Operations