(One in a series of posts about the system recovery)

I've started fixing up more issues with the mail system and bringing it up to 2024 standards. Now we have DKIM, ARC and DMARC installed and configured. DMARC is set to reporting mode but in another week or so I will set it to reject mode if the reports I get are favorable. I've also reinstalled clamav-milter and clamd for better anti-malware protection on mail passed through obscure.org. I've started the installation process for mailman3 to restore mailing list functionality.

I have also managed to get the bareos backup volumes online and am preparing to configure the software. That should make it possible to recover backups of some of the last things that have not yet been recoved, such as the mysql databases and configurations of some of the other support systems that have been offline since the 2022 disaster.

Slowly but surely we are getting there! You can call or SMS me at +1 (571) 236-0938 if you have questions or need urgent help.

(One in a series of posts about the system recovery)

Since the new system upgrade, changing passwords via the Roundcube webmail program has been broken, but thanks to recent fixes in poppassd-ceti this has been resolved.

It has been a bit of a slog to get the last bits of functionality back online, but I'm still working on it. You can call or SMS me at +1 (571) 236-0938 if you have questions or need urgent help.

(One in a series of posts about the system recovery)

A recent system update tightened crypto security policies on the main Obscure server. As a result, we have re-generated the SSH host keys for tiamat.obscure.org; you will need to update your local `ssh` configuration to accept the new keys.

The new SSH ED25519 fingerprint is SHA256:PcjtHUn2X+SGwXgnr4gSEOWDZjdM/SVnl17ssv54Toc. I also re-generated the RSA key, which had been a 1024 bit key. The new SSH RSA fingerprint is SHA256:f+Q+q80voBpW0nUlVRR0YYPLyOe+Vd18ptlyWMBpRq4.

I still plan on diving into these areas next: mailing lists with Mailman, establishing new backups, and any additional software requested by users.

I am going to do whatever I can to restore service. You can call or SMS me at +1 (571) 236-0938 if you have questions or need urgent help.

(One in a series of posts about the system recovery)

Obscure now has Roundcube webmail operational again! I have tested this for receiving and sending mail, and the address book and identity settings should now be fixed. The old experimental webmail installation is no longer needed given our new Dovecot IMAP configuration that handles both Maildir and mbox format mailboxes.

I managed to restore the PostgreSQL datbases, including the membership database and the Roundcube database. I also have the Icinga monitoring service working reasonably well to monitor basic operations on the server. I also recently installed some software requested by uesrs, including mutt, emacs, and tcsh.

I plan on diving into these areas next: mailing lists with Mailman, establishing new backups, and any additional software requested by users.

I am going to do whatever I can to restore service. You can call or SMS me at +1 (571) 236-0938 if you have questions or need urgent help.

(One in a series of posts about the system recovery)

Obscure is now listening for mail clients on IMAP and POP3. The standard ports (110 and 995 for POP3 and POP3S and 143 and 993 for IMAP and IMAPS) are all working. Also the "special" ports we had allocated for the Maildir-formatted-mailbox experiment (10110/10143/10993/10993) are working. Folks who have had their mailbox format converted to Maildir should not need to do anything to their mail clients.

Folks still using the legacy mbox mail format may need to change their mail client configuration to remove the IMAP namespace prefix setting (previously ~mail) to get their mail clients to see mail folders other than the inbox.

I plan on diving into these areas next: webmail with Roundcube, mailing lists with Mailman, the PostgreSQL membership database, the icinga monitoring service, establishing new backups, and any software requested by users.

It looks like the home directories and mail spools were 100% intact, and I am so relieved that I got mail running before most other servers started bouncing mail - we should see more mail deliveries soon.

I am going to do whatever I can to restore service. You can call or SMS me at +1 (571) 236-0938 if you have questions or need urgent help.

(One in a series of posts about the system recovery)

Obscure is now accepting mail from the outside world, and it looks like it is getting delivered ok. I have not restored IMAP or POP3 service yet however. You may be able to SSH in and view mail via alpine, depending on how your mail is configured.

I hope to restore IMAP and POP3 service next.

I am going to do whatever I can to restore service. You can call or SMS me at +1 (571) 236-0938 if you have questions or need urgent help.

(One in a series of posts about the system recovery)

I've made some progress on recovering the Obscure main server after the catastrophic system crash last week, see below for details.

Basic web service is restored, including HTTP and HTTPS for the main site, and HTTP for all other sites. However, but mail is still completely offline. Users may be able to log in via SSH, but only a few home directories are restored at this point.

I am going to do whatever I can to restore service. You can call or SMS me at +1 (571) 236-0938 if you have questions or need urgent help.

On Thursday, August 18th, the main server running obscure.org suffered a series of power failures due to a faulty uninterruptible power supply. This resulted in multiple damaged filesystems that prevented the hypervisor from booting, among other problems. Given the age of that server (a 2008-vintage HP DL360 G5), the previous obsolete operating system (CentOS 5), and the extent of the filesystem damage recovering it in place is not going to be possible. I have procured some new hardware and I am in the process of restoring services. Mail is not yet flowing, it is going to take a significant effort to set that up again. I hope to have mail running in the next couple of days, but it may be a tall order.

The main backup system was a virtual machine on the same physical hardware, one of the many decisions I am regretting now, as it will be challenging to rebuild it enough to be able to do restores of any damaged files.

Initial inspections are showing that user home directories and mail spools are intact, fortunately. I am rebuilding using Rocky Linux 9 which will be supported for many years to come. This incarnation is going to be much simpler, with no hypervisor, at least to start.

I am going to do whatever I can to restore service. You can call or SMS me at +1 (571) 236-0938 if you have questions or need urgent help.

I'm pleased to announce that The Obscure Organization is now running on a new Verizon FiOS gigabit-class fiber optic Internet connection. This represents a major speed upgrade from the 75 megabit Verizon FiOS connection we have been running for the past few years.

As a part of this transition, the main IP address for tiamat.obscure.org has changed to 71.163.169.18. If you have IP address whitelists that reference obscure.org, you may need to update them. You proably will not have to change DNS server registrations, as I have already updated the IP address for dns1.obscure.org with Network Solutions, the registrar responsible for posting the glue record for dns1.obscure.org to the parent domain.

We've come a long way since 1995 - when we had a full-time SLIP connection (the predecessor to PPP) running at 28Kbps. The new connection is approximately 35,000 times faster than the one we started with 25 years ago!

25 years ago I registered the domain name obscure.org, on March 7, 1995.

Many things have changed since I started giving out free accounts to people that asked nicely. The Obscure Organization incorporated and became a full-fledged 501(c)(3) non-profit organization 18 years ago. We've provided free web hosting for multiple other community organizations. We've taught classes and made the materials available for people to build on. But most importantly, we've connected people who have a desire to learn with other people that want to use technology in creative ways.

I'd like to hear from folks who have used the services we've provided.

1. What do you value about Obscure?
2. How have you benefited?
3. What should we aim to do over the next 1, 5, 10, and 25 years?

I'll start:

1. I value the connections I've made with people, and the help that we've given community organizations. I also value that the organization has been able to provide an alternative to the advertising-driven "free" model that the big commercial service providers and social networks have settled on to provide basic services such as email and social communications.
2. I have made friends around the DC area, and around the world through my work leading obscure.org. The community involvement even helped me meet my lovely wife Patricia.
3. I hope to continue to serve the communities and people we have been serving, but I'd also like to find a way to expand our services to a broader audience. This is going to mean getting more and likely different people involved in both governance and operations. Over the next year I want to do a hardware and software refresh for Obscure, and set goals for the next 5, 10, and 25 years based on community feedback.

Please feel free to contact me by email with your feedback.

The article Decentralised SMTP is for the greater good from Gilles Chehade from poolp.org explains why having a decentralized email system is preferable to the alternatives. The Obscure Organization runs its mail server on hardware we own on premises. In a world where cloud giants have a huge share of SMTP users and traffic, this is still important.

I’m proud to announce the release of several new scripts from The Obscure Organization’s vaults. This repository has a mixture of scripts for both server and client environments.

Server-focused scripts in the repository help us maintain servers and 3rd party integrations, ranging from Cloud Init scripts to enable IPv6 on CentOS on AWS EC2, to Icinga2 watchdog scripts, and Uptime Robot firewall manipulation scripts.

Client-focused scripts focus on ssh and authentication agent forwarding, and session management.

The obscureorganization/obscure-scripts repo contains one of the only examples of Harry Potter parody fan fiction written in bash: patronus.sh (Used to banish all interactive user sessions for a user except the one you are currently running)

https://github.com/obscureorganization/obscure-scripts

The Obscure Organization promotes creativity and community through technology - and is approaching its 25th year in existence and 18th year as a 501(c)(3) public charity.

On Monday, June 3^rd, obscure.org will have some downtime due to planned installation of a gigabit-class Internet connection, via Verizon FiOS. They have to install new network equipment to support the faster connection - 980Mbps downstream and 860Mbps upstream.

The new connection will necessitate re-assigning IP addresses to all our hosts. We don't have the new IP addresses yet, but will announce them in email and on the obscure.org home page as soon as we do. Most people won't need to take any action regarding this update.

We are still seeking donations to cover the cost of new hardware to replace our 10-year-old main server, please consider donating.

We are pleased to announce the availability of lj2wp, a toolkit that allows you to export LiveJournal posts and import them into WordPress in an isolated environment.

The current WordPress LiveJournal importer was broken as of 2017-04-09, but an older path exists that can enable you to import your LiveJournal to WordPress.

The tool ljdump still works to export LiveJournal content. It contains a converter called convertdump that exports that to an older WordPress XML import format. Unfortunately, WordPress.com and recent versions of WordPress no longer import this format. In order to import it, you need an older version of WordPress, sucn as 2.7.1. If you have that on hand, you can go through the process described in lj2wp to get a Wordpress-formatted import file you can use in a modern version of WordPress, or other software that can read WordPress export files.

I used this myself to migrate my LiveJournal account to wordpress in 2017. In April 2019, I secured permission from both of the authors of the upstream code to relicense this under the MIT License with an Obscure Organization copyright.

On Fri, Nov 16^th, I restored service to Obscure. The disk volumes used for online operations of the main VM, tiamat.obscure.org, are now on redundant storage. Due to a compound error I accidentally knocked the server offline completely last Tuesday, so some mail may have bounced. There might be some minor data loss in the /var partition, though I think I got all the critical user files migrated, including mail and databases.

The hardware we are running on is 9 years old, and it's time for a major upgrade. Please consider donating to support our long-term stability and ability to serve the community.

Obscure is having some hardware trouble. One of the disks that is not part of a redundant set is failing and one of the network ports on the main server has not been reliable for the past week. Expect intermittent downtime as I attempt repairs over the next few days. On Friday, a power surge and outage exacerbated the problems.

We have switched to using LetsEncrypt SSL certificates for the Obscure web site and mail servers. These are free, and have been set up to renew automatically using EFF's CertBot software. Please let us know if you have trouble with the web site or SMTP / IMAP / POP certificates.

I just updated the Obscure User Home Page Directory to add users that have non-trivial public home pages, and to remove home pages of users who have closed or removed their pages.

Curating these by hand every 5 to 10 years seems to be about the speed of updates we can manage. I am OK with that. This is an HTML page that has been lovingly updated by hand for more than 20 years.

The IP address for the dns2.obscure.org server has changed to 35.163.133.59.

This restores DNS redundancy which we had lost due to a compound failure of monitoring systems, and a provider who discontinued services without stopping billing for those services for over 6 months.

No action should be required on the part of people who host domains on obscure.org.

The set of DNS servers for domains hosted on obscure.org has just been updated to this set of servers:

dns1.obscure.org
dns2.obscure.org

If you host a domain on obscure.org, and it is registered with Network Solutions, and Richard Bullington-McGuire is the technical contact, all changes have been made for you. If you use another registrar, please make sure that the set of domain servers assigned for your domain matches the above list exactly.

Many thanks to go Tripp Lilley, who for many years provided a third domain name server for free for obscure.org domains as a service to the community.

Recently we've added the ability to "greylist" mail, that is, delay mail delivery for unknown senders in order to reduce incoming spam volume. It may result in mail delivery being delayed by 15 minutes to an hour for legitimate mail from addresses that don't email Obscure users often.

I've applied this to every Obscure account that has its mail forwarded elsewhere, in order to reduce the amount of spam that we relay. If you are forwarding your mail and would like greylisting disabled for your account, please ask me.

I'm pleased to announce that we have a new chat system for the Obscure community, provided by Slack.

You can sign up for it with your Obscure.org email address at:

https://obscureorg.slack.com/

This is one of the new breed of persistent, searchable chat systems with support for public and rivate groups, emoji, chat bots, and 3rd party integrations. It's tons of fun and several Obscure users have been kicking the tires for a while—please join us!. If you are part of our community and want an invite but don't have a current working Obscure.org email address, write me or better yet send me a text message at +1 571 236 0938 and I will add you to Slack.

Keep in mind that this system is hosted by a third party (Slack) and is not directly under our administrative control - it doesn't have the same level of privacy that the central Obscure systems offer.