existing hub not recognized

TempMussol

Thu, 01 Nov 2018 10:13:54 -0400

!Hubzilla Support Forum Having solved my 502/504 problem with the able assisstance of @M. Dent, I now get the following when I go to my hub:

Hubzilla Server - Setup
  System check

I am running 3.6.1 and have not changed anything to do with Hubzilla config since it was last running. The restart was due to a full disk and that service mysql status at the time told me that the no space error happened while trying to write a database file though I can't remember which table. Possible corruption causing this?

show all 12 comments

TempMussol

Thu, 01 Nov 2018 11:20:16 -0400 last edited: Thu, 01 Nov 2018 11:49:35 -0400

The database is there:

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| hubzilladb         |
| information_schema |
| maildb             |
| mysql              |
| nextclouddb        |
| performance_schema |
+--------------------+
6 rows in set (0.00 sec)

But mysqlcheck hubzilladb gives me nothing (and not a stream of checked tables as I would expect). So I check /var/lib/mysql/hubzilladb and it's empty. What! I haven't deleted it (I would've remembered that).

edit: Not actually empty:

root@pendle:/var/lib/mysql/hubzilladb# ls -al
total 12
drwx------ 2 mysql mysql 4096 Oct 22 10:38 .
drwxr-xr-x 7 mysql mysql 4096 Nov  1 14:35 ..
-rw-rw---- 1 mysql mysql   65 Mar  8  2018 db.opt

I believe 08 March was the day I transitioned from RedMatrix to Hubzilla.

Philipp

Thu, 01 Nov 2018 11:27:42 -0400

I had the same thing some days ago, without any obvious reason. At my server, .htconfig.php was missing and the system therefore thought it to be a new install.

TempMussol

Thu, 01 Nov 2018 11:44:31 -0400

My database seems to have completely disappeared:

MariaDB [(none)]> use hubzilladb;
Database changed
MariaDB [hubzilladb]> show tables;
Empty set (0.00 sec)

I am now officially panicking. I have definitely not dropped the db (or why would it still show in show databases;?) and I have definitely not deleted the files from /var/lib/mysql/.

cb7f604332cf39

Thu, 01 Nov 2018 12:05:44 -0400

are the files still exist under /var/lib/mysql/ ?
Are the files corrupt?

TempMussol

Thu, 01 Nov 2018 12:08:52 -0400

I have been through the 1,000 commands each of the readline for bash and mysql and I have done nothing that would indicate I have done anything to delete Hubzilla. I am seriously confused and anxious.

Daniel

Thu, 01 Nov 2018 12:45:11 -0400

[wild guesses incoming]
Something may have gone wrong with mysql during the time when you were low on disk space. I remember something similar maybe ~8 years ago with mysql. For me that was just an inconvinience, as that database didn't contain any live data. I was also rather inexperienced with mysql at that time, so that memory may well be false.
Maybe the tables got corrupted. I believe Debian runs an automatic recovery when bringing the database up, so have another look at your logs, especially of the recovery after you free'd up some space.

Unless you have filesystems snapshots, you probably have to restore your backup of the database.

TempMussol

Thu, 01 Nov 2018 14:05:09 -0400

This is bad news. I do a daily backup that is rsynced to another box. However because the database completely disappeared rather than just wouldn't work because it was corrupted, I have been getting happy messages from the dbdump and rsync scripts as it dumps and copies nothing essentially.

I therefore have seven lovely backups of a non database. The "incident" happened over seven days ago.

I am completely baffled as to why a disk space problem would delete a whole multitable database and not just leave one table corrupted

Thu, 01 Nov 2018 15:07:41 -0400

My heart sank as I read your most recent message. I've been in those shoes, and there is nothing I can say to make it better at all. Hopefully, maybe, you have another backup to restore from?

I don't remember clearly enough the details about INNODB and MYSQL, but open write file descriptors and overwriting disk space typically do not play well together. It was likely the auto-repair that is ultimately to blame for deleting them. I try (though not always successfully) to turn stuff like that off - I want to remain in control of the "thinking" - and the machine to do my bidding. If a decision is involved - the system should wait for an operator to make it. Old School, I know. But from my own similar experiences, I know the ruthless efficiency of automation and "automatic defaults".

Daniel

Thu, 01 Nov 2018 16:52:57 -0400

Looking at this part...

root@pendle:/var/lib/mysql/hubzilladb# ls -al
total 12
drwx------ 2 mysql mysql 4096 Oct 22 10:38 .
drwxr-xr-x 7 mysql mysql 4096 Nov  1 14:35 ..
-rw-rw---- 1 mysql mysql   65 Mar  8  2018 db.opt

This means that the last time an entry (file, directory, link, ...) was added or removed from the directory /var/lib/mysql/hubzilladb was on Oct 22 at 10:38 and gives you a chance to find the incident in the logs.
Any file backup after that date will contain nothing.
If you've used mysqldump or similar, it might still make sense to check backups after that timestamp, in case mysql still had the database files open, but if it was an automatic recovery on service restart or a reboot it's rather unlikely.

Another thing to verify: Are all filesystems mounted?

TempMussol

Thu, 01 Nov 2018 17:56:49 -0400 last edited: Thu, 01 Nov 2018 18:01:58 -0400

The back up box is not mine. There is a possibility that the owner of it has a back up of my back up (preferably from between 0100 22 October when the database dump was rsynced and 1038 when the incident occured).

cb7f604332cf39

Fri, 02 Nov 2018 03:17:12 -0400

I keep the fingers crossed.

elmussol

Sun, 04 Nov 2018 17:11:12 -0500

Thank you to everyone who helped on this. Its seems that elsmussols.org is not retrievable. You live and learn. I will be relaunching as elsmussols.net shortly. I have written more on my own timeline.