[Mayan EDMS: 2541] Migrate sqlite DB to postgres

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 2541] Migrate sqlite DB to postgres

hubertus.peters
I have updated mayan to 3.0 via docker and I am getting the message, that i should not use sqlite in production. But when I start the docker image with postgresql, the database is empty. i assume, mayan will not automatically convert any existing sqlite databases.
I tried to export the sqlite data and import on the posgresql installation with mayan-edms.py loaddata , but im getting errors about duplicat keys, etc. (i did a mayan-edms.py flush   before.).

How do I properly migrate my database from >3.0 mayan db.sqlite3 to posgresql?

--
Hubi

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 2549] Re: Migrate sqlite DB to postgres

rosarior
Administrator
Django itself doesn't provide a method for direct database migrations. What you tried to do is the recommended procedure by the Django project. However for it to work there are some stuff that need to be added to the models so they use unique natural keys and not numeric primary keys which can be repeated (which is the error you encountered). We started adding natural keys a while back but that work is unfinished. What apps reported repeated keys when you did the loaddata command? We can try to work on those to see if the migration finishes without error.

On Tuesday, July 3, 2018 at 5:19:57 PM UTC-4, [hidden email] wrote:
I have updated mayan to 3.0 via docker and I am getting the message, that i should not use sqlite in production. But when I start the docker image with postgresql, the database is empty. i assume, mayan will not automatically convert any existing sqlite databases.
I tried to export the sqlite data and import on the posgresql installation with mayan-edms.py loaddata , but im getting errors about duplicat keys, etc. (i did a mayan-edms.py flush   before.).

How do I properly migrate my database from >3.0 mayan db.sqlite3 to posgresql?

--
Hubi

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 2557] Re: Migrate sqlite DB to postgres

Michael Claassen
Hi,

I'm running in the same issue, that I'm using the SQLite3 DB and getting this message on every site in the WebUI.
I'm wondering if there is a standard routine to migrate the DB from SQLite3 to PostgreSQL, mySQL or MariaDB?
Maybe other users already done so and have a migration path or useful hints, which migration works best?

Kind regards,
Michael

Am Donnerstag, 5. Juli 2018 00:51:58 UTC+2 schrieb Roberto Rosario:
Django itself doesn't provide a method for direct database migrations. What you tried to do is the recommended procedure by the Django project. However for it to work there are some stuff that need to be added to the models so they use unique natural keys and not numeric primary keys which can be repeated (which is the error you encountered). We started adding natural keys a while back but that work is unfinished. What apps reported repeated keys when you did the loaddata command? We can try to work on those to see if the migration finishes without error.

On Tuesday, July 3, 2018 at 5:19:57 PM UTC-4, <a href="javascript:" target="_blank" gdf-obfuscated-mailto="t4pZBzo1CAAJ" rel="nofollow" onmousedown="this.href=&#39;javascript:&#39;;return true;" onclick="this.href=&#39;javascript:&#39;;return true;">hubertu...@... wrote:
I have updated mayan to 3.0 via docker and I am getting the message, that i should not use sqlite in production. But when I start the docker image with postgresql, the database is empty. i assume, mayan will not automatically convert any existing sqlite databases.
I tried to export the sqlite data and import on the posgresql installation with mayan-edms.py loaddata , but im getting errors about duplicat keys, etc. (i did a mayan-edms.py flush   before.).

How do I properly migrate my database from >3.0 mayan db.sqlite3 to posgresql?

--
Hubi

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 2559] Re: Migrate sqlite DB to postgres

rosarior
Administrator
The method that most approaches a migration solution is using dumpdata and loaddata (https://docs.djangoproject.com/en/2.0/ref/django-admin/#dumpdata) (https://docs.djangoproject.com/en/2.0/ref/django-admin/#django-admin-loaddata). dumpdata create a platform independent JSON file of the data using serialization. This method requires that support for natural keys is added to all database models (https://docs.djangoproject.com/en/2.0/topics/serialization/#topics-serialization-natural-keys). This process was started but is not yet complete.

If you tried to do a dumpdata/loaddata migration and it failed, reporting the apps that failed will help us complete support for this and provide an official migration process.

On Friday, July 6, 2018 at 7:39:01 AM UTC-4, Jay 1972 wrote:
Hi,

I'm running in the same issue, that I'm using the SQLite3 DB and getting this message on every site in the WebUI.
I'm wondering if there is a standard routine to migrate the DB from SQLite3 to PostgreSQL, mySQL or MariaDB?
Maybe other users already done so and have a migration path or useful hints, which migration works best?

Kind regards,
Michael

Am Donnerstag, 5. Juli 2018 00:51:58 UTC+2 schrieb Roberto Rosario:
Django itself doesn't provide a method for direct database migrations. What you tried to do is the recommended procedure by the Django project. However for it to work there are some stuff that need to be added to the models so they use unique natural keys and not numeric primary keys which can be repeated (which is the error you encountered). We started adding natural keys a while back but that work is unfinished. What apps reported repeated keys when you did the loaddata command? We can try to work on those to see if the migration finishes without error.

On Tuesday, July 3, 2018 at 5:19:57 PM UTC-4, [hidden email] wrote:
I have updated mayan to 3.0 via docker and I am getting the message, that i should not use sqlite in production. But when I start the docker image with postgresql, the database is empty. i assume, mayan will not automatically convert any existing sqlite databases.
I tried to export the sqlite data and import on the posgresql installation with mayan-edms.py loaddata , but im getting errors about duplicat keys, etc. (i did a mayan-edms.py flush   before.).

How do I properly migrate my database from >3.0 mayan db.sqlite3 to posgresql?

--
Hubi

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 2593] Re: Migrate sqlite DB to postgres

Nicholas Loscalzo
Unfortunately running mayan-edms.py dumpdata from inside the docker container gives me [CommandError: Unable to serialize database: unable to open database file and I'm unsure what's going on.  Is it because Mayan has a lock on the database file ?

The warning messages against using SQLite got me a bit scared, so I'd like to move over to Postgres from SQLite, but I already have a good number of documents in Mayan.  In the worst case scenario, can the database be re-built from scratch if I move the document cache folder into place ?  In that scenario I'd understand if it lost the cabinets, etc

Thanks !
-Nick

On Friday, July 6, 2018 at 4:43:11 PM UTC-4, Roberto Rosario wrote:
The method that most approaches a migration solution is using dumpdata and loaddata (<a href="https://docs.djangoproject.com/en/2.0/ref/django-admin/#dumpdata" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Fref%2Fdjango-admin%2F%23dumpdata\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGTGu7GcIP5ZqipJGtTTX0duGowxw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Fref%2Fdjango-admin%2F%23dumpdata\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGTGu7GcIP5ZqipJGtTTX0duGowxw&#39;;return true;">https://docs.djangoproject.com/en/2.0/ref/django-admin/#dumpdata) (<a href="https://docs.djangoproject.com/en/2.0/ref/django-admin/#django-admin-loaddata" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Fref%2Fdjango-admin%2F%23django-admin-loaddata\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFAQ6N3rODPQJ2fFDk967d9gFN1Uw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Fref%2Fdjango-admin%2F%23django-admin-loaddata\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFAQ6N3rODPQJ2fFDk967d9gFN1Uw&#39;;return true;">https://docs.djangoproject.com/en/2.0/ref/django-admin/#django-admin-loaddata). dumpdata create a platform independent JSON file of the data using serialization. This method requires that support for natural keys is added to all database models (<a href="https://docs.djangoproject.com/en/2.0/topics/serialization/#topics-serialization-natural-keys" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Ftopics%2Fserialization%2F%23topics-serialization-natural-keys\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4p2vrTGKt-nFgReTp2ti_t-3AqA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Ftopics%2Fserialization%2F%23topics-serialization-natural-keys\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4p2vrTGKt-nFgReTp2ti_t-3AqA&#39;;return true;">https://docs.djangoproject.com/en/2.0/topics/serialization/#topics-serialization-natural-keys). This process was started but is not yet complete.

If you tried to do a dumpdata/loaddata migration and it failed, reporting the apps that failed will help us complete support for this and provide an official migration process.

On Friday, July 6, 2018 at 7:39:01 AM UTC-4, Jay 1972 wrote:
Hi,

I'm running in the same issue, that I'm using the SQLite3 DB and getting this message on every site in the WebUI.
I'm wondering if there is a standard routine to migrate the DB from SQLite3 to PostgreSQL, mySQL or MariaDB?
Maybe other users already done so and have a migration path or useful hints, which migration works best?

Kind regards,
Michael

Am Donnerstag, 5. Juli 2018 00:51:58 UTC+2 schrieb Roberto Rosario:
Django itself doesn't provide a method for direct database migrations. What you tried to do is the recommended procedure by the Django project. However for it to work there are some stuff that need to be added to the models so they use unique natural keys and not numeric primary keys which can be repeated (which is the error you encountered). We started adding natural keys a while back but that work is unfinished. What apps reported repeated keys when you did the loaddata command? We can try to work on those to see if the migration finishes without error.

On Tuesday, July 3, 2018 at 5:19:57 PM UTC-4, [hidden email] wrote:
I have updated mayan to 3.0 via docker and I am getting the message, that i should not use sqlite in production. But when I start the docker image with postgresql, the database is empty. i assume, mayan will not automatically convert any existing sqlite databases.
I tried to export the sqlite data and import on the posgresql installation with mayan-edms.py loaddata , but im getting errors about duplicat keys, etc. (i did a mayan-edms.py flush   before.).

How do I properly migrate my database from >3.0 mayan db.sqlite3 to posgresql?

--
Hubi

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 2630] Re: Migrate sqlite DB to postgres

rosarior
Administrator
We are working on adding and official method to convert the database use from SQLite to another like Postgres. Some changes already made it to the next version branch and we've execute successful migrations with some large document databases.

We have included a "convertdb" command (https://gitlab.com/mayan-edms/mayan-edms/blob/versions/next/mayan/apps/common/management/commands/convertdb.py) and will available in the next version.

On Tuesday, July 24, 2018 at 1:33:08 AM UTC-4, Nicholas Loscalzo wrote:
Unfortunately running mayan-edms.py dumpdata from inside the docker container gives me [CommandError: Unable to serialize database: unable to open database file and I'm unsure what's going on.  Is it because Mayan has a lock on the database file ?

The warning messages against using SQLite got me a bit scared, so I'd like to move over to Postgres from SQLite, but I already have a good number of documents in Mayan.  In the worst case scenario, can the database be re-built from scratch if I move the document cache folder into place ?  In that scenario I'd understand if it lost the cabinets, etc

Thanks !
-Nick

On Friday, July 6, 2018 at 4:43:11 PM UTC-4, Roberto Rosario wrote:
The method that most approaches a migration solution is using dumpdata and loaddata (<a href="https://docs.djangoproject.com/en/2.0/ref/django-admin/#dumpdata" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Fref%2Fdjango-admin%2F%23dumpdata\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGTGu7GcIP5ZqipJGtTTX0duGowxw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Fref%2Fdjango-admin%2F%23dumpdata\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNGTGu7GcIP5ZqipJGtTTX0duGowxw&#39;;return true;">https://docs.djangoproject.com/en/2.0/ref/django-admin/#dumpdata) (<a href="https://docs.djangoproject.com/en/2.0/ref/django-admin/#django-admin-loaddata" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Fref%2Fdjango-admin%2F%23django-admin-loaddata\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFAQ6N3rODPQJ2fFDk967d9gFN1Uw&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Fref%2Fdjango-admin%2F%23django-admin-loaddata\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFAQ6N3rODPQJ2fFDk967d9gFN1Uw&#39;;return true;">https://docs.djangoproject.com/en/2.0/ref/django-admin/#django-admin-loaddata). dumpdata create a platform independent JSON file of the data using serialization. This method requires that support for natural keys is added to all database models (<a href="https://docs.djangoproject.com/en/2.0/topics/serialization/#topics-serialization-natural-keys" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Ftopics%2Fserialization%2F%23topics-serialization-natural-keys\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4p2vrTGKt-nFgReTp2ti_t-3AqA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fdocs.djangoproject.com%2Fen%2F2.0%2Ftopics%2Fserialization%2F%23topics-serialization-natural-keys\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNH4p2vrTGKt-nFgReTp2ti_t-3AqA&#39;;return true;">https://docs.djangoproject.com/en/2.0/topics/serialization/#topics-serialization-natural-keys). This process was started but is not yet complete.

If you tried to do a dumpdata/loaddata migration and it failed, reporting the apps that failed will help us complete support for this and provide an official migration process.

On Friday, July 6, 2018 at 7:39:01 AM UTC-4, Jay 1972 wrote:
Hi,

I'm running in the same issue, that I'm using the SQLite3 DB and getting this message on every site in the WebUI.
I'm wondering if there is a standard routine to migrate the DB from SQLite3 to PostgreSQL, mySQL or MariaDB?
Maybe other users already done so and have a migration path or useful hints, which migration works best?

Kind regards,
Michael

Am Donnerstag, 5. Juli 2018 00:51:58 UTC+2 schrieb Roberto Rosario:
Django itself doesn't provide a method for direct database migrations. What you tried to do is the recommended procedure by the Django project. However for it to work there are some stuff that need to be added to the models so they use unique natural keys and not numeric primary keys which can be repeated (which is the error you encountered). We started adding natural keys a while back but that work is unfinished. What apps reported repeated keys when you did the loaddata command? We can try to work on those to see if the migration finishes without error.

On Tuesday, July 3, 2018 at 5:19:57 PM UTC-4, [hidden email] wrote:
I have updated mayan to 3.0 via docker and I am getting the message, that i should not use sqlite in production. But when I start the docker image with postgresql, the database is empty. i assume, mayan will not automatically convert any existing sqlite databases.
I tried to export the sqlite data and import on the posgresql installation with mayan-edms.py loaddata , but im getting errors about duplicat keys, etc. (i did a mayan-edms.py flush   before.).

How do I properly migrate my database from >3.0 mayan db.sqlite3 to posgresql?

--
Hubi

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.