[Mayan EDMS: 1710] Mayan EDMS Docker image updates

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 1710] Mayan EDMS Docker image updates

rosarior
Administrator
As part of the round of improvements to the next version of the Docker image, a production stack using Docker compose has been included.
This stack will launch 4 containers:

- A database container using Postgres
- A Celery broker container using RabbitMQ
- A Celery results container using Redis
- A Mayan EDMS container that uses the above service containers

If you have experience building Docker images and using Docker compose, please help test this stack on your hardware.

Other improvements added:

Allowing changing the database backend using the environment variables:

- MAYAN_DATABASE_DRIVER, default: None
- MAYAN_DATABASE_NAME, default : 'mayan'
- MAYAN_DATABASE_USER, default: 'mayan'
- MAYAN_DATABASE_PASSWORD, default: ''
- MAYAN_DATABASE_HOST, default: None
- MAYAN_DATABASE_PORT, default: None

The Celery broker and Celery results backend can also be changed using the environment variables:

- MAYAN_BROKER_URL, default: 'redis://127.0.0.1:6379/0'
- MAYAN_CELERY_RESULT_BACKEND, default: 'redis://127.0.0.1:6379/0'

If the MAYAN_BROKER_URL and MAYAN_CELERY_RESULT_BACKEND are specified, the built in
REDIS server is disabled in favor of the external Celery result backend.

A new dedicated image converter worker was added to supervisor. This allows for previews to be generated
and displayed even during high loads (such as when doing OCR on large documents).

The installation process was simplified and reduced to a single step:

docker run -d --name mayan-edms --restart=always -p 80:80 -v mayan_data:/var/lib/mayan mayanedms/next

Gone is the initialization step. The image now keeps track of the state of the data in the volume to determine whether to initialize and run 
or upgrade and run when launched.

The volume setup was also simplified and reduced to a single volume. The settings and document storage files now co-exists in a single tree
inside a sngle volume.

The NGINX web server limits for uploading files were increased to a max file size of 500 MB and the upload timeout was increased to
 600 seconds (10 minutes). These are web server settings and don't affect the overall memory usage of the container.

If an external database is used, database connection keepalive is automatically enabled with a default of 60 seconds.
This Django settings allows reusing database connections instead of closing and opening database connection for each
HTTP request/response cycle. 

Finally a health check was added to allow other Docker images like reverse proxies or load balancers to know when the Mayan EDMS container
is ready for traffic.

To follow and test these improvements follow the 'development' branch of the Docker repository at: https://gitlab.com/mayan-edms/mayan-edms-docker/tree/development 

Please help test these changes to move them to the stable master branch as soon as possible.

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 1747] Re: Mayan EDMS Docker image updates

David Kornahrens
Hello Roberto.  We are new to your project, but have spun up a Docker image of Myan.  We are attempting to run this in a production environment.

Few questions here:

-Even though we are on the latest right now (we believe), what is the command to update the docker image without losing our data?
-Is this ready for primetime?

Thanks,
David

On Thursday, May 18, 2017 at 2:02:21 AM UTC-4, Roberto Rosario wrote:
As part of the round of improvements to the next version of the Docker image, a production stack using Docker compose has been included.
This stack will launch 4 containers:

- A database container using Postgres
- A Celery broker container using RabbitMQ
- A Celery results container using Redis
- A Mayan EDMS container that uses the above service containers

If you have experience building Docker images and using Docker compose, please help test this stack on your hardware.

Other improvements added:

Allowing changing the database backend using the environment variables:

- MAYAN_DATABASE_DRIVER, default: None
- MAYAN_DATABASE_NAME, default : 'mayan'
- MAYAN_DATABASE_USER, default: 'mayan'
- MAYAN_DATABASE_PASSWORD, default: ''
- MAYAN_DATABASE_HOST, default: None
- MAYAN_DATABASE_PORT, default: None

The Celery broker and Celery results backend can also be changed using the environment variables:

- MAYAN_BROKER_URL, default: 'redis://<a href="http://127.0.0.1:6379/0" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;">127.0.0.1:6379/0'
- MAYAN_CELERY_RESULT_BACKEND, default: 'redis://<a href="http://127.0.0.1:6379/0" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;">127.0.0.1:6379/0'

If the MAYAN_BROKER_URL and MAYAN_CELERY_RESULT_BACKEND are specified, the built in
REDIS server is disabled in favor of the external Celery result backend.

A new dedicated image converter worker was added to supervisor. This allows for previews to be generated
and displayed even during high loads (such as when doing OCR on large documents).

The installation process was simplified and reduced to a single step:

docker run -d --name mayan-edms --restart=always -p 80:80 -v mayan_data:/var/lib/mayan mayanedms/next

Gone is the initialization step. The image now keeps track of the state of the data in the volume to determine whether to initialize and run 
or upgrade and run when launched.

The volume setup was also simplified and reduced to a single volume. The settings and document storage files now co-exists in a single tree
inside a sngle volume.

The NGINX web server limits for uploading files were increased to a max file size of 500 MB and the upload timeout was increased to
 600 seconds (10 minutes). These are web server settings and don't affect the overall memory usage of the container.

If an external database is used, database connection keepalive is automatically enabled with a default of 60 seconds.
This Django settings allows reusing database connections instead of closing and opening database connection for each
HTTP request/response cycle. 

Finally a health check was added to allow other Docker images like reverse proxies or load balancers to know when the Mayan EDMS container
is ready for traffic.

To follow and test these improvements follow the 'development' branch of the Docker repository at: <a href="https://gitlab.com/mayan-edms/mayan-edms-docker/tree/development" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgitlab.com%2Fmayan-edms%2Fmayan-edms-docker%2Ftree%2Fdevelopment\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG3H-JgLK1oRIH0HuDQ9OppWQBZEA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgitlab.com%2Fmayan-edms%2Fmayan-edms-docker%2Ftree%2Fdevelopment\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG3H-JgLK1oRIH0HuDQ9OppWQBZEA&#39;;return true;">https://gitlab.com/mayan-edms/mayan-edms-docker/tree/development 

Please help test these changes to move them to the stable master branch as soon as possible.

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 1750] Re: Mayan EDMS Docker image updates

rosarior
Administrator
Hi David,

When using Docker the database, documents and settings are kept in separate volumes. The volumes are persistent and remain after version upgrades.
The current image is being used in production without issues, in fact users report more success with the Docker image than trying the advanced deployment themselves and the OS, Django and Python experience it requires.

As mentioned in the original post, new improvements are being made to the image including being able to use a Postgresql database or container instead of the default SQLite to improve performance and data protection.

On Thursday, June 1, 2017 at 9:24:29 PM UTC-4, David Kornahrens wrote:
Hello Roberto.  We are new to your project, but have spun up a Docker image of Myan.  We are attempting to run this in a production environment.

Few questions here:

-Even though we are on the latest right now (we believe), what is the command to update the docker image without losing our data?
-Is this ready for primetime?

Thanks,
David

On Thursday, May 18, 2017 at 2:02:21 AM UTC-4, Roberto Rosario wrote:
As part of the round of improvements to the next version of the Docker image, a production stack using Docker compose has been included.
This stack will launch 4 containers:

- A database container using Postgres
- A Celery broker container using RabbitMQ
- A Celery results container using Redis
- A Mayan EDMS container that uses the above service containers

If you have experience building Docker images and using Docker compose, please help test this stack on your hardware.

Other improvements added:

Allowing changing the database backend using the environment variables:

- MAYAN_DATABASE_DRIVER, default: None
- MAYAN_DATABASE_NAME, default : 'mayan'
- MAYAN_DATABASE_USER, default: 'mayan'
- MAYAN_DATABASE_PASSWORD, default: ''
- MAYAN_DATABASE_HOST, default: None
- MAYAN_DATABASE_PORT, default: None

The Celery broker and Celery results backend can also be changed using the environment variables:

- MAYAN_BROKER_URL, default: 'redis://<a href="http://127.0.0.1:6379/0" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;">127.0.0.1:6379/0'
- MAYAN_CELERY_RESULT_BACKEND, default: 'redis://<a href="http://127.0.0.1:6379/0" rel="nofollow" target="_blank" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;">127.0.0.1:6379/0'

If the MAYAN_BROKER_URL and MAYAN_CELERY_RESULT_BACKEND are specified, the built in
REDIS server is disabled in favor of the external Celery result backend.

A new dedicated image converter worker was added to supervisor. This allows for previews to be generated
and displayed even during high loads (such as when doing OCR on large documents).

The installation process was simplified and reduced to a single step:

docker run -d --name mayan-edms --restart=always -p 80:80 -v mayan_data:/var/lib/mayan mayanedms/next

Gone is the initialization step. The image now keeps track of the state of the data in the volume to determine whether to initialize and run 
or upgrade and run when launched.

The volume setup was also simplified and reduced to a single volume. The settings and document storage files now co-exists in a single tree
inside a sngle volume.

The NGINX web server limits for uploading files were increased to a max file size of 500 MB and the upload timeout was increased to
 600 seconds (10 minutes). These are web server settings and don't affect the overall memory usage of the container.

If an external database is used, database connection keepalive is automatically enabled with a default of 60 seconds.
This Django settings allows reusing database connections instead of closing and opening database connection for each
HTTP request/response cycle. 

Finally a health check was added to allow other Docker images like reverse proxies or load balancers to know when the Mayan EDMS container
is ready for traffic.

To follow and test these improvements follow the 'development' branch of the Docker repository at: <a href="https://gitlab.com/mayan-edms/mayan-edms-docker/tree/development" rel="nofollow" target="_blank" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgitlab.com%2Fmayan-edms%2Fmayan-edms-docker%2Ftree%2Fdevelopment\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG3H-JgLK1oRIH0HuDQ9OppWQBZEA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgitlab.com%2Fmayan-edms%2Fmayan-edms-docker%2Ftree%2Fdevelopment\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG3H-JgLK1oRIH0HuDQ9OppWQBZEA&#39;;return true;">https://gitlab.com/mayan-edms/mayan-edms-docker/tree/development 

Please help test these changes to move them to the stable master branch as soon as possible.

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 1751] Re: Mayan EDMS Docker image updates

David Kornahrens
In reply to this post by rosarior
Thank you Roberto.  We have spun up a Docker image with your product and have been building out our instance to fit our needs.  We are impressed with the product so far.  I'm sure I'll have some other questions.  Once we get everything up and running, we plan to purchase a support contract with you for business use as we will most likely need some assistance.

On Thursday, May 18, 2017 at 2:02:21 AM UTC-4, Roberto Rosario wrote:
As part of the round of improvements to the next version of the Docker image, a production stack using Docker compose has been included.
This stack will launch 4 containers:

- A database container using Postgres
- A Celery broker container using RabbitMQ
- A Celery results container using Redis
- A Mayan EDMS container that uses the above service containers

If you have experience building Docker images and using Docker compose, please help test this stack on your hardware.

Other improvements added:

Allowing changing the database backend using the environment variables:

- MAYAN_DATABASE_DRIVER, default: None
- MAYAN_DATABASE_NAME, default : 'mayan'
- MAYAN_DATABASE_USER, default: 'mayan'
- MAYAN_DATABASE_PASSWORD, default: ''
- MAYAN_DATABASE_HOST, default: None
- MAYAN_DATABASE_PORT, default: None

The Celery broker and Celery results backend can also be changed using the environment variables:

- MAYAN_BROKER_URL, default: 'redis://<a href="http://127.0.0.1:6379/0" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;">127.0.0.1:6379/0'
- MAYAN_CELERY_RESULT_BACKEND, default: 'redis://<a href="http://127.0.0.1:6379/0" target="_blank" rel="nofollow" onmousedown="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;" onclick="this.href=&#39;http://www.google.com/url?q\x3dhttp%3A%2F%2F127.0.0.1%3A6379%2F0\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNFqfdzNqQtEaZWAsQlAy1i5F1LFPw&#39;;return true;">127.0.0.1:6379/0'

If the MAYAN_BROKER_URL and MAYAN_CELERY_RESULT_BACKEND are specified, the built in
REDIS server is disabled in favor of the external Celery result backend.

A new dedicated image converter worker was added to supervisor. This allows for previews to be generated
and displayed even during high loads (such as when doing OCR on large documents).

The installation process was simplified and reduced to a single step:

docker run -d --name mayan-edms --restart=always -p 80:80 -v mayan_data:/var/lib/mayan mayanedms/next

Gone is the initialization step. The image now keeps track of the state of the data in the volume to determine whether to initialize and run 
or upgrade and run when launched.

The volume setup was also simplified and reduced to a single volume. The settings and document storage files now co-exists in a single tree
inside a sngle volume.

The NGINX web server limits for uploading files were increased to a max file size of 500 MB and the upload timeout was increased to
 600 seconds (10 minutes). These are web server settings and don't affect the overall memory usage of the container.

If an external database is used, database connection keepalive is automatically enabled with a default of 60 seconds.
This Django settings allows reusing database connections instead of closing and opening database connection for each
HTTP request/response cycle. 

Finally a health check was added to allow other Docker images like reverse proxies or load balancers to know when the Mayan EDMS container
is ready for traffic.

To follow and test these improvements follow the 'development' branch of the Docker repository at: <a href="https://gitlab.com/mayan-edms/mayan-edms-docker/tree/development" target="_blank" rel="nofollow" onmousedown="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgitlab.com%2Fmayan-edms%2Fmayan-edms-docker%2Ftree%2Fdevelopment\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG3H-JgLK1oRIH0HuDQ9OppWQBZEA&#39;;return true;" onclick="this.href=&#39;https://www.google.com/url?q\x3dhttps%3A%2F%2Fgitlab.com%2Fmayan-edms%2Fmayan-edms-docker%2Ftree%2Fdevelopment\x26sa\x3dD\x26sntz\x3d1\x26usg\x3dAFQjCNG3H-JgLK1oRIH0HuDQ9OppWQBZEA&#39;;return true;">https://gitlab.com/mayan-edms/mayan-edms-docker/tree/development 

Please help test these changes to move them to the stable master branch as soon as possible.

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 1753] Re: Mayan EDMS Docker image updates

David Kornahrens
Roberto - Is there a method in your program to verify we are on the latest product?  And how do we check this via command line or some other interface?

Thx,
David

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 1755] Re: Mayan EDMS Docker image updates

rosarior
Administrator
Excellent, glad to hear it! Hope it fulfill your needs. 

A commit landed about a week ago to automatic the check for an update via the UI: https://gitlab.com/mayan-edms/mayan-edms/commit/579595fe7fde062e6b1143ea5115446853573fbd

For the moment you can also do:

    pip search mayan-edms

to check for the latest Python version published in PyPI, the Python Package Index.


On Friday, June 2, 2017 at 10:15:27 AM UTC-4, David Kornahrens wrote:
Roberto - Is there a method in your program to verify we are on the latest product?  And how do we check this via command line or some other interface?

Thx,
David

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

Screenshot from 2017-06-02 12-41-36.png (21K) Download Attachment
Screenshot from 2017-06-02 12-42-05.png (9K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 1755] Re: Mayan EDMS Docker image updates

David Kornahrens
Outstanding.  If the commit hasn't been packaged yet, me running the command will not do anything correct?

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 1757] Re: Mayan EDMS Docker image updates

David Kornahrens
In reply to this post by rosarior
One more question Roberto -- Because I used the Docker deploy method, do I still update the same way "pip search mayan-edms" ?

THank you,
David

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|

[Mayan EDMS: 1765] Re: Mayan EDMS Docker image updates

rosarior
Administrator
A commit means that the code passed the design/discussion phase, has been coded, is already submitted in the project's repository and will be included in the next release.

The Docker image is a separate project with it's own separate repository. The full documentation for the Docker install (including upgrade, backup and restore procedures) is here:
 https://gitlab.com/mayan-edms/mayan-edms-docker/blob/master/README.md

The Mayan EDMS Manual project seeks to merge all these documents into a single canonical manual and single point of reference (https://gitlab.com/mayan-edms/manual). Any help achieving this goal is appreciated.

On Friday, June 2, 2017 at 2:08:48 PM UTC-4, David Kornahrens wrote:
One more question Roberto -- Because I used the Docker deploy method, do I still update the same way "pip search mayan-edms" ?

THank you,
David

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.