Quantcast

[Mayan EDMS: 1543] Upgrade to 2.2b2 and thumbnails

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Mayan EDMS: 1543] Upgrade to 2.2b2 and thumbnails

Sebastian Tänzer
I managed to perform an upgrade to 2.2b2. 

Is there anything I have to do to use the new thumbnail caching or isn't that implemented yet?
Thumbnails are still generated with every page call.

Best
Sebastian

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Mayan EDMS: 1551] Re: Upgrade to 2.2b2 and thumbnails

rosarior
Administrator
Nothing is needed to use the new thumbnail caching. Enable the DEBUG mode by adding this to you settings/local.py file

DEBUG=True

Go to a view that shows document images and take a look at the debug messages in the console. You should entries like this:

documents.models <8408> [DEBUG] "generate_image() transformations cache file "page-cache-bea37f94-57c4-4789-a5b3-f501dc678fcd-24-55-118690ab21ab21be" found"

The long filename is the page UUID followed by the transformations if serialized format (zoom, resize, rotation). This means that the page images are being cached in their
final resized dimensions. The system still uses a background process to check is a document thumbnail is already available and this causes a slight overhead (not realtime).
If runing using the `runserver` command the task broker is disabled and all background tasks run in the same process as the UI, this will cause the appearance that the
thumbnails are being generated from scratch. A screen refresh should show that they are not being regenerated and will display faster than the first time they displayed,
this should indicate that they are being cached. With this in mind give it another try to see a difference. You can also delete all cached images using the "Clear document image cache"
button in the Tools menu or by manually deleting all file in the mayan/media/document_cache folder and doing a view refresh. The time to display the thumbnails should be very 
different in the first and subsequent refreshes.




On Friday, March 17, 2017 at 8:59:04 AM UTC-4, Sebastian Tänzer wrote:
I managed to perform an upgrade to 2.2b2. 

Is there anything I have to do to use the new thumbnail caching or isn't that implemented yet?
Thumbnails are still generated with every page call.

Best
Sebastian

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Mayan EDMS: 1601] Re: Upgrade to 2.2b2 and thumbnails

Sebastian Tänzer
Hello Roberto,

sorry for the late response and thanks for the info. I upgraded my virtualenv/pip setup to the latest git development branch and did some tests.

When running Mayan in the foreground using runserver and Debug=True it shows this:

documents.models <24377> [DEBUG] "generate_image() transformations cache filename: page-cache-bd7290ba-4f11-4b1f-ad1d-5a69fdd9a619-77-163-1d20f688585d3c09"
documents.models <24377> [DEBUG] "generate_image() transformations cache file "page-cache-bd7290ba-4f11-4b1f-ad1d-5a69fdd9a619-77-163-1d20f688585d3c09" found"

and thumbnails are beeing generated really fast and then loaded from cache the second time.

When running Mayan using supervisor/nginx this is not the case. All thumbs are regenerated everytime.

Any ideas why that is?

Best
Sebastian

Am Montag, 20. März 2017 06:43:56 UTC+1 schrieb Roberto Rosario:
Nothing is needed to use the new thumbnail caching. Enable the DEBUG mode by adding this to you settings/local.py file

DEBUG=True

Go to a view that shows document images and take a look at the debug messages in the console. You should entries like this:

documents.models <8408> [DEBUG] "generate_image() transformations cache file "page-cache-bea37f94-57c4-4789-a5b3-f501dc678fcd-24-55-118690ab21ab21be" found"

The long filename is the page UUID followed by the transformations if serialized format (zoom, resize, rotation). This means that the page images are being cached in their
final resized dimensions. The system still uses a background process to check is a document thumbnail is already available and this causes a slight overhead (not realtime).
If runing using the `runserver` command the task broker is disabled and all background tasks run in the same process as the UI, this will cause the appearance that the
thumbnails are being generated from scratch. A screen refresh should show that they are not being regenerated and will display faster than the first time they displayed,
this should indicate that they are being cached. With this in mind give it another try to see a difference. You can also delete all cached images using the "Clear document image cache"
button in the Tools menu or by manually deleting all file in the mayan/media/document_cache folder and doing a view refresh. The time to display the thumbnails should be very 
different in the first and subsequent refreshes.




On Friday, March 17, 2017 at 8:59:04 AM UTC-4, Sebastian Tänzer wrote:
I managed to perform an upgrade to 2.2b2. 

Is there anything I have to do to use the new thumbnail caching or isn't that implemented yet?
Thumbnails are still generated with every page call.

Best
Sebastian

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Mayan EDMS: 1611] Re: Upgrade to 2.2b2 and thumbnails

rosarior
Administrator
Hi Sebastian,

I'm drawing a blank because it is the same process either way. For example when running using supervisor and using a background worker you can check the output of the worker servicing the converter queue and the same output about `generate_image()` should be displayed. You might need to edit the supervisor line of the worker to add -l DEBUG instead of -l INFO to increase the verbosity. This means that the same logic is being applied to the generation of thumbnails. 

The only thing that comes to mind is that the speed of the message queue is creating an overhead that makes it looks like the thumbnails are being generated.

The process is as follows: 

Frontend will always ask the document model to see if a cached representation (image + transformation[resize, rotation]) exists. The thumbnail generation request travels from the frontend to the celery then to the broker (Redis or Rabbit), then the worker's celery pickups the message from the broker checks to see if a thumbnail is available, generates one if not, sends back the filename of the generated of cached thumbnail back to the worker's celery to the result storage (Redis), the frontend's celery gets the message, decodes it and passes it to the waiting frontend display code.

In DEBUG mode this happens very fast because it is the same process running both sides of the code (frontend and model+converter, minus the broker and result storage).

My guess is that there is a bottleneck in the deployment configuration, broker getting choked, not enough workers, worker not running in multi threaded or multi process mode (servicing messages sequentially), uWSGI process servicinong HTTP requests sequentially and thus showing thumbnails synchronously.

Are the thumbnails being displayed in sequence with one displaying only after the previous one is displayed? Or are they being displayed in random order only slow?

On Tuesday, April 18, 2017 at 8:38:57 AM UTC-4, Sebastian Tänzer wrote:
Hello Roberto,

sorry for the late response and thanks for the info. I upgraded my virtualenv/pip setup to the latest git development branch and did some tests.

When running Mayan in the foreground using runserver and Debug=True it shows this:

documents.models <24377> [DEBUG] "generate_image() transformations cache filename: page-cache-bd7290ba-4f11-4b1f-ad1d-5a69fdd9a619-77-163-1d20f688585d3c09"
documents.models <24377> [DEBUG] "generate_image() transformations cache file "page-cache-bd7290ba-4f11-4b1f-ad1d-5a69fdd9a619-77-163-1d20f688585d3c09" found"

and thumbnails are beeing generated really fast and then loaded from cache the second time.

When running Mayan using supervisor/nginx this is not the case. All thumbs are regenerated everytime.

Any ideas why that is?

Best
Sebastian

Am Montag, 20. März 2017 06:43:56 UTC+1 schrieb Roberto Rosario:
Nothing is needed to use the new thumbnail caching. Enable the DEBUG mode by adding this to you settings/local.py file

DEBUG=True

Go to a view that shows document images and take a look at the debug messages in the console. You should entries like this:

documents.models <8408> [DEBUG] "generate_image() transformations cache file "page-cache-bea37f94-57c4-4789-a5b3-f501dc678fcd-24-55-118690ab21ab21be" found"

The long filename is the page UUID followed by the transformations if serialized format (zoom, resize, rotation). This means that the page images are being cached in their
final resized dimensions. The system still uses a background process to check is a document thumbnail is already available and this causes a slight overhead (not realtime).
If runing using the `runserver` command the task broker is disabled and all background tasks run in the same process as the UI, this will cause the appearance that the
thumbnails are being generated from scratch. A screen refresh should show that they are not being regenerated and will display faster than the first time they displayed,
this should indicate that they are being cached. With this in mind give it another try to see a difference. You can also delete all cached images using the "Clear document image cache"
button in the Tools menu or by manually deleting all file in the mayan/media/document_cache folder and doing a view refresh. The time to display the thumbnails should be very 
different in the first and subsequent refreshes.




On Friday, March 17, 2017 at 8:59:04 AM UTC-4, Sebastian Tänzer wrote:
I managed to perform an upgrade to 2.2b2. 

Is there anything I have to do to use the new thumbnail caching or isn't that implemented yet?
Thumbnails are still generated with every page call.

Best
Sebastian

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Mayan EDMS: 1612] Re: Upgrade to 2.2b2 and thumbnails

Sebastian Tänzer
Hey Roberto,

thanks for the detailed response! 

The thumbnails are beeing displayed in a random order and very slow.
I have attached the logs from the worker when running with supervisor.

The performance difference is huge compared to the runserver standalone.

I really have no idea what could lead to this. I try to record a screen video.

Best
Sebastian

Am Mittwoch, 19. April 2017 18:00:44 UTC+2 schrieb Roberto Rosario:
Hi Sebastian,

I'm drawing a blank because it is the same process either way. For example when running using supervisor and using a background worker you can check the output of the worker servicing the converter queue and the same output about `generate_image()` should be displayed. You might need to edit the supervisor line of the worker to add -l DEBUG instead of -l INFO to increase the verbosity. This means that the same logic is being applied to the generation of thumbnails. 

The only thing that comes to mind is that the speed of the message queue is creating an overhead that makes it looks like the thumbnails are being generated.

The process is as follows: 

Frontend will always ask the document model to see if a cached representation (image + transformation[resize, rotation]) exists. The thumbnail generation request travels from the frontend to the celery then to the broker (Redis or Rabbit), then the worker's celery pickups the message from the broker checks to see if a thumbnail is available, generates one if not, sends back the filename of the generated of cached thumbnail back to the worker's celery to the result storage (Redis), the frontend's celery gets the message, decodes it and passes it to the waiting frontend display code.

In DEBUG mode this happens very fast because it is the same process running both sides of the code (frontend and model+converter, minus the broker and result storage).

My guess is that there is a bottleneck in the deployment configuration, broker getting choked, not enough workers, worker not running in multi threaded or multi process mode (servicing messages sequentially), uWSGI process servicinong HTTP requests sequentially and thus showing thumbnails synchronously.

Are the thumbnails being displayed in sequence with one displaying only after the previous one is displayed? Or are they being displayed in random order only slow?

On Tuesday, April 18, 2017 at 8:38:57 AM UTC-4, Sebastian Tänzer wrote:
Hello Roberto,

sorry for the late response and thanks for the info. I upgraded my virtualenv/pip setup to the latest git development branch and did some tests.

When running Mayan in the foreground using runserver and Debug=True it shows this:

documents.models <24377> [DEBUG] "generate_image() transformations cache filename: page-cache-bd7290ba-4f11-4b1f-ad1d-5a69fdd9a619-77-163-1d20f688585d3c09"
documents.models <24377> [DEBUG] "generate_image() transformations cache file "page-cache-bd7290ba-4f11-4b1f-ad1d-5a69fdd9a619-77-163-1d20f688585d3c09" found"

and thumbnails are beeing generated really fast and then loaded from cache the second time.

When running Mayan using supervisor/nginx this is not the case. All thumbs are regenerated everytime.

Any ideas why that is?

Best
Sebastian

Am Montag, 20. März 2017 06:43:56 UTC+1 schrieb Roberto Rosario:
Nothing is needed to use the new thumbnail caching. Enable the DEBUG mode by adding this to you settings/local.py file

DEBUG=True

Go to a view that shows document images and take a look at the debug messages in the console. You should entries like this:

documents.models <8408> [DEBUG] "generate_image() transformations cache file "page-cache-bea37f94-57c4-4789-a5b3-f501dc678fcd-24-55-118690ab21ab21be" found"

The long filename is the page UUID followed by the transformations if serialized format (zoom, resize, rotation). This means that the page images are being cached in their
final resized dimensions. The system still uses a background process to check is a document thumbnail is already available and this causes a slight overhead (not realtime).
If runing using the `runserver` command the task broker is disabled and all background tasks run in the same process as the UI, this will cause the appearance that the
thumbnails are being generated from scratch. A screen refresh should show that they are not being regenerated and will display faster than the first time they displayed,
this should indicate that they are being cached. With this in mind give it another try to see a difference. You can also delete all cached images using the "Clear document image cache"
button in the Tools menu or by manually deleting all file in the mayan/media/document_cache folder and doing a view refresh. The time to display the thumbnails should be very 
different in the first and subsequent refreshes.




On Friday, March 17, 2017 at 8:59:04 AM UTC-4, Sebastian Tänzer wrote:
I managed to perform an upgrade to 2.2b2. 

Is there anything I have to do to use the new thumbnail caching or isn't that implemented yet?
Thumbnails are still generated with every page call.

Best
Sebastian

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.

mayan-celery.conf (876 bytes) Download Attachment
worker-stderr.log (121K) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[Mayan EDMS: 1613] Re: Upgrade to 2.2b2 and thumbnails

Sebastian Tänzer
In reply to this post by rosarior
I did 2 screencasts to show the difference:

1) slow (supervisor/nginx/wsgi): https://www.dropbox.com/s/4cfdvs7wo7buiit/mayan-slow-supervisor.mp4?dl=0
2) fast (runserver): https://www.dropbox.com/s/pdpildht80ihw2v/mayan-runserver.mp4?dl=0

Am Mittwoch, 19. April 2017 18:00:44 UTC+2 schrieb Roberto Rosario:
Hi Sebastian,

I'm drawing a blank because it is the same process either way. For example when running using supervisor and using a background worker you can check the output of the worker servicing the converter queue and the same output about `generate_image()` should be displayed. You might need to edit the supervisor line of the worker to add -l DEBUG instead of -l INFO to increase the verbosity. This means that the same logic is being applied to the generation of thumbnails. 

The only thing that comes to mind is that the speed of the message queue is creating an overhead that makes it looks like the thumbnails are being generated.

The process is as follows: 

Frontend will always ask the document model to see if a cached representation (image + transformation[resize, rotation]) exists. The thumbnail generation request travels from the frontend to the celery then to the broker (Redis or Rabbit), then the worker's celery pickups the message from the broker checks to see if a thumbnail is available, generates one if not, sends back the filename of the generated of cached thumbnail back to the worker's celery to the result storage (Redis), the frontend's celery gets the message, decodes it and passes it to the waiting frontend display code.

In DEBUG mode this happens very fast because it is the same process running both sides of the code (frontend and model+converter, minus the broker and result storage).

My guess is that there is a bottleneck in the deployment configuration, broker getting choked, not enough workers, worker not running in multi threaded or multi process mode (servicing messages sequentially), uWSGI process servicinong HTTP requests sequentially and thus showing thumbnails synchronously.

Are the thumbnails being displayed in sequence with one displaying only after the previous one is displayed? Or are they being displayed in random order only slow?

On Tuesday, April 18, 2017 at 8:38:57 AM UTC-4, Sebastian Tänzer wrote:
Hello Roberto,

sorry for the late response and thanks for the info. I upgraded my virtualenv/pip setup to the latest git development branch and did some tests.

When running Mayan in the foreground using runserver and Debug=True it shows this:

documents.models <24377> [DEBUG] "generate_image() transformations cache filename: page-cache-bd7290ba-4f11-4b1f-ad1d-5a69fdd9a619-77-163-1d20f688585d3c09"
documents.models <24377> [DEBUG] "generate_image() transformations cache file "page-cache-bd7290ba-4f11-4b1f-ad1d-5a69fdd9a619-77-163-1d20f688585d3c09" found"

and thumbnails are beeing generated really fast and then loaded from cache the second time.

When running Mayan using supervisor/nginx this is not the case. All thumbs are regenerated everytime.

Any ideas why that is?

Best
Sebastian

Am Montag, 20. März 2017 06:43:56 UTC+1 schrieb Roberto Rosario:
Nothing is needed to use the new thumbnail caching. Enable the DEBUG mode by adding this to you settings/local.py file

DEBUG=True

Go to a view that shows document images and take a look at the debug messages in the console. You should entries like this:

documents.models <8408> [DEBUG] "generate_image() transformations cache file "page-cache-bea37f94-57c4-4789-a5b3-f501dc678fcd-24-55-118690ab21ab21be" found"

The long filename is the page UUID followed by the transformations if serialized format (zoom, resize, rotation). This means that the page images are being cached in their
final resized dimensions. The system still uses a background process to check is a document thumbnail is already available and this causes a slight overhead (not realtime).
If runing using the `runserver` command the task broker is disabled and all background tasks run in the same process as the UI, this will cause the appearance that the
thumbnails are being generated from scratch. A screen refresh should show that they are not being regenerated and will display faster than the first time they displayed,
this should indicate that they are being cached. With this in mind give it another try to see a difference. You can also delete all cached images using the "Clear document image cache"
button in the Tools menu or by manually deleting all file in the mayan/media/document_cache folder and doing a view refresh. The time to display the thumbnails should be very 
different in the first and subsequent refreshes.




On Friday, March 17, 2017 at 8:59:04 AM UTC-4, Sebastian Tänzer wrote:
I managed to perform an upgrade to 2.2b2. 

Is there anything I have to do to use the new thumbnail caching or isn't that implemented yet?
Thumbnails are still generated with every page call.

Best
Sebastian

--

---
You received this message because you are subscribed to the Google Groups "Mayan EDMS" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [hidden email].
For more options, visit https://groups.google.com/d/optout.
Loading...