Page Load Test

In this test I would like to test the initial page load. One way to test this is the time between request is sent and response is received. For Dash this will be an ok measure. But for Streamlit and Panel this is only part of the answer. Because they don't transfer everything in the request-response cycle. Streamlit only transfers a minimum amount of data to open a web socket connection. The rest is transfered through that. Panel is somewhere in between Streamlit and Dash. It transfers parts of the response initially. But opens a web socket to transfer the rest. So I figured I would do a first test of how long time it takes before the user actually sees a rendered page.

Request - Response Test

Dash - Close up

docker run --rm -it -p 80:80 -p 8089:8089 dataappperformance:latest -c 'invoke page-load.dash'

Panel - Fails big time

docker run --rm -it -p 80:80 -p 8089:8089 dataappperformance:latest -c 'invoke page-load.dash'

As you can see in the screenshot Panels performance is orders of magnitude lower. One minor cause is probably that it sends the most data across. But as you can also see the max response time is +1min. As you can see in the video below the server is not responding at all for more than a minute. As Streamlit and Panel are both running on the Tornado server I would expect them both to be pretty quick. I've filed a bug report with Panel here holoviz/panel#3105. I believe something in the implementation of the Bokeh/ Panel server must be wrong.

Streamlit - Most Performant

docker run --rm -it -p 80:80 -p 8089:8089 dataappperformance:latest -c 'invoke page-load.streamlit'

Discussion

If you are making apps for 1-30 users that are not refreshing often as I am the Panel issue is not that significant. But it will be expensive to scale to a lot of concurrent users currently.

Videos

The videos have been speeded up 4 times to be able to upload them.

Page Refresh Test

Here we just click the refresh button as fast as we "feel" its possible to do.

• Dash: CLEAR WINNER
• Panel: Seems relative fast and faster than Streamlit. But just as in previous test suddenly the server blocks making this the WORST PERFORMER.
• Streamlit seems to load consistently slow compared to the others. But its better to be consistent than to block for longer periods of time.

docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.dash'
docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.panel'
docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.streamlit'

Promo Gif

Drag Slider with Plot Updating Performance Test

I've created a basic application with a slider and a Plotly plot. I want to explore my data/ model interactively by dragging the slider fast back and forth.

As you can see below only Panel can deliver the performance needed.

You can find the source code here.

Test Results

Discussion

• Dash: As you can see Dash quickly gives up. I believe that the combination of Flask and http will never be able to provide really snappy updates. I've been told that an additional issue is that dash sends a lot of data back and forth on each update.
• Panel: I've always been impressed by the performance of Bokeh. Panel builds on Bokeh and is capable of achieving similar performance. My hypothesis is that Panel would be even more snappy with Bokeh/ Holoviews plots.
• Streamlit: I could not find a way to react to slider dragging. I believe the run script top to bottom architecture of Streamlit will never be able to provide really snappy updates. See this discussion Make st.slider and st.color_picker only rerun script on mouse up and Improve performance of Streamlit to update plots.

For example Dask Dashboard is developed in Bokeh and is very, very performant. I don't believe its possible to build a performant application like that in Streamlit or Dash. It would be in Panel though.

I would expect interactive applications with streaming to perform much better in Panel than Dash. Streamlit does not really support interactive applications with streaming.

Another noticeable difference is just how much code it takes to make a basic Dash application.

Source

https://user-images.githubusercontent.com/42288570/149649005-24f9f4d2-b84f-45d2-888a-c2b4b618c66b.mp4
https://user-images.githubusercontent.com/42288570/149649006-44307849-273b-4b35-b3f1-bf657f8b5a36.mp4
https://user-images.githubusercontent.com/42288570/149649007-eca75cf9-fdf6-4304-85ef-6eea26ccd3c0.mp4

Promo Gif

Promo Video

Commands

docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.panel'
docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.streamlit'
docker run --rm -it -p 80:80 dataappperformance:latest -c 'invoke slider-plot.dash'