Raspberry Pi and Redis
This summer we did something amazing: we took our enterprise appliance, a very powerful server and we tried to run same software, same analytic kernel and all the other things in something smaller and lot simpler, like this:
And we did it ! Being powered by OpenResty and Lua we were able to size our appliance easily and be up and running quickly. Low factor, low power consumption, very cheap, these ARM based devices are getting more and more attention and become more popular nowadays:
Yep. First of all, we wanted to experiment with ARM based boards using few number of hosts and few data messages per host as input. The immediate trouble was that we should fit in 512MB RAM with all our software and customer data. Then these boards don't run on latest Intel specs but something like 700 Mhz CPU as a system on chip with 100Mbps NIC and shared USB ports.
A central part of our analytic software is Redis, a very popular and powerful NoSQL in cache memory data structure server. It runs on majority of Linux distributions, *BSD and many other UNIXes. It has a nice and friendly community and you get lots of help around. For our analytic software Redis was a natural choice: several ways to structure data in memory, very fast, no threading and simple.
The RBPi model B+ used for this test:
As storage we have selected these two cards: SanDisk and Kingston 16GB , Class 10. We used default ext4 file system on both card, with default options.
After several weeks of testing and testing again, we found several issues with Redis on Raspberry PI when using Kingston cards:
Redis has something amazing called Latency doctor. All things explained here: Redis latency monitoring. Using that we were able to see that majority of all our timeouts were caused by a slow storage media, in fact the SD and MicroSD cards. And we start asking the doctor:
After all tuning we were still seeing things like these and lots of timeouts on our Lua clients. The usage of the appliance was kind of high for 5 messages per second.
We changed cards. This is SanDisk 16GB. This time we were able to run without no Lua timeouts nor any other errors in the error log. Still visible a high CPU utilization:
 |
| Light, mobile appliance |
What ? Analytics on a Raspberry PI ... !?
Yep. First of all, we wanted to experiment with ARM based boards using few number of hosts and few data messages per host as input. The immediate trouble was that we should fit in 512MB RAM with all our software and customer data. Then these boards don't run on latest Intel specs but something like 700 Mhz CPU as a system on chip with 100Mbps NIC and shared USB ports.
But still we were thrilled by this idea of having, few hosts and messages per appliance and be able to deploy such appliance quicker than a larger one for different setups. For example
- Enterprise Appliance: up to 5000 hosts, 5 messages per host, 25.000 messages
- PI Appliance: up to 5 hosts, 2 messages per host, 10 messages
Redis on Raspberry PI
A central part of our analytic software is Redis, a very popular and powerful NoSQL in cache memory data structure server. It runs on majority of Linux distributions, *BSD and many other UNIXes. It has a nice and friendly community and you get lots of help around. For our analytic software Redis was a natural choice: several ways to structure data in memory, very fast, no threading and simple.
The RBPi model B+ used for this test:
As storage we have selected these two cards: SanDisk and Kingston 16GB , Class 10. We used default ext4 file system on both card, with default options.
 |
| king-msdhc6-c10 |
[ 2.112778] mmc0: SDHCI controller on BCM2708_Arasan [platform] using platform's DMA
[ 2.123339] mmc0: BCM2708 SDHC host at 0x20300000 DMA 2 IRQ 77
[ 2.233281] Waiting for root device /dev/mmcblk0p2...
[ 2.284667] mmc0: read SD Status register (SSR) after 3 attempts
[ 2.305649] mmc0: new high speed SDHC card at address 0007
[ 2.319414] mmcblk0: mmc0:0007 SD16G 14.9 GiB
[ 2.327567] mmcblk0: p1 p2 |
| SanDisk 16GB Class 10 |
[ 2.131742] mmc0: SDHCI controller on BCM2708_Arasan [platform] using platform's DMA
[ 2.142791] mmc0: BCM2708 SDHC host at 0x20300000 DMA 2 IRQ 77
[ 2.262561] Waiting for root device /dev/mmcblk0p2...
[ 2.293773] mmc0: read SD Status register (SSR) after 2 attempts
[ 2.316876] mmc0: new high speed SDHC card at address aaaa
[ 2.332197] mmcblk0: mmc0:aaaa SU16G 14.8 GiB
[ 2.345618] mmcblk0: p1 p2
 |
| Transcend 32GB Class 10 MLC |
[ 2.167920] mmc0: SDHCI controller on BCM2708_Arasan [platform] using platform's DMA
[ 2.180580] mmc0: BCM2708 SDHC host at 0x20300000 DMA 2 IRQ 77
[ 2.358084] mmc0: read SD Status register (SSR) after 2 attempts
[ 2.381255] Waiting for root device /dev/mmcblk0p2...
[ 2.392431] mmc0: new high speed SDHC card at address 59b4
[ 2.418636] mmcblk0: mmc0:59b4 USDU1 29.7 GiB
[ 2.430805] mmcblk0: p1 p2
After several weeks of testing and testing again, we found several issues with Redis on Raspberry PI when using Kingston cards:
- slow storage
- lots of Lua client errors
- high CPU Utilization
Types of Errors
2014/09/11 11:53:57 [error] 2590#0: *43736 lua tcp socket read timed out, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx" 2014/09/11 11:53:57 [error] 2590#0: *43736 attempt to send data on a closed socket: u:B6925060, c:00000000, ft:0 eof:0, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx"
[3305] 18 Oct 12:52:31.024 * Asynchronous AOF fsync is taking too long (disk is busy?). Writing the AOF buffer without waiting for fsync to complete, this may slow down Redis.
2014/10/26 21:17:58 [error] 2712#0: *14598 lua tcp socket read timed out, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx" 2014/10/26 21:17:58 [error] 2712#0: *14598 [lua] messages.lua:154: deposit_stats_data(): Error setting expiration for key: stats:3f87c584-2312-42e5-67e4-93005181870d:eth0:cpd-linux-nicrec:rxpkt:2338:10800:MIN | expire at: 1415232000 | err: timeout, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx" 2014/10/26 21:17:58 [error] 2712#0: *14598 attempt to send data on a closed socket: u:B5FA41B8, c:00000000, ft:0 eof:0, client: 127.0.0.1, server: , request: "POST /api/private/send_data HTTP/1.0", host: "xxx.xxx.xxx.xxx"
Latency Doctor
Redis has something amazing called Latency doctor. All things explained here: Redis latency monitoring. Using that we were able to see that majority of all our timeouts were caused by a slow storage media, in fact the SD and MicroSD cards. And we start asking the doctor:
1. aof-write: 160 latency spikes (average 1312ms, mean deviation 726ms, period 19.63 sec). Worst all time event 5686ms.2. aof-write-alone: 160 latency spikes (average 1355ms, mean deviation 749ms, period 20.49 sec). Worst all time event 5686ms.3. aof-write-active-child: 11 latency spikes (average 1088ms, mean deviation 656ms, period 614.18 sec). Worst all time event 4160ms.
After all tuning we were still seeing things like these and lots of timeouts on our Lua clients. The usage of the appliance was kind of high for 5 messages per second.
 |
| Kingston 16GB, 3-5 messages per second |
We changed cards. This is SanDisk 16GB. This time we were able to run without no Lua timeouts nor any other errors in the error log. Still visible a high CPU utilization:
 |
| SanDisk 16GB, 3-5 messages per second |
Final configuration
The final settings OS and Redis for Raspberry Pi B+ model with SanDisk 16 GB MicroSD:
Running on a RBPi board is cool since we can look even deeper how things could be improved and tuned for better performance and reduce utilization. We learned:
| Item | Status | Description |
|---|---|---|
| /etc/sysctl.conf |
vm.swappiness=1
vm.overcommit_memory=1
vm.min_free_kbytes=16384
|
Virtual Memory Tuning
|
| /etc/fstab |
noatime
| /dev/mmcblk0p2 root |
| redis.conf |
port 0
unixsocket kernel.sock
unixsocketperm 755
appendfsync no
no-appendfsync-on-rewrite yes
|
Redis configurations
|
Conclusions
Running on a RBPi board is cool since we can look even deeper how things could be improved and tuned for better performance and reduce utilization. We learned:
- Running Redis on MicroSD, SD cards is a bit complicated
- You need to measure what is slow, welcome Latency Doctor
- To measure where we waste time: Lua, Redis, both ?
- To profile all our Lua scripts
your modification noatime - could you be a little more explicit about that one? I'm not a Linux whizz - did the others ok.
ReplyDeletenoatime option on the file system mount, this means the OS doesn't record an access time every time a file is read. Here is my mountpoint from fstab on the pi. It would be interesting to see the difference on a pi2 with quad core cpu and 1G ram.
Delete/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
Ron.. a little help for the slower amongst us... in /etc/fstab on my Raspberry Pi I have this
Deleteproc /proc proc defaults 0 0
#/dev/mmcblk0p1 /boot vfat defaults 0 2
#/dev/mmcblk0p2 / ext4 defaults,noatime 0 1
So actually only one line is doing anything - just what mod should I make?
And on my Orange Pi...
# OrangePI fstab
/dev/mmcblk0p2 / ext4 errors=remount-ro,noatime,nodiratime 0 1
/dev/mmcblk0p1 /media/boot vfat defaults 0 0
tmpfs /tmp tmpfs nodev,nosuid,mode=1777 0 0
Should I be modifying either?
We feel we can conquer almost any feat in this industry. In keeping with this mission, we offer a wide variety of services.
ReplyDeletehttp://www.cosmopolitanmechanical.com
http://www.cosmopolitanmechanical.com/heating-and-cooling-aurora.html