[BUG] imx6qdl: degraded eMMC write performance

[BUG] imx6qdl: degraded eMMC write performance

[Hubert Feurstein]

Hi,

I've updated barebox for our custom platform from v2015.06.0 to
v2019.12.0. With the new version I have noticed a much worse write
performance onto the onboard eMMC.

With v2015.06.0 the indicated progress of the copy command is very
smooth. Calling "cp -v /dev/zero /dev/mmc3.root" takes about 80
seconds for 256MB. But with v2019.12.0 the progress is very bumpy and
the copy takes about 280 seconds.

I've tracked this down to this commit which destroys the performance:
"block: Adjust cache sizes" (b6fef20c1215c6ef0004f6af4a9c4b77af51dc43)

Regards
Hubert

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: [BUG] imx6qdl: degraded eMMC write performance

[Sascha Hauer]

Hi Hubert,

On Tue, Dec 10, 2019 at 03:44:52PM +0100, Hubert Feurstein wrote:
> Hi,
> 
> I've updated barebox for our custom platform from v2015.06.0 to
> v2019.12.0. With the new version I have noticed a much worse write
> performance onto the onboard eMMC.
> 
> With v2015.06.0 the indicated progress of the copy command is very
> smooth. Calling "cp -v /dev/zero /dev/mmc3.root" takes about 80
> seconds for 256MB. But with v2019.12.0 the progress is very bumpy and
> the copy takes about 280 seconds.
> 
> I've tracked this down to this commit which destroys the performance:
> "block: Adjust cache sizes" (b6fef20c1215c6ef0004f6af4a9c4b77af51dc43)

Well the commit message says it:

| commit b6fef20c1215c6ef0004f6af4a9c4b77af51dc43
| Author: Sascha Hauer <s.hauer@pengutronix.de>
| Date:   Thu Mar 29 13:49:45 2018 +0200
| 
|     block: Adjust cache sizes
|     
|     Use four times more cache entries and divide the memory for each entry
|     by four. This lowers the linear read throughput somewhat but increases
|     the access speed for filesystems.

The barebox block layer is quite simple. It always handles data in chunks of
a hardcoded size. For a high throughput these should be large as this increases
the buffer size we can write in the MMC controller in one go. Increasing the
size is bad for filesystems though, as for each small block a fs wants
to read at minimum a whole chunk must be read, or for write,
read-modified-written.

I remember I once tried to make the caching more intelligent. It seems I didn't
even get far enough to store a branch in my working copy :(

I just saw that in block_op_write() we must first read any partially
written chunk, then modify it and write it back. Only full chunks can be
written without reading them first. This means we should gain
performance when we enter block_op_write() with only full chunks. We
could adjust RW_BUF_SIZE (used by copy_file as buffer size) to a full
chunk size (16KiB). Does this give you back some lost performance?

Sascha

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: [BUG] imx6qdl: degraded eMMC write performance

[Hubert Feurstein]

Hi Sascha,

Am Di., 10. Dez. 2019 um 16:10 Uhr schrieb Sascha Hauer
<s.hauer@pengutronix.de>:
[...]
> Well the commit message says it:
> | commit b6fef20c1215c6ef0004f6af4a9c4b77af51dc43
> | Author: Sascha Hauer <s.hauer@pengutronix.de>
> | Date:   Thu Mar 29 13:49:45 2018 +0200
> |
> |     block: Adjust cache sizes
> |
> |     Use four times more cache entries and divide the memory for each entry
> |     by four. This lowers the linear read throughput somewhat but increases
> |     the access speed for filesystems.
Yes, I know. I've read the commit message. But this patch even costs me ~200ms
in boottime (loading kernel and dts from ext4 root partition).

[...]
> We could adjust RW_BUF_SIZE (used by copy_file as buffer size) to a full
> chunk size (16KiB). Does this give you back some lost performance?
No, changing RW_BUF_SIZE did not help.

Regards
Hubert

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: [BUG] imx6qdl: degraded eMMC write performance

[Sascha Hauer]

On Tue, Dec 10, 2019 at 04:43:54PM +0100, Hubert Feurstein wrote:
> Hi Sascha,
> 
> Am Di., 10. Dez. 2019 um 16:10 Uhr schrieb Sascha Hauer
> <s.hauer@pengutronix.de>:
> [...]
> > Well the commit message says it:
> > | commit b6fef20c1215c6ef0004f6af4a9c4b77af51dc43
> > | Author: Sascha Hauer <s.hauer@pengutronix.de>
> > | Date:   Thu Mar 29 13:49:45 2018 +0200
> > |
> > |     block: Adjust cache sizes
> > |
> > |     Use four times more cache entries and divide the memory for each entry
> > |     by four. This lowers the linear read throughput somewhat but increases
> > |     the access speed for filesystems.
> Yes, I know. I've read the commit message. But this patch even costs me ~200ms
> in boottime (loading kernel and dts from ext4 root partition).
> 
> [...]
> > We could adjust RW_BUF_SIZE (used by copy_file as buffer size) to a full
> > chunk size (16KiB). Does this give you back some lost performance?
> No, changing RW_BUF_SIZE did not help.

And I also see why :-/

block_op_write() works around block sizes (512bytes), not around chunk
sizes. This means we always read before we write. This could probably be
optimized somehow, but this would only speed up the write case you
described in your initial mail. It seems what you are more interested in
is the read performance.

We could make the number of chunks and the chunk size configurable
during runtime. This is less than ideal, but at least we could get a
better performance with manageable effort.

Sascha

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: [BUG] imx6qdl: degraded eMMC write performance

[Hubert Feurstein]

Am Di., 10. Dez. 2019 um 17:18 Uhr schrieb Sascha Hauer
<s.hauer@pengutronix.de>:
[...]
> > > We could adjust RW_BUF_SIZE (used by copy_file as buffer size) to a full
> > > chunk size (16KiB). Does this give you back some lost performance?
> > No, changing RW_BUF_SIZE did not help.
>
> And I also see why :-/
>
> block_op_write() works around block sizes (512bytes), not around chunk
> sizes. This means we always read before we write. This could probably be
> optimized somehow, but this would only speed up the write case you
> described in your initial mail. It seems what you are more interested in
> is the read performance.
Well, I'm interested in both. Of course the read-performance is important
for booting. But the write-performance is important for production. It simply
matters if you have to wait for another 200 seconds for a device to
get programmed.

Regards
Hubert

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: [BUG] imx6qdl: degraded eMMC write performance

[Sascha Hauer]

On Tue, Dec 10, 2019 at 03:44:52PM +0100, Hubert Feurstein wrote:
> Hi,
> 
> I've updated barebox for our custom platform from v2015.06.0 to
> v2019.12.0. With the new version I have noticed a much worse write
> performance onto the onboard eMMC.
> 
> With v2015.06.0 the indicated progress of the copy command is very
> smooth. Calling "cp -v /dev/zero /dev/mmc3.root" takes about 80
> seconds for 256MB. But with v2019.12.0 the progress is very bumpy and
> the copy takes about 280 seconds.
> 
> I've tracked this down to this commit which destroys the performance:
> "block: Adjust cache sizes" (b6fef20c1215c6ef0004f6af4a9c4b77af51dc43)

We could just revert this patch. I can't find any workload that gets
faster with b6fef20c1215. It's rather the other way round.

Sascha

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox