Продолжаем разбираться с прожорливым кэшем Chromium -> disk_cache.h -> WebKit -> MemoryCache class

Как работает кэш броузера Chrome? Эта навязчивая идея заставляет меня раз в два месяца копаться в настройках браузера. Вчера вечером я так увлекся, что сегодня потратил еще полдня на конспектирование документации chromium.org/developers, trac.webkit.org/wiki/MemoryCache. Здесь более десятка ссылок, схема файлов кэша, сэема класса MemoryCache... Начал с перевода с китайского, который слегка сбил с толку... но надеюсь что найду все в "сырцах" описанияй классов ... Defines the public interface of the disk cache.

wiki:MemoryCache - MemoryCache module is a part of the bigger mechanism responsible for page loading called loader. You can learn more about loader from this article (​https://www.webkit.org/blog/427/webkit-page-cache-i-the-basics/). This article focuses only on Memory Cache aspects and its internal components without interaction with loader.
WebKit Page Cache I – The Basics
WebKit Page Cache II – The unload Event
The cache mechanism of Chromium/WebKit
Welcome to the ​WebKit Wiki!
Disk Cache
Contents of /trunk/src/net/disk_cache/disk_cache.h - Defines the public interface of the disk cache
HTML5 link prefetching severely broken in Chrome
Showing 1-10 of 18 results for prefetch
Chromium Contents of /trunk/src/net/base/cache_type.h
Chrome Prerendering
Link prefetching FAQ
MemoryCache module
[Required information in an Excessive Memory Use Bug(https://trac.webkit.org/wiki/Memory%20Use) - здесь есть рекомендации по диагностике Launch Safari from the cygwin terminal, with standard output enabled: Tools/Scripts/run-safari --debug 2>&1 | cat

WebKit page cache(Page Cache)¶

Здесь косноязычный перевод с китайского на английский. Есть три вида памяти: WebKit page cache(Page Cache) The memory cache WebKit(Memory Cache) Disk cache Chromoium(Disk Cacke) И, по словам автора, страница проходит путь к удалению последовательно перегружаясь в дисковую память..., эта модель слишком упрощена..., на самом деле, как мы увидим ниже, там буферов значительно больше.

Disk Cache from chromium.org¶

Здесь в основном описана структура файлов, потому я пропустил описание отсюда и сначала прочитал то, что в конце поста. Но это описание из первых рук, потому его надо внимательно изучить.

In [11]:

Image('https://www.chromium.org/developers/design-documents/network-stack/disk-cache/files4.PNG')

Out[11]:

This diagram shows a disk cache with 7 files on disk: the index file, 5 block-files and one separate file.
data_1 and data_4 are chained together so they store blocks of the same size (256 bytes), while data_2 stores blocks of 1KB and data_3 stores blocks of 4 KB.
The depicted entry has its key stored outside the EntryStore structure, and given that it uses two blocks, it must be between one and two kilobytes.
This entry also has two data streams, one for the HTTP headers (less than 256 bytes) and another one for the actual payload (more than 16 KB so it lives on a dedicated file). All blue arrows indicate that a cache address is used to locate another piece of data.

Implementation Notes Chromium has two different implementations of the cache interfaces: while the main one is used to store info on a given disk, there is also a very simple implementation that doesn’t use a hard drive at all, and stores everything in memory.
The in-memory implementation is used for the Incognito mode so that even if the application crashes it will be quite difficult to extract any information that was accessed while browsing in that mode.

There are a few different types of caches (see net/base/cache_type.h), mainly defined by their intended use:
there is a media specific cache,
the general purpose disk cache, and another one that serves as
the back end storage for AppCache, in addition to
the in-memory type already mentioned.
All types of caches behave in a similar way, with the exception that the eviction algorithm used by the general purpose cache is not the same LRU used by the others.
The regular cache implementation is located on

disk_cache/backend_impl.cc and disk_cache/entry_impl.cc.

Most of the files on that folder are actually related to the main implementation, except for a few that implement the
in-memory cache:
disk_cache/mem_backend_impl.cc and disk_cache/entry_impl.cc.

In [ ]:

MemoryCache module from Safari trac.webkit.org¶

This cache holds subresources used by Web pages: images, scripts, stylesheets, etc. The cache keeps a flexible but bounded window of dead resources that grows/shrinks depending on the live resource load. Here's an example of cache growth over time, with a min dead resource capacity of 25% and a max dead resource capacity of 50%:

In [ ]:

            |-----|                   Dead: -
            |----------|              Live: +
          --|----------|              Cache boundary: | (objects outside this mark have been evicted)
          --|----------++++++++++|
     -------|-----+++++++++++++++|
     -------|-----+++++++++++++++|+++++

The behavior of the cache changes in the following way if shouldMakeResourcePurgeableOnEviction returns true.

1. Dead resources in the cache are kept in non-purgeable memory.
   
2. When we prune dead resources, instead of freeing them, we mark their memory as purgeable and keep the resources until the kernel reclaims the purgeable memory.

Eviction - выселение, purge - чистка, prune - подрезать, удалять; reclaim - затребовать¶

By leaving the in-cache dead resources in dirty resident memory, we decrease the likelihood of the kernel claiming that memory and forcing us to refetch the resource (for example when a user presses back).

And by having an unbounded number of resource objects using purgeable memory, we can use as much memory as it is available on the machine.
The trade-off here is that the CachedResource object (and its member variables) are allocated in non-purgeable TC-malloc'd memory so we would see slightly more memory use due to this.

In [ ]:

What exactly means that some memory is purgeable/non-purgeable? Well, some operating systems allows to mark memory as purgeable.
It means that since it is marked the content may disappear (the system expropriatse it) at any point of time (until it gets unmarked) without informing the application.

So the application may know if the memory is purged after it asks the system for the resource (this may be implemented differently on separate systems). This leads to serious consequences in object life cycle management:

voc: consequence - следствие, disallow - отвергать¶

this kind of memory disallows to store objects that depends on its (or related by inheritance/consistency) destructor

ownership/references to the memory should never be passed to the modules that wouldn't treat this memory as a “special object”.

known and widely used techniques of managing object lifetime (like smart pointers) are not relevant to objects stored in purgable memory

Now, after you know about consequences of storing objects in purgeable memory let's take a look on how the cache module is constructed¶

In [5]:

from IPython.display import Image
Image(url='https://trac.webkit.org/attachment/wiki/MemoryCache/MemoryCache.png')

Out[5]:

In [9]:

Image('/home/kiss/Desktop/MemoryCache.png', retina=True )

Out[9]:

MemoryCache class¶

This class is designed to store and manage objects (web resources) lifecycle. Stored resources may be live or dead. A resource is dead when there are no references from web pages. And vice versa resource is live when there is at least one reference to it from web page.

MemoryCache client may have influence on RAM usage using three values:

**min dead bytes** - minimum size of dead resources when purging<br/>
**max dead bytes** - maximum size of dead resources when purging<br/>
**total capacity bytes** - sum of live and dead resources capacity sizes 

Don't panic. Total capacity does not determine upper boundary of resources size that WebKit would keep in memory therefore will not be a situation when only half of your favourite page will be loaded.
It is only a threshold to decide when to prune resources.
But what exactly means to prune?
First, let's explain the meaning of live and dead resources. When a resource is downloaded it is stored in memory as raw data (it can be an image, CSS, JavaScript etc.).
Such data needs to be decoded (parsed) to make it usable so when the resource is decoded (it means implicitly that some CachedResourceClient needs it) it starts to be a live resource.

It will be dead after last CachedResourceClient stops using it. Anyway live or dead resources may contain decoded data that (depending on external factors) unnecessarily occupies memory.
Therefore prune() method is provided. In first step MemoryCache will prune dead resources (to meet minimum and maximum dead resources size boundary) and then live resources one by one until all resources size exceeds total capacity.

voc: threshold - порог, unnecessarily - необязательно, prune - подрезать, quaint -причудливыйб Eviction - выселение, regain - восстановление, redundant - избыточный¶

Prune procedure provides one more mechanism to reduce memory consumption. In the beginning of this article I've mentioned about quaint technique - purgeable memory.
CachedResource has ability to move stored raw data to special buffer called PurgeableBuffer. Since data is located in such buffer the operating system may take over (purge) memory allocated for it.
So if a resource buffer was purged the prune procedure would also remove all CachedResource data related to the buffer - it's called eviction.
Evict will also remove a resource from MemoryCache internal structures and after that the only way to restore lost data is to regain it basing on related ResourceRequest.

LRU lists¶

MemoryCache keeps references to resources in three different structures:

m_resources - a LRU-based map of all resources kept in cache
m_allResources - Vector of LRU lists with fixed size (32).
Basing on CachedResource access counter a resource is located in appropriate list. Prune procedure will start evict beginning from last recently used resources.

m_liveDecodedResources - live resources with decoded data.
If stored resources significantly exceeds total capacity MemoryCache tries to remove decoded data one by one until boundary assumptions will be fulfilled or there would be nothing redundant data to release.

Посты чуть ниже также могут вас заинтересовать