Comments (6)
TCMalloc is routinely used with Transparent Huge Pages and it works out-of-the-box.
For backing memory with hugetlbfs, you can install a custom system allocator (tcmalloc::MallocExtension::SetRegionFactory
).
from tcmalloc.
So I am working on an application where I want to exclusively force hugepages to be used instead of transparent HugePages.
As mentioned by your reply, I want to back memory with hugetlbfs, however in the tcmalloc::MallocExtension::SetRegionFactory
there is no explicit call/Region set to hugetlbfs or to dev/hugepages/
.
Please help me understand if I am missing something or how I must approach setting up hugepages in this way using the SetRegionFactory allocator.
from tcmalloc.
Just checking in, so I tried running one of the test files: tcmalloc/huge_page_aware_allocator_test.cc
however when I check the size of the pages being used on cat /proc/meminfo
; it shows that there are no huge pages being used with hugetlbfs
; which probably suggests that it's only using transparent hugepages.
If you could please let me know how and where I can back memory with hugetlbs
using the tcmalloc::MallocExtension::SetRegionFactory
system allocator.
from tcmalloc.
So I have made a rough code that creates a region that has been mapped to hugetlb using mmap
and have attempted to set this region using SetRegionFactory
; I have attached the code below:
#include <iostream>
#include <cstddef>
#include "tcmalloc/malloc_extension.h"
#include <stddef.h>
#include <sys/mman.h>
#include "tcmalloc/common.h"
#include <unistd.h>
inline constexpr size_t kMinMmapAlloc = 1 << 30; //1GiB region
#define PROTECTION (PROT_READ | PROT_WRITE)
#define LENGTH (100UL * 1024 * 1024)
#define ADDR (void *) (0x0UL)
#define FLAGS (MAP_PRIVATE | MAP_ANONYMOUS | MAP_HUGETLB | MAP_POPULATE)
using namespace std;
//Creating a class ExtraRegion to create a new region and allocate hugePages
class ExtraRegion : public tcmalloc::AddressRegion {
public:
explicit ExtraRegion(AddressRegion* under) : under_(under) {}
std::pair<void*, size_t> Alloc(size_t size, size_t alignment) override {
size_t big = size + alignment + alignment;
// Can't pad if allocation is within 2 * alignment of region size.
if (big > kMinMmapAlloc) {
return under_->Alloc(size, alignment);
}
void* ptr;
size_t actual_size;
std::tie(ptr, actual_size) = under_->Alloc(big, alignment);
if (!ptr) return {nullptr, 0};
actual_size = actual_size - alignment * 2;
return {static_cast<char*>(ptr) + alignment, actual_size};
}
private:
AddressRegion* under_;
};
//Creating a class ExtraRegionFactory to create a new region and allocate hugePages
class ExtraRegionFactory : public tcmalloc::AddressRegionFactory {
public:
explicit ExtraRegionFactory(AddressRegionFactory* under) : under_(under) {}
tcmalloc::AddressRegion* Create(void* start, size_t size, UsageHint hint) override {
tcmalloc::AddressRegion* underlying_region = under_->Create(start, size, hint);
CHECK_CONDITION(underlying_region);
void* region_space = MallocInternal(sizeof(ExtraRegion));
CHECK_CONDITION(region_space);
return new (region_space) ExtraRegion(underlying_region);
}
size_t GetStats(absl::Span<char> buffer) override {
return under_->GetStats(buffer);
}
private:
AddressRegionFactory* under_;
};
int main(int argc, char *argv[]) {
//using mmap on buf to create a hugetlb region
void * buf;
buf = mmap(NULL, LENGTH, PROTECTION, FLAGS, 0, 0);
if (buf == MAP_FAILED) {
perror("mmap");
return 1;
}
char cmd[128];
const int pid = getpid();
snprintf(cmd, sizeof(cmd),"cat /proc/%d/maps", pid);
//searching for the respective maps file of the code
system(cmd);
cout << "MMap done, buf is " << pid << " " << std::hex << buf << " done " << std::endl;
//Setting the new region factory
ExtraRegionFactory* extra_;
tcmalloc::AddressRegionFactory* before_;
tcmalloc::AddressRegionFactory::UsageHint hint = tcmalloc::AddressRegionFactory::UsageHint::kNormal;
before_ = tcmalloc::MallocExtension::GetRegionFactory();
extra_ = new ExtraRegionFactory(before_);
double *ptr1 = (double*) malloc(sizeof(double));
cout << "Malloc done ptr1: " << std::hex << ptr1 << " done " << std::endl;
//Creating the hugePages region
extra_->Create(buf, LENGTH, hint);
tcmalloc::MallocExtension::SetRegionFactory(extra_);
//Allocating memory from the hugePages region
double *ptr1 = (double*) malloc(sizeof(double));//kSize
cout << "Malloc done: " << std::hex << ptr1 << " done " << std::endl;
return 0;
}
Here I have printed the maps
file as well as the variable buf
which has been used as a variable for mmap
: I have also printed ptr1
which has been declared after calling SetRegionFactory
.
The output is as follows:
.
.
7fbbf2800000-7fbbf8c00000 rw-p 00000000 00:0f 43468 /anon_hugepage (deleted)
.
.
MMap done, buf is 5603 0x7fbbf2800000 done
Malloc done: ptr1: 0x25c33fa00040 done
As we can see above: 7fbbf2800000-7fbbf8c00000
refers to the hugepage region however in the final line of the output we can see that ptr1
does not take memory from this region. Would like to know if there is some error that has been made in the understanding of the code and how it can be changed to fix this.
from tcmalloc.
My recollection is that MmapAligned
(in system-alloc.cc
) governs the placement and choice of virtual address space. Given a particular piece of virtual address space chosen by MmapAligned
, the region/factory logic determines how to appropriately back it. I'm not sure it can take an externally chosen virtual address, since RegionManager::Allocate
is going from desired target address to region.
Once installed, though, we might not immediately need more virtual address space. For example, malloc(sizeof(double))
is very likely hitting the per-CPU cache and reusing memory that has already been allocated, rather than obtaining more.
from tcmalloc.
@ckennelly how does the region factory logic tell MmapAligned how to back this memory ? We don't see a flag or any argument when creating the region factory to tell it to use hugetlbfs.
To give you a bit more background, I am more familiar with the memfs implementation in gperfs/tcmalloc, over there it allows us to specify TCMALLOC_MEMFS_MALLOC_PATH which helps us initialize the memfs allocator rather than the default system_allocator (which allocates from small page pool). See this for reference - https://github.com/gperftools/gperftools/blob/master/src/memfs_malloc.cc#L232
For that matter, for gperftools/tcmalloc we even updated this code to ensure that we disable fallback option, so that all allocations from tcmalloc only comes from the hugepages backed region. See - gperftools/gperftools@b7607ee
We have been meaning to experiment with the per-cpu cache effort that is available with this fork of tcmalloc but for us to make any progress with this I would first like to ensure that we can get all objects exclusively allocated from the hugepage pool.
from tcmalloc.
Related Issues (20)
- segmentation fault when build with tsan HOT 2
- failed mater branch build on Fedora 38 HOT 1
- bazel build error
- MemoryLimit issue on CentOS
- OSS-Fuzz issue 63111
- OSS-Fuzz issue 63140
- OSS-Fuzz issue 63195 HOT 1
- Following QuickStart does not Build Successfully HOT 2
- tcmalloc central cache too large memory HOT 2
- Is there any release branch or tag of current project? HOT 2
- issue for "improve malloc fast path codegen" HOT 1
- Possibly crash with kernel disabling CONFIG_PROC_PID_CPUSET HOT 2
- tcmalloc is trying to keep everything allocated for longer HOT 3
- Disabling tcmalloc cache HOT 2
- OSS-Fuzz issue 65919 HOT 1
- OSS-Fuzz issue 66172 HOT 1
- MADV_COLLAPSE
- Failed to build tcmalloc, Error info: junk at end of line HOT 1
- Quick start guide broken
- How to access tcmalloc MallocExtensions? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from tcmalloc.