Glib-ERROR about libcachesim

[1a1a11a]

It looks like you ran out of memory, how much DRAM do you have?

…

On Aug 17, 2022, at 7:03 PM, Gwanil Jeong ***@***.***> wrote: The following error occurred when I opened trace while running the program. What kind of problem? (process:8155): GLib-ERROR **: 10:10:40.250: ../../../../glib/gmem.c:105: failed to allocate 49463296 bytes Trace/breakpoint trap (core dumped) — Reply to this email directly, view it on GitHub <#9>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBP4TZOXINEXULEUZKH27TVZWKXRANCNFSM563QACOA>. You are receiving this because you are subscribed to this thread.

[gwanil]

Memory on the server.

root@dbms:~# free -h
total used free shared buff/cache available
Mem: 62G 475M 62G 16K 148M 61G
Swap: 9.0G 102M 8.9G

[1a1a11a]

hmm, it would be good if you have stack trace.

…

On Aug 17, 2022, at 8:11 PM, Gwanil Jeong ***@***.***> wrote: Memory on the server. ***@***.***:~# free -h total used free shared buff/cache available Mem: 62G 475M 62G 16K 148M 61G Swap: 9.0G 102M 8.9G — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBP4TZ2T5OKTA7OLX7HUC3VZWSW3ANCNFSM563QACOA>. You are receiving this because you commented.

[gwanil]

I tried to figure out the stack with gdb, but it says I can't.

So I looked at the problem with dmeseg.

(gdb) p main
$1 = {int (int, char **)} 0x555555554c90

(gdb) r
Starting program: /root/libCacheSim/_build/c18.back.out
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
filename = /hdd/cluster22.000

Program terminated with signal SIGKILL, Killed.
The program no longer exists.
(gdb) bt
No stack.
(gdb)

[188077.776581] [14560] 0 14560 20101354 16210629 150704128 2311952 0 c18.back.out
[188077.776582] [14597] 0 14597 1546 16 53248 0 0 sleep
[188077.776584] Out of memory: Kill process 14560 (c18.back.out) score 956 or sacrifice child
[188077.777441] Killed process 14560 (c18.back.out) total-vm:80405416kB, anon-rss:64842516kB, file-rss:0kB, shmem-rss:0kB

I don't know why it holds 80gb...
Is the test.c file that I wrote wrong...?

[1a1a11a]

is the problem solved?

[gwanil]

No, some run for each trace and others don't. I haven't solved it yet.

[1a1a11a]

How do you setup the reader?

[gwanil]

I wrote the test in this way and opened the reader in various formats.

#include <libCacheSim.h>

#define NUM_TRACE_FILES         88
#define MEASUREMENT_FILES       72

int main(int argc, char *argv[]) {
    unsigned long total = 0;
    long lru_miss_cnt, fifo_miss_cnt;
    int nfiles;
    reader_t *reader;
    request_t *req = new_request();
    char filename[64];

    common_cache_params_t cc_params = {.cache_size = 1024*1024*1024, .hashpower = 26};
    cache_t *lru_cache = create_cache("LRU", cc_params, NULL);               
 //   cache_t *fifo_cache = create_cache("FIFO", cc_params, NULL);

    int64_t req_byte = 0, miss_byte = 0;

    lru_miss_cnt = fifo_miss_cnt = 0;

    for (nfiles = 0; nfiles < NUM_TRACE_FILES; nfiles++) {

        sprintf(filename, "/hdd/cluster18.%03d", nfiles);
        printf("filename = %s\n", filename);

        reader_init_param_t init_params_csv = {.delimiter=',', .real_time_field=1, .obj_id_field=2, .ttl_field=7, .has_header=FALSE};
       // reader = open_trace(filename, CSV_TRACE, OBJ_ID_STR, &init_params_csv);
        // reader = open_trace(filename, PLAIN_TXT_TRACE, OBJ_ID_NUM, NULL);

        while (read_one_req(reader, req) == 0) { 
         //   printf("req_size = %u\n", req->obj_size);
        
		if (nfiles > MEASUREMENT_FILES) total++;
        if (lru_cache->get(lru_cache, req) == cache_ck_miss) {
            if (nfiles > MEASUREMENT_FILES) lru_miss_cnt++;
            }
 //           if (fifo_cache->get(fifo_cache, req) == cache_ck_miss) {
 //               if (nfiles > MEASUREMENT_FILES) fifo_miss_cnt++;
//                }
        }
        close_trace(reader);
    }

    free_request(req);
    lru_cache->cache_free(lru_cache);
//    fifo_cache->cache_free(fifo_cache);
    
    printf("========LRU=======\n");
    printf("total:%lu\n", total);
    printf("hit_rate:%f\n", ((float) total - (float) lru_miss_cnt) / (float) total * 100);
    printf("miss_rate:%f\n", (float) lru_miss_cnt / (float) total * 100);
    printf("======================\n");
    printf("========FIFO=======\n");
    printf("total:%lu\n", total);
    printf("hit_rate:%f\n", ((float) total - (float) fifo_miss_cnt) / (float) total * 100);
    printf("miss_rate:%f\n", (float) fifo_miss_cnt / (float) total * 100);
    printf("======================\n");

    return 0;
}

[1a1a11a]

you did not specify obj_size_col, then each object will be size 1, and you give a cache size of 102410241024, which means the cache will have 102410241024 objects, each object will use 10s bytes, so it may use more than 64 GB DRAM.
You can try to 1. specify the obj_size_col 2. reduce the cache size

[gwanil]

Thank you. I'll try again in the way you told me.

[gwanil]

Can I know the location of obj_size_col? I can't find it.

[1a1a11a]

reader_init_param_t init_params_csv = {.delimiter=',', .real_time_field=1, .obj_id_field=2, .ttl_field=7, .has_header=FALSE};
add .obj_size_field=v

[gwanil]

Oh!
I was originally going to add this, but I couldn't because I didn't know how to specify it because the trace has cache_size_field with key and value.

How should I write the field when the trace is like this?

[1a1a11a]

1. you can write a new trace reader
2. you can preprocess the trace and add a new column
3. you can use the value size to approximate object size (not recommend)

[gwanil]

Which reader format should I use to use Method 1? Originally, I tried to use txt, but it was not available due to a similar error, and there was an error in other formats.

[1a1a11a]

you need to create a trace reader under libCacheSim/traceReader/ that can parse the data set you use

[gwanil]

I'll give it a try.

[1a1a11a]

It looks like you do not specify object size field? In this case, each object is 1 byte, 1GB cache can store 1e9 objects. Is this what you want?

…

On Aug 17, 2022, at 11:08 PM, Gwanil Jeong ***@***.***> wrote: `#include <libCacheSim.h> #define W 20 // total trace read #define F 72 // use one-day trace int main(int argc, char argv[]) { unsigned long i = 0; // total check long miss_cnt = 0; long miss_cnt2 = 0; int a; reader_t reader; request_t* req = new_request(); char filename[50]; common_cache_params_t cc_params = {.cache_size=1024*1024*1024}; // cache_size check cache_t *cache = create_cache("LRU", cc_params, NULL); // alg cache_t *cache2 = create_cache("FIFO", cc_params, NULL); int64_t req_byte = 0, miss_byte = 0; for (a = 0; a <= W; a++) { sprintf(filename, "/hdd/cluster22.%03d", a); printf("filename = %s\n ", filename); reader_init_param_t init_params_csv = {.delimiter=',', .real_time_field=1, .obj_id_field=2, .ttl_field=7, .has_header=FALSE}; reader = open_trace(filename, CSV_TRACE, OBJ_ID_STR, &init_params_csv); if (a == W) { while (read_one_req(reader, req) == 0) { i++; if (cache->get(cache, req) == cache_ck_miss) { miss_cnt++; } if (cache2->get(cache2, req) == cache_ck_miss) { miss_cnt2++; } } printf("========LRU=======\n"); printf("total:%lu\n", i); printf("hit_rate:%f\n", ((float)i-(float)miss_cnt)/(float)i*100); printf("miss_rate:%f\n",(float)miss_cnt/(float)i*100); printf("======================\n"); printf("========FIFO=======\n"); printf("total:%lu\n", i); printf("hit_rate:%f\n", ((float)i-(float)miss_cnt2)/(float)i*100); printf("miss_rate:%f\n",(float)miss_cnt2/(float)i*100); printf("======================\n"); }else if(a >= F){ while (read_one_req(reader, req) == 0) { i++; if (cache->get(cache, req) == cache_ck_miss) { miss_cnt++; } if (cache2->get(cache2, req) == cache_ck_miss) { miss_cnt2++; } } }else{ while (read_one_req(reader, req) == 0) { if (cache->get(cache, req) == cache_ck_miss) { } if (cache2->get(cache2, req) == cache_ck_miss) { } } } close_trace(reader); } free_request(req); cache->cache_free(cache); cache2->cache_free(cache2); return 0; } ` — Reply to this email directly, view it on GitHub <#9 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ACBP4T2H5MUJKIO2DTB6GJ3VZXHPTANCNFSM563QACOA>. You are receiving this because you commented.

[gwanil]

At first, I tried to specify an object size field, but I didn't write it because I didn't know how to assign both sizes because the object size is key and value, as shown in the image above.

[1a1a11a]

what image?

[1a1a11a]

somehow the previous message was delayed
let me know if you have any question, if not I will close this.