Comments (5)
To recap our earlier discussion: The benchmark appears to be calling pwrite(), which, as you noted, wasn't supported by Assise. I've now added support to this syscall. Please pull and re-compile LibFS/KernFS.
I haven't used Assise with MPI workloads, but hopefully this patch will help get you there.
from assise.
Hi @wreda,
We have tried to run the benchmark again, but the problem still persists.
This time we ran the benchmark with strace
to produce a list of the system calls used in hope to find a system call that is not yet supported in Assise.
We have come up with the following list, containing system calls that the benchmark uses, but do not exist in Assise:
arch_prctl
bind
brk
chdir
clone
connect
epoll_create
epoll_ctl
eventfd2
execve
exit_group
faccessat
futex
getcwd
getegid
geteuid
getgid
getpgrp
getrlimit
getsockname
getuid
ioctl
listen
mknod
mprotect
newfstatat
pipe
poll
readlink
readlinkat
recvfrom
recvmsg
rt_sigaction
rt_sigprocmask
rt_sigreturn
sched_getaffinity
sched_setaffinity
sendto
set_robust_list
set_tid_address
setsockopt
shmat
shmctl
shmdt
shmget
shutdown
socket
socketpair
statfs
uname
wait4
From this list, several system calls came up (there could be more) that are related to file I/O: statfs
, newfstatat
(similar system calls like lstat
, fstat
do exist), and perhaps chdir
and getcwd
.
From the man page of newfstatat
:
... The underlying system call employed by the glibc fstatat() wrapper
function is actually called fstatat64() or, on some architectures,
newfstatat(). ... "
These system calls implementation do not exist as well.
Do you think that this could be the reason for the benchmark not to work properly?
from assise.
Getting to this now.
The use of unsupported file system calls would indeed explain why your benchmark is not working. statfs
in particular will not work with Assise since we don't have an implementation of mounts. It appears your benchmark is also using execve
. This is typically done after a call to fork
, which is also not supported.
Adding support for these calls is currently not in our roadmap. However, you might find it possible to tweak your benchmark to avoid using these particular syscalls with Assise. Alternatively, you could try to add support for some of them yourself. For instance, it might be possible to quickly implement newfstatat
by simply re-using our existing implementations of fstat
(which can be found here).
To use Assise with fork
/execve
, one workaround is to disable automated initialization of LibFS (which does not work properly with fork
). This can be done by setting the environment variable MLFS_DISABLE_INIT
to 1 when running your benchmark. You will then need to call init_fs
/shutdown_fs
manually for each of your benchmark's processes (these functions are declared here). We applied this same workaround to get Assise working with Filebench.
If you manage to add proper support to any of the aforementioned syscalls, I'd be more than happy to integrate it.
Hope this helps!
from assise.
I followed your suggestion and edited the source code of our program, so that for each process a call to init_fs()
and shutdown_fs()
will be performed. It seems to work properly in initiating Assise for each process.
I am now experiencing an inconsistent behavior from Assise when performing basic file I/O operations such as read and write.
For instance, echo "some test" > /mlfs/file
doesn't seem to produce any output at all (usually the logs by LibFS are shown, but not when running this specific command).
When running our benchmark, I got the following error:
incorrect fd -2: file /mlfs/btio.mpi
mca_fbtl_posix_pwritev: error in writev:Bad file descriptor
I have tried to first create the file because I though it might help, but I am also getting an error when trying to touch
a file:
touch: cannot touch '/mlfs/btio.mpi': Bad file descriptor
I have also tried to run 5 python processes in parallel, with each one writing to a different file in /mlfs. At first, only 3 files were created (for processes 2-4). Then, when I tried to run the script again, all of them were created normally.
Have you experienced this type of inconsistent behavior by Assise in your work?
Is the cause of this related to Assise or perhaps to something that is wrong with our setup?
Thanks,
Kfir
from assise.
I am now experiencing an inconsistent behavior from Assise when performing basic file I/O operations such as read and write.
For instance, echo "some test" > /mlfs/file doesn't seem to produce any output at all
This is likely because LibFS hasn't been preloaded. You can try this command instead (with MLFS_DISABLE_INIT set to 0):
echo "foo" | sudo LD_PRELOAD=../build/libmlfs.so tee -a /mlfs/file
I have tried to first create the file because I though it might help, but I am also getting an error when trying to touch a file:
touch: cannot touch '/mlfs/btio.mpi': Bad file descriptor
The bad file descriptor error is likely due to the touch
command using an unsupported syscall to force a certain file descriptor (namely dup2
). This could also be why the benchmark fails. I'd recommend creating these files using your own custom script.
I have also tried to run 5 python processes in parallel, with each one writing to a different file in /mlfs. At first, only 3 files were created (for processes 2-4). Then, when I tried to run the script again, all of them were created normally.
If multiple processes are creating files in the same directory, you will need to enable leases in order to observe consistent behavior. This can be done by enabling the flag -USE_LEASE
in the LibFS/KernFS makefiles.
from assise.
Related Issues (20)
- about zookeeper HOT 6
- Not using Assise's libpmem during runtime. HOT 2
- registeration memory failed with errno: Cannot allocate memory HOT 2
- Compile errors HOT 1
- Confused about function compute_log_blocks in libfs/src/log/log.c HOT 2
- How to set up log recovery in Assise? HOT 4
- Access control in Assise HOT 2
- Segmentation fault when leases are enabled HOT 5
- Cleanup of threads managing connections to SharedFS HOT 1
- Updation of number of inode links not persisted HOT 4
- Hitting assertion failure during replication HOT 3
- Setting up a cluster with 8 nodes HOT 6
- Setting up Cluster with Multiple Nodes - Segmentation Fault HOT 12
- Memory Region Failure ibv_reg_mr failed [error code: 14] HOT 3
- Benchmark Details HOT 5
- Multi-thread Filebench HOT 1
- Running the warm replicas experiment from the paper
- Segmentation fault on pthread_create when initializing connections
- I see that the global lru list "g_fcache_head" is not protected by locks, which may cause something wrong? HOT 1
- Can we run assise directly on dram? HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from assise.