Comments (1)
@twang15 Thanks for your comments. This benchmark is still at the early stage and it is great if users could provide us with any suggestions.
It is nice to have these AI benchmarks.
From an academic perspective, this benchmark can be improved as follows:
- Input datasets.
For input-sensitivity study, a lot of datasets are needed. Since this benchmark originates from industry, collecting datasets should be relatively easy to be addressed by the Alibaba than anyone else.
We basically thought about this issue and have some internal discussion. One of our tasks in the future release is to provide some datasets for available benchmarks, especially some Alibaba applications. We are working on it.
- Correctness/Accuracy criteria.
With compiler involved in the optimization process, it is easy to have an incorrect compiled binary. Therefore, it is extremely important to have a correctness checking feature for a successful benchmark suite. For example, SPEC CPU 2006/2017 have built-in correctness checking feature as part of its scripted tool chain; many HPC benchmarks, such as Cloverleaf/Cleverleaf also have these kind of features.
For approximated computation, especially on machine-learning, numerical correctness may not be applicable. Instead, accuracy may be a better criterion. Again, this domain-specific criterion is easy for Alibaba to provide and critical for researchers in other domains.
Good suggestion here. We are actually aware of this issue and needs from others. The work is ongoing and tries to make it happen on layer-based benchmark first.
- Automated installation and report.
Installation of big programs on main-stream Linux distribution, especially without root privilege, can be very challenging. Reporting the benchmark results could also be an interesting feature to include.
So far, SPEC seems to be most successful in this aspect than any other benchmark suites I have tried.
User-space software package management tools such as linuxBrew, spack (LLNL), are very useful to automate installation.
As another example, this on-going exascale computing benchmark suite (https://proxyapps.exascaleproject.org/ecp-proxy-apps-suite/) is supported by spack (https://spack.readthedocs.io/en/latest/package_list.html) for automatic installation, not only the package itself but also its dependency, all in user space.
The applications are collected from other open source software. Different applications will have different dependency and it takes time to set them up in automated installation process. Our solution will leverage the docker image to help users get rid of the annoying installation issues.
from ai-matrix.
Related Issues (14)
- the model files cannot be downloaded via git lfs
- DIEN implementation is not aligned with DIEN paper for some reason HOT 2
- why the CNN_Tensorflow have no script for dataset prepare HOT 1
- DeepInterestNetwork opens HOT 4
- DeepInterestNetwork: inconsistant between code and result HOT 1
- typo in micro_benchmark/gemm/test_allgemm.sh:13&15 HOT 1
- Why DeepSpeech is removed in macro_benchmark directory ? HOT 1
- Is there DIEN pretrained model provided? HOT 1
- DIEN got lower AUC than original paper
- module 'mlperf_compliance.mlperf_log' has no attribute 'setdefault' in SSD_ResNet34_PyTorch
- DIEN_TF2 .contrib is used by FP16 and mode=test HOT 1
- Bugs in DIEN and DIEN_TF2, both got nan when training with prepare_data.sh HOT 1
- maskrcnn generates images/second but result is secs HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ai-matrix.